Winappdriver: sendkeys fails with error An unknown error occurred in the remote end while processing the command

[shankarkc]

My testcode will launch microsoftword and tries to type inside.
Here is the code I am using

        String application="C:\\Program Files (x86)\\Microsoft Office\\root\\Office16\\WINWORD.EXE";
        String appWorkingDir="c:\\tmp";
        //To suppress the splash screen and want to use blank word doc. so we use flags /q and /w flags respectively
        String appArguments = "/q /w";
...
....
....
WebElement element =  ((WindowsDriver) webDriver).findElement("name", "Page 1 content");
        if ( element != null ) {
                    element.sendKeys(keysToSend);
               }

When I debug I notice that webelement is not NULL...But sendKeys is failing on it.
I get exception "An unknown error occurred in the remote end while processing the command." I am not sure how to proceed..

WinAppDriver log

POST /wd/hub/session/7D092E3B-44FD-4C0F-957C-4F99363BEF69/element/42.1115954.2.2/value HTTP/1.1
Accept-Encoding: gzip,deflate
Connection: Keep-Alive
Content-Length: 55
Content-Type: application/json; charset=utf-8
Host: 10.173.75.51:4723
User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_112)

{"id":"42.1115954.2.2","value":["This is sample text"]}
HTTP/1.1 500 Internal Error
Content-Length: 133
Content-Type: application/json

{"status":13,"value":{"error":"unknown error","message":"An unknown error occurred in the remote end while processing the command."}}

It will be helpful for us if you print what exactly went wrong. Please help me to proceed further.

[shankarkc]

I even tried attaching from desktop session

                DesiredCapabilities desktopCapabilities = new DesiredCapabilities();
                desktopCapabilities.setCapability("app", "Root");
                WindowsDriver rootSession = new WindowsDriver(new URL(wadHubURL), desktopCapabilities);
                Assert.assertNotNull(rootSession);


                RemoteWebElement wordShellWindow = (RemoteWebElement) rootSession.findElementByName("Document1 - Word");
                String wordTopLevelWindowHandle = "0x"+ Integer.toHexString(Integer.decode(wordShellWindow.getAttribute("NativeWindowHandle"))).toString();

                DesiredCapabilities appCapabilities = new DesiredCapabilities();
                appCapabilities.setCapability("appTopLevelWindow", wordTopLevelWindowHandle);
                webDriver = new WindowsDriver<WindowsElement>(new URL(wadHubURL), appCapabilities);
                Assert.assertNotNull(webDriver);

              Assert.assertNotNull(((WindowsDriver)webDriver).findElement("name", "Page 1 content"));
              RemoteWebElement wordPage = ((WindowsDriver)webDriver).findElement("name", "Page 1 content");
            wordPage.click();
             wordPage.sendKeys("Test");

It fails for last 2 lines...I cannot make it work. Please suggest how to proceed further. My team is blocked bcz of this issue. It would be great if you take a look at this and help us.

[shankarkc]

@hassanuz / @timotiusmargo These are some observations that might help you to find the rootcause....

Also Please make sure you are running the test on a remote machine and not on local machine. My observation is that when it is running on local machine it works fine ...but when we run on a remote machine we face this issue. On remote machine I am running using below command. On this machine I have MS word installed. I am trying to open it using WAD.

WinAppDriver.exe 10.xxx.xx.51 4723/wd/hub

One more observation is I created similar test for notepad application on remote machine...It works fine..Its able to send keys and click works fine. It fails only on application like microsoft word and that too when executed on a remote machine.

[shankarkc]

@hassanuz / @timotiusmargo any updates on this?

[shankarkc]

@hassanuz / @timotiusmargo any updates on this? My team is kind of blocked bcz of this issue. We cannot run the tests in CI environment as we need to run WAD on a remote machine. Your fast response highly appreciated.

[timotiusmargo]

Hi @shankarkc,

Thank you for bringing up the issue along with all the information. I reproduced this issue and it came down to an exception that happened when attempting to set focus on the Microsoft Word Page 1 content Edit Control. The UIAutomationClient.IUIAutomationElement.SetFocus system call invoked at this element returned "Operation is not valid due to the current state of the object." error.

Looking at this issue, it doesn't warrant a fix with WinAppDriver as we need to enforce focus on the target element before sending the keyboard input. Your observation that this issue only happens on remote machine (running as administrator) may indicate that there's an access problem when WinAppDriver or the application under test is run as administrator. However, I also replicated the issue even when Microsoft Word is not run as administrator.

Nevertheless, I found that using a generic (non-element targeting) SendKeys works well in the scenario you described. You may need to find what the Java command looks like, but in C# it is as follows:

```c#
webDriver.Keyboard.SendKeys("Test");

Expected JSON Wire Protocol command:

POST /session/FC8AA401-30BC-4B22-B5D1-6C729171198C/keys HTTP/1.1

In addition, I also find element click to work fine with the Edit Control you are targeting. If this is also the case with your setup, you can simply use it to enforce the focus on the right element. Otherwise, I would attempt to click on other element within the window to set the focus instead.

A quick tip that I don't get to share enough is that you can set focus on a window by [switching](https://github.com/SeleniumHQ/selenium/wiki/JsonWireProtocol#sessionsessionidwindow) to it. I find this to be the most reliable way to enforce focus on an application window. E.g.

```c#
session.SwitchTo().Window(session.CurrentWindowHandle);

Would you try the workaround above and let us know?

[shankarkc]

@timotiusmargo thanks for the investigation. I will gibe a try using switchto windows.. And non-element targeted send keys and update you.

[shankarkc]

@timotiusmargo I tried this. But it didnt give me desired results.

webDriver = new WindowsDriver(new URL(wadHubURL), capabilities);
            Thread.sleep(5000);
            webDriver.switchTo().window(webDriver.getWindowHandle());
            ((WindowsDriver)webDriver).findElement("name", "Page 1 content").click();

Click failed for me. Here is WAD log

==========================================
POST /wd/hub/session/F3FD0991-1620-4527-A2BF-8017F466E6C8/element HTTP/1.1
Accept-Encoding: gzip,deflate
Connection: Keep-Alive
Content-Length: 41
Content-Type: application/json; charset=utf-8
Host: 10.173.75.51:4723
User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_112)

{"using":"name","value":"Page 1 content"}
HTTP/1.1 200 OK
Content-Length: 99
Content-Type: application/json

{"sessionId":"F3FD0991-1620-4527-A2BF-8017F466E6C8","status":0,"value":{"ELEMENT":"42.853088.2.2"}}


==========================================
POST /wd/hub/session/F3FD0991-1620-4527-A2BF-8017F466E6C8/element/42.853088.2.2/click HTTP/1.1
Accept-Encoding: gzip,deflate
Connection: Keep-Alive
Content-Length: 22
Content-Type: application/json; charset=utf-8
Host: 10.173.75.51:4723
User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_112)


HTTP/1.1 500 Internal Error
Content-Length: 133
Content-Type: application/json

{"status":13,"value":{"error":"unknown error","message":"An unknown error occurred in the remote end while processing the command."}}


==========================================

I tried to use sendkeys too as you suggested without targeting elements. But it didnt work too.

                        webDriver = new WindowsDriver(new URL(wadHubURL), capabilities);
            Thread.sleep(5000);
            webDriver.switchTo().window(webDriver.getWindowHandle());
                ((WindowsDriver)webDriver).getKeyboard().sendKeys("Test");

Here is wad log for same

==========================================
GET /wd/hub/session/694BEFE6-167E-4A09-B213-9C7FE242BCA5/window_handle HTTP/1.1
Accept-Encoding: gzip,deflate
Cache-Control: no-cache
Connection: Keep-Alive
Host: 10.173.75.51:4723
User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_112)


HTTP/1.1 200 OK
Content-Length: 84
Content-Type: application/json

{"sessionId":"694BEFE6-167E-4A09-B213-9C7FE242BCA5","status":0,"value":"0x00070432"}


==========================================
POST /wd/hub/session/694BEFE6-167E-4A09-B213-9C7FE242BCA5/window HTTP/1.1
Accept-Encoding: gzip,deflate
Connection: Keep-Alive
Content-Length: 21
Content-Type: application/json; charset=utf-8
Host: 10.173.75.51:4723
User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_112)

{"name":"0x00070432"}
HTTP/1.1 200 OK
Content-Length: 63
Content-Type: application/json

{"sessionId":"694BEFE6-167E-4A09-B213-9C7FE242BCA5","status":0}


==========================================
POST /wd/hub/session/694BEFE6-167E-4A09-B213-9C7FE242BCA5/keys HTTP/1.1
Accept-Encoding: gzip,deflate
Connection: Keep-Alive
Content-Length: 18
Content-Type: application/json; charset=utf-8
Host: 10.173.75.51:4723
User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_112)


HTTP/1.1 500 Internal Error
Content-Length: 133
Content-Type: application/json

{"status":13,"value":{"error":"unknown error","message":"An unknown error occurred in the remote end while processing the command."}}

It would be great if you unblock my team.

[shankarkc]

what puzzles me is why it works for notepad and fails for MSWord? As this is not opensource i cannot debug beyond this boundary. WAD doesnt handle apps with splash screens? I have disable splash screen. Still WAD has difficulty to interacting with UI elements.

[shankarkc]

@timotiusmargo can you pls help me here...can you pls post a code that opens winword on a remote machine and types helloworld? why it works for notepad and fails for MSWord?

[ashton721]

I can also verify this issue. It only occurs for us when running tests from a tfs build agent

I can create a test that executes perfectly on the machine. If I execute the same test on the same machine via a tfs build, it fails at sendkeys with an "unknown error" it appears to be able to click find other elements without issue. SendKeys() and clear() both appear to fail

[shankarkc]

@timotiusmargo can we get some solution for this issue. Its a blocker for my team. If no solution in sight we have to evaluate some other tool for our testing. Your faster resolution is highly appreciated.

[shankarkc]

@hassanuz / @timotiusmargo any updates on this?

[timotiusmargo]

Hi @shankarkc,

The issue you saw is caused by the _"Page 1 Content"_ Edit Control that is a part of a custom control used in Microsoft Word, This particular UI control is not keyboard focusable and unlike other non-actionable element, it actively refuse a focus event. For this reason, both set text and click hit the same error.

I understand that you attempted sending non-element targeting keys without success, In addition to this, I also had another workaround below that always succesfuly send the text to Microsoft Word in a remote machine as long as it is interactive (has keyboard and mouse focus).

c# // Send the input to the first parent node of "Page 1 Content" that can receive focus instead RemoteWebElement element = session.FindElementByName("Document1"); if (element != null) { element.Click(); element.SendKeys("Test"); }

Would you try this instead and let us know how it goes? This perform the send text without issue in my setup.

[shankarkc]

@timotiusmargo thanks for the pointers. Its working now. I realized now that there are some UI elements that cannot take keyboard inputs and if we send keybord inputs it will fail. As per your suggestion when i used Document1 it works and I am able to type text inside word document.

[timotiusmargo]

Thanks for the update @shankarkc!. I'm glad it worked out well. Note that you may still need to refer to _"Page 1 Content"_ when you are trying to retrieve the text value. E.g.

```c#
// Send the text input
RemoteWebElement element = session.FindElementByName("Document1");
if (element != null)
{
element.Click();
element.SendKeys("Test");
}

// Get the text value
RemoteWebElement element1 = session.FindElementByName("Page 1 content");
if (element1 != null)
{
var text = element1.Text;
}
```

[filepi]

Unfortunately this didn't work for me =/

[didhddldlq]

didn't work for me...

[444ghost]

me neither, I think it's losing the focus or the keyboard interface is being used by previous test runs.

[didhddldlq]

What makes me crazy is it sometimes work fine. But Once fails, It continuously fails. if I wait few minutes or restart computer, it works( or sometimes not!).

[444ghost]

What makes me crazy is it sometimes work fine. But Once fails, It continuously fails. if I wait few minutes or restart computer, it works( or sometimes not!).

Exactly the same...

[PArns]

Having the same issue .... Also on a local maschine ...

Tried to switch the Window, also with Click

[peteferraro]

Ditto on this issue. It has me stopped dead in the water. It seems like a bigger deal than anticipated.

[andreea-qa]

I get the same issue. It happens intermittently, so far only when I run all the tests together, and not when I run them individually.
Also, I get the error on different tests, on Click method.
I am running the tests on my local machine, the app has only one window and the button that should be clicked is visible and enabled.
Any help is welcome

[444ghost]

You can just use sendkeys.sendwait() in c# instead I guess. Focus the
session using switchTo() before you do that.

[retrogemHK]

What makes me crazy is it sometimes work fine. But Once fails, It continuously fails. if I wait few minutes or restart computer, it works( or sometimes not!).

I had exactly same issue. I think that is Windows10 multi language(IME) input mode issue.
When I set Windows10 to only English input mode, solved this problem.
Try setting "ONLY single language" keyboard input mode of Windows10

[Jteve-Sobs]

@retrogemHK What exactly have you set in the language input mode?
I can't seem to find a setting besides changing the input method.
And is it important that the only language is set to english?

[retrogemHK]

@Jteve-Sobs
You can find Windows Settings -> Time & Language -> Language -> Add a preferred language
I don't know setting to English is important or not. In my cases, when set to my local keyboard(It can toggle English and Korean using ALT key) I got the error. But English(United States) is working

[grey-ha]

@Jteve-Sobs
You can find Windows Settings -> Time & Language -> Language -> Add a preferred language
I don't know setting to English is important or not. In my cases, when set to my local keyboard(It can toggle English and Korean using ALT key) I got the error. But English(United States) is working

Thank you.
The key input is Korean, but an error occurred when entering English.