Appium: Socket Timeout - Too many instances of this services are already running

[andreiaionitoaiesv]

The problem

After 5-6 test runs (i ran only one test) the WebDriverAgent seems to be stucked in a weird state. The only thing that I can do here is to restart the phone. Not a viable solution. Any thoughts?

Environment

• Appium version - 1.6.1-beta
• OS X El Capitan 10.11.6
• Node.js version 7.0.0
• iOS 10.1.1
• Real device

Link to Appium logs

https://gist.github.com/andreiaionitoaiesv/ed88032ed99e301ba32e2af344bbf6f6

[mykola-mokhnach]

Try to uninstall the app with bundle ID "com.apple.test.WebDriverAgentRunner-Runner" using ios-deploy tool before starting the driver. Probably, this can workaround the problem.

[andreiaionitoaiesv]

So you try to tell me that I should uninstall de the AgentRunner and install again during each teardown? I'm asking because the tests already run like 4-5 times slower than before, if i do this step will only increase the running time.
Instead can I send a command to kill the proxy or what is stucked?
Also, fyi. if I try to uninstall manually the app and then run the test again, it will NOT fix the problem. Only solution that i found until now is to restart the phone. Not cool.

Any other suggestions?

[mykola-mokhnach]

@andreiaionitoaiesv Does it help if you just reconnect the device or only reboot fixes the problem?

[andreiaionitoaiesv]

@mykola-mokhnach Just tried to reconnect the device, still reproducing. So seems that only reboot fixes the problem.

[mykola-mokhnach]

I'd rather say it looks like WDA issue to me ( @imurchie correct me if I'm wrong ) and you can report it to https://github.com/facebook/WebDriverAgent/issues

It's also possible to restart iOS device from command line, but I'm not sure how it can help you to fully workaround the problem:

idevicediagnostics restart

[andreiaionitoaiesv]

I opened an issue on WebDriverAgent github. If you can confirm this is exclusively a WebDriverAgent issue, please feel free to close this ticket.

[andreiaionitoaiesv]

@mykola-mokhnach @imurchie

Any hints about how to increase tests execution speed? Before 60 seconds test now takes about 250 seconds. Simply this new approach with the xcuitest that performs all kinds of the request through the proxy, slows down a lot the test execution speed. If you have any idea or a little bit of hope in a future release, please let me know. You will make my day! :)

Thanks.

[imurchie]

Yes, things are much slower. Part of this is Apple, part is that we need to proxy through a third-party system because of the way that Apple architected their automation system.

We are working on trying to improve speed.

[ssynix]

WDA responded: https://github.com/facebook/WebDriverAgent/issues/355

Is it possible to reuse an already running WDA server for subsequent Appium sessions? It's my understanding that WDA is restarted for every session.

[jlipps]

I think it's definitely something we should look into @ssynix! It would definitely shave some time off the startup

[imurchie]

I tried it early in my testing and it was ridiculously unstable. It's worth another shot, though.

[imurchie]

Can you try the latest? Update by re-installing 1.6.1-beta. I've added some logic to try to handle this, though it doesn't happen for me so it's difficult to say if it fixes the problem.

npm uninstall -g appium
npm install -g appium@beta

[andreiaionitoaiesv]

Hey

I've already updated to 1.6.1-beta, a few days ago. And the problem is still there, but seems to appear at bigger intervals of time. In 1.6.0 after 5-6 runs but now in 1.6.1-beta the occurance is like in 20 runs or something like that...

I don't know...Do you need some logs or some aditional information?

Thanks

[imurchie]

The change was published this morning. So unless you update after that, there will be nothing.

[andreiaionitoaiesv]

Receive this when trying to install:

Error: ENOENT: no such file or directory, uv_cwd
at Object.resolve (path.js:1142:25)
at Function.Module._resolveLookupPaths (module.js:393:17)
at Function.Module._resolveFilename (module.js:463:31)
at Function.Module._load (module.js:420:25)
at Module.require (module.js:500:17)
at require (internal/module.js:20:19)
at /Users/P4793/.npm-packages/lib/node_modules/npm/bin/npm-cli.js:19:21
at Object. (/Users/P4793/.npm-packages/lib/node_modules/npm/bin/npm-cli.js:79:3)
at Module._compile (module.js:573:32)
at Object.Module._extensions..js (module.js:582:10)

[imurchie]

Are you in a directory you deleted?

[andreiaionitoaiesv]

So I've tried right now and fails when running the second test from the test suite. Here is the stacktrace.

https://gist.github.com/andreiaionitoaiesv/abb8ae76547d4a400961aa614bb37886

Unfortunately, for me this is a blocker...

[imurchie]

I understand it is important. I wish that I could reproduce it so I could fix it. Alas.

Can you provide the Appium logs, so I can see what is happening?

[andreiaionitoaiesv]

Hey @imurchie

Here are the logs:
https://gist.github.com/andreiaionitoaiesv/bc9530ded2880f18c7892ff8f10e2967

You also find the end of the previous test. As far as I can tell the previous session ends as expected. What I noticed is the fact that the issue itself is with the WebDriverAgentRUNNER, because the WebDriverAgent seems to start without any problems. Weird.... I'm expecting that the Runner should be closed once the session was closed, but for some reasons seems to remain in stuck state.

[XCUITest] WebDriverAgent started at url 'http://192.168.2.2:8100'

[imurchie]

It seems like the process is not correctly being killed. I've changed things a bit in https://github.com/appium/appium-xcuitest-driver/pull/301, which better handles xcodebuild in real devices. I do not know if this will help, but we can try again when that is in. :(

I will update here when that happens.

[andreiaionitoaiesv]

Hmm code looks good and the logs adition might help if this is not fixed. Thanks. Waiting to merge.

[andreiaionitoaiesv]

FYI @imurchie
I noticed this extra info when I stop the test execution with CTRL+C from terminal, when the process remains stuck. Then I restart the device from line command with idevicediagnostics restart and this error appears.

https://gist.github.com/andreiaionitoaiesv/1a7f75b5ee8a9750ea423885ee16a745

This clearly confirm that the process doesn't get killed, since WDA still try to proxy the remote server.

[imurchie]

That would be the iproxy process. I think the xcodebuild process was failing to be killed, and in doing so we got bumped out of trying to kill the other processes. Perhaps all three (xcodebuild, iproxy, logger) should be killed individually instead of concurrently. Ever-so-slightly less efficient, but more reliable.

[imurchie]

Published as [email protected]. Can you update and try again?

[andreiaionitoaiesv]

Hey @imurchie

So after updating to 2.0.37 version, I've noticed new behaviour.
Here are the logs starting from the end of the previous test:
https://gist.github.com/andreiaionitoaiesv/072ca1e9e307e57194829dbc45779b34

As you can see at the end of the first test, the issue remains with the WDA.

[XCUITest] Shutting down WebDriverAgent
[XCUITest] xcodebuild exited with code 'null' and signal 'SIGTERM'
[XCUITest] System log exited with code 'null'
[XCUITest] iproxy exited
[debug] [XCUITest] WebDriverAgent process did not end in a timely fashion. Sending SIGHUP signal...

Also I've noticed after x seconds from when the session got stuck, the WDA tries again to compile and run but with the same result. In the end the test stops.
But a weird-ish thing, is that if I try to stop the session with CTRL-C from terminal, the WDA keeps trying to compile again and again, basically I cannot user stop running the WDA.

Hope this new info will help you.

Thanks

[imurchie]

It does not look like the upgrade worked

[debug] [XCUITest] XCUITestDriver version: 2.0.35

[andreiaionitoaiesv]

That's weird. I used npm install [email protected]

[imurchie]

You'll need to do that inside your appium installation.

[andreiaionitoaiesv]

Hey @imurchie

Unfortunately, I'm back. Happened again. Here are the logs:
https://gist.github.com/andreiaionitoaiesv/9f3fb1bb2f8b9fd7f734481766d39e5a

As you can see my test has failed and this is the output for the processes:
[XCUITest] xcodebuild exited with code 'null' and signal 'SIGTERM'
[XCUITest] Shutting down Logger process (pid 27374)
[XCUITest] System log exited with code 'null'
[XCUITest] Shutting down iproxy process (pid 27645)
[XCUITest] iproxy exited with code 'null'.

Then I run the same test and the WDA got stuck again. Do you think will it help to kill the proxy, xcodebuild and the logger process with pkill before each test?

[andreiaionitoaiesv]

Captured this on the Activity Monitor for the device. So that is the process that blocks my way towards happiness. Do you know how to kill it without restart (and fire)?

[imurchie]

I do not know. There is no general way to kill things on a device.

I'm concerned about the errors that are here:

[debug] [WebDriverAgent] Device: Nov 21 13:50:43 MyPhone XCTRunner[274] <Notice>: Running tests...
[debug] [WebDriverAgent] Device: Nov 21 13:50:43 MyPhone XCTRunner[274] <Notice>: Unable to load configuration data from specified path /var/folders/z0/mpvc9gcs5ljcy1rvttz0ygp80000gp/T/com.apple.dt.XCTest/C06EAC0C-15AB-4CFC-8DA3-39C77B6A42B8/remote-container/tmp/WebDriverAgentRunner-C06EAC0C-15AB-4CFC-8DA3-39C77B6A42B8.xctestconfiguration; error: The file doesn\M-b\M^@\M^Yt exist.
[debug] [WebDriverAgent] Device: Nov 21 13:50:43 MyPhone XCTRunner[274] <Notice>: Looking for test bundles in /var/containers/Bundle/Application/FA201768-20E0-4907-8CFC-F992D7391FE2/WebDriverAgentRunner-Runner.app/PlugIns
[debug] [WebDriverAgent] Device: Nov 21 13:50:43 MyPhone XCTRunner[274] <Notice>: Found test bundle at /var/containers/Bundle/Application/FA201768-20E0-4907-8CFC-F992D7391FE2/WebDriverAgentRunner-Runner.app/PlugIns/WebDriverAgentRunner.xctest
[debug] [WebDriverAgent] Device: Nov 21 13:50:43 MyPhone XCTRunner[274] <Notice>: Looking for configurations in /var/containers/Bundle/Application/FA201768-20E0-4907-8CFC-F992D7391FE2/WebDriverAgentRunner-Runner.app/PlugIns/WebDriverAgentRunner.xctest
[debug] [WebDriverAgent] Device: Nov 21 13:50:43 MyPhone XCTRunner[274] <Notice>: No configurations found, creating a default configuration that will run all tests.

[andreiaionitoaiesv]

That's weird. I don't provide any configuration files. Seems like a WDA issue, should I open a ticket for this separately at https://github.com/facebook/WebDriverAgent/issues ?

[kenmcc]

I'm seeing the same issue as andreiaionitoaiesv, similar reported frequency of circa 5 test startups and the same required reboot-the-device recovery. Am also removing the webdriveragent-runner binary from the ipad before attempting to launch it, so that it's a clean load.

Have seen the issue on real devices from 9.3.5, 10.0.1, 10.1, 10.2
all on OSX 10.11.6 but I don't think that makes a difference
Have also seen it on webdriveragents built with XCode 8.0. 8.1 and 8.2, so it appears to be not related to whatever is on the mac, rather the ios device.

Once launched it appears to be pretty stable, I've been running for almost 8 hours now with lots of taps, swipes etc but the start is incredibly flakey.

[andreiaionitoaiesv]

@imurchie
Any updates?

[mykola-mokhnach]

@andreiaionitoaiesv

Try the newly added _usePrebuiltWDA_ capability for your iOS tests (available since Appium 1.6.1), which prevents WDA driver from being rebuilt after each driver recreation. This is how I use it in our automated tests:

      if (!capabilities.is(CAPABILITY_NAME_FORCE_RESET) && FBDriverAPI.isAlive()) {
            capabilities.setCapability(CustomizedIOSDriver.CAPABILITY_NAME_USE_PREBUILT_WDA, true);
        }

....

 public static boolean isAlive(URL siteURL) {
        try {
            final HttpURLConnection connection = (HttpURLConnection) siteURL.openConnection();
            connection.setRequestMethod("HEAD");
            connection.setConnectTimeout(IS_ALIVE_VERIFICATION_TIMEOUT_MS);
            connection.setReadTimeout(IS_ALIVE_VERIFICATION_TIMEOUT_MS);
            connection.connect();
            final int responseCode = connection.getResponseCode();
            log.debug(String.format("Response code from %s: %s", siteURL.toString(), responseCode));
            return (responseCode == HttpURLConnection.HTTP_OK);
        } catch (Exception e) {
            // Just ignore
            // e.printStackTrace();
        }
        return false;
    }

WDA URL is usually http://localhost:8100

[Prakash-Saravanan]

http://stackoverflow.com/questions/39464682/dtdevicekit-could-not-start-house-arrest-service-for-app-identifier-xxx

[Prakash-Saravanan]

The newly added _usePrebuiltWDA_ capability is not working. It still tries to build and install WDA on the device. Using another capability _webDriverAgentUrl_ may help

[MaisGurbanov]

Hello guys

I have the similar problem. Is there any solution instead of restart the device?

Thanks in advance

[ohadschn]

@mykola-mokhnach I can confirm that uninstalling com.apple.test.WebDriverAgentRunner-Runner from the device doesn't help unfortunately. I have to restart the device.

[ohadschn]

I implemented the restart workaround as follows:

#!/bin/bash
set -e -o pipefail

idevicediagnostics restart -u $DEVICE_ID

waitAttempts=100
for i in $(seq 1 $waitAttempts)
do      
    echo "Checking if device is up... (attempt #$i / $waitAttempts)"

    set +e
    idevicediagnostics diagnostics -u $DEVICE_ID
    idevicediagnosticsExitCode=$?
    set -e

    if (( idevicediagnosticsExitCode == 0 )); then
        echo "device is back up."
        break
    else
        echo "device is still down..."
        sleep 1
    fi
done

Unfortunately sometimes after the restart the device seems to lose the computer's trust, at which point manual intervention is necessary (on both the computer and the device): https://github.com/libimobiledevice/libimobiledevice/issues/372.

[ohadschn]

It seems that this only happens on specific devices (iPad 4th Gen and iPhone 5c in my case). With other devices, it doesn't seem to a be problem, or at least the problem takes longer to manifest. I've been running dozens of tests in the space of a few days now without encountering this even once on an iPad mini (both gen 2 and 3).

[ohadschn]

Looks like the devices exhibiting this behavior I mentioned above are a special case of the more general problem, namely 32-bit devices: http://stackoverflow.com/a/42264316/67824. So running on 64-bit devices should avoid this issue (at least until it's fixed by Apple).

[ohadschn]

Update - I have now seen this issue on a 64-bit device (iPad mini 2). Maybe it just takes longer to manifest on 64-bit.

[Koen-George]

I'm also encountering this issue with 64-bit iOS devices after a number of tests. My temporary solution is to reboot the iOS device using idevicesiagnostics and just to be sure, the Appium server process as well. However, the dreaded "Do you trust this computer" popup demands human intervention.

Appium version (or git revision) that exhibits the issue: Appium 1.6.3
Desktop OS/version used to run Appium: MACOS 10.12.3 Sierra
Node version (unless using Appium.app|exe): v7.6.0
Npm Version : 4.1.2
Mobile platform/version under test: IOS 10.2.1
Real device or emulator/simulator: Real device: iPad Mini 4, iPhone 7, iPhone 6 Plus
java version: not installed since not needed

[kenmcc]

What sort of number of tests are we talking about? I've not seen it yet, and thought it was resolved ok....

[Koen-George]

After InitDriver, my test script performs many resetApp calls. After 70, the iPhone screen stays black (with the battery icon still visible) and the Appium server console shows the error:
Could not start house arrest service for app identifier com.apple.test.WebDriverAgentRunner-Runner [Error Domain=com.apple.dtdevicekit Code=-402653093 "Too many instances of this service are already running." UserInfo={NSLocalizedFailureReason=Too many instances of this service are already running., com.apple.dtdevicekit.stacktrace=(...

[kenmcc]

in that 8083 issue i linked yesterday, i document the findings i had for a simple loop test of just opening youtube, taking a screenshot and closing youtube.

performance degrades with the increasing number of successful test runs until we hit a point where the fail to launch WDA for a period before recovering and running ok again for a while.
After a bunch of times of repeating this, it then hits the black screen. Although I got in the region of 240 successful starts (in total, not consecutively) before the black screen/too many instances of this running was encountered.

[pluie-cloud]

@andreiaionitoaiesv
1.use iproxy connect iphone by usb
2.when you start app,config : capabilities.setCapability("webDriverAgentUrl", http://localhost:8100);
3.keep booting wda terminal

[mykola-mokhnach]

@pluie-cloud this approach is, actually, documented: https://github.com/appium/appium/blob/master/docs/en/advanced-concepts/wda-custom-server.md

[BalabN]

Does this approach solves this issue and can it be used for parallel test execution on real devices?

[pluie-cloud]

@mykola-mokhnach yes however i use this method to do my ios auto test now,but i foud a problem,when i use iproxy 8100 8100 udid,i can use 127.0.0.1 to connect iphone but can't use my mac's ip to connect iphone,it bother me long time.

[jlipps]

Given the existence of a workaround and the experimentation happening with https://github.com/appium/appium-xcuitest-driver/pull/476, which should keep WDA from being reinstalled and xcodebuild from being restarted, this should be resolved.

[MaheshZebpay]

I faced the same problem.Then i followed the below steps.
1.clean xcode
2.delete derived data
3.switch off and on iphone.
4.test it.
It worked for me.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.