Testcafe: Fix browser restart for BrowserStack

[AndreyBelym]

TestCafe cannot restart browsers on BrowserStack when heartbeat is lost.

[rob4629]

I would love for this to be done in the near future. We're constantly having our overnight regression tests stop due to a loss of connection with BS... and it's just creates a delay in investigating whether a bug has been introduced.

[AlexKamaev]

@rob4629
I would like to research this issue. However, the cause of the initial issue can be different from that of the issue you reported. Could you please share your test or help me reproduce it? I checked some tests that emulate disconnections on BrowserStack, but every time the browser was restarted successfully.

[rob4629]

I haven't managed to _consistently_ reproduce this error, but we do still see it (albeit, we're still on v1.9.0). I'll run some tests on the latest version, and get back to you.

[rob4629]

I've upgraded to the latest version (1.9.4), and will be running our suite over the weekend. So far, the testing I have done hasn't reproduced the issue, but I should have more confidence Monday morning (PST).

UPDATE:
I'll find the test which caused the failure, and see if I can replicate. I'll then share the steps with you.

ERROR The Internet Explorer 11.0 / Windows 10 (https://automate.browserstack.com/builds/4b5b1b031c44402913826a4f56aa6a5127db5ccf/sessions/fcb6aa4e78482b74ec51e30146fa2a699b3e81ff) browser disconnected. This problem may appear when a browser hangs or is closed, or due to network issues.

Type "testcafe -h" for help.
npm ERR! Test failed.  See above for more details.

[AlexKamaev]

@rob4629 Thank you for your input. I prepared a draft pull request with the fix of the initial issue. You can see the details at https://github.com/DevExpress/testcafe/pull/5731

[rob4629]

Sounds good, I'll check out your PR this morning (PST), and let you know 🕵🏻‍♂️ .

Thanks Alex!

[rob4629]

UPDATE:
I'm still seeing this error/disconnect occur, during a BrowserStack run. The actual failing test changes between runs, so I'll give an overview of my test scenario:

_Setup_
Browser: Opera
Max BrowserStack (concurrent) Instances: 5
Command: testcafe 'browserstack:opera' -c 5 -q

I've noticed that the error might be related to BS, failing to load the page. The last successful session ended midway through an assertion... then failed to re-connect:

I'm not really sure what TestCafe should do in this instance... it seems like this happened in all 5 active instances:

[AlexKamaev]

@rob4629 Thank you for the additional information. It looks like your issue is not related to the restarting mechanism. We will research your case as well. It seems that your issue may be related to BrowserStack, not TestCafe.

Look at the following article: https://github.com/DevExpress/testcafe-browser-provider-browserstack#usage :

If you have troubles starting multiple browsers at once, or get browserstack-local related errors like #27, try setting the BROWSERSTACK_PARALLEL_RUNS environment variable to the number of browsers you want to run simultaneously, or to 1 if you want to run just one browser.

Also, you can run BrowserStackLocal itself as described in the User managed tunnel connection section of this article: https://www.browserstack.com/docs/automate/selenium/getting-started/nodejs/testcafe#set-up-the-browserstacklocal-connection article.

You need to start BrowserStackLocal with the following command:

BrowserStackLocal --key your_key --local-identifier TestCafeIdentifier --daemon start --logFile logs.txt

Before you run TestCafe, you need to set the BROWSERSTACK_LOCAL_IDENTIFIER environment variable to TestCafeIdentifier as in the previous command.

Then, please run your tests on BrowserStack and check the logs.txt file to see if it has any errors.

[rob4629]

It looks like your issue is not related to the restarting mechanism.

I'm of a similar opinion. Thanks for the additional info. I raised a ticket with BS yesterday, but I'll try your suggestions today, and get back to you.