Testcafe: Runner hangs when error occurs (Error in Role initializer) in reusable role

[darkowic]

What is your Test Scenario?

Using a simple role before each test to prepare the environment.

What is the Current behavior?

From time to time, connection to tested service fails (internet problem, server restart, application restart etc). When it fails, the whole tests runner pipeline hangs.

Looks like it is a similar issue to https://github.com/DevExpress/testcafe/issues/4297 (@TauPan) and https://github.com/DevExpress/testcafe/issues/4892 (@sijosyn).

What is the Expected behavior?

When such error in role occurs, the test should be marked as failed and the test runner should continue execution.

Further in quarantine mode:

• the tests should be retried
• Would be great to define some delay between failures in such a case - a bit longer than in case of quarantine mode?

Steps to Reproduce:

I was able to reproduce it - you can find the code in my fork branch. Use yarn runner or npm run runner to run the tests.
I use a URL that does not exist to test it (there is no server running at http://localhost:1234/`.

reuse Role

const url = "http://localhost:1234/";
// Here is the important line - create role which is reused for all tests
const role = Role(url, () => {}, { preserveUrl: true });

fixture`A set of examples that illustrate how to use TestCafe API`
.beforeEach(
    async (t) => {
        await t.useRole(role);
    }
)

1. Quarantine mode - no warning, just hangs...

➜  basic git:(master) ✗ yarn runner
yarn run v1.22.4
$ node runner.js
 Running tests in:
 - Chrome 83.0.4103.116 / macOS 10.15.5

 A set of examples that illustrate how to use TestCafe API
^C

1. Normal mode: there is just one warning that connection failed.

➜  basic git:(master) ✗ yarn runner
yarn run v1.22.4
$ node runner.js
 Running tests in:
 - Chrome 83.0.4103.116 / macOS 10.15.5

 A set of examples that illustrate how to use TestCafe API
 ✖ Text typing basics

   1) - Error in Role initializer -
      A request to "http://localhost:1234/" has failed.
      Use quarantine mode to perform additional attempts to execute this test.
      You can find troubleshooting information for this issue at "https://go.devexpress.com/TestCafe_FAQ_ARequestHasFailed.aspx".

      Error details:
      Failed to find a DNS-record for the resource at "http://localhost:1234/".

      Browser: Chrome 83.0.4103.116 / macOS 10.15.5

^C

New Role for each test

const url = "http://localhost:1234/";

fixture`A set of examples that illustrate how to use TestCafe API`
.beforeEach(
    async (t) => {
        await t.useRole(Role(url, () => {}, { preserveUrl: true }));
    }
)

In this case, the result is different and, I would say, expected. For each test, the role is executed:

See tests log:

➜  basic git:(hanging-reproduction-issue) ✗ yarn runner
yarn run v1.22.4
$ node runner.js
 Running tests in:
 - Chrome 83.0.4103.116 / macOS 10.15.5

 A set of examples that illustrate how to use TestCafe API
 ✖ Text typing basics

   1) - Error in Role initializer -
      A request to "http://localhost:1234/" has failed.
      Use quarantine mode to perform additional attempts to execute this test.
      You can find troubleshooting information for this issue at "https://go.devexpress.com/TestCafe_FAQ_ARequestHasFailed.aspx".

      Error details:
      Failed to find a DNS-record for the resource at "http://localhost:1234/".

      Browser: Chrome 83.0.4103.116 / macOS 10.15.5


 ✖ Click an array of labels and then check their states

   1) - Error in Role initializer -
      A request to "http://localhost:1234/" has failed.
      Use quarantine mode to perform additional attempts to execute this test.
      You can find troubleshooting information for this issue at "https://go.devexpress.com/TestCafe_FAQ_ARequestHasFailed.aspx".

      Error details:
      Failed to find a DNS-record for the resource at "http://localhost:1234/".

      Browser: Chrome 83.0.4103.116 / macOS 10.15.5


 ✖ Dealing with text using keyboard

   1) - Error in Role initializer -
      A request to "http://localhost:1234/" has failed.
      Use quarantine mode to perform additional attempts to execute this test.
      You can find troubleshooting information for this issue at "https://go.devexpress.com/TestCafe_FAQ_ARequestHasFailed.aspx".

      Error details:
      Failed to find a DNS-record for the resource at "http://localhost:1234/".

      Browser: Chrome 83.0.4103.116 / macOS 10.15.5



 3/3 failed (16s)
✨  Done in 35.67s.

Your Environment details:

• testcafe version: latest - 1.8.7
• node.js version: v12.14.1
• command-line arguments: See runner in reproduction example
• browser name and version: Chrome 83.0.4103.116
• platform and version: macOS 10.15.5

[Ogurecher]

Hello,

Thank you for pointing out to this problem. We've reproduced it in your example.
We will update this thread once it's fixed.

[darkowic]

Test | Master | Test branch | Time execution difference [%]
-- | -- | -- | --
batch 1 | 18min21s | 25min28s | 39
batch 2 | 18min55s | 28min17s | 50
batch 3 | 17min39s | 25min45s | 46
batch 4 | 18min9s | 26min27s | 46
storybook 1 | 15min22s | 20min15s | 32
storobook 2 | 15min27s | 20min24s | 32

I just wanted to point out that applying the workaround to generate the new role for each test to avoid test cafe hanging in quarantine mode is quite expensive - the time to execute all the tests increases significantly :/

[alexey-lin]

Thanks for sharing your finding. Yes, creating a new role for every test actually diminishes all the performance increment that the roles feature should provide. We would not consider this workaround as a permanent solution. Please bear with us.