Testcafe: Allow configuring the quarantine mode

[arijo]

What is your Test Scenario?

Some test are flaky in CI and we would like to increase the number of retries when running in quarantine mode.

What are you suggesting?

A way to configure the number of retries when running the tests in quarantine mode: -q n

What alternatives have you considered?

Disabling the tests and trying to make the less flaky - but flakiness can depend on other factors like cpu / memory that are not possible to change.

Additional context

Would be nice to know the number of successful tries but not really a priority.

[aleks-pro]

Hello @arijo ,

Thank you for sharing your idea with us. We have decided to add this option in future Testcafe releases. In the meantime, you are welcome to submit your PR.

[jiteshsojitra]

This is very important because default retry for -q is 3, which may be high for someone and takes longer to mark test as failed. Ideal retry for test is one more time for majority of the people. Can we prioritize this issue?

[AndreyBelym]

From our testing experiences, running a failed test just one more time doesn't reduce flakiness well. Current settings for quarantine mode feels balanced enough between robustness and time requirements. So we will be happy to accept a PR about this feature, but our team is busy with more important tasks now.

[dlangerenken]

What I think would be helpful is to have an option to run the failed test again later and not directly after the first initial fail - given that network hiccups usually resolve itself in a few seconds/minutes.

Also, if my test succeeds only once in five runs, I'm still happy enough and would like to proceed with the next tests... But I guess that's something not many teams would accept as "stable"

[aleks-pro]

Hello @dlangerenken ,

Thank you for sharing your idea with us. We will take it into account.

[Farfurix]

For the team: https://stackoverflow.com/questions/60700277/how-to-configure-testcafe-quarantine-mode-to-pass-if-there-is-at-least-1-success

[rodrigotolledo]

From my own experience, the most useful behaviour would be:

If a test succeeded in Quarantine Mode at least once, we should mark it as a PASS (even if majority of executions failed).

The main thinking behind this: If a test is in Quarantine Mode, it's because it's flaky, so it's not very realistic to expect that it will Pass majority of times. If it passes at least once, that's enough to conclude the functionality is working.

Yesterday I was having a quick look at TestCafe code and looks like this is the area that may need to be changed:

_shouldKeepInQuarantine () {
        const errors         = this.testRun.errs;
        const hasErrors      = !!errors.length;
        const attempts       = this.quarantine.attempts;
        const isFirstAttempt = this._isFirstQuarantineAttempt();

        attempts.push(errors);

        // Code below is just a draft/idea to finish Quarantine if there is at least 1 successful run. Not a final solution.
        if (errors.length == 0 && hasErrors == false && attempts > 0 && isFirstAttempt == false) {
            await this._endQuarantine();
        }

        return isFirstAttempt ? hasErrors : !this.quarantine.isThresholdReached();
    }

[rodrigotolledo]

Hello @Farfurix,

Just wondering if this work will be prioritised at some point?

Thank you!

[VadimKrim]

Same question here

[AlexSkorkin]

We didn't plan to address this issue in the nearest sprints. In the meantime, you are welcome to submit your PR with this functionality.