Cypress: Option to abort on first failure

[HugoGiraudel]

Current behavior:

All tests are run even if one fails.

Expected behavior:

Providing an option to abort the test suite as soon as a test fails.

How to reproduce the current behavior:

Make a test fail.

Environment

• Cypress Version: 0.19.2

[HugoGiraudel]

In the mean time, this does the trick:

Cypress.on('fail', () => Cypress.runner.abort())

[brian-mann]

In 0.20.0 this became:

Cypress.on('fail', () => {
  Cypress.stop()
})

[HugoGiraudel]

From what we’ve seen, Cypress.stop() exits the process with a 0 code, which means even if a test fails, this will make it behave like everything went fine. We believe Cypress.runner.abort() did exit with a non-0 code in case of a failure.

Are we mistaken?

[brian-mann]

I didn't try it, but Cypress.stop() does do more things than aborting just the runner.

Try this:

Cypress.runner.stop()

That's where the abort code moved to, but even this does a bit more than it used it. May work though.

[tomaswitek]

Any update here?
Both Cypress.runner.stop() and Cypress.stop() behave like everything went fine.

[brian-mann]

Nope. PR's are welcome.

[ThomasDecavele]

This is a much desired feature!

In the meanwhile I managed to get the workaround working though, here's what's in my "support/index.js":

import './commands'
import { OperationCanceledException } from 'typescript';

Cypress.on('fail', () => {
    Cypress.stop();
    throw new OperationCanceledException();
});

[nanoflat]

I think it will work in support/index.js, not in plugins/index.js

[01taylop]

The use-case I have for this is that I need my authentication tests to pass first - if they don't all subsequent tests will fail. Although the other tests will fail in the before hooks, it still slows things down unnecessarily (30 seconds per test file).

I have to name my test 1auth so that it runs first and whitelist a cookie which contains a token required for the other tests to pass.

Cypress.Cookies.defaults({
  whitelist: 'token',
})

I was hoping to do something like this in my auth tests to stop running tests on fail;

  before(() => {
    Cypress.on('fail', () => {
      Cypress.runner.stop()
    })
  })

It does stop the test runner, but it doesn't fail any tests so I get 3/3 tests passed and a green build which is not the intended outcome. I also tried adding am intentional failing test before stopping cypress, but then the test runner continues.

[DanH91]

I have a temporary solution that I think can help someone.

I just add an after each in my utilities file imported by all my specs.

afterEach(function() {
  if (this.currentTest.state === 'failed') {
    Cypress.runner.stop()
  }
});

I take advantage of Mocha offering a way to check the state of the last test.
And the nice part is that the process exit with a failing code (not 0)

[sanderdejong88]

@DanH91 I've been following this thread for months, and finally someone found a way! Awesome!
I've put your code in the support file which also seems to work.

[harvitronix]

This no longer works for me with Cypress 3.x.

[jennifer-shehane]

@harvitronix We changed how specs are run in 3.0 so that the files are run isolated. This means the before runs before each spec file - so I am guessing that it is stopping and failing that 1 spec file and then continuing to the next file. Is this what is happening?

[brian-mann]

Yes we've waited to implement this feature due to how aborting is much more complex now.

We likely need to allow for multiple abort strategies

abortStrategy: 'spec' | 'run'

One would abort just the spec, one would abort the entire run.

This becomes even more complex with parallelization.

[harvitronix]

@jennifer-shehane that's exactly right. Totally understand it gets much more complex now. Would love to have the options you describe, @brian-mann.

[vedmant]

Is there a way to stop execution but run after() method?

[mattvb91]

:+1:

[mohsenny]

I tried your solutions and the runner correctly stops.
However, I'm looking for a way to only stop the current test scenario/test file/describe(), but let the runner keep running to execute the rest of scenarios/test files/describe()s.
Any suggestion?

An example:

As you can see, the 3rd test file (Device Type) failed and the runner correctly didn't execute the rest of the test cases in that file, but it also didn't go any further with the 4th test file (Device) either.

[drumheadrob]

Hi, loving Cypress, but not being able to stop our CI build for a failing test is stopping me from being able to use this in CI, and will especially be a problem in CD.

I've searched quite thoroughly and hope I'm missing something somewhere, but it seems like this is related to this thread?

If so, is there any progress / planned work on this solution? Cheers.

[bahmutov]

@drumheadrob your issue is very different - you are saying that the cypress run does NOT exit with code 1 after all tests complete and some fail?!

[jennifer-shehane]

We are currently working on the underlying processes needed to 'cancel' a run and any other queued spec files, which is a precursor to the work needed to 'abort' the rest of the tests on first failure.

[pacop]

Some news right here? I am facing the very same problem, we were not able to use cypress on our CI.

[stclairdaniel]

I'm also interested in this.

[nantoniazzi]

Are you still working on this feature ?
The more we add tests with Cypress, the more it becomes unusable in our CI

[cmcnicholas]

We have the same situation, paying for cypress dashboard on private repos but our tests (now ~400 or so) are taking more and more time to complete. Much needed to stop early so our pipeline is freed up for the next PR to execute.

[vladarefiev]

It will be cool feature

[gurre]

In order to fail the test make sure to throw the error.

// cypress/support/index.js
Cypress.on('fail', error => {
  Cypress.runner.stop()
  throw error; // throw error to have test fail
});

[itslenny]

@gurre is this working for you?

I'm doing this in my main describe block. The test run is stopping, but it's exiting as if it was successful.

  before(() => {
    Cypress.on('fail', error => {
      Cypress.runner.stop()
      throw error;
    });
  });

edit: nvm... I totally missed @DanH91 's answer above ( https://github.com/cypress-io/cypress/issues/518#issuecomment-373369129 ) which works perfectly with no need to explicitly throw.

[TrueMan777]

I wanted to do the fail-fast-all type of approach so I've ended up adding the following to my cypress/support/index.js file. Hopefully, it helps someone else.
```js
// Fail-fast-all-files
before(function () {
cy.getCookie('has-failed-test').then(cookie => {
if (cookie && typeof cookie === 'object' && cookie.value === 'true') {
Cypress.runner.stop();
}
});
});

// Fail-fast-single-file
afterEach(function () {
if (this.currentTest.state === 'failed') {
cy.setCookie('has-failed-test', 'true');
Cypress.runner.stop();
}
});

[vesper8]

@TrueMan777

Hi and thanks for sharing your solution! I'm rather new to Cypress. I tried dumping the above code inside my cypress/support/index.js but that is throwing an unexpected error. Do you need to wrap it or do something extra?

This is my index.js before and after

// Import commands.js using ES2015 syntax:
import './commands';

// Import commands.js using ES2015 syntax:
import './commands';

// Fail-fast-all-files
before(function () {
    cy.getCookie('has-failed-test').then(cookie => {
        if (cookie && typeof cookie === 'object' && cookie.value === 'true') {
            Cypress.runner.stop();
        }
    });
});

// Fail-fast-single-file
afterEach(function () {
    if (this.currentTest.state === 'failed') {
        cy.setCookie('has-failed-test', 'true');
        Cypress.runner.stop();
    }
});

I am now receiving this error:

/node_modules/@babel/runtime-corejs2/helpers/esm/typeof.js:1
import _Symbol$iterator from "../../core-js/symbol/iterator";
^
ParseError: 'import' and 'export' may appear only with 'sourceType: module'

This occurred while Cypress was compiling and bundling your test code. This is usually caused by:

- A missing file or dependency
- A syntax error in the file or one of its dependencies

Fix the error in your code and re-run your tests.

Any idea what's wrong? Thanks!!

[TrueMan777]

A wild guess, could change the before clause to :

before(function () {
    cy.getCookie('has-failed-test').then(function(cookie) {
        if (cookie && typeof cookie === 'object' && cookie.value === 'true') {
            Cypress.runner.stop();
        }
    });
});

[vesper8]

I just removed the typeof cookie === 'object' since that was causing problem and I don't think it's absolutely necessary

My specs only contain a single test, I want all specs to be aborted on first fail, and this isn't causing this to happen for me.. is it working like that for you?

[TrueMan777]

I just removed the typeof cookie === 'object' since that was causing problem and I don't think it's absolutely necessary

My specs only contain a single test, I want all specs to be aborted on first fail, and this isn't causing this to happen for me.. is it working like that for you?
Interesting, I would never think that condition could trigger ParseError: 'import' and 'export' may appear only with 'sourceType: module' error.

@vesper8, it won't fail-fast the following tests but skip them. So you will still have the tests in the table but the execution time is now 1-2s. This works fine with me, at least for now. It is a workaround anyway.

[pawelrudnicki]

Did you find anything that would work with 3.3.1 version?

[bellsenawat]

I try to using this script and Cypress not stop when fist test failed :(
But It skipped and continue test another spec

[oskargustafsson]

The following works for me in Cypress 3.3.2
(solution based on prev. comments in this issue)

// in cypress/support/index.js

switch (Cypress.env('abort_strategy')) {
  case 'run':
    // eslint-disable-next-line no-undef
    before(function onBeforeEach() {
      // Skips any subsequent specs, if the run has been flagged as failed
      cy.getCookie('has_failed_test').then(cookie => {
        if (cookie && typeof cookie === 'object' && cookie.value === 'true') {
          Cypress.runner.stop();
        }
      });
    });
  /* fallthrough */
  case 'spec':
    afterEach(function onAfterEach() {
      // Skips all subsequent tests in a spec, and flags the whole run as failed
      if (this.currentTest.state === 'failed') {
        cy.setCookie('has_failed_test', 'true');
        Cypress.runner.stop();
      }
    });
    Cypress.Cookies.defaults({
      whitelist: 'has_failed_test',
    });
    break;
  default:
}

This lets you control the abort behavior using the cypress_abort_strategy env var, e.g. export cypress_abort_strategy=spec && cypress run.

Would be nice not to have to hack it like this though!

[Undistraction]

Same use-case as a lot of others here. We now have a large-enough test-suite that it is becoming impractical to let the CI run on past a failing test. We need to way of aborting on the first failure.

[vedmant]

Is there a way to stop tests after first fail but still run after() hook? I use to to clean temp data that created for test.

[ericorruption]

Bummer that this is open for over a year with no official solution. :/
None of the workarounds suggested worked for me.

[philjones88]

I too want this, when cypress flake happens I want to be able to stop the run.

The problem is I use a pool of machines so the workarounds all seem targeted at just 1 cypress machine?

[ericorruption]

Not even in this case. I don't use parallel runs and it still doesn't work.

On Thu, Jul 25, 2019 at 12:24 PM Phil Jones [e-mail]
wrote:

I too want this, when cypress flake happens I want to be able to stop the
run.

The problem is I use a pool of machines so the workarounds all seem
targeted at just 1 cypress machine?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/cypress-io/cypress/issues/518?email_source=notifications&email_token=AASDX7QBEIYC7V6AYO2L5WTQBF5MDA5CNFSM4DNFROL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2ZBZAI#issuecomment-514989185,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AASDX7TLECVHK4OZC3Z3SPLQBF5MDANCNFSM4DNFROLQ
.

--
Eric Quanz
Front-end focused developer
ericquanz.com / [phone]

[vesper8]

unbelievable it's been 2 years and a half and such a simple thing that others have long included is still missing?? there's seriously still no built-in way to run ./node_modules/cypress/bin/cypress run and abort on first failure??

[vaske]

what about in cypress.json:

{
  "maxFailedTests": 1
}

[AnshuChaudhari]

I tried your solutions and the runner correctly stops.
However, I'm looking for a way to only stop the current test scenario/test file/describe(), but let the runner keep running to execute the rest of scenarios/test files/describe()s.
Any suggestion?

An example:

As you can see, the 3rd test file (Device Type) failed and the runner correctly didn't execute the rest of the test cases in that file, but it also didn't go any further with the 4th test file (Device) either.

@mohsenny I am searching a solution for exactly this same issue, where the rest of the test cases dint execute if one of the iteration fails. Did you found any solution for this ?

[ksmith-karma-police]

also interested in this feature (stop current test scenario/file)!

[jpike88]

Yeah this is a major pain. It would be nice if the cypress team at least had an officially recommended workaround inside of people shooting in the dark.

This snippet didn't work.

afterEach(function () {
    if (this.currentTest && this.currentTest.state === 'failed') {
        (Cypress as any).runner.stop();
    }
});

[dwelle]

I've modified one of the recommendations here and posted it as an answer on SO a week back: https://stackoverflow.com/a/58660504/927631

Here's the code:

// cypress/plugins/index.js

let shouldSkip = false;
module.exports = ( on ) => {
  on('task', {
    resetShouldSkipFlag () {
      shouldSkip = false;
      return null;
    },
    shouldSkip ( value ) {
      if ( value != null ) shouldSkip = value;
      return shouldSkip;
    }
  });
}

// cypress/support/index.js

function abortEarly () {
  if ( this.currentTest.state === 'failed' ) {
    return cy.task('shouldSkip', true);
  }
  cy.task('shouldSkip').then( value => {
    if ( value ) this.skip();
  });
}

beforeEach(abortEarly);
afterEach(abortEarly);

before(() => {
  if ( Cypress.browser.isHeaded ) {
    // Reset the shouldSkip flag at the start of a run, so that it 
    //  doesn't carry over into subsequent runs.
    // Do this only for headed runs because in headless runs,
    //  the `before` hook is executed for each spec file.
    cy.task('resetShouldSkipFlag');
  }
});

Will skip all further tests once a failure is encountered. The output will look like:

[jmfn]

@dwelle thanks for this.

I've found this works well w/ cypress run invocations and CI use cases, but not when using the GUI runner. Appears the shouldSkip state scoped at the plugin module is not reset between runs. Haven't determined a way yet to reset it without exiting and reopening the runner.

[dwelle]

My bad --- haven't tested that scenario. I've updated the answer. It's an easy fix --- reset the flag on each run (must be done only for headed runs. See the comments in the code, for more).

[seyfer]

@dwelle would be great to stop it, not showing any output in the end. Now it goes through every file after the first fail and spends 1s for each, just to mark it skipped. Given I have 60 files - I need to wait 1 minute just to get failed output at the end.

[Ciniok]

Is there any workaround for cypress 3.8 ?
I don't have much time for coming up with my own and it's super imporant to be able to fail fast when in the CI

[Mumcio]

Any update here?

[Arver1]

Hi, cypress 4.2.0.
That is quick and easy solution:

afterEach(function () {
        if (this.currentTest.state === 'failed') {
            throw Error();
        }
    });

https://www.npmjs.com/package/cy-webpack-plugin
or you can write me and i can write node.js script if you do not use webpack.

[Mumcio]

@Arver1 yes but in this way, by throwing an error you will lose the reason why test failed

[dialex]

FYI this code snippet no longer works with the latest Cypress version, when running tests visually. Documented issue at https://github.com/cypress-io/cypress/issues/6414#issuecomment-612933329

(...) For that mechanism to work, the cookie needs to be whitelisted, so that it persists between tests of the same test run. After we get one failure, the test suite is aborted. When we try to rerun it, the cookie is still there (whitelisted), the before sees its latest value (true) and immediately aborts the test run without running any test.

[biiiipy]

I can't imagine how can anyone use cypress without the ability to stop on error. It's insane that this issue is open for 3 years

[Mumcio]

Indeed, I fully agree with you @biiiipy. I can not believe it is not resolved so long too.
In addition, the lack of possibility to stop tests on fail will extend pricing plans on any CI.

It is elementary. Basic. Do you have this on your list @jennifer-shehane ?

[PierreCavalet]

As a workaround we use a script and manually list all the test we want to use:

  cypress run --spec="cypress/integration/test1.js" &&
  cypress run --spec="cypress/integration/test2.js" &&
  cypress run --spec="cypress/integration/test3.js" 
  exit $? # in case this is not the end of your script because you are in a "if" or something

It's not perfect but it works so you can use this as a workaround. I did not evaluate the cost of relaunching cypress everytime compared to the execution time you loose when it fails but if anyone wants to provide feedbacks, you are more than welcome.

Hopefully we will have a solution for this by then.

[CodeTroopers]

Could we have a status on this feature by the cypress team? After 3 years and the criticality of the need, it will be nice to have a view on the next. @jennifer-shehane

[biiiipy]

The weird thing is that this "feature" is 1) super essential 2) should be quite easy to implement in the right place and 3) it hasn't been implemented for 3 years (especially, imo, it should have been the default behavior of the test runner since the beginning).

[drumslave-git]

I used such hack for now

describe(`Some title" app`, function () {
    let failedTest;
    beforeEach(function () {
        cy.wrap(failedTest).should('be.undefined')
    });
    afterEach(function () {
        if (this.currentTest.state === 'failed') {
            failedTest = this.currentTest.title;
        }
    });
})

So as soon as one of tests will fail - beforeEach check also will be failed.

[sesam]

@biiiipy I'm doing like this:

Cypress.on("fail", (error, runnable) => {
  if (runnable.type === "test") {
    // (here we do some statistics on which specific step failed)

    runnable.parent._bail = true; // WORKS. Now, to find the officially sanctioned way to do this... :)
  }
  throw error; // throw error to have test still fail
});

[asandoval1]

I tried your solutions and the runner correctly stops.
However, I'm looking for a way to only stop the current test scenario/test file/describe(), but let the runner keep running to execute the rest of scenarios/test files/describe()s.
Any suggestion?
An example:

As you can see, the 3rd test file (Device Type) failed and the runner correctly didn't execute the rest of the test cases in that file, but it also didn't go any further with the 4th test file (Device) either.

@mohsenny I am searching a solution for exactly this same issue, where the rest of the test cases dint execute if one of the iteration fails. Did you found any solution for this ?

@mohsenny , @AnshuChaudhari
I'm looking for a solution to this issue as well. Have you found anything useful so far?

[asandoval1]

@jennifer-shehane, @bahmutov
Are there any updates on this one? This is a huge inconvenience that's preventing us from moving towards a nice and smooth CI integration for testing modules with multiple specs.

[boredland]

hi, is there a workaround that works with cypress 4.7.0? when I do

if (this.currentTest.state === 'failed') {
  throw Error();
}

this only seems to stop the current spec, not the cypress run.

[mduft]

Would also require a solution expecially for CIs - our suite runs ~30 minutes, and most failures occur in the first 5 minutes... Thats just a massive waste of resources and time.

[jakedowns]

same here. been using cypress for a week and a half now and it's one "wont-fix" after another. i really love the concept, and the tooling provided, but these limitations are very frustrating.

:bow:

[jennifer-shehane]

From https://github.com/cypress-io/cypress/issues/1599 - expressed a want to set a maxFailures option instead of exit on exact first, so could maybe make this configurable.

@jakedowns This feature is not a 'wont-fix', it is labeled as a 'proposal', which means no work has been done on this issue as of today but it is still a consideration for future feature work. We do not have an estimate on when this will be delivered.

[drumslave-git]

@jakedowns also please remember that cypress is open source and very flexible, means we are able to achieve any required behavior in theory. My hack that Iv posted here works pretty fine for me.

[jennifer-shehane]

Went through the thread of comments to consolidate some workarounds. I don't think any of them completely meets the requirements that everyone wants for this feature, but you would be able to partially abort on first failure.

Tests that are skipped or pending do not count as test recordings for billing.

abort using this.skip

This workaround is probably the most comprehensive. Use this.skip to properly skip any remaining tests run after the failed test. Thanks @dwelle https://github.com/cypress-io/cypress/issues/518#issuecomment-552382781

What it does:

• This will log the correct failing tests message to stdout during cypress run
• Will show all remaining tests of the current spec file as Pending (or just not run when running via cypress open)
• Will show all remaining tests in spec files following the failure as Pending.

Add this to your plugins/index.js file

let shouldSkip = false;

module.exports = (on) => {
  on('task', {
    resetShouldSkipFlag() {
      shouldSkip = false;
      return null;
    },
    shouldSkip(value) {
      if (value != null) shouldSkip = value;
      return shouldSkip;
    }
  });
}

Add this to your support/index.js file

function abortEarly() {
  if (this.currentTest.state === 'failed') {
    return cy.task('shouldSkip', true);
  }
  cy.task('shouldSkip').then(value => {
    if (value) this.skip();
  });
}

beforeEach(abortEarly);
afterEach(abortEarly);

before(() => {
  if (Cypress.browser.isHeaded) {
    // Reset the shouldSkip flag at the start of a run, so that it 
    //  doesn't carry over into subsequent runs.
    // Do this only for headed runs because in headless runs,
    //  the `before` hook is executed for each spec file.
    cy.task('resetShouldSkipFlag');
  }
});

abort_strategy using cookies

You can stop the tests with an afterEach hook when the state of the current test has failed, then set a cookie that can later be read at the beginning of every spec file that is run to also stop the specfile. Thanks @oskargustafsson https://github.com/cypress-io/cypress/issues/518#issuecomment-508731869

What it does:

• This will log the correct failing tests message to stdout during cypress run
• Will show all remaining tests of the current spec file as Skipped (or just not run when running via cypress open)
• Will show all remaining tests in spec files following the failure as passed and skip the execution of the tests.

Add this to your support/index.js file

switch (Cypress.env('abort_strategy')) {
  case 'run':
    before(function onBeforeEach() {
      // Skips any subsequent specs, if the run has been flagged as failed
      cy.getCookie('has_failed_test').then(cookie => {
        if (cookie && typeof cookie === 'object' && cookie.value === 'true') {
          Cypress.runner.stop();
        }
      });
    });
  /* fallthrough */
  case 'spec':
    afterEach(function onAfterEach() {
      // Skips all subsequent tests in a spec, and flags the whole run as failed
      if (this.currentTest.state === 'failed') {
        cy.setCookie('has_failed_test', 'true');
        Cypress.runner.stop();
      }
    });
    Cypress.Cookies.defaults({
      whitelist: 'has_failed_test',
    });
    break;
  default:
}

Run either strategy to exit early on the single spec or on the entire run.

cypress_abort_strategy=run cypress run

cypress_abort_strategy=run cypress spec

afterEach stop runner

You can stop the tests for each spec file ran with an afterEach hook when the state of the current test has failed. Thanks to @DanH91 https://github.com/cypress-io/cypress/issues/518#issuecomment-373369129

What it does:

• This will log the correct failing tests message to stdout during cypress run
• Will show all remaining tests as Skipped (or just not run when running via cypress open)
• This will not abort the entire run, it will only exit the currently running spec, so all other spec files will run.

You can add this to your support/index.js file for it to be universal across all spec files.

afterEach(function () {
  if (this.currentTest.state === 'failed') {
    Cypress.runner.stop()
  }
});

[jakedowns]

@jennifer-shehane Thank you for compiling this list. It was tough to tell which proposed workarounds were still valid since a lot of comments mentioned them not working on newer/latest version

@drumslave-git thank you for pointing out your workaround, i had overlooked it and I think it's the one I need for my usecase

Sorry for the negative comment, frustrated about other things. Thanks for a great tool and a responsive support community. I hope to contribute something constructive to this project as I learn more about it's ins and outs.

[loren138]

I have a PR which will bail once a test fails #7892.

The mocha bail flag is a boolean so I've built this as a boolean rather than accepting a count https://mochajs.org/api/mocha. By defaulting count to 1, it would be possible to extend this to accept a count and be backwards compatible in the future.

[loren138]

Similar to @PierreCavalet above, we are now running our tests one file at a time. This works pretty well except cypress takes several seconds to start up before it starts running tests so with all our test files this wastes several minutes of time on successful runs.

./bootServer & SERVER_PID=$!

npmbin=$(npm bin)
for file in ./tests/e2e/*.test.js
do
  ${npmbin}/cypress run --spec="${file}" || ${npmbin}/cypress run --spec="${file}" // try twice due to flake
  CYPRESS_RESULT=$?
  [ $CYPRESS_RESULT -eq 0 ] || break
done

kill -2 $SERVER_PID

exit $CYPRESS_RESULT

I am also using the afterEach stop runner code from above to skip tests in the current spec once a single test fails.

I would love to find a way to "cache" the cypress bootup or otherwise make that faster to make this more viable with a large number of files.

[sesam]

We run specs through a custom spec-file-packer runner that takes all specs,
then if one fails we track it in an array, and finally repeats broken tests
from last run up to two more times. That might be an option for others
reading this ticket too.

This hack allows us to stay "green" while working slowly on making specs
less flaky.

(Next level will be to mock away external APIs which will probably get us
to "5 nines".)

>

[dillingham]

@jennifer-shehane can we have a simple (and somewhat common) test config option?

{
    "baseUrl": "http://app.wip",
    "stopOnFirstFailure": true
}

[Lvlynfnt]

afterEach(function () {
  if (this.currentTest.state === 'failed') {
    Cypress.runner.stop()
  }
});

No longer working using version 5.0. Test stops even if it's passed.

[lonewarrior556]

Wow this has been highly request feature for 3 years... since Cypress Version: 0.19.2
I'm sure it'll be added any day now...

[dvsoukup]

I was wanting to abort early as well, as often when the first test fails in some spec, we don't want to continue along since the rest of the DOM is likely to not be in a state to allow for the other tests to pass. Also, due to some flakyness once in awhile, it can be useful to use retries. However current suggestion didn't work well with setting retries in the cypress.json. Thus, here's something that seems to work well!

As of version 5, you can set test retries. In cypress.json:

    "retries": {
        "runMode": 3,
        "openMode": 0
    },

In my case, I only want to retry in headless mode.

Then, in support/index.js:

function abortEarly() {
    if (this.currentTest.state === 'failed' && this.currentTest.currentRetry() === this.currentTest.retries()) {
        Cypress.runner.stop();
    }
}

beforeEach(abortEarly);
afterEach(abortEarly);

Be sure to use function and not fat arrow syntax, to preserve the state of what is on this. This one will check the currentTest (as defined within the Mocha framework...) current retry count according to what the max is allowed, and only stop the runner at that point. Worked like a champ!

[iozz]

Thanks @dvsoukup it works great. I just use the afterEach() hook.

I've combined your approach with the use of a cookie to be able to avoid the remaining spec files as well. Then, in support/index.js:

export const FAILED_TEST_COOKIE = 'has-failed-test';

// CYPRESS_ABORT_STRATEGY environment variable:
// run: Play the whole run
// spec: Stop on first error
if (Cypress.env('ABORT_STRATEGY') === 'spec') {
  // Do not clear the cookie between tests, to be able to shunt down all remaining spec files
  Cypress.Cookies.defaults({
    preserve: FAILED_TEST_COOKIE
  });
}

before(function () {
  if (Cypress.env('ABORT_STRATEGY') === 'spec') {
      cy.getCookie(FAILED_TEST_COOKIE).then(cookie => {
      if (cookie && Boolean(cookie.value) === true) {
        Cypress.runner.stop();
      }
    });
  }
});

afterEach(function () {
  if (Cypress.env('ABORT_STRATEGY') === 'spec') {
    // Check if it is the last retry, and if it has failed
    if (this.currentTest.state === 'failed' && this.currentTest.currentRetry() === this.currentTest.retries()) {
      // Set cookie to abort remaining spec files
      cy.setCookie(String(FAILED_TEST_COOKIE), 'true');
      // Stop the current spec file
      Cypress.runner.stop();
    }
  }
});

[ryanjyost]

@iozz 's solution seems to do the trick, but I needed to change currentRetry() to _currentRetry and retries() to _retries.

I also replaced Cypress.runner.stop() in before hook to this.skip() so it's clear that we are skipping tests after the failed one.

My project has cypress": "^5.0.0"

export const FAILED_TEST_COOKIE = 'has-failed-test'

// CYPRESS_ABORT_STRATEGY environment variable:
// run: Play the whole run, spec: Stop on first error
if (Cypress.env('ABORT_STRATEGY') === 'spec') {
  // Do not clear the cookie between tests, to be able to shut down all remaining spec files
  Cypress.Cookies.defaults({
    preserve: FAILED_TEST_COOKIE,
  })
}

before(function() {
  if (Cypress.env('ABORT_STRATEGY') === 'spec') {
    cy.getCookie(FAILED_TEST_COOKIE).then((cookie) => {
      if (cookie && Boolean(cookie.value) === true) {
        this.skip()
      }
    })
  }
})

afterEach(function() {
  if (Cypress.env('ABORT_STRATEGY') === 'spec') {
    // Check if it is the last retry, and if it has failed
    if (
      this.currentTest.state === 'failed' &&
      this.currentTest._currentRetry === this.currentTest._retries
    ) {
      // Set cookie to abort remaining spec files
      cy.setCookie(String(FAILED_TEST_COOKIE), 'true')
      // Stop the current spec file
      Cypress.runner.stop()
    }
  }
})

[jpike88]

I feel I am taking crazy pills reading this thread. It's like a car without brakes. Instead of stabbing at workarounds that may break as soon as tomorrow, why isn't there a clear, single flag to just fix this? I assume it's pretty damn easy to implement. We will fork if this isn't going to be addressed properly

[csvan]

I assume it's pretty damn easy to implement

That's pretty rude. If you don't understand the internals, don't make assumptions about how easy something is to do.

But if you are sure, why not submit a PR?

[leettaylor]

I assume it's pretty damn easy to implement

That's pretty rude. If you don't understand the internals, don't make assumptions about how easy something is to do.

But if you are sure, why not submit a PR?

Maybe because there's already a PR with the functionality (#7892) that has been waiting _deeper evaluation_ from the Cypress product team since July.

[jennifer-shehane]

The option to abort on first (or a specified) number of failures is still under consideration and we are gathering interest from this thread to help us weigh this feature against others. We are reading the feedback and it does help in our evaluation of what to work on.

Delivering this requires work across both our Test Runner and Dashboard teams in order to deliver a good experience for both of our products. So it's a bit more complex than the initial PR that was proposed.

Priorities can change as we move forward, but our Roadmap shows the larger features that we plan to work on in both products: http://on.cypress.io/roadmap

[biiiipy]

The option to abort on first (or a specified) number of failures is still under consideration and we are gathering interest from this thread to help us weigh this feature against others. We are reading the feedback and it does help in our evaluation of what to work on.

Delivering this requires work across both our Test Runner and Dashboard teams in order to deliver a good experience for both of our products. So it's a bit more complex than the initial PR that was proposed.

Priorities can change as we move forward, but our Roadmap shows the larger features that we plan to work on in both products: on.cypress.io/roadmap

Thank you, we really needed more PR BS here...

[mattvb91]

Is it just me or are comments getting deleted? There was a comment here explaining this issue would cut into the "time for runs" business model of cypress if it got merged.

Edit: I would just like to add in that if it is a business case decision that from my side its a valid reason to choose against it but it could just be communicated that way.

[jennifer-shehane]

The Cypress team has not deleted any comments in this thread. If comments are deleted in GitHub there is a timeline event that is posted within the thread that shows that a comment was deleted. This would be viewable to everyone.

From GitHub docs: https://docs.github.com/en/free-pro-team@latest/github/building-a-strong-community/managing-disruptive-comments#deleting-a-comment

People are able to edit/delete their own comments however, and you should be able to see an edit history of any edits on each individual comment.

[lonewarrior556]

People are able to edit/delete their own comments however, and you should be able to see an edit history of any edits on each individual comment.

And yet it is gone

[QuentinFarizon]

I'm surprised such a great testing tool doesn't have this baked-in from day one ...

[jennifer-shehane]

@lonewarrior556 yes, people can delete their own comments. This would remove the comment entirely without showing it in the history. If it is deleted by a our team it shows the history of all edits/deletions.

[dillingham]

I deleted my comment lol
Didn't seem accurate after looking at their pricing.

[csvan]

@biiiipy can you teach me how to solve other problems through nothing but whining?

[lonewarrior556]

@lonewarrior556 yes, people can delete their own comments. This would remove the comment entirely without showing it in the history. If it is deleted by a our team it shows the history of all edits/deletions.

Wasn't trying to accuse anyone.
Soo should we all delete out comments regarding this or would this loop continue?

[jennifer-shehane]

Please be mindful of our Code of Conduct when interacting with other commenters in our issues. https://github.com/cypress-io/cypress/blob/develop/CODE_OF_CONDUCT.md We'd like to keep the thread open and on the topic of specific requests for the feature or existing workarounds. Thanks!

[csvan]

@vesper8 it has already been explained why.
https://github.com/cypress-io/cypress/issues/518#issuecomment-733573085

Also, that accusation was deleted by @dillingham after actually checking their pricing model.

[jakedowns]

well, if the interest level isn't clear enough, consider this a 👍 from me.
maybe a uservoice page with pure votes, or an upvote system on your own roadmap page would be useful :)

that said, i've been using the afterEach stop just fine

[javierbrea]

If anyone is interested, I have just published a Cypress plugin that can do the trick while the Cypress team works on this feature. The solution is based in some of the proposals I found on this thread, specially those from @DanH91 @dvsoukup @dwelle @jennifer-shehane, etc. so thanks to everybody.

I have added e2e tests to the package, and tested it with Cypress versions from v5.0.0 to v5.6.0 and it works well. I am going to test it also with Cypress v6.x versions, and will add the correspondent tests, so I hope it can be stable enough until the Cypress team adds native support for this feature.

Github: https://github.com/javierbrea/cypress-fail-fast
NPM: https://www.npmjs.com/package/cypress-fail-fast

I hope it might help.