Cypress: Support `cypress run --no-exit`

[Bkucera]

Option to keep browser open after test run, useful for

• quickly finding errors that happened in CI, without having to use debugger

[jennifer-shehane]

Hey @Bkucera, can you add this new flag to our docs?

[Bkucera]

@jennifer-shehane sure

[jennifer-shehane]

I was just looking over the new docs on this and thinking. Is this flag also going to be supported when passed as an option through the Module API?

[Bkucera]

should be, any option passed via the Module API is passed right through the same code as the CLI

[brian-mann]

Released in 3.0.2.

[nwshane]

@Bkucera Can you describe how you would use the --no-exit flag to find/debug errors that happen in CI?

Also, would it be possible / make sense to add a --no-exit-on-failure method which will keep the browser open and prevent it from moving onto subsequent tests when a test fails?

I've been debugging certain test failures that only happen 1/20 (or even 1/100) times by tweaking my test suite so that the test runs 100s of times in CI. This allows me to 1. reproduce the failure, 2. add cy.log statements to see what's happening, and 3. confirm (with a high level of confidence) that I've fixed the error when the test runs 100s of times without the failure. I think it could be quite useful to have the --no-exit-on-failure (or something like it) available, so that I could get access to browser logs when the failure occurs.

[Bkucera]

@nwshane it's nothing magical, it just allows you to keep the browser open and use see the failed commands after a failed test using cypress run. Otherwise the browser would just close and you wouldn't be able to inspect anything

[nwshane]

@Bkucera but how would you be able to see what's happening in the browser if it's running in CI?

[Bkucera]

@nwshane test fails in CI, you then run cypress run --spec failed.spec.js --headed --no-exit
This assumes the test fails the same way in ci as it does locally

[bahmutov]

What Ben means you run the command locally hoping it fails the same way as it does on CI

Sent from my iPhone

On Jul 6, 2018, at 16:06, Ben Kucera [e-mail] wrote:

@nwshane test fails in CI, you then run cypress run --spec failed.spec.js --headed --no-exit
This assumes the test fails the same way in ci as it does locally

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[nwshane]

@bahmutov @Bkucera Okay thanks, I understand. Did you see my question about --no-exit-on-failure (or maybe --pause-on-failure)? This could allow me to run the test locally 100 times and automatically pause when the test fails - I think it'd actually be quite useful. Ideally there'd also be some sort of API for specifying what kind of failure it should pause on, in case there are multiple intermittent test failures happening (which I've experienced in my test suite).

[bahmutov]

Yes we are thinking of adding flags or even single test option to catch these intermittent failures that are so hard to debug normally one run at a time

Sent from my iPhone

On Jul 6, 2018, at 16:20, Nathan Shane [e-mail] wrote:

@bahmutov @Bkucera Okay thanks, I understand. Did you see my question about --no-exit-on-failure (or maybe --pause-on-failure)? This could allow me to run the test locally 100 times and automatically pause when the test fails - I think it'd actually be quite useful. Ideally there'd also be some sort of API for specifying what kind of failure it should pause on, in case there are multiple intermittent test failures happening (which I've experienced in my test suite).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[jennifer-shehane]

@nwshane Could you open a new issue for these flags you would like to see per failure? Thanks!

[nwshane]

@jennifer-shehane I can open a new issue for the --pause-on-failure flag, but thinking again, I'm not sure that it would address the problem I'm trying to fix. I'd like it to be easier to diagnose and fix intermittent cypress test failures that happen in CI. The --pause-on-failure flag would allow me to run tests locally many times and to see browser logs when a failure occurs, but as the local and CI testing environments are different, the CI failures might not occur locally.

What I'd really love would be to have access to browser logs in the dashboard, i.e. https://github.com/cypress-io/cypress/issues/448

[andezzat]

I have to second this.

I have issues with flaky tests in Cypress that are very difficult to debug as they ONLY show up when running via cypress run and not cypress open. So my only solution right now is to keep doing cypress run --headed --no-exit, and then I have SOME chance to debug...

Even then, it's cumbersome as I would love a no exit flag ONLY on failure, this way I can repeat the run as many times as needed knowing it'll pause when failed.