Cypress: Allow for random spec ordering when running tests

[joelzimmer]

Current behavior:

When running specs in cypress, the files run in alphabetical order, and tests within each spec run in order. See #2901.

Desired behavior:

It'd be great to pass a flag into cypress (or for it to do some sort of optimization like Jest does) to randomize the order of test runs. This would allow tests that rely on other state to fail and could help increase confidence in how the app works.

Versions

3.1.3, Alpine Linux, Electron

[Strajk]

Just noticed there is a library for randomizing tests in Mocha by the almighty @bahmutov, would be great to integrate it to Cypress :)

https://github.com/bahmutov/rocha

[bahmutov]

One thought: it would be nice to have a callback from Cypress into
userspace code with the list of specs and tests in each. The user code then
can return the list of tests - in any desired order. I kind of do this in
https://github.com/bahmutov/cypress-select-tests projects where I get a
callback of all tests found in the spec file and can remove all tests I
don't want to run

On Tue, Apr 16, 2019 at 11:20 AM Strajk [e-mail] wrote:

Just noticed there is a library for randomizing tests in Mocha by the
almighty @bahmutov https://github.com/bahmutov, would be great to
integrate it to Cypress :)

https://github.com/bahmutov/rocha

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/cypress-io/cypress/issues/2908#issuecomment-483707749,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACHApsM-jXyR5MgkhhuybvFWTHANHjBJks5vhepFgaJpZM4ZHKIm
.

--
Dr. Gleb Bahmutov, PhD

Schedule video chat / phone call / meeting with me via
https://calendly.com/bahmutov
gleb.[email protected] @bahmutov https://twitter.com/@bahmutov
https://glebbahmutov.com/ https://glebbahmutov.com/blog
https://github.com/bahmutov

[charlex]

Is there an update on this @ Cypress Team?

[heathd]

+1

[jaysonesmith]

+1

[sumitngupta]

+1

[terencechow]

+1

If you share where the location is in this repo that grabs spec files I can take a stab at this. Would it be enough to:

1) randomize the spec files order
2) use https://github.com/bahmutov/cypress-select-tests to randomize the tests within the spec files?

Or is there another reason that randomizing spec file order might break things? (Does something depend on spec file ordering?)

[mncharlton]

I've had a go at randomising the order of the tests (it blocks) within a spec, it's not perfect yet, but is a start - https://www.npmjs.com/package/cypress-random-test-order

[jennifer-shehane]

@mncharlton Could you open a pull request to add this plugin to our documentation?

Instructions here as well as a PR Template Checklist. Thanks!

[mncharlton]

Thanks for the prompt @jennifer-shehane! PR is here: https://github.com/cypress-io/cypress-documentation/pull/3024