Cypress: --spec flag should accept array of files or glob pattern

[brian-mann]

Currently the --spec argument accepts only a single spec file to run, and this could be upgraded to accept an array of files and/or a glob pattern.

Related to #64

[lukemadera]

Would really love this feature - any ETA on this @brian-mann ?

[mvandebunt]

would love this feature.
Maybe use wildcards?
cypress run --spec cypress/integration/*_mocked_spec would run all specs ending with _mocked_spec

[paulfalgout]

Also related https://github.com/cypress-io/cypress/issues/263

[bencao]

Would vote for this feature. One great use case for this feature is that we could add tags as a part of the file name, such as:

user-list.smoke.js
user-update.stubbed.js

and then use cypress run --spec cypress/**/*.smoke.* to trigger only smoke tests.

[wasmithee]

This is an essential feature for us. We need to be able to tailor the tests that are executed on the fly. Do you have an ETA yet?

[jagreehal]

Would it be possible to use globs in the config file as well?

[lukemadera]

@brian-mann Any update on this? It is becoming a deal breaker for us. Parallelization aside, if we run tests one at a time they will often pass, but when we try to run the whole suite, things lag and tend to flake fail. Some may be caused from tests not properly cleaning up after themselves, but in order to nail those down and fix them, we need a MUCH quicker way to run the subset of tests in question. Currently our whole test suite is taking over 10 minutes to run, and often will just hang and get stuck. Snapshots are missing (apparently if more than 50 tests they're not captured?) so when things do fail, we can't really see why. But when we run one off, they pass. So we're stuck. We NEED a way to specify a set of tests to run so we can run more than one at a time, but not all.

[brian-mann]

@lukemadera I am sympathetic to your situation - unfortunately internally for our team there are higher priority items to get released ahead of this - like open sourcing all of Cypress. That will solve the bigger challenges we face - which is that we cannot possibly keep up with all feature demands without getting community support. Supporting multiple specs is on our list, but it's not quite high enough to have work done on it right now.

We looked into it and although it's not that difficult to do, its still days or possible a week-ish of work for someone full time.

As for the snapshotting - the Desktop GUI mode is not really designed to run all of your tests. We've actually internally discussed updating the GUI to either straight up refuse running if there are too many tests, or adding a note at the top warning you it's not built for this mode (and to prevent capturing any snapshots).

You're really meant to run them all headlessly from cypress run. There are a slew of internal optimizations we make when running headlessly (like bypassing snapshots altogether). The GUI mode is really meant for a to be used on a single test (or handful of tests) as you're developing your application.

We only take snapshots for up to 50 tests and then start purging them because the memory continually adds up and would eventually OOM. But even this is not aggressive enough.

Another note here is we've discussed adding an option something like runSpecFilesInIsolation: true which automatically restarts the renderer process when running between spec files - which can help prevent situations where state is built up / modified between spec files. This would actually match much closer to how you're likely already using the GUI today.

[lukemadera]

Thanks for the prompt reply @brian-mann and understood. I wrote a node script that does this for us for the time being. Given that our tests are flaky, I had to wrap the entire thing in a loop that will try to run all tests, keep track of the failures, then re-run the failures, hoping the failure set gets smaller each time and eventually they all pass. I think it will wreak havoc on the dashboard since tests are run multiple times until they pass, but it's the only thing I can think of right now. We MUST have our CI be green reliably (tests all pass 100% or 99.9% of the time). It would be awesome if the full suite just passed the first time without flake, but unfortunately that is not the case. I know how hard this is to do - Cypress is better than solutions I've used in the past and this is always a pain. If we could get Cypress to remove the flake even better than it already does, be faster / not lag / time out, and run in parallel, it would truly be a huge step forward for e2e testing!

I would strongly encourage preventing the GUI from running with all tests (e.g. just remove the "run all tests" button), as you suggested. If it wasn't meant for that, and it does not work well for it (which it doesn't), don't allow it it (at least by default) as it just leads to frustration.

[brian-mann]

We've internally discussed this as a team and will be opening a new issue for a proposal. There's a lot of complexity around this that touches many areas of Cypress that we think we may have an elegant solution for.

I'll reference that issue once the proposal is done.

[brian-mann]

@lukemadera by the way with the release of 0.20.0 we've written Cypress as a node module which means your script finding all of your specs and then running Cypress on each could be made a bit simpler. We plan on expanding the modules API's to basically do what you're doing yourself: loop through all the specs and then run each one in isolation and then aggregate the results at the end.

[lukemadera]

Thanks Brian for the update! My script is working fine so we'll just keep using it as is, but great to hear this may be made easier for others in the future!

[brian-mann]

In case anyone is interested in running your specs in isolation - it is fairly trivial to do this now with the module API.

Here is an example of looping through each of your specs and then iteratively running them...

const cypress = require('cypress')
const Promise = require('bluebird')

const glob = Promise.promisify(require('glob'))

const started = new Date()
let numFailed = 0

return glob('cypress/integration/**/*', {
  nodir: true,
  realpath: true,
})
.tap((specs) => {
  console.log('Found ', specs.length, ' specs:\n\n', specs)
})
.each((spec) => {
  console.log('\nRunning spec:', spec)

  return cypress.run({
    spec: spec,

    // an example of more config stuff
    // record: true,
    // reporter: 'junit',
    // browser: 'chrome',
    // config: {
    //   baseUrl: 'http://localhost:8080',
    //   chromeWebSecurity: false,
    // },
    // env: {
    //   foo: 'bar',
    //   baz: 'quux',
    // }
  })
  .then((results) => {
    numFailed += results.failures
  })
})
.then(() => {
  const duration = new Date() - started

  console.log('\n--All Done--\n')
  console.log('Total duration:', duration) // format this however you like
  console.log('Exiting with final code:', numFailed)

  process.exit(numFailed)
})
.catch((err) => {
  console.error(err)
  throw err
})

[nickclicksco]

@brian-mann your example of running tests helps tremendously. Having said that it does take quite a long time to run as it needs to restart cypress.run each time. Is there a timeframe on being able to supply an array to the spec: object without needing to use this hack?

Thanks

[EirikBirkeland]

@nickclicksco As a temporary workaround, you could modify the .each step in the above example to use Bluebird's .map to run in parallel:

.map((spec) => {
  console.log('\nRunning spec:', spec)

  return cypress.run({
    spec: spec,
  }, { concurrency: 2 })

And if you want to programmatically use all available CPU cores:

const os = require('os')

<code omitted>

.map((spec) => {
  console.log('\nRunning spec:', spec)

  return cypress.run({
    spec: spec,
  }, { concurrency: os.cpus().length })

[brian-mann]

@EirikBirkeland this works sure... but it's highly not recommended for a few reasons:

CPU is not the bottleneck here - RAM is. Spinning up multiple browser instances will chew your up apart and max it out. When it runs out of RAM, it will begin to swap to disk and tests will fail or the process itself will be killed by the CI provider for exceeding RAM limits.

The second problem is focusability. Browsers will behave differently whether or not they are focused. I believe this problem is mitigated by xvfb but we need to do some internal tests to ensure that's the case. If it's not the case, then the last browser will retain focus, and the other ones won't - and it will cause them to behave differently.

This today will only work for Electron - because in Chrome we utilize a single Cypress profile, and it will not spawn a separate browser instance which means trying to run multiple specs will only open a new "tab" instead of a new "window", which will cause the other tabs to be throttled.

If you are doing any kind of DB seeding, you'd have to account for this, else things will squash and affect one another. You'd have to come up with a way to isolate and run a separate server + DB process from each other.

For those reasons - its not ideal to try to parallelize on a single machine. Instead parallelization should happen across isolated machines, each with their own server/db setup. Then resources will automatically be pooled and allocated for better performance, and you will avoid the focus/blur problem, or the CPU/RAM bottleneck.

Summary

• Don't run tests in parallel on the same machine, run them serially
• Do run them in parallel across machines, each with its own DB/Server setup

[EirikBirkeland]

@brian-mann I have 60 gb of RAM, 32 vcores and running Electron -- so for my specific scenario, I suppose this works. But I'll keep an eye on the RAM, thanks.

I currently simply seed the DB without doing any cleanups. These cleanups naturally take place when the CI instance is finished.

So I'll keep in mind spinning up Docker containers for this; perhaps just a couple of them, to ensure the entire suite can be completed in 5-10 min (if I ever get to the point of having a ton of time-consuming tests).

[brian-mann]

@EirikBirkeland oh 60gb of RAM... if only you had 70 it would probably work :P

There is actually something we need to do in your case - all of the Electron instances share the same "Chromium Profile" which means that they will share local storage and cookies (which could conflict). What we need to do is create new profiles whenever there is a new cypress run.

[EirikBirkeland]

@brian-mann Lately, I'm running things in serial using Chrome, and I won't be running in parallel on CI anytime soon. I'll be able to optimize a lot before eventually needing parallelization. I'll keep in mind Docker containers for now perhaps. I could use your own setup as a template, perhaps!

I'll keep in mind the localStorage and cookies, thanks!

[brian-mann]

Fixed by PR https://github.com/cypress-io/cypress/pull/1583

[brian-mann]

Released in 3.0.0.