Cypress: Run selected subset of files from Cypress interface (multiple tests)

[mattmazzola]

Current behavior:

As far as I know cypress only allows you to run in 3 types of ways.

• UI
  
  
  
  • Run all specs located in the tests/integration folder
  
  
  • Run an individual test by clicking it
  
  
• CI / Command line
  
  
  
  • Run all specs
  
  
  • Run a folder of test using --spec option and specifying folder
  
  

Desired behavior:

It would be nice if we could run tests in more flexible combinations. Below is picture of what I'm imagining:

I assume this feature could be hidden by default so it doesn't overwhelm users, but maybe enabled with a setting such as "Advanced Test Running" or something to opt-in to.

Each test and folder would have a check box. Modifying check of folder affects all children folders/tests.

There is button / warning to indicate when your selection deviates from normal and option to clear it in case people have checkbox selected but scroll down and might forget about it. In worse case I think they would see it trying to run test that had accidentally remained checked, stop the tests, uncheck it and continue.

Related option would be that the "Run All Specs" option only runs the tests that are satisfying the search text filter.

Versions

Cypress: Beta Version 3.0.3
Windows 10: 10.0.17763 Build 17763
Chrome: Version 72.0.3626.119 (Official Build) (64-bit)

[jennifer-shehane]

You should be able to do this via the command log using the --spec option. Like: cypress run --spec 'cypress/integration/login/**/*'

Also, make sure to be aware the issues with the 'Run all tests' implementation here https://github.com/cypress-io/cypress/issues/1586

The code to update the specs list, to add checkboxes, etc can be found here: https://github.com/cypress-io/cypress/blob/develop/packages/desktop-gui/src/specs/specs-list.jsx

We would however have to update how the specs are executed per the url when it is then opened in the Test Runner on click.

[mattmazzola]

You should be able to do this via the command log using the --spec option. Like: cypress run --spec 'cypress/integration/login/**/*'

Oh, my understanding of the spec option is that this line would only run tests in the login folder, but what if we want to run tests outside that folder as well?

On that issue: #1586 it's mentioned:

Enable clicking folders to automatically run a subset of specs

This would help a lot and be inline with checking the box for the folder and clicking run selected. Also this also seems very similar to using the --spec option and specifying the folder path.

My understanding of the issues you were referring to in (#1586) is that when running a lot of tests in interactive mode it will take up too much memory and it's advised to limit the set of tests run.

If we had check boxes and only ran those checked or only those in a folder this would also be limiting the number of tests so it seems also to help with that issue.

Can you explain a bit more about the url issue, I didn't quite understand?

I know when you click a single test that file name is put in the url like:
http://localhost:58054/__/#/tests/integration\CreateModels\WaitVsNoWaitActions.js
and my understanding of your suggesting is that if we did this change the url would need to be updated as the interactive mode switched form testA to testB, etc

When you click "Run all specs" it seems to show:
http://localhost:58054/__/#/tests/__all

So I thought we could so something like:
http://localhost:58054/__/#/tests/__filtered

Which would prevent having to deal with updating the url

[jennifer-shehane]

Oh, my understanding of the spec option is that this line would only run tests in the login folder, but what if we want to run tests outside that folder as well?

You should be able to comma separate globs as well so you can run multiple separate directories or files, like:

cypress run --spec 'cypress/integration/examples/**/*,cypress/integration/other-folder/**/*,cypress/integration/other-file.js'

Can you explain a bit more about the url issue, I didn't quite understand?

Yeah, what you guessed is on the right train of thought. Cypress currently will look for an exact match to 1 file (like when http://localhost:58054/__/#/tests/integration\CreateModels\WaitVsNoWaitActions.js it will look for a single file matching CreateModels\WaitVsNoWaitActions.js within your integrationFolder) or will look for __all and run all of the tests within your integrationFolder. We'd have to have some logic here for it to know how to run a subset of specs from a url.

[yagudaev]

Would love to see this added as soon as possible. This is a simple enough feature that will have a huge positive impact on developer experience.

Scenarios like "I only changed the customer app, so I don't need to run the admin specs" would be easy. It will also make it easier to debug leaky tests that fail when they run together.

[jennifer-shehane]

Related issue with how we want to change the behavior of 'run all specs' in general https://github.com/cypress-io/cypress/issues/1586

[atopper]

I'd like to see annotations on the test level. So you could add a test to multiple test groups (smoke, daily, full, etc.) and then run Cypress specifying the group.

[jennifer-shehane]

From @bahmutov suggestion in #6581

Our users are asking us sometimes to run some specs when using the interactive mode. I think the simplest way we could make them happy is to use the search pattern when the user clicks "Run All Specs" button.

[jennifer-shehane]

We've released the ability to run a selected subset of files based on the search filter in 4.12.0 - closing https://github.com/cypress-io/cypress/issues/6581.

Full instructions here: https://on.cypress.io/writing-and-organizing-tests#Configuration

I don't believe this exactly covers the original criteria of this feature request, so this issue will remain open, but I thought it may be helpful to some people following this issue.