Jest: Running tests in a specific order

[danvc]

🚀 Feature Proposal

Allow define which files should run before others.

Motivation

Some times, we need to define a specific test to be ran before others. As we want to maintain different test cases in different files, would be essential define the order for some tests. This case implies on environment tests when we are using Jest with other frameworks, like Selenium, for example.

Example

I need to test an email client (browser-oriented). As Jest is a robust framework, we'll use it with Selenium for environment, unit and integration coverage. The following tests are focused only in environment coverage with Selenium. The tests are:

• Add an account and make sure that it was added successfully;
• Check messages in the account;
• Perform other actions (Open an message, download file, perform filtering, etc);
• Remove the account.

The idea is to define in the Jest settings an array containing the file names and the order of files that should be ran firstly. For example:

{
    runFirstly: [
        'A.test.js',
        'B.test.js',
        'C.test.js',
        'D.test.js'
}

So, the that would run firstly would be A.test.js, B.test.js, etc.

Once these tests are done, then jest could run other tests sequentially or not.

Pitch

Why does this feature belong in the Jest core platform?

Because we already have a feature that run sequentially --runInBandbut we don't have a way to define which files should run firstly.

I believe that the way that I described before, is goes beyond the --runInBand feature, allowing define files that must ran first and then, keep the tests according the default behavior of Jest.

[zGrav]

I would like to see this implemented in a future version of jest.

[lukeapage]

We have a similar need ; we have a bug somewhere in our 600 spec files where changing the order of execution changes code coverage %.. but we have no way to narrow down the problem...

[sandorvasas]

+1.

[steventle]

@zGrav @sandorvasas @lukeapage work around would be to have an index file, call it index.spec.js, that dictates the order in which you'd execute:

require('./test1.js)
require('./test2.js)
require('./test3.js)

just make sure that there are no other related tests that match against the jest default testMatch unless o/w custom defined

[sapjax]

@steventle
It's not good enough, include all tests into one file, will break the http://jestjs.io/docs/en/configuration.html#bail-boolean option, so the test process can only be stoped after all tests run.

[steventle]

@sapjax bail option for jest is only after each test suite. To the original ask of this feature proposal-it was to gain control of the order in which tests are run, not to break at the granularity of a breaking test within a test suite.

If you're looking for that type of feature, try Mocha for your test runner as they support that with their bail option & no additional set up.

[lukeapage]

I've had some success with an index file - for my use case I am trying to track down why code coverage changes with different runs. So I've made one script that finds spec files, makes an index file importing them all, runs jest, then reverse the order of the imports, runs jest again and compares the coverage.. I use this to narrow down which files have a problem, then another script runs two at a time in different orders to narrow it down to a set of two tests.

The problems are with importing in a index files:

• jest.mock doesn't work if one file expects a mock and one file doesn't
• I get environmental conflicts between tests that don't happen within jest - we haven't been forced to make sure tests are 100% torn down because it seems like imported files are reset between files, which doesn't happen inside jest.

What would allow me to test my tests consistently without work-arounds is if jest had an order parameter (only making sense in runInBand?) that allowed say alphabetical order and reverse alphabetical order. Then I could run the tests one way and then the other and confirm it passes both directions and coverage is consistent.

[lukeapage]

Note, reading the original issue, I wouldn't want a field in the package.json - it would be workable / better than nothing but not ideal. Maybe what I really need is a --testTheTests option that does everything within jest.

For the original reporter I would define 1/2/3 jest setups and then run them one after another. wouldn't that work?

[danvc]

@lukeapage the idea is let the programmer define:

• A procedural test which may respect a order of tests according a flow until some point;
• The order of few tests as primary tests;
• The order should respect what was defined in a array of filenames, and not alphabetically or numerally way;
• That would respect a flow (when your tests need a flow);
• After running these primary tests in the order that was defined, all other tests would run in any order asynchronously or even serially (using the --runInBand parameter);

Also, avoid writing huge tests.

I believe that using a index is not a good approach for many reasons, but, most focusing in maintenance, allowing me to test or perform small pieces of code as well.

Defining a 1/2/3 jest setups would work for small projects, but, for more complexes tests, I don't think that it would be a good approach.

[MarisaAfuera]

I agree with DanZeuss about runFirstly: array to allow the programmer defines the order in which the tests will be done. I'm arranging everything to implement the DanZeuss solution. Do everybody agree?

As you can see, I'm new here. In order to begin the development I cloned the repository locally. Now I'm trying to execute Jest locally. Which way you develop? Any IDE or only and editor like notepad++ and command line?

[chadacious]

Something else you could consider is to name your files something like:

testStuff1.set1.spec.js
testStuff2.set1.spec.js
// these next tests depend on stuff (like data in a database for example) from the first set
// so they have to run 'after' set1
testStuff3.set2.spec.js
testStuff4.set2.spec.js

then you can run your jest command (from your script in package.json or command line) as:
jest .set1.spec && jest .set2.spec

[sahil5695]

I got a hold on this by creating files of the functions then importing them in the main tests.spec.js describing all the tests in order and assigning them with the imported functions:

test.spec.js

import { signuptests } from './signup'
import { logintests } from './login'

describe('Signup', signuptests)
describe('Signup', signuptests)

signup.js

export const signuptests = () => {
     it('Should have login elements', () => {});
     it('Should Signup', () => {}});
}



md5-6d234afa6cab498005a3113adcc80dfd



login.js

export const logintests = () => {
    it('Should Login', () => {}});
}

[SimenB]

I doubt we'll implement this, see https://github.com/facebook/jest/issues/4032#issuecomment-315210901

[Vadorequest]

Well, that's too bad because it could definitely help, especially when testing crud operations to run create-read-update-read-delete in a specific order, and avoid lots of boilerplate from using beforeEach everywhere, and alike.

[Zovi343]

Well I would very much appreciate such a feature as I am currently in desperate need of it, because I use Jest together with Puppeteer and it would help me lot. Was someone able to come up at least with some reliable workaround ?

[palmerj3]

If you need tests to run in a specific order you can automated this yourself with a bash script to execute specific tests/patterns in a particular order. Jest likely won't provide this functionality because it can absolutely lead to situations where tests HAVE to be run in a particular order and it also makes it insanely hard to parallelize tests.

But you could, yourself, write a npm/bash script similar to so:

testInOrder: "jest path/to/test1.js && jest path/to/test2.js"

[regularfry]

Just to chip in to support a previously-made point: it's all very well trying to prevent people from building in a reliance on a test order, but the trade-off is that given an accidental test-order problem, I am now denied a particularly useful tool to find out where it is. That's less helpful than it could be.

[pyrogenic]

@palmerj3 unfortunately that prevents those becoming a single run for report & coverage purposes.

[jeysal]

PR #8209 implements this (and it looks like a lot of effort put in, thanks @WeiAnAn), but I do share the concerns about adding this feature stated by @palmerj3 .
To propose an alternative that might help move this forward: What about making TestSequencer (which is the key component that sorts the tests, where most of the changes in PR #8209 are) pluggable so that users can provide a custom implementation that could do stuff like this?
I know there were ideas of doing this with TestScheduler - for TestSequencer it should be way easier because it is more isolated.
cc @SimenB

[SimenB]

I agree making it pluggable _might_ make sense, if we wanna support this. I don't think it makes sense in core, so I think plugability is the thing we should explore.

@cpojer @scotthovestadt thoughts?

[scotthovestadt]

Providing something almost exactly like the filter API that sorts the tests is something I'd be comfortable with and should be a very simple implementation.

[jeysal]

Or moving TestScheduler into @jest/test-scheduler and making it possible to swap out just like runner etc is what I thought of

[cpojer]

I agree that this shouldn't be a default feature part of core but I wouldn't mind if the scheduler can be adjusted via plugins.

[jeysal]

(I meant moving TestSequencer into @jest/test-sequencer, sorry)

[WeiAnAn]

I think move TestSequencer to individual package is great. It keep jest-core clean.
Should I just move the TestSequencer or making different sequencer to handle different situation.

[jeysal]

@WeiAnAn the custom sequencer would be an external package, Jest itself would only need to have the default sequencer extracted to a package and a config option to allow using a custom one, similar to runner. @scotthovestadt this is not quite like filter but it's something we have for runner, testEnvironment, testRunner etc. Any objections?

[regularfry]

This needn't be complex. I've spiked out something which looks like this:

import TestSequencer from './TestSequencer';

export default class NullSequencer extends TestSequencer {
  sort(tests: Array<Test>): Array<Test> {
    return tests;
  }
}

...and the CLI flag to select it. That's enough for my use case. The order's arbitrary, but I can control it from the command line by explicitly naming files, in the order I want them to run.

[SimenB]

Creating a @jest/test-scheduler, and no other changes, would be a great first step. Will make the diff adding an option for it to be replaced by something else trivial

[SimenB]

I'll call this closed via #8223. Will be available whenever the next release of Jest happens (no timeline)

[SimenB]

Released in 24.7.0

[tomassommar]

Have jest-test-sequencer been released as well?

[SimenB]

Yes https://yarnpkg.com/en/package/@jest/test-sequencer

[jeysal]

Note that this package contains the default implementation though, one that orders tests in a specific order will have to be user-provided (I'm sure someone will publish an alphabetic test sequencer to npm 😅)

[behrouz-s]

Yes https://yarnpkg.com/en/package/@jest/test-sequencer

How should it be used?

[SimenB]

Please refer to the docs https://jestjs.io/docs/en/next/configuration#testsequencer-string

[behrouz-s]

Please refer to the docs https://jestjs.io/docs/en/next/configuration#testsequencer-string
Thanks @SimenB
How to run "it" synchronous in integration tests?

[Carnevali]

Hey guys,

Someone could send a clear example of how can I use the test sequencer? I need to run in order my test but the --runInBand is not working properly.

I will really appreciate it.

Cheers!

[WeiAnAn]

Hey guys,

Someone could send a clear example of how can I use the test sequencer? I need to run in order my test but the --runInBand is not working properly.

I will really appreciate it.

Cheers!

Suppose you have 3 files, a, b, c, and you want to run in the order of b, a, c.

CustomSequencer.js

const TestSequencer = require('@jest/test-sequencer').default;

class CustomSequencer extends TestSequencer {
  sort(tests) {
    const orderPath = ['b', 'a', 'c'];
    return tests.sort((testA, testB) => {
      const indexA = orderPath.indexOf(testA.path);
      const indexB = orderPath.indexOf(testB.path);

    if (indexA === indexB) return 0; // do not swap when tests both not specify in order.

    if (indexA === -1) return 1;
    if (indexB === -1) return -1;
    return indexA < indexB ? -1 : 1;
  } 
}

module.exports = CustomSequencer;

note: you need to ensure you orderPath is in absolute path, you can do it with path.join(__dirname, './path/to/test')

You can refer to jest default TestSequencer sort implementation .

And run

jest --testSequencer=<path-to-CustomSequencer>

Or set in jest config

testSequencer: "<path-to-CustomSequencer>"

If you want to make sure the next test start running after previous test completed, add --runInBand option.

[Carnevali]

Hey guys,
Someone could send a clear example of how can I use the test sequencer? I need to run in order my test but the --runInBand is not working properly.
I will really appreciate it.
Cheers!

Suppose you have 3 files, a, b, c, and you want to run in the order of b, a, c.

CustomSequencer.js

const TestSequencer = require('@jest/test-sequencer').default;

class CustomSequencer extends TestSequencer {
  sort(tests) {
    const orderPath = ['b', 'a', 'c'];
    return tests.sort((testA, testB) => {
      const indexA = orderPath.indexOf(testA.path);
      const indexB = orderPath.indexOf(testB.path);

    if (indexA === indexB) return 0; // do not swap when tests both not specify in order.

    if (indexA === -1) return 1;
    if (indexB === -1) return -1;
    return indexA < indexB ? -1 : 1;
  } 
}

module.exports = CustomSequencer;

note: you need to ensure you orderPath is in absolute path, you can do it with path.join(__dirname, './path/to/test')

You can refer to jest default TestSequencer sort implementation .

And run

jest --testSequencer=<path-to-CustomSequencer>

Or set in jest config

testSequencer: "<path-to-CustomSequencer>"

If you want to make sure the next test start running after previous test completed, add --runInBand option.

Thank you very much! Helped me a lot :)

[akash8663]

I got a hold on this by creating files of the functions then importing them in the main tests.spec.js describing all the tests in order and assigning them with the imported functions:

test.spec.js

import { signuptests } from './signup'
import { logintests } from './login'

describe('Signup', signuptests)
describe('Signup', signuptests)

signup.js

export const signuptests = () => {
     it('Should have login elements', () => {});
     it('Should Signup', () => {}});
}

```
login.js

export const logintests = () => {
it('Should Login', () => {}});
}
`` Is there any way that i can pass parameters from the describes to the IT(tests)? Like this below: describe('Signup', signuptests('name', 'username'))`
I have tried to do that. My test passes but i get the below error:

Invalid second argument, [object Promise]. It must be a callback function.

If it is possible to pass arguments, that would be great. I will be able to write reusable tests

[DjordjePetrovic]

@akash8663 just do describe('Signup', () => signuptests('name', 'username'))

[rodoabad]

How do you use the test sequencer to randomize tests? instead of the usual sort?

[SimenB]

import TestSequencer from '@jest/test-sequencer';

export default class CustomSequencer extends TestSequencer {
  sort(tests) {
    return tests.sort(() => Math.random() > 0.5);
  }
}

[rodoabad]

@SimenB here's a ❤️ for you. Thanks.

Is this per file? Or per test block?

[SimenB]

Per file, tests within a file is always ran in declaration order

[reddypdl]

@SimenB Any solution to sequence tests per test block?

[basickarl]

Would it be an idea to add a parameter to the it and describe blocks to declare that they should be run in order and the ones which do not run in parrarell?

I have tests in a file, some of which I wish to run in sequence and some that I do not care and could be run in parrarell to anything.

it('example test to NOT be run in sequal', () => {

});

it('example test to be run in sequal', () => {

}, true); // <-- if boolean, treat as run in parrarell or sequal, if number, treat as timeout

it('example test to be run in sequal', () => {

}, 3000, true); // <-- both timeout and run in sequal