Jest: Document how parallelization works in Jest

[callumlocke]

🚀 Feature Proposal

A piece of official documentation stating exactly how parallelization works in Jest, including whether it's safe to assume that all suites and test cases _within a single file_ will always run serially in source order.

Motivation

It is basic information that should be officially documented.

Pitch

I've searched online for answers to this, and I can only find equivocal Stack Overflow threads and disagreement. From experimenting, I think it works like this:

• Individual files are run in parallel (unless you enable the runInBand option). So it is not safe for multiple test files to share a mutable data store.
• All describe and test blocks within a file always run in serial, in declaration order. So you can safely mutate a module-scoped variable across several tests with predictable results (even when some tests are nested more deeply than others in describe trees). This can be useful when testing a series of mutations to a piece of state.

That's how Jest seems to work today. But I want to see docs stating if that's the intended behaviour, so I can be sure it won't suddenly change without warning in a minor performance update.

[SimenB]

Your points are correct. We have test.concurrent for running tests concurrently within a single file, but it's quite buggy, so it's not documented.

Wanna send a PR? 🙂

One thing to note about runInBand is that Jest will also switch into that mode if it thinks running tests that way will be quicker. I don't think we wanna document _how_ that's decided as it's an implementation detail, but might be a good idea to mention it?

[Luanf]

Hi! I'm interested in picking up and trying to solve this issue.
It would be my first contribution here. :blush:

Now, I see some work was done in #6979 but it was never completed.
I would be glad to pick it up from there. Is there a decision on which document to put this information into?

Thanks

[SimenB]

We have since gotten an architecture page: https://jestjs.io/docs/en/architecture

Adding it there would be perfect 🙂

[Luanf]

Finished a first attempt at #7984
I also added some information regarding test executing order which I thought made sense to be in the same place as the parallelization topic.
Would be glad to have some feedback on this, I'm unsure if I modified the correct places. :smile:

[SimenB]

Merged the above PR (as code comments)

[yashaka]

Hi @SimenB ,

You wrote:

We have test.concurrent for running tests concurrently within a single file, but it's quite buggy, so it's not documented.

So, it's possible to run tests from same file in parallel, but it was buggy in the past... Have anything changed since then?

I saw this commit: https://github.com/facebook/jest/pull/7408/files
Regarding worker-threads support... So maybe things were improved in context of parallelisation... Any more details on that?

Thank you a lot!

You answer will be very helpful, because we need such a feature on our project, and trying to decide between something more mature like jest and something new and experimental like toundra that was created specifically to support such type of parallelization...

[SimenB]

7408 does not affect parallelization in a single test, just across test files (which Jest has had for years, and will use node workers when it's less buggy (see #7681)). However, relatively recent fixes like #7770 (which came in 24.1.0) does. So this is something we want to support properly at some point. I've created a label with a few of the main missing features: https://github.com/facebook/jest/labels/Area%3A%20Concurrent

However, almost all of Yarn's tests use concurrent, so if you can live with some warts and missing features, you are safe to use it

[yashaka]

Hi @SimenB ,

Just to clarify, sorry if I am asking stupid questions...

Will ever jest support "running tests inside one suite/file in parallel"? Or is this supported even now?

[jeysal]

@yashaka this is somewhat supported, see test.concurrent

[ericsoco]

I'm finding that tests run concurrently even within a describe block ([email protected]):

describe('concurrency test', () => {
  let a = 1;
  test('should be one', () => expect(a).toBe(1));
  a = 2;
  test('should be two', () => expect(a).toBe(2));
});

-->

  concurrency test
    ✕ should be one (6ms)
    ✓ should be two

  ● concurrency test › should be one

    expect(received).toBe(expected) // Object.is equality

    Expected: 1
    Received: 2

      54 | describe('concurrency test', () => {
      55 |   let a = 1;
    > 56 |   test('should be one', () => expect(a).toBe(1));
         |                                         ^
      57 |   a = 2;
      58 |   test('should be two', () => expect(a).toBe(2));
      59 | });

This runs contrary to my expectations, and to @callumlocke's "Pitch" above. Unless I'm missing something...

I see this ticket is closed, but the Architecture page @SimenB mentioned doesn't actually describe the general rules for parallelizing tests.

[SimenB]

That's expected behaviour - tests do not execute synchronously. We execute test and it etc, and collect the test implementation. We then run the tests later (this makes filtering by test names work, etc). So if you change a between test definitions, it'll "leak" to all tests

[ericsoco]

Ok, thanks for the explanation. Is this behavior documented anywhere? It was a surprise to me, and took me some time to figure out why my tests like the above example (ported from tape) were failing.

EDIT: this is probably a good enough explanation, though the information about collecting test/it first and executing them later could be a good addition.

[andreabisello]

So the only way to runs tests in parallel way is to create one testsuite (one file) for every test? i thought jest tries always to runs test in a parallel way if --runInBand is not set, but making a try it looks like it runs in parallel testsuites, not tests.

You can understand the suites are running in a parallel way because the command prompt

say RUNS a lot of file together

even if you put more test in a file (a testsuite) they are runned sequentially both they are in a describe block or N describe block :

i make this experiment : every test wait 5000ms before ends

in a single describe block, it tooks more than 10 000 ms to ends

describe('describe block 1', () => {

    test("1", async () => {
        await new Promise(resolve => setTimeout(resolve, 5000));
        expect(1).toBe(1)
    })

    test("2", async () => {
        await new Promise(resolve => setTimeout(resolve, 5000));
        expect(1).toBe(1)
    })

})

splitting in two describe block,

describe('describe block 1', () => {

    test("1", async () => {
        await new Promise(resolve => setTimeout(resolve, 5000));
        expect(1).toBe(1)
    })


})

describe('describe block 2', () => {

    test("2", async () => {
        await new Promise(resolve => setTimeout(resolve, 5000));
        expect(1).toBe(1)
    })

})

nothing changed

splitting in two different files, they runs in a parallel way (there are 3 seconds of overhead)

but i don't like so much to put one test in one file , because it's very time consuming. what i'm doing wrong?

thanks.

[octalmage]

@andreabisello https://github.com/facebook/jest/issues/6957#issuecomment-473374831

[andreabisello]

as suggestede by @octalmage , test.concurrent looks what we needed.
this is not well documented (what will happens in a describe block with some test.concurrent and some other test ? what happens if the test.concurrent are mixed in order with other normal test?)

i suggest to open a "parallelization" section in "guide", due parallelization is a good topic

anyway i make a try , webstorm looks confused about the execution time required, but it looks working as espected : there is a test that will wait 6 seconds and other 3 tests wait less, the total execution time is 6 seconds because i have 12 cpu and 4 test are runs in parallel.

[mgwidmann]

The above example using test.concurrent does not have any method for running a single test. This is an issue for me using jest for integration testing since I want all of them to run concurrently when the whole suite is run but when I'm testing/developing I want to run just one at a time. Instead I have to remove test.concurrent for all tests and do test.only for the ones I want to run. Going to test.concurrent makes it a pain to run a single test, is there a workaround I'm not doing? I've tried doing just test.concurrent.only on the one I want to run and while it "skips" the other tests, I still see output from their execution (side effect of running the test still exists).