Jest: Ability to run tests within a file in a random order

[k-shan]

Do you want to request a feature or report a bug?
Feature

There have been cases where a test passes but when reordering the test in a file it actually fails. To prevent this false positive it would beneficial to randomize the order of tests within a specific file.

Jasmine has enabled this (https://jasmine.github.io/2.8/node.html#section-22), would it be possible to expose a configuration to specify the order or just randomize it by default?

[cpojer]

To isolate tests in Jest, you can use jest.resetModules() and then re-require modules for each individual tests. If you are strict about this, you won't need to randomize tests, and we aren't considering this feature for inclusion at this point.

[felixc]

Thanks for the pointer @cpojer, but as per https://github.com/facebook/jest/issues/3236 it seems that this isn't actually possible when using ES6 import syntax. Is there anything in the works for that? I suspect increasing numbers of folks will be trying out the nice new imports...

[cpojer]

Right, you'll have to use require calls inside of your tests, but the rest of your code can still use ES modules.

If you create your own ES module plugin that allows imports within functions (instead of only on the top level), you can use ES imports there as well. It's a small change to the flavor of babel that you are using, and would be limited to tests.

[thisismydesign]

In general, this would be a very useful feature that can root out many issues. I don't think the proposed workaround is realistic for existing bigger code bases. So +1 for this feature.

Edit: I haven't tested this myself yet but dynamic imports should be possible with babel-plugin-dynamic-import-node as shown here: https://github.com/facebook/jest/issues/2567#issuecomment-344827804

[theblang]

@cpojer Exposing the Jasmine 2 random config would actually be super helpful for us because we are using jest-image-snapshot along with Jest to write Puppeteer screenshot and integration tests, so being able to run a suite in random order would really help root out tests that might pass in a certain order but fail in isolation. I know that unit tests should never behave that way and thus it'd be a code smell in that class of tests, but Jest is also working great for us as a test runner for screenshot and integration tests where that behavior is more likely.

[SimenB]

IIRC the the randomizing implementation was ripped out of jest-jasmine2

[theblang]

@SimenB Do you know if that is something that would be considered to be put back in (i.e. would it be worth me trying to do a PR for)?

[jeysal]

@theblang jest-jasmine2 is going to be phased out, jest-circus is planned to become the default in Jest 25. As for adding randomization to jest-circus, I personally don't think this is a good solution for discovering order-dependent tests, plus it would have to be opt-in or a huge breaking change.

[SimenB]

I wonder if it makes sense as a feature for jest-circus the agnostic test runner, but not necessarily something jest the test framework does?

We have guarantees today about execution order within the files, and I highly doubt we'll break those, so it'd have to be opt-in, yeah.

[jeysal]

Maybe there's some other arguments I didn't consider, but I think in the case of order-dependent tests randomization IMO makes the situation worse. I'd rather notice the inter-dependency when reordering some tests than have them run in an order that happens to pass a few times locally and on CI and then once it's in master on the 5th run they randomly break. The proper solution to discovering tests depending on order would be to run all possible orders, which is obviously not feasible.

[felixc]

As an opt-in feature it would still be hugely helpful!

Since yes, running all possible orders is not feasible, I'd be curious to hear about what other strategies folks have other than randomization for discovering these bugs in their tests.

[theblang]

@felixc What I'm about to suggest isn't very pertinent to the ticket since suites are already run in a random order, but one thing we do with our screenshot tests (which utilize Puppeteer and jest-image-snapshot) is close and reopen Puppeteer between each suite run. Then at least you are getting clean client state between each suite run. Before Puppeteer we were using CasperJS without a proper test runner and would use the same instance for all the suites, which could get really annoying when a test change in one suite affected an entirely different suite. And that would happen a fair amount since the suite order wasn't random either and client state could linger between suites.

@jeysal The reason that randomization helps is that the screenshot test causing the problem will be discovered much sooner, potentially even during the development of the feature branch that the problematic test is written in if your CI tests are running for branches. Then the test can hopefully be addressed closer to writing the feature that it was initially for, and with the various state and code involved fresh on the mind. As it stands right now, intermittent failures can collect for a long time then creep up on someone down the road when they make a change that somehow disturbs the order and exposes the problematic test.

I definitely want to highlight the screenshot word, because this is mostly an issue with doing integration / screenshot tests where there's a lot of state, side effects, and other things going on, and where you can't easily reset state between each test. It's easy to unknowingly mutate something that lingers and affects other tests when writing this class of tests.

[SimenB]

test.each(randomize([
  ['title 1', ()=> { setup(); return screenshot() }],
  ['title 2', ()=> { setup2(); return screenshot() }],
]))('%s', (_title, func) => func());

Written in freehand on a phone, so excuse syntax/API errors.

Would something like that work you?

[alex-hall]

+1 for the ability to run tests in a random order.

I understand best practices like clearing mocks between tests and such, but i've run into a lot of test pollution or even tests accidentally passing due to certain data being created (persistent stores like local storage comes to mind). Randomizing tests helps engineers not be bitten by test number 3 creating data/state that test 1034 relies upon and having no real way of tracking down what's going on.

I'd be interested in taking a stab at a PR with the idea that the whole thing would be opt in and wouldn't break any existing contracts/tests.

I'd also like to propose that if a random run were to be accepted that it also be shipped with a deterministic seed value so you could run a bisect over the suite to figure out the minimum number of tests that would re-produce the problem.

[mockdeep]

Likewise. RSpec in Ruby has randomized test order and it has helped us find a lot of tests that inadvertently impacted others for one reason or another. When I go about refactoring Jest specs I start to come across these sorts of issues where the test is passing when it shouldn't be due to previous tests mistakenly modifying some global state. When RSpec runs randomized, it prints out the seed which you can use to reproduce an issue and fix it.

[theblang]

@cpojer @jeysal @SimenB Just wanted to see if there's a possibility of this feature being reconsidered after the amount of interest that has been expressed, and the benefits it could bring to the visual and integration testing domains (for which some nice tools exist that make use of Jest, like jest-image-snapshot from AmEx).

[jeysal]

I am still of the opinion that even as an opt-in this would do more harm than good across all the users that would enable it, based on the reason I stated above.
If I really really wanted this, I'd probably enable jest-circus (see docs) and write a custom event handler that does what it tells you not to do: on finish_describe_definition shuffle (mutate) (currentDescribeBlock.children.slice(-1)[0] || currentDescribeBlock).tests (and .children to also randomize order of describe blocks).
I think it goes without saying that this must be extremely important to your project to warrant relying on things that could easily break in the future - I am already hating myself for giving tips on how to best do this :sweat_smile: but I empathize with the situation some of you may be in and would rather you don't go with an even more fragile approach.

[gregplaysguitar]

@jeysal I don't understand how randomising the run order would make things more fragile - on the contrary, doesn't it force our tests to be more robust, to ensure they don't break when the run order changes?

The key issue IMO is _reproducibility_ - i.e., when the tests do fail due to ordering, there needs to be a way to reproduce that order. @alex-hall's suggestion hits the nail on the head there:

if a random run were to be accepted that it also be shipped with a deterministic seed value so you could run a bisect over the suite to figure out the minimum number of tests that would re-produce the problem.

My 2c, I would imagine it working like this: a new --random flag added which randomises the test runs, and optionally accepts a seed, i.e.

jest --random
jest --random {SEED}

The seed would be auto-generated if not provided, and if the tests fail, the seed value used would be part of the output - so they could easily be reproduced.

[SimenB]

_If_ we were to ship this, it would 100% be with a seed that can be set and that'd be prominently printed if tests fail.

I think it's far less likely to cause hard to track down errors than say test.concurrent which we have today. Those are way more nondeterministic than a seeded "randomization" of test order.

@alex-hall with no promises on it being merged (although I personally would use such a feature at work 😅), are you still open to taking a stab at this? At least an initial exploratory try at an implementation might reveal if it slots nicely into Circus or not.

[thisismydesign]

@jeysal You sound like you take this as some experimental idea. It's not. It's been around as practically a standard in many other frameworks. In most places I've seen it's considered a code smell not to run tests in random order. There's no question about the validity of this feature.

To address your specific concern, most order-dependent tests will fail right away or within the first few tries and the devs will learn not to write them anymore. In the rare cases, it can root out subtle edge cases and bugs. I was never unhappy to have found them.

[alex-hall]

@SimenB lll do some jest archaeology and take a look this weekend.

[SimenB]

Wonderful, thanks! Feel free to open an early PR with hacky WIP code for feedback as you go. 🙂
Also note that just adding it to jest--circus is fine (maybe even preferred) - we'll be phasing out jasmine (although it'll probably stick around for a long time yet).

[langri-sha]

This would be an incredibly useful feature! I wouldn't also mind if this could be implemented by simply running tests within a module concurrently, because it would help expose other kinds of problems as well.