Jest: Speedup slow e2e tests

[thymikee]

Here's a totally unscientific list of our slowest tests (times doesn't really matter, it all depends on machine, but if you're curious this is on MBP 2015, 8 cores, 2nd run with warm cache):

 e2e/__tests__/jasmineAsync.test.js (36.086s)
 e2e/__tests__/snapshot.test.js (56.125s)
 e2e/__tests__/toMatchInlineSnapshot.test.js (58.722s)
 e2e/__tests__/errorOnDeprecated.test.js (83.333s)
 e2e/__tests__/failures.test.js (44.999s)
 e2e/__tests__/toMatchSnapshot.test.js (88.268s)
 e2e/__tests__/coverageReport.test.js (96.158s)
 e2e/__tests__/onlyChanged.test.js (98.087s)
 e2e/__tests__/coverageThreshold.test.js (45.632s)
 e2e/__tests__/transform.test.js (60.83s)
 e2e/__tests__/mockNames.test.js (36.544s)
 e2e/__tests__/globals.test.js (43.55s)
 e2e/__tests__/each.test.js (33.104s)
 e2e/__tests__/findRelatedFiles.test.js (36.346s)
 e2e/__tests__/customReporters.test.js (45.161s)
 e2e/__tests__/globalTeardown.test.js (33.038s)
 e2e/__tests__/testPathPatternReporterMessage.test.js (30.985s)
 e2e/__tests__/filter.test.js (20.721s)
 e2e/__tests__/env.test.js (24.019s)
 e2e/__tests__/globalSetup.test.js (27.017s)
 e2e/__tests__/stackTrace.test.js (27.964s)
 e2e/__tests__/detectOpenHandles.js (24.307s)
 e2e/__tests__/config.test.js (24.529s)
 e2e/__tests__/toThrowErrorMatchingInlineSnapshot.test.js (26.213s)
 e2e/__tests__/timeoutsLegacy.test.js (23.311s)
 e2e/__tests__/watchModePatterns.test.js (21.548s)
 e2e/__tests__/toThrowErrorMatchingSnapshot.test.js (21.85s)
 e2e/__tests__/jestChangedFiles.test.js (36.407s)
 e2e/__tests__/console.test.js (22.244s)
 packages/jest-runtime/src/__tests__/runtime_cli.test.js (23.2s)
 e2e/__tests__/setupFilesAfterEnvConfig.test.js (19.104s)
 e2e/__tests__/testFailureExitCode.test.js (19.079s)
 e2e/__tests__/multiProjectRunner.test.js (104.954s)

The way to speed them up is to simply split them into smaller chunks – maybe 2, maybe 3 or 4, it all depends – to better use parallelization.

Example:
onlyChanged.test.js -> onlyChangedGit.test.js + onlyChangedHg.test.js

If anyone feels adventurous enough to split any of the tests, let us know in the comments and send a PR :)

[nicollbryla]

Hi, I would like to give it a shot. Do you have more suggestions? Or should I trust my own intuition?

[jeysal]

Hi, we would love to accept your contribution! As far as splitting tests goes, looking at the existing describe/it/test blocks to see which belong together and using your intuition to determine where a split could be made should work well - there's often multiple valid points to split anyway ;)

[vitkarpov]

Here's the sorted list, probably it's better to start from the "slowest" items:

e2e/__tests__/multiProjectRunner.test.js (104.954s)
e2e/__tests__/onlyChanged.test.js (98.087s)
e2e/__tests__/coverageReport.test.js (96.158s)
e2e/__tests__/toMatchSnapshot.test.js (88.268s)
e2e/__tests__/errorOnDeprecated.test.js (83.333s)
e2e/__tests__/transform.test.js (60.83s)
e2e/__tests__/toMatchInlineSnapshot.test.js (58.722s)
e2e/__tests__/snapshot.test.js (56.125s)
e2e/__tests__/coverageThreshold.test.js (45.632s)
e2e/__tests__/customReporters.test.js (45.161s)
e2e/__tests__/failures.test.js (44.999s)
e2e/__tests__/globals.test.js (43.55s)
e2e/__tests__/mockNames.test.js (36.544s)
e2e/__tests__/jestChangedFiles.test.js (36.407s)
e2e/__tests__/findRelatedFiles.test.js (36.346s)
e2e/__tests__/jasmineAsync.test.js (36.086s)
e2e/__tests__/each.test.js (33.104s)
e2e/__tests__/globalTeardown.test.js (33.038s)
e2e/__tests__/testPathPatternReporterMessage.test.js (30.985s)
e2e/__tests__/stackTrace.test.js (27.964s)
e2e/__tests__/globalSetup.test.js (27.017s)
e2e/__tests__/toThrowErrorMatchingInlineSnapshot.test.js (26.213s)
e2e/__tests__/config.test.js (24.529s)
e2e/__tests__/detectOpenHandles.js (24.307s)
e2e/__tests__/env.test.js (24.019s)
e2e/__tests__/timeoutsLegacy.test.js (23.311s)
packages/jest-runtime/src/__tests__/runtime_cli.test.js (23.2s)
e2e/__tests__/console.test.js (22.244s)
e2e/__tests__/toThrowErrorMatchingSnapshot.test.js (21.85s)
e2e/__tests__/watchModePatterns.test.js (21.548s)
e2e/__tests__/filter.test.js (20.721s)
e2e/__tests__/setupFilesAfterEnvConfig.test.js (19.104s)
e2e/__tests__/testFailureExitCode.test.js (19.079s)

@nicollbryla @SimenB hey guys, have you already taken some of these files?

[SimenB]

I haven't (and won't). Feel free to pick some!

[vitkarpov]

Gotcha, thanks! (pinged you since you left 👀, sorry if it's irrelevant)

[vitkarpov]

Guys, probably you can elaborate a little bit about the following.

I run jest within e2e folder (just run ../jest since it's pointing to jest-cli) and get a bunch of errors like "it's not plain JavaScript, check that you're using babel plugin" but I see that package.json in e2e folder has the needed transform:

"transform": {
  "^.+\\.js$": "<rootDir>/../packages/babel-jest"
}

What am I missing?

[SimenB]

You should run Jest from the root. Just yarn jest e2e/__tests__/multiProjectRunner.test.js (or ./jest e2e/__tests__/multiProjectRunner.test.js) for instance

[thymikee]

Yep, I see. We could split the suits into smaller ones but the overall amount would increase so the actual running time shouldn't decrease dramatically.
https://github.com/facebook/jest/pull/7826#issuecomment-461485273

Splitting tests doesn't seem to help much. It's still good for faster feedback. I'll leave this open until we find a good way to speed them up :) (or close if we don't 😅)

[vitkarpov]

Guys, btw, I'm wondering which way is legacy: writeFiles or having an actual folder in e2e?

[thymikee]

I'd say neither. writeFiles was introduced to easily add more cases, so you don't have to manually create files.

[vitkarpov]

Gotcha, thanks! 👍 Basically, it means that writeFiles is a more convenient way and having actual files is a legacy one.

[pmansukhani]

Is this still open? I'm new to open source. Can I take this up?

[thymikee]

Sure, if you can make them faster, that's always welcome!

[Dhruvi16]

Hey! this looks interesting to me. Can I take this up?

[SimenB]

That'd be awesome, please do 🙂

[Dhruvi16]

Can anyone help me to figure out where to start?

[vitkarpov]

Can anyone help me to figure out where to start?

@Dhruvi16 I tried to split the tests up (even more) so we could benefit from parallelism but failed (it didn't work out much).

[dhuang612]

Hi,

Is this available to work on?

[shreyashbhatt]

Hey, can i work on it, if it is still open. It seems an interesting first issue for getting familiar with oss

[vitkarpov]

I'll open a bottle of champaign when it's done 🎉