Jest: Investigate speed regressions

[rogeliog]

💥 Regression Report

Speed regression in [email protected]

Last working version

Worked up to version:

[email protected]

Stopped working in version:

A big chunk of the regression was introduced with this PR https://github.com/facebook/jest/pull/5932

To Reproduce

Steps to reproduce the behavior:

I've been using a small repo, https://github.com/rogeliog/jest-benchmark(I can give you access if you need to) to run simple benchmarks across Jest versions.

Expected behavior

Test should run faster

Here are is progress that I've made on the speed regression investigation.

1. A big chunk of the speed regression was introduced here https://github.com/facebook/jest/pull/5932
2. I've been testing with https://github.com/rogeliog/jest-benchmark and the differences that I get between jest@22 and jest@23 are the following
   ------------------- Jest@22 ------------------------- 0m2.338s 0m2.317s ------------------- Jest@23 ------------------------- 0m6.068s 0m6.048s

Initial findings

1. In jest@22 https://github.com/facebook/jest/blob/v22.4.4/packages/jest-config/src/normalize.js#L128 returns null which causes https://github.com/facebook/jest/blob/v22.4.4/packages/jest-config/src/normalize.js#L130-L132 not to get executed.
2. In jest@23 https://github.com/facebook/jest/blob/master/packages/jest-config/src/normalize.js#L143 returns a path and https://github.com/facebook/jest/blob/master/packages/jest-config/src/normalize.js#L144-L146 gets executed.

3. If I comment out https://github.com/facebook/jest/blob/master/packages/jest-config/src/normalize.js#L144-L146 then I get the following stats.

   ------------------- Jest@22 -------------------------
   0m2.537s
   0m2.331s
   ------------------- Jest@dev -------------------------
   0m3.658s
   0m3.633s
   

4. I'm not to familiar the reasoning behind it, but it seems that since the original there were some comments about them returning different values https://github.com/facebook/jest/pull/5932#discussion_r179545920.

cc: @cpojer, @arcanis

[arcanis]

It seems to me the problem is that babel-jest is now always used to run the tests, is that correct? Whereas before it was only used if it was listed as a dependency in the root package.json.

I'm not sure what's the behavior that Jest wants to keep, but a possible solution might be to use the paths option from require.resolve (Node 8+, iirc) to specifically make the lookup relative to options.rootDir.

[SimenB]

We used babel-jest all the time previously as well, see this example with jest 22: https://github.com/SimenB/jest-22-babel

[JKillian]

We've found that switching from Jest 23.5 -> 23.6, upgrading babel-jest, and switching from Babel 6 -> Babel 7 has somehow caused our testing time to increase by about 30% when using --no-cache. It doesn't appear babel is the culprit because webpack builds and other tooling that uses babel are actually slightly faster now.

Still trying to narrow down what's causing the perf regression, haven't been able to figure out what jest (or other) package is causing the issue.

Any advice on tracking down the cause? I can try to build out a minimal repro if it would be useful.

[rickhanlonii]

@JKillian thanks for the info - typically the way to track this down is to run with a local copy of jest (there are instructions in CONTRIBUTING.md) and doing a git bisect to track down the PR responsible for the regression

[JKillian]

For what it's worth, turns out I misdiagnosed the cause of the problem. It was actually an upgrade from ts-jest 23.1.4 -> 23.10.5 which was causing the large slowdown and memory usage increase (babel and jest proper were red herrings). We found switching our jest config from:

    globals: {
      'ts-jest': {
        diagnostics: false,
      },
    },

to

    globals: {
      'ts-jest': {
        diagnostics: false,
        isolatedModules: true,
      },
    },

resolved the performance issues for us! Just posting the info here to absolve jest of the blame I put on it 😁 and to help anyone else if they have similar issues with ts-jest

[rickhanlonii]

Oh great find, thanks for following up @JKillian!

[nasreddineskandrani]

please check my comment here:
problem: https://github.com/facebook/jest/issues/6783#issuecomment-445560998
investigation: https://github.com/facebook/jest/issues/6783#issuecomment-445660665
cause: https://github.com/facebook/jest/issues/6783#issuecomment-446123323
solution: https://github.com/facebook/jest/pull/7518

I am sure 100% that the TestScheduler is creating the regression for the case i reported in the issue. I don't know yet if solving this will help with the run on all files => if you think i should isolate this in a specific issue let me know.

[massimonewsuk]

Is the advice to stick to older versions of Jest for now?