Jest: Jest spends half its time importing jest-snapshot even when it isn’t used

[andersk]

🐛 Bug Report

I did some profiling on a directory of 500 empty tests and discovered that Jest is spending 50% of its time importing jest-snapshot into every test environment. Then I wrote a quick and dirty patch that rips jest-snapshot out of jest-jasmine2 and jest-runtime, and observed that it makes my tests run twice as fast (39.706s → 19.941s)!

Obviously we can’t actually rip jest-snapshot out, but if we could import it lazily, we could get a large speedup on projects with many small test files that don’t use snapshot testing.

To Reproduce

In an empty directory:

mkdir __tests__
for i in {000..499}; do echo "test(\"test $i\", () => {});" > __tests__/test$i.js; done
echo '{ "testEnvironment": "node" }' > jest.config.json
jest --runInBand

(Running tests serially with --runInBand gave me much more stable benchmark results.)

Expected behavior

Jest should be faster! :slightly_smiling_face:

Link to repl or repo (highly encouraged)

https://github.com/andersk/500-empty-jest-tests

envinfo

  System:
    OS: Linux 5.5 NixOS 20.03 (Markhor) 20.03pre212208.8130f3c1c2b (Markhor)
    CPU: (12) x64 Intel(R) Core(TM) i7-10710U CPU @ 1.10GHz
  Binaries:
    Node: 12.15.0 - ~/.nix-profile/bin/node
    Yarn: 1.22.0 - ~/.yarn/bin/yarn
    npm: 6.13.4 - ~/.nix-profile/bin/npm
  npmPackages:
    jest: ^25.1.0 => 25.1.0 

[SimenB]

Hey @andersk, thanks for digging into this! We do most of our importing lazily, but the modules that are evaluated inside the user's sandbox (like jest-snapshot) are explicitly blacklisted from this:
https://github.com/facebook/jest/blob/017264f6730d7e99fe0f054d799574e47a802c08/scripts/build.js#L43

For some background, you can read through #3786, tl;dr being that users might mess with the globals (like Array.prototype) and that should never affect Jest's internals. And we need to evaluate these modules in the user's sandbox so instanceof works correctly etc.

---

That said, the speedup you're seeing is _very_ tempting, so I think we should definitely investigate this. I wonder if making the implementations of the matchers and snapshot state lazy would help - e.g. only setting up serializers, state etc if snapshot matchers are actually called. I'll try to find some time digging into this, thank you for a short and sweet reproduction!

[andersk]

Yeah, ignoring the blacklist for @babel/plugin-transform-modules-commonjs similarly halves the running time on this benchmark:

--- a/scripts/build.js
+++ b/scripts/build.js
@@ -143,7 +143,8 @@ function buildFile(file, silent) {
       options.plugins.push(
         require.resolve('./babel-plugin-jest-native-globals')
       );
-    } else {
+    }
+    {
       options.plugins = options.plugins.map(plugin => {
         if (
           Array.isArray(plugin) &&

So now I’m curious, what exactly does this blacklist accomplish in the way of preventing test code from messing with Jest internals? It means that lots of Jest code gets loaded before the test code has a chance to run—but the loaded code still refers to globals that the test code could interfere with, so how does that help?