Jest: Is jest fully scanning node_modules and then ignoring its content by default ?

[fungiboletus]

⚡ Performance improvement ?

Jest has been slow to start on my computer, which has a relatively slow filesystem (windows ntfs and a few layers of encryption).

Reading the default configuration, I'm wondering whether jest scans every file in the <rootDir> and then ignores the ones in node_modules. I replaced my roots with <rootDir>/src/ instead of <rootDir> and it feels faster.

To Reproduce

• Develop a javascript project with a lot of npm dependencies with many files on a slow filesystem.
• Run jest with no config.
• Run jest with roots: ['<rootDir>/src/'] in its config.

Expected behavior

Jest should scan the files to find the tests relatively fast. But by default I'm not sure it's the case.

envinfo

System:
    OS: Windows 10 10.0.18363
    CPU: (8) x64 Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz
  Binaries:
    Node: 12.13.0 - C:\Program Files\nodejs\node.EXE
    npm: 6.12.0 - C:\Program Files\nodejs\npm.CMD
  npmPackages:
    jest: ^25.4.0 => 25.4.0

[fungiboletus]

https://github.com/facebook/jest/blob/32aaff83f02c347ccd591727544002490fb4ee9a/packages/jest-core/src/SearchSource.ts#L139-L144

I'm wondering whether this._context.hasteFS.getAllFiles() returns all files in <rootDir> including the files from node_modules.

[sibelius]

what is the right configuration in monorepo to improve this?

[jufemaiz]

I'm certain this is what I'm seeing right now.

ps aux gives me the following file scan (docker linux):

find /app -type f ( -iname *.snap -o -iname *.js -o -iname *.json -o -iname *.jsx -o -iname *.ts -o -iname *.tsx -o -iname *.node )

This is causing around a 30s run for a single test file in /app/src/test/simple.test.js:

describe('simple', () => {
  it('should be sane', () => {
    expect(false).not.toBe(true);
  });
});

[lexaurin]

Just figured out how it works and it's a sad story.. 😞

First it checks if you have watchman installed..

https://github.com/facebook/jest/blob/ac73de8989ca06675858b4c8b4f3e6f864a071f0/packages/jest-haste-map/src/index.ts#L738

and if not...

https://github.com/facebook/jest/blob/ac73de8989ca06675858b4c8b4f3e6f864a071f0/packages/jest-haste-map/src/crawlers/node.ts#L233

it either uses native find (your usecase @jufemaiz ) for all files in rootDir!!!

https://github.com/facebook/jest/blob/ac73de8989ca06675858b4c8b4f3e6f864a071f0/packages/jest-haste-map/src/crawlers/node.ts#L236

or if find is not available, on Windows for example if I understand correctly, then it travers recursively all directories in Node.js code.. Seems like @fungiboletus case

[lexaurin]

And the worst part is that all this Jest do completely silently.. No message like "Hey install watchman" or "Warning: Native find is not available on your system"..

[sibelius]

is it much faster using watchman?

[lexaurin]

In our project, with watchman, startup time is almost instantaneous.
In comparison without watchman, it takes 10s to start first test - find output includes even files from node_modules so in our case it is more than 8MB just this file list..

[fungiboletus]

Sounds like having a warning on the console output would be a good improvement.

[jufemaiz]

Definitely @fungiboletus !

[lexaurin]

Also worth mentioning that last saved index must be available in jest cache and cache key must be valid otherwise it builds index again.

And completely agree that it should be evident from CLI that indexing is happening and in documentation I think it should be stated how to avoid it.

From jest-haste-map code:

 * The HasteMap is created as follows:
 *  1. read data from the cache or create an empty structure.
 *
 *  2. crawl the file system.
 *     * empty cache: crawl the entire file system.
 *     * cache available:
 *       * if watchman is available: get file system delta changes.
 *       * if watchman is unavailable: crawl the entire file system.

[beckerei]

Is the startup time going only up when you are actually running in watch mode or is it sufficient to have watchman installed?

We have a quite big mono repo with > 3k tests and quite some dependencies. It takes 20-40s for jest to actually start a test run.
Tests itself have expected speed.