Jest: Haste map is sometimes missing dotfiles

[lencioni]

🐛 Bug Report

At Airbnb we are using jest haste map directly in some of our projects. When toggling on the useWatchman option to true, we've noticed that it sometimes includes dotfiles (e.g. .eslintrc.js) and sometimes does not. Removing the haste map cache and running watchman watch-del-all seems to resolve the problem. I've been able to reproduce this bug sometimes by restarting my machine and running the program again.

I've read through some issues that seem to be somewhat relevant to this problemspace:

• https://github.com/facebook/jest/issues/7267
• https://github.com/facebook/jest/pull/3006
• https://github.com/facebook/watchman/issues/647

From that last issue on watchman, I suspect we maybe need to add glob_includedotfiles somewhere, maybe in this vicinity? https://github.com/facebook/jest/blob/9ffd368330a3aa05a7db9836be44891419b0b97d/packages/jest-haste-map/src/crawlers/watchman.ts#L98-L147

Maybe here: https://github.com/facebook/jest/blob/9ffd368330a3aa05a7db9836be44891419b0b97d/packages/jest-haste-map/src/crawlers/watchman.ts#L129

To Reproduce

Steps to reproduce the behavior:

We are calling HasteMap like this:

const haste = new HasteMap({
    name: 'dependencies',
    platforms: [],
    retainAllFiles: true,
    extensions: ['js', 'ts', 'jsx', 'tsx'],
    rootDir,
    roots: options.roots,
    maxWorkers: os.cpus().length,
    dependencyExtractor: require.resolve(path.join(__dirname, 'jestDependencyExtractor')),
    useWatchman: false,
    watch: false,
    ignorePattern: options.honorGitIgnore ? createIgnorer(rootDir) : undefined,
    console: {
      ...global.console,
      warn() {},
    },
  });

Our directories contain a mix of files, and some are named .eslintrc.js.

This could also be related to our ignorePattern setting here, but the caching seems suspect.

Expected behavior

The list of files should be consistent regardless of cache state.

envinfo

  System:
    OS: macOS 10.15.4
    CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
  Binaries:
    Node: 10.16.0 - ~/.nvm/versions/node/v10.16.0/bin/node
    Yarn: 1.22.4 - /usr/local/bin/yarn
    npm: 6.14.2 - ~/.nvm/versions/node/v10.16.0/bin/npm
  npmPackages:
    jest: ^25.5.4 => 25.5.4 

[grosto]

Cheers for a great bug report!

I tested and adding glob_includedotfiles: true seems to consistently retrieve dotfiles on macOS with Watchman. I will open a PR soon.

[vjpr]

Just bumped into this with pnpm. Pnpm uses node_modules/.pnpm/ to hold its dependencies. jest-haste-map ignores them, preventing them being included in the haste map which causes "SHA-1 missing" errors (https://github.com/facebook/metro/issues/330).

[vjpr]

@lencioni sometimes includes dotfiles (e.g. .eslintrc.js) and sometimes does not.

I've noticed this sporadic behaviour too and I've seen reports of similar sporadic behaviour too in metro with react-native.

The code below would explain this behaviour I think. Depending on whether the clock is available, if glob is not used, then you are not going to get any dotfiles or dot dirs included.

const relativeRoot = fastPath.relative(rootDir, root);
const query = clocks.has(relativeRoot) // Use the `since` generator if we have a clock available
  ? {
      expression,
      fields,
      since: clocks.get(relativeRoot)
    } // Otherwise use the `glob` filter
  : {
      expression,
      fields,
      glob
    };
const response = yield cmd('query', root, query);

[majames]

@vjpr is totally spot on! When a clock value is NOT present and, therefore, glob is used the Watchman query supplied will miss dot files and directories. I assume this is a bug as we'd want the same set of files to be produced for both queries (when all files have changed since the clock value).

@grosto is there any chance you could include a fix for this as well please?