Rules_nodejs: Missing @npm/:node_modules/ input files because repo rule isn't being triggered

[thundergolfer]

🐞 bug report

Affected Rule

The issue is caused by the rule: yarn_install

Is this a regression?

Don't think so.

Description

For some reason when I do bazel build //... or bazel test //... I get a bunch of failures because my yarn_install(name = "npm" ...) isn't working and thus I'm getting a bunch of "missing input files" that look like:

bazel test //...
--
  | INFO: Analyzed 38 targets (38 packages loaded, 193 targets configured).
  | INFO: Found 36 targets and 2 test targets...
  | ERROR: missing input file '@npm//:node_modules/@types/eslint-visitor-keys/index.d.ts'
  | ERROR: missing input file '@npm//:node_modules/@types/json-schema/index.d.ts'
  | ERROR: missing input file '@npm//:node_modules/@types/lodash/add.d.ts'
  | ERROR: missing input file '@npm//:node_modules/@types/lodash/after.d.ts'
  | ERROR: missing input file '@npm//:node_modules/@types/lodash/ary.d.ts'
  | ERROR: missing input file '@npm//:node_modules/@types/lodash/attempt.d.ts'

If I do `bazel sync` and then try `bazel build //...` again things work. ## 🔬 Minimal Reproduction It's difficult to reproduce locally (I've only had it fail once) but it's reliably failing in my CI here: https://buildkite.com/thundergolfer-inc/example-bazel-monorepo/builds/202#b2a69efe-7e3e-4a48-a95e-bd0c37f08ee8 The branch is https://github.com/thundergolfer/example-bazel-monorepo/tree/jonathon/fix-node-build-in-CI and the node typescript is in `frontend/`. As said above, I ssh'd onto the failing CI instance and manually ran `bazel sync` and then ran `build //...` and it succeeded. ## 🔥 Exception or Error

...
hundreds of other ERROR: lines
...

  | ERROR: missing input file '@npm//:node_modules/react-dom/umd/react-dom.production.min.js'
  | ERROR: missing input file '@npm//:node_modules/react-dom/umd/react-dom.profiling.min.js'
  | ERROR: missing input file '@npm//:node_modules/react-dom/unstable-native-dependencies.js'
  | ERROR: /var/lib/buildkite-agent/builds/buildkite-i-09aaa116bea63be4f-1/thundergolfer-inc/example-bazel-monorepo/frontend/src/demo/components/Snackbar/BUILD.bazel:5:1 509 input file(s) do not exist
  | INFO: Elapsed time: 7.517s, Critical Path: 0.08s
  | INFO: 0 processes.

## 🌍 Your Environment **Operating System:** Happens on Mac OSX Mojave very rarely and Amazon Linux 2 (CI). **Output of `bazel version`:**

  
Build label: 2.0.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Dec 19 12:30:18 2019 (1576758618)
Build timestamp: 1576758618
Build timestamp as int: 1576758618
  

Rules_nodejs version:

(Please check that you have matching versions between WORKSPACE file and @bazel/* npm packages.)

  
1.0.1
  

Anything else relevant?

[alexeagle]

I think we've seen this flake with yarn on Mac in the past, but I don't think I've seen it on linux.
It's strange on that buildkite log that we don't see any output from running yarn at all. Seems like you say, that Bazel thinks the repository rule is already up-to-date. Is there any FS caching going on here? I don't see anything in your config files that seems suspicious.

[thundergolfer]

Is there any FS caching going on here? I don't see anything in your config files that seems suspicious.

I don't think so. My CI setup on this project is pretty janky, but it's straightforward. This problem presents on a fresh instance. Most of the time my CI instances are brought up fresh because I don't keep a long-running pool around.

It's not uncommon for network issues to leave external bzl repos in a bad state, but I don't think that's happening here. Would have to look into more though.

[alexeagle]

This thread is kinda similar
https://github.com/bazelbuild/rules_nodejs/issues/1527#issuecomment-584798369
I wonder if adding the managed_directories = {"@npm": ["3rdparty/typescript/node_modules"]} to your workspace() declaration makes a difference?

[sgammon]

i'm getting caught by this locally and in CI, generally on targets unrelated to rules_nodejs, which are (for some reason) pulling in their dependencies from the @npm//:node_modules/<my_workspace>/some/path/here.js-style paths.

_EDIT:_ also worth saying, at least in my case, it is usually solved by a bazel clean --expunge and re-build using the same invocation.

[sgammon]

okay, this just became a rather critical issue. it's happening reliably when ibazel detects changes in a target and tries to rebuild. in my case, it's failing (presumably) while trying to access postcss which is being run through a nodejs_binary rule.

here's the output i get (uploaded as screenshot because the colorized text is much easier to read):

what's weird is, it's touching a bunch of Java stuff under node_modules, which feels inappropriate, but perhaps that's just a side-effect of using rules_nodejs?

[sgammon]

@alexeagle / @gregmagolan this issue is making Bazel completely unusable for anyone who so much as mentions rules_nodejs

are we going to fix this, or are we going to not care about this project / external users that depend on it?

[DavidANeil]

This is a regression introduced somewhere between 0.38.1 and 1.1.0. Based on this issue the bad side can be further restricted to 1.0.1

[sgammon]

it's breaking java builds lol wtf

[sgammon]

it happens intermittently. re-running the build sometimes works. cleaning seems to help.

[thundergolfer]

It's breaking java...

I've also weirdly seen Java stuff in the errors. The .jars inside nodule_modules folders is odd.

[sgammon]

@thundergolfer it seems to be symlinking all project targets under node_modules/

[sgammon]

ha ha okay that has to be a joke, it's trying to load bazel_tools/ from node_modules/, and can't find it, and is failing, on something that _does not even touch rules_nodejs_

pardon my, ahem, attitude, but guys, you're blocking a critical project on this, and this issue has seen no activity from Google for ~two weeks.

can someone please help? the only way to fix this for us (reliably) is a bazel clean --expunge which is adding immense time to our builds. this error is unavoidable when using ibazel (as well), and occurs intermittently for no apparent reason when changes are detected.

[sgammon]

cc / @alexeagle, @gregmagolan - if you can't repro i can work with you to converge on a sample

[alexeagle]

Yes please, a repro is needed here, I haven't seen this happen.

it looks to me like there is some wires crossed. you have external/npm/covidmap/external/bazel_tools which looks like a combination of the node_modules layout external/npm and your project (I assume 'covidmap' is the name of your workspace) covidmap/external/bazel_tools

How is that path appearing as if it's installed from npm? is there actually a node_modules/covidmap directory? It feels like there's an undesirable interaction between bazel's repository layout and the npm installed one.

[sgammon]

@alexeagle yes, covidmap is the workspace (it's over here)

the folder does exist

i'm installing rules_nodejs via this code in my WORKSPACE, not via package.json/npm:

## NodeJS
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
    name = "build_bazel_rules_nodejs",
    sha256 = "2eca5b934dee47b5ff304f502ae187c40ec4e33e12bcbce872a2eeb786e23269",
    urls = ["https://github.com/bazelbuild/rules_nodejs/releases/download/1.4.1/rules_nodejs-1.4.1.tar.gz"],
)

load("@build_bazel_rules_nodejs//:index.bzl",
     "node_repositories",
     "yarn_install")

node_repositories(
    package_json = ["//:package.json"],
    node_version = "10.13.0",
    yarn_version = "1.12.1")

yarn_install(
    name = "npm",
    package_json = "//:package.json",
    yarn_lock = "//:yarn.lock")

load("@npm//:install_bazel_dependencies.bzl",
     "install_bazel_dependencies")

install_bazel_dependencies()

i'll try and be available tomorrow for some debugging on this

[nazariyv]

Not sure how relevant this is, but I get this error when I try to serve yarn serve in examples/react_webpack after successfully building (bazelisk build //...) in that directory with bazelisk (picks up bazel version 2.2.0). However, if I checkout from this commit c5d09095ba4653b4cd5837360e13ff2b105ba3be, all works fine.

[Toxicable]

I've seen this error a bunch.
We only run on Linux, never seen it in CI only locally.
It usually happens when im upgrading or changing our npm deps which I do with a globally installed yarn
We'll get an error like

ERROR: missing input file 'external/npm/node_modules/braces/lib/compile.js', owner: '@npm//:node_modules/braces/lib/compile.js'
ERROR: missing input file 'external/npm/node_modules/braces/lib/constants.js', owner: '@npm//:node_modules/braces/lib/constants.js'

Which would indicate that the file has been moved or deleted but the repository rule hasn't been rerun

workspace(
    name = "<SNIP>",
    managed_directories = {"@npm": ["node_modules"]},
)
<SNIP>
yarn_install(
    name = "npm",
    data = [ "//:.npmrc",],
    included_files = [  <SNIP>   ],
    package_json = "//:package.json",
    quiet = True,
    yarn_lock = "//:yarn.lock",
)

I usually just do a npx bazel clean --expunge and carry on with my work, but ill see if I can narrow it turn further

[Toxicable]

Ok just managed to reproduce it in our repository:

1. checkout to master.
2. run npx bazel test ...
3. wait for it to start running tests - all repository rules have been run
4. check to some other branch
5. use yarn to remove a module or some other modification to your node_modules
6. checkout to master
7. run npx bazel test ...
8. :

ERROR: missing input file 'external/npm/node_modules/picomatch/lib/.DS_Store', owner: '@npm//:node_modules/picomatch/lib/.DS_Store'
ERROR: missing input file 'external/npm/node_modules/mkdirp/examples/pow.js', owner: '@npm//:node_modules/mkdirp/examples/pow.js'
ERROR: missing input file 'external/npm/node_modules/mkdirp/test/chmod.js', owner: '@npm//:node_modules/mkdirp/test/chmod.js'
ERROR: missing input file 'external/npm/node_modules/mkdirp/test/clobber.js', owner: '@npm//:node_modules/mkdirp/test/clobber.js'
ERROR: missing input file 'external/npm/node_modules/mkdirp/test/mkdirp.js', owner: '@npm//:node_modules/mkdirp/test/mkdirp.js'
ERROR: missing input file 'external/npm/node_modules/mkdirp/test/opts_fs.js', owner: '@npm//:node_modules/mkdirp/test/opts_fs.js'

My guess is that since your yarn.lock has not changed changed, yarn_install dosen't rerun and update your node_modules on disk, however you have modified your node_modules externally

[alexeagle]

Ah! I reproduced this yesterday. It looks to me that the linker is running outside of a sandbox. That means that it creates a node_modules/<workspace_name> symlink in your source tree!
It's the normal situation on Windows, which has no sandbox, but I think I broke it on Mac/Linux with the same root cause as #1803 - the typescript compiler is both linked and is a worker and that's spawning things in this way that pollutes the environment and consumes resources.

In my case --worker_sandboxing would prevent it, I think the action that did the linking was a tsc_wrapped

https://github.com/bazelbuild/rules_nodejs/pull/1797 is the principled fix I think