Yarn: node_modules is missing after successful yarn install

[NickHeiner]

yarn install exits successfully, but node_modules does not exist.

Repro steps

1. Clone https://github.com/nickheiner/camelot-web/tree/b6e1924a1aeb611ec4fc9e21d97597d6451234c5.
2.

λ yarn install --verbose
yarn install v1.5.1
verbose 0.471 Checking for configuration file "/Users/nickheiner/Code/camelot-web/.npmrc".
verbose 0.471 Checking for configuration file "/Users/nickheiner/.npmrc".
verbose 0.471 Found configuration file "/Users/nickheiner/.npmrc".
verbose 0.472 Checking for configuration file "/usr/local/etc/npmrc".
verbose 0.472 Checking for configuration file "/Users/nickheiner/Code/camelot-web/.npmrc".
verbose 0.472 Checking for configuration file "/Users/nickheiner/Code/.npmrc".
verbose 0.472 Checking for configuration file "/Users/nickheiner/.npmrc".
verbose 0.472 Found configuration file "/Users/nickheiner/.npmrc".
verbose 0.473 Checking for configuration file "/Users/.npmrc".
verbose 0.475 Checking for configuration file "/Users/nickheiner/Code/camelot-web/.yarnrc".
verbose 0.475 Checking for configuration file "/Users/nickheiner/.yarnrc".
verbose 0.475 Found configuration file "/Users/nickheiner/.yarnrc".
verbose 0.476 Checking for configuration file "/usr/local/etc/yarnrc".
verbose 0.476 Checking for configuration file "/Users/nickheiner/Code/camelot-web/.yarnrc".
verbose 0.476 Checking for configuration file "/Users/nickheiner/Code/.yarnrc".
verbose 0.476 Checking for configuration file "/Users/nickheiner/.yarnrc".
verbose 0.476 Found configuration file "/Users/nickheiner/.yarnrc".
verbose 0.477 Checking for configuration file "/Users/.yarnrc".
verbose 0.48 current time: 2018-03-10T22:32:40.308Z
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[##################################################################################################################################################################################################################################-] 902/904

λ echo $?
0

Expected: node_modules exists
Actual:

λ ls node_modules
ls: node_modules: No such file or directory

It is odd that the final package counter says 902/904.

Workaround: Delete yarn.lock and regenerate it by running yarn again. This time, node_modules exists, but of course any state you were persisting in yarn.lock is gone.

Versions

λ yarn -v
1.5.1
λ node -v
v9.8.0

macOS 10.13.3

I also tried on [email protected].

[rally25rs]

Are you able to get it into this broken state again? If so, can you share your yarn.lock file? I wonder if this is #5491 where if a package download times out then yarn will just exit with no error.

[NickHeiner]

Hi – following the repro steps in the OP consistently put me in the broken state. The lockfile you get from following the repro steps is here: https://github.com/NickHeiner/camelot-web/blob/b6e1924a1aeb611ec4fc9e21d97597d6451234c5/yarn.lock.

Thanks!

[rally25rs]

Tried cloning your repo:

~/Projects 🐒   git clone [email protected]:NickHeiner/camelot-web.git
Cloning into 'camelot-web'...
remote: Counting objects: 1115, done.
remote: Compressing objects: 100% (76/76), done.
remote: Total 1115 (delta 36), reused 38 (delta 12), pack-reused 1021
Receiving objects: 100% (1115/1115), 2.44 MiB | 2.65 MiB/s, done.
Resolving deltas: 100% (734/734), done.

~/Projects 🐒   cd camelot-web/

~/Projects/camelot-web (master) 🐒   git co b6e1924a1aeb611ec4fc9e21d97597d6451234c5
Note: checking out 'b6e1924a1aeb611ec4fc9e21d97597d6451234c5'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at b6e1924... Massive and sloppy update.

~/Projects/camelot-web ((b6e1924...)) 🐒   yarn
yarn install v1.5.1
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
✨  Done in 6.29s.

Seems to work ok for me. Odd that it would consistently fail for you.

I tried it with a clean cache too so that it would have to download packages:

~/Projects/camelot-web ((b6e1924...)) 🐒   yarn cache clean
yarn cache v1.5.1
success Cleared cache.
✨  Done in 41.87s.

~/Projects/camelot-web ((b6e1924...)) 🐒   rm -rf node_modules/

~/Projects/camelot-web ((b6e1924...)) 🐒   yarn
yarn install v1.5.1
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
✨  Done in 17.98s.

If you redirect the verbose output to a file, does it contain anything interesting? i.e. yarn --verbose > yarn.log should list every package its downloading.

...
verbose 0.341 Checking for configuration file "/Users/.yarnrc".
verbose 0.343 current time: 2018-03-13T01:43:09.495Z
[1/4] Resolving packages...
[2/4] Fetching packages...
verbose 0.914 Performing "GET" request to "https://registry.yarnpkg.com/autobind-decorator/-/autobind-decorator-2.0.0.tgz".
...

[NickHeiner]

Hmm, that's odd.

I just tried again, and the output of yarn --verbose is the same as what I put in the OP:

yarn install v1.5.1
verbose 0.476 Checking for configuration file "/Users/nickheiner/Code/camelot-web/.npmrc".
verbose 0.477 Checking for configuration file "/Users/nickheiner/.npmrc".
verbose 0.477 Found configuration file "/Users/nickheiner/.npmrc".
verbose 0.477 Checking for configuration file "/usr/local/etc/npmrc".
verbose 0.478 Checking for configuration file "/Users/nickheiner/Code/camelot-web/.npmrc".
verbose 0.478 Checking for configuration file "/Users/nickheiner/Code/.npmrc".
verbose 0.478 Checking for configuration file "/Users/nickheiner/.npmrc".
verbose 0.478 Found configuration file "/Users/nickheiner/.npmrc".
verbose 0.478 Checking for configuration file "/Users/.npmrc".
verbose 0.48 Checking for configuration file "/Users/nickheiner/Code/camelot-web/.yarnrc".
verbose 0.481 Checking for configuration file "/Users/nickheiner/.yarnrc".
verbose 0.481 Found configuration file "/Users/nickheiner/.yarnrc".
verbose 0.481 Checking for configuration file "/usr/local/etc/yarnrc".
verbose 0.481 Checking for configuration file "/Users/nickheiner/Code/camelot-web/.yarnrc".
verbose 0.481 Checking for configuration file "/Users/nickheiner/Code/.yarnrc".
verbose 0.481 Checking for configuration file "/Users/nickheiner/.yarnrc".
verbose 0.482 Found configuration file "/Users/nickheiner/.yarnrc".
verbose 0.482 Checking for configuration file "/Users/.yarnrc".
verbose 0.484 current time: 2018-03-13T03:32:52.502Z
[1/4] Resolving packages...
[2/4] Fetching packages...

After yarn cache clean, I have a longer output to yarn --verbose, but it still does not create a node_modules directory: https://gist.github.com/NickHeiner/384f0c277bd0f596ee820ae2e20e9cc6.

This line at the end is intriguing:

info There appears to be trouble with your network connection. Retrying...

[jishi]

I'm seeing similar behavior when running yarn in a docker build.

[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
There appears to be trouble with your network connection. Retrying...

It never logs step 4 or 5, and it seems to exit with code 0 for some weird reason. This only seem to happen for certain installations, and occasionally it goes away? Can't really give reproducable steps either because if the intermittent behavior (and our dependencies contains private packages)

[cecton]

Had the exact same issue than @jishi my company has sometimes issue with the network but that success code when exiting made the CI go to the next steps.

[NickHeiner]

I believe I'm seeing this issue again. This time, it's not repo-specific. When I do yarn info, the registry GET times out. This is an issue with my network connectivity that I've resolved separately. However, yarn should exit with a non-zero exit code in this case.

λ time yarn info @netflix/cl-logger --verbose
yarn info v1.12.3
verbose 0.384 Checking for configuration file "/Users/nheiner/code/tvui/.npmrc".
verbose 0.385 Found configuration file "/Users/nheiner/code/tvui/.npmrc".
verbose 0.385 Checking for configuration file "/Users/nheiner/.npmrc".
verbose 0.385 Found configuration file "/Users/nheiner/.npmrc".
verbose 0.385 Checking for configuration file "/usr/local/etc/npmrc".
verbose 0.386 Checking for configuration file "/Users/nheiner/code/tvui/.npmrc".
verbose 0.386 Found configuration file "/Users/nheiner/code/tvui/.npmrc".
verbose 0.386 Checking for configuration file "/Users/nheiner/code/.npmrc".
verbose 0.386 Checking for configuration file "/Users/nheiner/.npmrc".
verbose 0.386 Found configuration file "/Users/nheiner/.npmrc".
verbose 0.387 Checking for configuration file "/Users/.npmrc".
verbose 0.389 Checking for configuration file "/Users/nheiner/code/tvui/.yarnrc".
verbose 0.389 Found configuration file "/Users/nheiner/code/tvui/.yarnrc".
verbose 0.39 Checking for configuration file "/Users/nheiner/.yarnrc".
verbose 0.39 Found configuration file "/Users/nheiner/.yarnrc".
verbose 0.39 Checking for configuration file "/usr/local/etc/yarnrc".
verbose 0.39 Checking for configuration file "/Users/nheiner/code/tvui/.yarnrc".
verbose 0.4 Found configuration file "/Users/nheiner/code/tvui/.yarnrc".
verbose 0.401 Checking for configuration file "/Users/nheiner/code/.yarnrc".
verbose 0.401 Checking for configuration file "/Users/nheiner/.yarnrc".
verbose 0.401 Found configuration file "/Users/nheiner/.yarnrc".
verbose 0.401 Checking for configuration file "/Users/.yarnrc".
verbose 0.405 current time: 2019-01-21T21:01:11.444Z
verbose 0.434 Performing "GET" request to "https://artifacts.netflix.com/api/npm/npm-netflix/@netflix%2fcl-logger".

real    0m30.694s
user    0m0.568s
sys 0m0.122s

λ echo $?
0

For comparison, this is npm's behavior:

λ npm info @netflix/cl-logger
npm ERR! code ETIMEDOUT
npm ERR! errno ETIMEDOUT
npm ERR! syscall connect
npm ERR! network connect ETIMEDOUT 54.187.210.157:443
npm ERR! network This is a problem related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network settings.
npm ERR! network
npm ERR! network If you are behind a proxy, please make sure that the
npm ERR! network 'proxy' config is set properly.  See: 'npm help config'

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/nheiner/.npm/_logs/2019-01-21T21_00_49_118Z-debug.log

λ echo $?
1

[michelalbers]

I encountered this problem when my private registry from which I pulled one package was down. Since Yarn did not exit with an error status my docker build continued to run but no node_modules were present.

[jcward]

Same as @michelalbers here: my yarn.lock file specifies a local caching repo server (in my case, 10.0.1.5)

When I try to run yarn install when the local repo is unreachable (e.g. outside my office network), obviously the fetches fail... Technically that's correct behavior, even though these are standard packages that have simply been cached on the local repo server. Ideally it could withstand local-caching-repo-failures if it checked the integrity SHA1 against the same-named-package on the public server.

But back to the bug at hand: upon failing to contact that local repo server, yarn seems to crash and exit with 0, indicating success:

>yarn install --verbose && echo "Ok, yarn exited with 0 indicating success"
yarn install v1.13.0
verbose 0.14 Checking for configuration file ...<snip>
...<snip>
[1/4] Resolving packages...
[2/4] Fetching packages...
verbose 0.209 Performing "GET" request to "http://10.0.1.5:4873/glob/-/glob-7.1.6.tgz".
verbose 0.215 Performing "GET" request to "http://10.0.1.5:4873/inherits/-/inherits-2.0.4.tgz".
verbose 0.224 Performing "GET" request to "http://10.0.1.5:4873/@types%2fnode/-/node-12.12.17.tgz".
verbose 0.226 Performing "GET" request to "http://10.0.1.5:4873/typescript/-/typescript-3.7.3.tgz".
[############----] 12/16Ok, yarn exited with 0 indicating success
>

Notice that it simply died while printing [############----] 12/16 and the chained echo command was immediately executed. Even though no node_modules directory was created.

Workarounds:

1) In my case, I simply removed the yarn.lock file - not an ideal workaround from a version management perspective.

2) For me, another workaround was to edit the yarn.lock file and replace my local repo URL with the public one (obviously, this only works when missing cached public modules):

> sed -i 's,http://10.0.1.5:4873/,https://registry.yarnpkg.com/,g' yarn.lock
> yarn install