Yarn: Yarn install quits silently if fetching one package times out

[geropl]

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
yarn install quits silently if fetching one of the packages times out during fetch phase due to package size and/or network lag.
Result:

• yarn does quit before reaching linking phase, therefore no packages are installed
• No error message
• Return code 0

(Shortened) example output:

$ yarn install --verbose
yarn install v1.5.1
[...]
[1/4] Resolving packages...
warning: [...]
[2/4] Fetching packages...
verbose 1.45 Performing "GET" request to "https://registry.yarnpkg.com/envfile/-/envfile-2.3.0.tgz".
[...]
verbose 20.696 Performing "GET" request to "https://registry.yarnpkg.com/command-exists/-/command-exists-1.2.2.tgz".
$ 

If the current behavior is a bug, please provide the steps to reproduce.

1. git clone https://github.com/theia-ide/theia-apps
2. cd theia-apps/theia-java-docker
3. mv next.package.json package.json
4. yarn install

(Retry until it fails, maybe use slow internet connection)
The large package here is @theia/java, which clocks in at around 33MB.

What is the expected behavior?
yarn install prints an error message with the timeout, the package name and url it failed to fetch. Finally, it quits with a return code != 0.

Please mention your node.js, yarn and operating system version.

• node 8.10
• yarn 1.5.1
• OS: docker image node:8.10
• (Host: Ubuntu 16.04)

Update:
Maybe related: #5055

Update 2:
Neither increasing --network-timeout nor setting --mutex network helped. Interestingly, the yarn install quits long before the network timeout is due (e.g., timeout 60000ms, yarn install quits after 35s).

Update 3:
I noticed that in the cases where it fails locally (via docker run ...), my system monitor shows a very slow download (~150Kb/s on a 50 Mb/s connection). The other runs go with ~5 Mb/s).

[rally25rs]

I attempted to reproduce this by setting the timeout passed to the request lib to 1 but it does throw an error in that case:

~/Projects/yarn-test 🐒   yarndev add @theia/java
yarn add v1.6.0-0
[1/4] 🔍  Resolving packages...
error An unexpected error occurred: "https://registry.yarnpkg.com/@theia%2fjava: ETIMEDOUT".
info If you think this is a bug, please open a bug report with the information provided in "/Users/jvalore/Projects/yarn-test/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.

Not sure how else to try to repro this 🤔 maybe I can simulate a slow network somehow...

[rally25rs]

Next attempt, used Apple's "Network Link Conditioner" to simulate DSL speed. This took >2min to download that package, but still finished

~/Projects/yarn-test 🐒   yarndev add @theia/java
yarn add v1.6.0-0
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
warning [email protected]: The engine "vscode" appears to be invalid.
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved 1 new dependency.
info Direct dependencies
└─ @theia/[email protected]
info All dependencies
└─ @theia/[email protected]
✨  Done in 147.88s.

then I tried turning off my network half way through the download. That also caught the error:

~/Projects/yarn-test 🐒   yarndev add @theia/java
yarn add v1.6.0-0
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
info There appears to be trouble with your network connection. Retrying...
error An unexpected error occurred: "https://registry.yarnpkg.com/@theia/java/-/java-0.3.7.tgz: connect ENETUNREACH 104.16.59.173:443 - Local (0.0.0.0:49668)".
info If you think this is a bug, please open a bug report with the information provided in "/Users/jvalore/Projects/yarn-test/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.

There must be some really weird edge case here... (or maybe it only happens on linux?)

[geropl]

That's really strange. Maybe it's related to running in docker? The build is executed with jenkins in a Docker build agent, but I can run the build in docker on my laptop and reproduce it in ~20% of the runs.

Update: Dockerfile to reproduce

FROM node:8.10
COPY . .                          # <- the files from the example above
RUN yarn install
RUN ls -al node_modules

[cjbturin]

This may be a scenario to reproduce similar problem on Windows env:
https://github.com/yarnpkg/yarn/issues/5512

[geropl]

Update 3:
I noticed that in the cases where it fails locally (via docker run ...), my system monitor shows a very slow download (~150Kb/s on a 50 Mb/s connection). The other runs go with ~5 Mb/s).

We don't use a yarn cache in our environment. Could it be that npmjs throttles/terminates repeated downloads from the same IP?

[ispringer]

Several of my teammates experienced this issue on OS X 10.13.3 with Node 5.12.0 and Yarn 1.5.1. Some comments in #764, an issue with some similarity to this one, said upgrading to Node >= 6.9.5 fixes that issue. So my teammates tried upgrading Node to the latest (9.8.0), and that did the trick! My suspicion is that the DNS caching that was recently adding to Yarn (#4447) may not work correctly with Node < 6.9.5, but it's only a hunch.

[damianb]

Maybe related to #5473?

[jneves]

@rally25rs I'm still getting it regularly, is there any specific options you'd like me to try, information to provide?

[jneves]

We run it from a docker container built from node:8.10-stretch.

[rally25rs]

@jneves I normally debug through the Chrome dev tools using node --inspect-brk ~/Projects/yarn/bin/yarn.js but doing it in a Docker container might be problematic. Does it happen outside a docker container? If not, then there might be some resource limit (like drive space?) being hit in docker?

[lili21]

same issue here, run it from a docker container.

[naderman]

Same issue here, also running from a docker container, would be happy to assist you in debugging this if you let me know how I can help?

[kondei]

I have the same issue when running it on a drone.io on a docker container.

[kondei]

In my case, docker container did not know our private npm registry ip address...
yarn install goes end, and it is resolved.
sorry

[b1rdex]

For me problem was in wrong definition of dependency:

"devDependencies": {
    "gulp": "git+https://github.com/gulpjs/gulp.git"
}

This is acceptable for npm btw.

[rojasmi1]

I'm experiencing the same issue using yarn 1.13.0, node 11.9.0, the issue seems to be caused by problems downloading a private package due to slow download speed using a VPN.

The installation was silently failing for me as well, only way I could figure the issue out was by setting --network-concurrency to 1. It'd nice to catch these kind of issues out of the box without any workaround, though.

[2ZZ]

I have the same problem with yarn v1.12.3
If packages fail to install the exit code is 0 with no error output

[gaplo917]

Same problem in v1.13.0

Step 1/11 : FROM node:10
...
Step 8/11 : RUN yarn install --frozen-lockfile
 ---> Running in 9174a622f137
yarn install v1.13.0
[1/4] Resolving packages...
[2/4] Fetching packages...
Removing intermediate container 9174a622f137
 ---> b151f7407bd8
Step 9/11 : COPY . /app
 ---> c014809a3bce

CI pipeline does not fail => the node_modules folder don't exist in the docker image.

The workaround to make sure the CI pipeline fail is to check whether node_modules is exist

Step 8/11 : RUN yarn install --frozen-lockfile && ls node_modules

[swayok]

For me the culprit was an antivirus (Dr.Web) that monitors http/https request and blocks "dangerous" domains silently.... In my case it was https://raw.githubusercontent.com. Adding it to whitelist of antivirus solved the issue.
Use --verbose to see which url is blocked and then try to open it in browser. If you see something like ERR_SSL_PROTOCOL_ERROR - then the culprit is your firewall or antivirus.

[philaw]

Had this when running yarn v1.1.3.0 inside docker, after upgrading to v1.17.3 (nodejs 12.7.0), it behaves normal now.

[puncha]

@philaw we also use yarn 1.13.0 and encountered this bug. I'll try to upgrade to 1.17.3.