Yarn: Yarn install for private module failing - unexpected end of file

[vincentdu101]

Hi all,

What is the current behavior?
I'm currently trying to install a private module using yarn. However, I get the following error when I do:

yarn install
yarn install v1.17.3
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
error An unexpected error occurred: "https://registry.yarnpkg.com/@private/ngffwd-node-processes/-/ngffwd-node-processes-1.0.123.tgz: unexpected end of file".
info If you think this is a bug, please open a bug report with the information provided in "/opt/atlassian/pipelines/agent/build/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

If the current behavior is a bug, please provide the steps to reproduce.
The commands I'm running are the following:

npm config set scripts-prepend-node-path auto
yarn install

What is the expected behavior?
We should be able to install the package with no issue. Previously we've been able to do, however, recently that has not been the case. We tried going back to previous versions with no seeming improvement. We don't want to use npm install as that will affect the yarn.lock, but we are getting notifications that the version is available from npm.

The yarn-error.log doesn't yield much either, so I don't want to paste it here.

Please mention your node.js, yarn and operating system version.
npm - v6.9.0
node - v10.16.3
yarn - 1.17.3

Thanks any help or suggestions would be appreciated!

[rvanmil]

Same issue here, seems pretty random. Both on Heroku and Netlify our builds are failing every now and then due to this issue. After restarting the build (a few times) installing will succeed.

[rkistner]

Also getting this error a lot over the last week or two. It appears like the normal retry mechanism is also not working for this specific error.

[meshulam]

This has been hitting us pretty frequently, I was able to reproduce it with curl and it seems like the server is randomly closing the connection:

❯ curl -H 'Authorization: Bearer <token>' https://registry.yarnpkg.com/@org/package.tgz -i > package.tgz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 47 6648k   47 3169k    0     0  2640k      0  0:00:02  0:00:01  0:00:01 2638k
curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)

The only other weird thing is that the headers have age: 0 on the failed request. Maybe the issue occurs when the proxy doesn't have the package cached?

[vincentdu101]

Hi all,

So just to update everyone, we figured out a way to fix the failures for now. What we have done so far was cache the node currently in our CLI tool (Bitbucket Pipelines) and that seem to throw yarn off when it was doing the install.

Instead, going forward we removed the cache for node and everything seems to work again. We're still not quite sure why that is the case, maybe something with the way yarn handles their node npm cache or the proxy as @meshulam mentioned doesn't have the package cached.

For us this is okay for now, but I'll keep this ticket opened since everyone has other weird error symptoms that I think is still worth addressing and understanding.

Thanks everyone again for the feedback!

[vincentdu101]

Hi all,

Sadly I'm back as it seems like the issue has resurfaced with the same error. I've tried also removing this line with no success:

npm config set scripts-prepend-node-path auto

Does anyone if Yarn is updating/modifying its registries or something?

Thanks!

[JavierPDev]

I've been running npm cache clean --force to overcome this problem for now

EDIT: Might have just been a fluke and rerunning yarn install worked

[kevinswarner]

We too are also having this problem - sometimes on 50% of our CI builds. What is frustrating is there is no helpful information in the error log or error message. It mostly surfaces in our CircleCI builds. But, I have also experienced it locally on my system once in a while. For us, it always happens with the same private package which we publish to NPM.

[Mustack]

I reproduced the problem with npm to try to get more information and found that it was failing because of an integrity check failure. The checksum it was expecting was correct so it looks like NPM is serving bad packages.

[vincentdu101]

Yes we also found what @Mustack noted to be the case. It seems our private repo was throwing the integrity check sum error. However, Yarn was not outputting the correct error messaging which threw us off the right trial. We have since transitioned back to npm at this point. We are currently working with NPM support to figure out the main cause of the issue.

[jmorrell]

The checksum it was expecting was correct so it looks like NPM is serving bad packages.

This reminds me of https://github.com/yarnpkg/yarn/pull/6400 CDN changes by npm caused intermittent failures in the yarn client

CloudFlare was returning a 500 response with an HTML error message as the body. This is not common, but I was able to recreate it by curling a particular tarball in the registry in a loop.

This was fixed in https://github.com/yarnpkg/yarn/pull/6413 and released in 1.11.0 but maybe there have been further CDN changes on npm's side?

If this only happens for private packages, maybe the registry / CDN is returning something other than a 408 or a 5xx https://github.com/yarnpkg/yarn/blob/master/src/util/request-manager.js#L422

[pleunv]

Yes we also found what @Mustack noted to be the case. It seems our private repo was throwing the integrity check sum error. However, Yarn was not outputting the correct error messaging which threw us off the right trial. We have since transitioned back to npm at this point. We are currently working with NPM support to figure out the main cause of the issue.

Do you happen to have heard anything back from them?

[kevinswarner]

Update: We have not seen the problem occur again lately. I don't think we did anything different. Fingers crossed.

[Fley]

Hi, we have the same issue and npm support answered it is due to the amount of versions we have for this package, causing their servers to fail.
We are working with them to remove old versions.

[vincentdu101]

@pleunv not yet at the moment, we did send over the info to them.

@Fley we also tried to create a new package off the existing one, but the new one package threw the issue up immediately.

At this point we're investigating to see if there is something wrong with our private package.

[pleunv]

Too bad. If there's anything I can do or test, please let me know. This is
a tough one to figure out and to reproduce it seems, but it is incredibly
annoying.

On Wed, Sep 25, 2019, 19:49 Vincent Du [e-mail] wrote:

@pleunv https://github.com/pleunv not yet at the moment, we did send
over the info to them.

@Fley https://github.com/Fley we also tried to create a new package off
the existing one, but the new one package threw the issue up immediately.

At this point we're investigating to see if there is something wrong with
our private package.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/yarnpkg/yarn/issues/7521?email_source=notifications&email_token=AABMOBPVDPD6KMTY67A3EADQLOQBHA5CNFSM4ITXOL5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7SYHRA#issuecomment-535135172,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABMOBPNBCKBDFLTQ4P5KTLQLOQBHANCNFSM4ITXOL5A
.

[rkistner]

I haven't reproduced this issue properly yet, but it appears like this issue is much less common when using the npm registry instead of the yarn one. I've seen with the yarn one that the same error often persists even after multiple retries.

Could it be that the yarn registry/CDN is caching an invalid response, and then keeps on returning the same invalid response for subsequent requests, while the NPM registry works on the next retry?

[loujaybee]

We've been running into this issue fairly consistently. I tried to setup a build that replicated, but could not. One of our actual builds replicated quite consistently. We're using Drone CI, a container build tool and running a "matrix" build (parallel) so each yarn install is running independently. We were seeing the issue failing in different branches of the matrix at different times.

No combination of yarn check --verify-tree, or setting the cache directory with yarn config set cache-folder differently seemed to help. Rather frustratingly, I continued to get the issue even when running yarn cache clean, yarn install --frozen-lockfile --force and then yarn check --verify-tree — this combination should in theory _always_ work, right?

In the end, we moved to npm ci and all our installs are working correctly now, and ironically we're using yarn to validate the install with yarn check --verify-tree which is showing green.

[pleunv]

Interesting! Unfortunately we can't use npm ci as we can't add a lock file and always want to fetch the latest version of the dependency, regardless of what it was at the time of creating the consuming package.

[rkistner]

npm ci / npm install would work because it ignores the yarn.lock file completely, and uses registry.npmjs.org.

Since this is an issue with the registry and not the local cache, no yarn install / check / cache command can fix this. The only solution with the yarn registry seems to be to wait until the issue with the affected package is resolved (e.g. wait 10 minutes to an hour), and retry.

The best solution I've found is to switch to registry.npmjs.org, while still using the yarn tooling. This can be done per repo:

echo 'registry "https://registry.npmjs.org"' > .yarnrc # in the repo, not the global file
sed -i 's#registry.yarnpkg.com#registry.npmjs.org#' yarn.lock

No change to the workflow is required: yarn install etc keeps on working as it used to.

It's a bit of a pain to do over dozens of repos and branches, but worth it to avoid this issue on CI builds.

There doesn't seem be any good reason to use registry.yarnpkg.com anymore (see https://github.com/yarnpkg/yarn/issues/5891 for some discussion) - the main reason it's kept is to not break existing builds (which are actually pretty-much broken because of this issue), so I'm doing this change for all my repos going forward.

Update: This does not fix the issue at all.

[florian-abtasty]

Unfortunately we did reproduce the issue using registry.npmjs.org. So it seems unrelated to registry.yarnpkg.com to me.

[2/4] Fetching packages...
error An unexpected error occurred: "https://registry.npmjs.org/@private/somepackage/-/somepackage-4.1.4.tgz: unexpected end of file".

[pleunv]

Indeed, we have exactly the same issue with npm install as with yarn install (only the error returned is different) so it seems to originate from the npm registry instead.

[loujaybee]

I believe the issue lies in Yarn's handling of the connection issues (integrity checks etc). The only way to reliably reproduce I believe would be to mock the network failures to try and see how yarn behaves under the different failure scenarios. I believe there are certain cases that are not correctly handled, nor successfully retried.

[andrewhughson]

For us, the issue seems to occur with all combinations of Yarn/NPM CLI and Yarn/NPM registries. Since the common denominator is the NPM registry it seems the issue is probably there.

I have been in contact with NPM support about this with one of our packages which had 2000+ versions and they advised us to unpublish some in order to fix the issue.

I would be interested to know if this is truly the cause, are other people with this problem having it with packages that have large numbers of published versions?

[AndriMar]

@bighuggies WE have only 14 versions and are seeing this error.

[rvanmil]

@bighuggies only 11 versions and seeing the error here.

[CameronJ26]

We have the same error, 2700 versions.

[jennale]

I've been encountering this a lot too. I can seemingly avoid it by adding a network timeout option:

yarn install --network-timeout 100000
yarn upgrade @package --network-timeout 100000
etc..

Seems to fix things locally and on the CI... But I can't tell if it's coincidence or not that this actually works. Hope that helps ppl as a temporary measure!

[achichenin]

This happens to us a lot in a k8s gitlab runner. And similar things happen with pypi (python) repository. But pip automatically retries the download in such cases.

I think it should not really matter what causes the issue - they are bound to happen if you have multiple virtualization/traffic routing layers - yarn should be able to handle such issues itself by default.

yarn install --network-timeout 100000
yarn install v1.17.3
[1/4] Resolving packages...
[2/4] Fetching packages...
info If you think this is a bug, please open a bug report with the information provided in ... 
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
error An unexpected error occurred: "https://registry.yarnpkg.com/antd/-/antd-3.24.2.tgz: unexpected end of file".

[rkistner]

So I've confirmed that changing the registry has no actual effect - it just appeared to help initially.

I'm also working with NPM's support, but didn't get anything not already mentioned in this thread.

In my case it's also happening with packages with relatively few published versions (less than 25). There is however a big correlation with big packages (e.g. > 2MB gzipped file) and failures.

Has someone figured out if yarn does automatically retry in these cases? My suspicion is that even retries are not enough to solve the problem, since it's not just isolated failures, but the requests failing a large percentage of the time. It's also bad enough by now that some of our builds are failing more than half the time.

What I did find interesting is that I only see this error on our CircleCI builds - never when running locally. Could it be an issue that the overall number/size of requests between the build service (e.g. CircleCI) and CloudFlare is too much, causing CloudFlare to then drop or rate-limit the requests?

[ostkrok]

We've also been seeing this error both locally and on CircleCI.
It only happens with one of our private packages. We're guessing due to its size (around 4MB).

We managed to reproduce what we're thinking is the same error using the npm client, although it prints a different message:

npm ERR! code EINTEGRITY
npm ERR! Verification failed while extracting @company/[email protected]:

So we went a bit further and tried to download the package via curl and we're seeing similar issues:

$ curl -H "Authorization: Bearer **REDACTED**" https://registry.npmjs.org/@company/package-name/-/package-name-8.0.11.tgz > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 79 4377k   79 3484k    0     0  1336k      0  0:00:03  0:00:02  0:00:01 1337k
curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)

Furthermore we've noted that if we try to download the same version of the package multiple times, it almost always succeeds every time besides the first one, regardless if we're using yarn, npm or curl.
If we then try another version it'll fail with the same error on the first run, and work on subsequent ones. This kinda leads us to believe that it's some kind of caching problem with the registry?

We've tried filing a support ticket over at npmjs.com, but haven't got any real help with solving the issue. So we've manually added a retry around yarn install in our CI pipeline until someone finds out what's causing this.

[pleunv]

Just throwing this out here as I thought about it before but haven't had the chance to investigate it yet: perhaps this has something to do with geographical locations/proxy/cdn locations on the side of npm? Delays with propagation of new packages across their whole network, cache updates, stuff like that.
We tend to have it mostly with newly published packages which seems to point in that direction, but a conversation with support regarding this didn't really help me much further either.

[briangonzalez]

@rkistner We came to this exact conclusion yesterday. We decreased the size of our package (from 18MB to 200KB) and haven't had an issue since.

[spiltcoffee]

@pleunv I've been seeing this issue when fetching a package from a local Nexus proxy to a Windows 10 VM on the same LAN, so in my experience geography seems to have little to do with things.

The similarities of the problem I've experienced are package size (~44mb) and the package being "private" (it's hosted by Nexus only rather than proxied from an npm/yarn registry).

I've found I can fairly reliably cause the issue by first running yarn cache clean followed by a yarn install. The first install attempt almost always fails, but the subsequent attempts work.

I've also tried tracking down where exactly the error was being thrown from, and noticed that the error "unexpected end of file" wasn't coming from a HTTP operation, but a zlib operation: https://github.com/nodejs/node/blob/8b4af64f50c5e41ce0155716f294c24ccdecad03/deps/zlib/gzread.c#L189

[MatissJanis]

This is a very irritating issue we are experiencing in CircleCI. Doesn't seem to happen locally.

[briangonzalez]

@MatissJanis Check the size of the package causing you issues.

[stevenweber]

Can also confirm getting intermittent eof and malformed response errors as part of installing a private package of ~6mb, that typically installs successfully after 1-3 successive retries.

error An unexpected error occurred: "https://registry.yarnpkg.com/@org/package/-/package-5.239.0.tgz: unexpected end of file".
info If you think this is a bug, please open a bug report with the information provided in "/Users/steve/projects/reverb/yarn-error.log".

error Received malformed response from registry for "@org%2fpackage". The registry may be down.
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.

Unfortunately, unlike @briangonzalez, we could only reduce the package size to 3.2mb and still have the issue.

One thing that I would like to point out is that npm seems to run into issues downloading the package as well. Unlike yarn, though, it automatically retries, making this only a problem when using yarn.

npm WARN tarball tarball data for @org/[email protected] (sha512-wlqFaTbXJcYK7kFgOxtjL9AEXgerHdPkAr4AAD5KGcNzN8Whq91j0X3qjQFKFMs5tAIR6+nfWQUSoDhEVRoj3A==) seems to be corrupted. Trying one more time.

For the time being we're just yarn'ing in a loop until it's successful.

[BYK]

Okay so this is clearly a "server terminating the connection early" issue from all the comments and I don't think sharing more anecdotes will do anything to help fix the problem or help others.

I'll either close and lock the issue to avoid more thrash or we can talk about trying to implement a retry mechanism if yarn can detect an unexpectedly closed connection.

Anyone volunteering to give the retry option a shot? Fetching is done here: https://github.com/yarnpkg/yarn/blob/6e5b0ea0dc15bef7d2abd67a46adbe7efe533404/src/fetchers/tarball-fetcher.js#L244-L297 and these lines are actually implementing some retry logic when the server fails "properly": https://github.com/yarnpkg/yarn/blob/6e5b0ea0dc15bef7d2abd67a46adbe7efe533404/src/util/request-manager.js#L422-L455

We can probably add a length check there by comparing the actual response size to the content-length header from the server or catch ungraceful TCP terminations somehow and catch them.

[rkistner]

I'm making an attempt at a fix, hoping to have a PR up soon.

[rkistner]

I added a PR which should help mitigate the issue. However, it will only reduce the failures, not eliminate them. The same error often happens multiple times in a row, and even 5 retries may not be enough.

The only real fix is fixing the npm registry.

[jamiemccrindle]

I'm not proud but this is working for me while we wait for the PR to be merged:

RUN for i in 1 2 3; do yarn install && break || sleep 1; done

[TheLudd]

@jamiemccrindle After almost a week of trying to solve this problem I have now pushed your solution to our repo and it was merged.

[jooj123]

I get this also - its becoming worse over time it seems. It happens sometimes locally but more often in CI environments (maybe due to more latency?)

[artemv]

I'm not proud but this is working for me while we wait for the PR to be merged:

RUN for i in 1 2 3; do yarn install && break || sleep 1; done

Until this is properly fixed by Yarn I've made a simple NPM module that works around it similarly. It's easy to tell it to retry e.g. 100 times: yarn-retry --attempts 100. Also it only retries on unexpected end of file error so it will not retry unnecessarily.

[benevbright]

Ok. It's becoming a competition.
Ours is yarn install || yarn install || yarn install

[jdmunro]

We are also experiencing this very frequently on a larger private module. Attempting to install multiple times can work around the problem but it shouldn't be required.

[markokeeffe]

Ok. It's becoming a competition.
Ours is yarn install || yarn install || yarn install

Why not yarn || yarn || yarn ? 😛

[Huespal]

https://dev.to/arcanis/introducing-yarn-2-4eh1 -> Yarn 2.0
Does anyone have tried it and can say now this 'private repositories fail' pain-in-the-ass is solved?

Thank you.

P.D: I am agnostic, but I will pray.

[CyriacBr]

Settingnetwork-timeout to 100000 worked for me.

#.yarnrc
network-timeout 100000

[Dico200]

I have the same issue.
I have a github action that deploys a project.
The project references two git repository URLs in package.json.
Errors similar to the ones referenced in this issue occur.
The problem does not occur on our build jobs.

My hypothesis is that this happens because yarn does not lock onto its cache. When yarn clones the repositories and installs there, it fails because the repositories share dependencies.
Github does not support caches for actions that run on release created. The cache works between jobs for other actions like build jobs, which is why it works fine for those. The errors, while similar, are random and intermittent as others have noted, which is indicative of running tasks in parallel without locking or similar (because it is non-deterministic).

[rkistner]

@Dico200 This issue is pretty-well understood by now.

The NPM registry (and by extension the yarn mirror) closes the connection in the middle of the download - not often for small packages, but quite often for large (multi-megabyte) private packages. After months they still don't appear to be doing anything about the issue.

This is not an issue with caches or parallel tasks. Caches may make the issue occur less, since it's not downloading from the registry each time.

Other workarounds like changing the timeout are not likely to have an actual effect. The issue is very intermittent, which could make something appear to help while in reality it didn't actually change anything.

I have a PR (#7663) to fix the issue by retrying the request, but it seems like the interest in getting this merged is too low. Yarn 2 has a complete rewrite of the network logic, and will likely handle this case better (but I haven't confirmed yet).

The best workaround available at the moment is retrying the install as in https://github.com/yarnpkg/yarn/issues/7521#issuecomment-550057386, but even this will not always be sufficient. My personal solution is using the GitHub package registry instead of the NPM one - also not issue-free, but I've had way less issues over the last couple of months there than with the NPM registry.

[Dico200]

@rkistner Thanks for explaining the cause that is generally understood to be underlying this issue.

That is good to know and it makes sense to me that this behaviour of the NPM registry could be impacting my use case.
However, I'd like you to re-evaluate my comment and see if you think there's any merit to what I'm suggesting.
This line here recursively calls the install command from the CLI after cloning a private git repository, which is referenced from package.json. If and only if a git repository reference is used, which means this code will run in our case. Errors like the ones mentioned in this issue occur in my project during CI scripts in particular, and they only occur when there aren't any cached modules, as I explained in my comment.

I am not an expert of this code base but I do not believe that any kind of locking is happening when running yarn install at this point in time, globally or on a per cache directory basis.

The git fetcher, and all other fetchers, are invoked here using the networkConcurrency. The problem there is that the install command is called as part of the fetch. The second install command should be queued to run after all the fetchers have completed (whereas right now, you technically have exponential networkConcurrency growth)

[axCodeInt]

I have the same issue with yarn install, till I remove the yarn.lock.
So maybe there is some wrong in the dependencies.