Yarn: "Incorrect integrity when fetching from the cache"

[xclidongbo]

yarn version: 1.19.0
Do you want to request a feature or report a bug?

bug:
error Incorrect integrity when fetching from the cache

What is the current behavior?
When I install thie lib, the error is show:
for example: yarn add react-native-zip-archive

yarn add v1.19.0
info No lockfile found.
warning package-lock.json found. Your project contains lock files generated by tools other than Yarn. It is advised not to mix package managers in order to avoid resolution inconsistencies caused by unsynchronized lock files. To clear this warning, remove package-lock.json.
[1/4] 🔍  Resolving packages...
warning react-native > create-react-class > fbjs > [email protected]: core-js@<2.6.8 is no longer maintained. Please, upgrade to core-js@3 or at least to actual version of core-js@2.
info There appears to be trouble with your network connection. Retrying...
warning jest > jest-cli > jest-config > jest-environment-jsdom > jsdom > [email protected]: use String.prototype.padStart()
[2/4] 🚚  Fetching packages...
error Incorrect integrity when fetching from the cache
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.
error Incorrect integrity when fetching from the cache

If the current behavior is a bug, please provide the steps to reproduce.

What is the expected behavior?
success install.
Please mention your node.js, yarn and operating system version.
node: v12.10.0
yarn: 1.19.0
macos: 10.14.6

[chuan6]

Strengthening of integrity check introduced in #7582 seems to surface the issue. As a temporary fix, I commented out the code doing integrity check.

My guess is that for some packages, while local yarn.lock file may contain integrity, the record on remote registry may not. In my case, it's [email protected].

Hope that @arcanis can help share some advice on resolving the issue @~@

[chuan6]

@xclidongbo Please re-title the issue to reflect the error being thrown, so that people encountering it can all arrive here. Thanks :)

[arcanis]

Some context: old lockfile entries (more than a year ago?) don't have the integrity field present. Back when it was added, we decided not to automatically migrate them, as it was generating a lot of changes on unsuspecting projects.

So what happens is that when installing from such a no-integrity entry, the integrity doesn't get stored in the cache, and then when later installing a project with a modern lockfile with an integrity field, we report that the cache got stored without integrity and refuse to use it.

The solution is to first enable the migration by adding the following in ~/.yarnrc:

unsafe-disable-integrity-migration false

Then cleaning the cache and reinstalling:

yarn cache clean
yarn install -f

I think I'll just make a patch release that will force-migrate the lockfiles... It was more than a year ago, hopefully the changes will be less disruptive now 🤔 Or I can wait for the next major which is scheduled to be released in a few months.

[Blasz]

The validation fix seems to be a breaking change since cache now cannot be shared across entries with and without integrity fields.

Any dependencies installed from a yarn.lock with unsafe-disable-integrity-migration true that aren't previously cached will now cause this error whenever another repo without that setting tries to install one of those dependencies...which is going to be pretty common given that there's a very high probability of different projects sharing at least one transitive dependency.

Force migrating the lockfiles also seems like a breaking change, since versions before/after the change won't be compatible anymore.

From my limited perspective, I would:

1. Update the error message, it doesn't provide any information making it incredibly hard to figure out what to do. I had to debug yarn code directly to figure out what the problem was.
2. Support both kinds of lockfile entries by either:
   a) Always adding the integrity field to the cache, regardless of whether lockfile entries have the integrity field or not, so that both kinds of yarn.lock entries can utilise the cache
   b) Storing separate cached versions for entries with and without integrity fields

If neither options for 2) work then I guess force migrating lockfiles is better than yarn not working at all.

[arcanis]

Any dependencies installed from a yarn.lock with unsafe-disable-integrity-migration true that aren't previously cached will now cause this error whenever another repo without that setting tries to install one of those dependencies...which is going to be pretty common given that there's a very high probability of different projects sharing at least one transitive dependency.

Note quite - only old lockfile entries. For example, if you create a new project now, you'll see the integrity field is properly populated. Similarly, if you take an old lockfile and add a new dependency the integrity field will also be there (but the existing entries won't have received a new one).

Update the error message, it doesn't provide any information making it incredibly hard to figure out what to do. I had to debug yarn code directly to figure out what the problem was.

Sounds reasonable.

b) Storing separate cached versions for entries with and without integrity fields

Hm 🤔 That sounds possible. I investigated using the integrity as cache entry before, and it proved quite challenging due to the timing of the integrity check. But if we were to just add a marker to disambiguate whether there's an integrity field or not ... yeah, it might work.

[vilic]

@arcanis We have experienced this issue with up-to-date yarn.lock files multiple times. And after cleaning the cache, the issue surface again under certain circumstances. Would corrupted cache due to unexpected termination of yarn or disconnection of the network cause this problem?

[arcanis]

Can someone use our latest nightly and tell me whether you still get this error?

[saimonmoore]

We are experiencing issues related to failing integrity checksums with yarn 1.19.0. Looks very much like this. I tested the nightly and it didn't seem to help much.

[vilic]

@arcanis Seems to work for my case.

[Blasz]

@arcanis I'm no longer experiencing the error after using latest nightly, thanks for the quick fix!

[Blasz]

Actually, I spoke too soon. I just experienced the same error but caused for a different reason:

The integrity between the cached and non-cached versions don't match.

From what I can see, the cached remote integrity is sha-1 whereas the integrity it is comparing to is sha-512, which I assume is causing a mismatch.

[arcanis]

Do you have a small repro I can use?

[nazimjamil]

I'm experiencing multiple project build failures due to the integrity check, currently having to bypass the lock file due to --no-lockfile.

This of course is defeating the purpose of the lock file altogether.

Are the steps set out here the recommended migration path https://github.com/yarnpkg/yarn/issues/7584#issuecomment-536437423?

[arcanis]

Are the steps set out here the recommended migration path #7584 (comment)?

Yes. You can also use the latest nightly, and if it doesn't work please share a reproduction case ("Run yarn cache clean, then yarn install on the following project, Yarn will fail to install", something like this).

[mark-r-cullen]

We suddenly starting seeing issue this on our build servers yesterday, with v1.19.0. We haven't seen it on any of the dev machines yet though. I guess this means a yarn cache clean would maybe fix it (except I don't have access to the build servers to actually try this)?

I suspect that installing a nightly version isn't going to be an acceptable solution. Is there any timeline for the next version which will include this fix? I saw a few posts back that you said the next major version is months away... but it seems like you've come up with a fix which could be released in a bug fix release now?

[arcanis]

I suspect that installing a nightly version isn't going to be an acceptable solution. Is there any timeline for the next version which will include this fix? I saw a few posts back that you said the next major version is months away... but it seems like you've come up with a fix which could be released in a bug fix release now?

I don't want to publish multiple incomplete patch releases in a short timeframe. Given that some people have reported having still some issues (cc @Blasz), I'm waiting to have more information before deciding whether to release now or later. Hence me asking for repros 🙂

If you need it now, the nightly is the best course of action (they're meant to be stable, as we only merge on master when the CI is green).

[mark-r-cullen]

I wouldn't be able to give you a repo, but I can give you the list of packages we have installed as dependencies and devDependencies if this is of any use?

"dependencies": {
"lodash.clonedeep": "4.5.0",
"react": "16.8.6",
"core-js": "3.2.1",
"isomorphic-fetch": "2.2.1",
"react-bootstrap": "0.32.4",
"react-bootstrap-toggle": "2.3.2",
"react-dom": "16.8.6",
"react-router-dom": "5.0.1",
"react-scripts": "3.0.1"
},

"devDependencies": {
"@rescripts/cli": "0.0.11",
"@rescripts/utilities": "0.0.5",
"@types/enzyme": "3.10.3",
"@types/enzyme-adapter-react-16": "1.0.5",
"@types/jest": "24.0.15",
"@types/lodash.clonedeep": "4.5.6",
"@types/node": "12.6.1",
"@types/react": "16.8.23",
"@types/react-bootstrap": "0.32.19",
"@types/react-dom": "16.8.4",
"@types/react-router-dom": "4.3.4",
"enzyme": "3.10.0",
"enzyme-adapter-react-16": "1.14.0",
"history": "4.9.0",
"jest-enzyme": "7.0.2",
"jest-junit": "6.4.0",
"tslint": "5.18.0",
"tslint-react": "4.0.0",
"typescript": "3.5.3",
"yarn-run-all": "3.1.1"
}

[arcanis]

Not really, no, if you don't use the nightly.

[torifat]

Hey @arcanis, in some CIs there's no good option to clear cache. And, some people are facing Incorrect integrity when fetching from the cache. What's your opinion about invalidating the cache when we hit this scenario?

[Blasz]

Actually, I spoke too soon. I just experienced the same error but caused for a different reason:

The integrity between the cached and non-cached versions don't match.

From what I can see, the cached remote integrity is sha-1 whereas the integrity it is comparing to is sha-512, which I assume is causing a mismatch.

@arcanis I haven't been able to create a small repro since I determined it is related to an internal npm registry that we use.

The registry that we use does not return an integrity field and so the cached versions of packages resolved from the registry uses sha1 hashes. When the same package is then resolved from npm instead of our own registry, it has a sha512 integrity field and causes a clash.

I'm not too familiar with internal registries so I'm not sure what the right solution is. Could we cache based on registry rather than using the same cache across different registries? Or just invalidate the cache as @torifat suggests.

[Blasz]

Also I'm not sure why @xclidongbo decided to close this since the issue isn't completely resolved, I'd suggest re-opening this. There's already another duplicate in #7589.

[Blasz]

For reference, we use an artifactory registry which does not support sha512 integrity fields yet: https://www.jfrog.com/jira/browse/RTFACT-20084

[arcanis]

Thanks for investigating, multi-registry are probably the reason why people experience issues.

Hey @arcanis, in some CIs there's no good option to clear cache

I think I'll bust the cache again in the next release, so CI will start fresh.

What's your opinion about invalidating the cache when we hit this scenario?

Although doable, I think a better solution would be to simply always generate the sha512 checksum, regardless of what the remote registry returns (as long as the other hashes matches). Since you can reproduce this problem would you have time to look into that @Blasz, by any chance? I wonder if simply prepopulating algorithms with sha512 wouldn't be enough.

[danechitoaie]

1. Update the error message, it doesn't provide any information making it incredibly hard to figure out what to do. I had to debug yarn code directly to figure out what the problem was.

Yes PLEASE! Had to debug the "transpiled" local version of yarn for hours line by line to see what was causing this and on which package.

With --verbose at least there should be better logging for what happens.

[arcanis]

@Blasz can you confirm that https://github.com/yarnpkg/yarn/pull/7591 fixes your problem?

[Luchanso]

@arcanis now, nightly build it fixes this problem, thank you

[Luchanso]

I use: yarn-v1.20.0-20191002.1501 (18 minutes ago)

[Luchanso]

I have same issue, but installation continue:

rbose 15.127 Request "http://binary/artifactory/api/npm/npm/electron-to-chromium" finished with status code 200.
verbose 16.053 Request "http://binary/artifactory/api/npm/npm/caniuse-lite" finished with status code 200.
[2/4] 🚚  Fetching packages...
verbose 31.592 Performing "GET" request to "http://binary:80/artifactory/api/npm/npm/arui-feather/-/arui-feather-13.11.0.tgz".
verbose 31.646 Error: Incorrect integrity when fetching from the cache
    at MessageError.ExtendableBuiltin (/Users/luchanso/.yarn/lib/cli.js:721:66)
    at new MessageError (/Users/luchanso/.yarn/lib/cli.js:750:123)
    at /Users/luchanso/.yarn/lib/cli.js:48147:15
    at Generator.next (<anonymous>)
    at step (/Users/luchanso/.yarn/lib/cli.js:304:30)
    at /Users/luchanso/.yarn/lib/cli.js:315:13
    at <anonymous>
error Incorrect integrity when fetching from the cache
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
verbose 31.703 Performing "GET" request to "http://mvn//artifactory/api/npm/npm/geoip-lite/-/geoip-lite-1.2.1.tgz".
verbose 31.795 Performing "GET" request to "http://mvn//artifactory/api/npm/npm/path-parse/-/path-parse-1.0.5.tgz".
verbose 31.797 Performing "GET" request to "http://mvn//artifactory/api/npm/npm/vary/-/vary-1.1.2.tgz".
...

[Luchanso]

and fail on finish 😞 yarn cache clean helping for first time

[arcanis]

@Luchanso This isn't a chat. Various people are subscribed to this topic and receive notification, and your posts won't help them. Please use our discord channel if you absolutely need to post a rapid fire of messages.

[blenoski]

FWIW, we started seeing this error during our yarn install for CI builds yesterday. Temporarily downgrading to yarn 1.18 seems to be a good workaround for us: yarn policies set-version 1.18

[Blasz]

@Blasz can you confirm that #7591 fixes your problem?

Unfortunately, it does not fix it completely. I believe it fixes the issue where the cached version was resolved from the 'low integrity' registry and the remote version is resolved from the 'high integrity' (npm) registry.

However, it does not solve the other way around where the cached version is from the high integrity registry and the package is resolved from the low integrity registry. In this case, remote.registry contains a sha1 hash and cacheIntegrity only has the sha512 hash if it was cached from a yarn.lock entry containing an integrity field. Interestingly, if the package was cached from the high integrity registry from a yarn.lock entry without an integrity field, the cacheIntegrity field contains both a sha1 and a sha512 hash.

I'll try to raise a PR today that fixes this scenario and improves the error messaging. I'm thinking that the fix could be adding sha1 to the default algorithms set alongside sha512.

[chingc]

@blenoski This issue started hitting our CI infrastructure as well. Downgrading with yarn policies set-version 1.18 worked great. Thanks!

[ErisDS]

Also started seeing this error causing failed builds in our Jenkins CI setup. Adding yarn policies set-version 1.18 fixes the issue.

[ttupper92618]

I can confirm - we immediately started seeing these failures in our pipelines when 1.19.0 was released. As above, we've found that adding yarn policies set-version 1.18.0 resolves the problems.

[Robula]

yarn policies set-version 1.18 works for the current repo, is there any way to set this globally?

[fhelwanger]

@Robula that's the goal of yarn policies set-version, enforcing that everyone uses the same yarn version for a specific repo.

To do this globally you can just install [email protected] instead of 1.19 I think.

[Stanzilla]

when can we expect a release that fixes this issue?

[Trainmaster]

Instead of "fixing" the problem via yarn policies set-version 1.18.0 there should be a hotfix version reverting the changes causing this issue.

[arcanis]

Closing - the relevant fixes have been cherry-picked into the 1.19.1 (now released).

[eselskas]

curl --compressed -o- -L https://yarnpkg.com/install.sh | bash

Installs v1.19.1 which solves the issue. :beer: @arcanis

[Offirmo]

I wish the CHANGELOG would flag v1.18 as bugged, as per https://semver.org/ recommendations.

Maybe an "npm deprecate" on this specific version as well?

It could save a lot of time for those late to the party.

[viceice]

on chocolatey release
v1.19.1 is missing

[rosskevin]

1.19.1 is yielding:

  Error: No algorithms available for ""
      at Integrity.pickAlgorithm (/Users/kross/.yarn/lib/cli.js:32619:13)
      at Integrity.match (/Users/kross/.yarn/lib/cli.js:32604:24)
      at /Users/kross/.yarn/lib/cli.js:48145:58
      at Generator.next (<anonymous>)
      at step (/Users/kross/.yarn/lib/cli.js:304:30)
      at /Users/kross/.yarn/lib/cli.js:315:13

I tried after continual failures this morning on 1.19.0. I went back to 1.18 and was able to install.

[Gzbox]

升级到v1.19.1，解决了此问题

[garyo]

Sadly Ubuntu 18.04 still has 1.19 as latest version. But adding the yarnpkg repo and upgrading as in https://phoenixnap.com/kb/how-to-install-yarn-ubuntu works fine for me.

[frederikocmr]

https://github.com/yarnpkg/yarn/issues/7584#issuecomment-536437423

The solution is to first enable the migration by adding the following in ~/.yarnrc:

You can also use:
yarn config set unsafe-disable-integrity-migration false.

And then yarn cache clean and yarn install -f.

[Leigh-M]

sudo resolved this for me, so:

sudo npx create-react-app myApp

[jedwards1211]

is there any way to upgrade a lockfile with the old sha-1 hashes to sha-256 without blowing away and recreating the entire lockfile?