Yarn: Spurious "Extracting tar content of undefined failed"

[jleclanche]

Followup on #6312. Potentially related issues: #7087, #6755, #6312

Details

• yarn --version: 1.15.2
• yarn.lock
• command-line: rm ~/.cache/yarn ~/.npm ./node_modules -rf && yarn install --production --frozen-lockfile
• Arch Linux. Also reproduced on node:latest docker image.

---

I can still reproduce the issue on the latest yarn version. I do have a git dependency ("@magicfind/wf-build-engine": "https://gitlab.com/magicfind/warframe/wf-build-engine.git") and I'm fairly certain it's related to that, because I've been messing with it, though I don't know what exactly caused this to start happening.
I've been hitting #5235 and trying to work around it, and while doing that started hitting #6312.

Locally I see three different behaviours:

• I hit the error and yarn aborts
• I hit the error and yarn continues
• I do not hit the error and the install succeeds.

This points to a race condition of some type?

Example outputs

All outputs below run the exact same command line, removing all the cache I can find. I've not found a reliable way to reproduce the issue 100% of the time.

yarn install v1.15.2
[1/4] Resolving packages...
[2/4] Fetching packages...
[1/4] Resolving packages...
[2/4] Fetching packages...
error https://registry.yarnpkg.com/require-main-filename/-/require-main-filename-1.0.1.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "ENOENT: no such file or directory, chmod '/home/adys/.cache/yarn/v4/npm-require-main-filename-1.0.1-97f717b69d48784f5f526a6c5aa8ffdda055a4d1/node_modules/require-main-filename/LICENSE.txt'"
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

yarn install v1.15.2
[1/4] Resolving packages...
[2/4] Fetching packages...
[1/4] Resolving packages...
[2/4] Fetching packages...
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
[4/4] Building fresh packages...
$ npm run build
npm WARN lifecycle The node binary used for scripts is /tmp/yarn--1555780098541-0.2687961721890739/node but npm is using /usr/bin/node itself. Use the `--scripts-prepend-node-path` option to include the path for the node binary npm was executed with.

> [email protected] build /home/adys/.cache/yarn/v4/.tmp/83dda1d0ded23f5a9d7b82676a04465a.c67a312e2664e1fa7b0ae1176d36d03b6e40e079.prepare
> rollup -c


src/index.ts → dist/index.js, dist/index.es.js...                                                                                                                                             
created dist/index.js, dist/index.es.js in 1.1s
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
warning " > [email protected]" has unmet peer dependency "prop-types@^15.6.0".
warning " > [email protected]" has unmet peer dependency "draft-js@^0.10.5".
warning "react-scripts > [email protected]" has incorrect peer dependency "[email protected]".
warning "react-scripts > [email protected]" has incorrect peer dependency "[email protected]".
warning " > [email protected]" has unmet peer dependency "prop-types@^15.5.0".
warning " > [email protected]" has unmet peer dependency "remark-parse@>=3.0.0".
warning " > [email protected]" has unmet peer dependency "webpack@^3.0.0 || ^4.0.0".
[4/4] Building fresh packages...
Done in 23.64s.

yarn install v1.15.2
[1/4] Resolving packages...
[2/4] Fetching packages...
[1/4] Resolving packages...
[2/4] Fetching packages...
error https://registry.yarnpkg.com/is-stream/-/is-stream-1.1.0.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "ENOENT: no such file or directory, open '/home/adys/.cache/yarn/v4/npm-is-stream-1.1.0-12d4a3dd4e68e0b79ceb8dbc84173ae80d91ca44/node_modules/is-stream/package.json'"
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

yarn install v1.15.2
[1/4] Resolving packages...
[2/4] Fetching packages...
[1/4] Resolving packages...
[2/4] Fetching packages...
error https://registry.yarnpkg.com/iconv-lite/-/iconv-lite-0.4.24.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "ENOENT: no such file or directory, chmod '/home/adys/.cache/yarn/v4/npm-iconv-lite-0.4.24-2022b4b25fbddc21d2f524974a474aafe733908b/node_modules/iconv-lite/encodings/utf16.js'"
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
[4/4] Building fresh packages...
$ npm run build
npm WARN lifecycle The node binary used for scripts is /tmp/yarn--1555780287695-0.048757454583545856/node but npm is using /usr/bin/node itself. Use the `--scripts-prepend-node-path` option to include the path for the node binary npm was executed with.

> [email protected] build /home/adys/.cache/yarn/v4/.tmp/83dda1d0ded23f5a9d7b82676a04465a.c67a312e2664e1fa7b0ae1176d36d03b6e40e079.prepare
> rollup -c


src/index.ts → dist/index.js, dist/index.es.js...                                                                                                                                             
created dist/index.js, dist/index.es.js in 1.1s

yarn install v1.15.2
[1/4] Resolving packages...
[2/4] Fetching packages...
[1/4] Resolving packages...
[2/4] Fetching packages...
error https://registry.yarnpkg.com/fast-json-stable-stringify/-/fast-json-stable-stringify-2.0.0.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "EEXIST: file already exists, mkdir '/home/adys/.cache/yarn/v4/npm-fast-json-stable-stringify-2.0.0-d5142c0caee6b1189f87d3a76111064f86c8bbf2/node_modules/fast-json-stable-stringify/test'"
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

[jleclanche]

Weird:

$ yarn cache list

yarn cache v1.15.2
error An unexpected error occurred: "There should only be one folder in a package cache (got  in /home/adys/.cache/yarn/v4/npm-@magicfind-wf-build-engine-0.1.0-c67a312e2664e1fa7b0ae1176d36d03b6e40e079/node_modules/@magicfind)".
info If you think this is a bug, please open a bug report with the information provided in "/home/adys/src/overframe/frontend/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/cache for documentation about this command.

tree ~/.cache/yarn/v4/npm-@magicfind-wf-build-engine-0.1.0-c67a312e2664e1fa7b0ae1176d36d03b6e40e079/
/home/adys/.cache/yarn/v4/npm-@magicfind-wf-build-engine-0.1.0-c67a312e2664e1fa7b0ae1176d36d03b6e40e079/
└── node_modules
    └── @magicfind

2 directories, 0 files

yarn-error.log

Edit: Relevant commit: b1f21a80275294afd4c8a2cb244676345868ff52

[jleclanche]

The undefined showing up in the error message made me wonder if there was another issue at play here but it seems to be a missing parameter which is only used for the error message.

https://github.com/yarnpkg/yarn/blob/4ad88165fdf607a70c75db556a3af95864b2189a/src/fetchers/tarball-fetcher.js#L244-L246

Passing tarballCachePath as a third parameter here will make the error message make more sense:

error https://registry.yarnpkg.com/glob/-/glob-7.1.3.tgz: Extracting tar content of "/home/adys/.cache/yarn/v4/npm-glob-7.1.3-3960832d3f1574108342dafd3a67b332c0969df1/node_modules/glob/.yarn-tarball.tgz" failed, the file appears to be corrupt: "ENOENT: no such file or directory, open '/home/adys/.cache/yarn/v4/npm-glob-7.1.3-3960832d3f1574108342dafd3a67b332c0969df1/node_modules/glob/changelog.md'"

It does nothing to actually help the error though.

[jleclanche]

I ran rm ~/.cache/yarn ~/.npm node_modules yarn.lock && yarn install --verbose, to see if anything else can be gleaned from there:

verbose 13.540595951 Error: https://registry.yarnpkg.com/fast-json-stable-stringify/-/fast-json-stable-stringify-2.0.0.tgz: Extracting tar content of "/home/adys/.cache/yarn/v4/npm-fast-json-stable-stringify-2.0.0-d5142c0caee6b1189f87d3a76111064f86c8bbf2/node_modules/fast-json-stable-stringify/.yarn-tarball.tgz" failed, the file appears to be corrupt: "ENOENT: no such file or directory, stat '/home/adys/.cache/yarn/v4/npm-fast-json-stable-stringify-2.0.0-d5142c0caee6b1189f87d3a76111064f86c8bbf2/node_modules/fast-json-stable-stringify/test/to-json.js'"
    at MessageError.ExtendableBuiltin (/usr/lib/node_modules/yarn/lib/cli.js:721:66)
    at new MessageError (/usr/lib/node_modules/yarn/lib/cli.js:750:123)
    at Extract.<anonymous> (/usr/lib/node_modules/yarn/lib/cli.js:62682:14)
    at Extract.emit (events.js:198:15)
    at Extract.module.exports.Extract.destroy (/usr/lib/node_modules/yarn/lib/cli.js:139624:17)
    at onunlock (/usr/lib/node_modules/yarn/lib/cli.js:139501:26)
    at /usr/lib/node_modules/yarn/lib/cli.js:45391:25
    at /usr/lib/node_modules/yarn/lib/cli.js:45357:23
    at /usr/lib/node_modules/yarn/lib/cli.js:62632:13
    at FSReqCallback.oncomplete (fs.js:158:21)
error https://registry.yarnpkg.com/fast-json-stable-stringify/-/fast-json-stable-stringify-2.0.0.tgz: Extracting tar content of "/home/adys/.cache/yarn/v4/npm-fast-json-stable-stringify-2.0.0-d5142c0caee6b1189f87d3a76111064f86c8bbf2/node_modules/fast-json-stable-stringify/.yarn-tarball.tgz" failed, the file appears to be corrupt: "ENOENT: no such file or directory, stat '/home/adys/.cache/yarn/v4/npm-fast-json-stable-stringify-2.0.0-d5142c0caee6b1189f87d3a76111064f86c8bbf2/node_modules/fast-json-stable-stringify/test/to-json.js'"
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

Full log: verbose.txt.gz

[melaku-z]

related to #6312

• Upgrade to 1.13+

[jleclanche]

This is on 1.15.2.

[melaku-z]

This is on 1.15.2.

Yes, my bad. "yarn install --network-concurrency 1" is working for me, a temporary, slow workaround.

[ue360]

I used the 1.7 version before today, it never happened, and today I upgraded to the latest version (1.15.2), it will happen: Extracting tar content of undefined failed, the file appears to be corrupt

[TikiTDO]

I was seeing this fairly consistently on 1.16.0, and was finally able to track this down.

Our setup is as follows:

1. A /package.json which defines a bunch of workspaces
2. A /workspace/package.json which references a git modules
3. A custom build of a node module in git, with a package.json with a scripts -> prepare key.

This was causing an extra instance of yarn to run that script, which then tried to fetch packages needed by this lib outside of the expected order. Since the lib in question is mostly dead, and barely needed in our case I was able to resolve the problem by removing the prepare script.

For everyone else having these problems I recommend removing any git packages you have referenced, and failing that to grep all the package.json files in your project for these reserved scripts since they may be the cause. This is almost certainly the case if you see this sort of duplication:

[1/4] Resolving packages...
[2/4] Fetching packages...
[1/4] Resolving packages...
[2/4] Fetching packages...

Hope this helps root cause this in a more reliable fashion.

[TikiTDO]

@arcanis You mentioned in #6312 that it was closed because there was no further details. I believe my post above this one may explain what's happening here. It look like (as an educated guess) that the instance of yarn being called to service the special scripts like install, postinstall, prepublish, and prepare is not respecting any sort of mutex config.

[foodforarabbit]

I was seeing this fairly consistently on 1.16.0, and was finally able to track this down.

Our setup is as follows:

1. A /package.json which defines a bunch of workspaces
2. A /workspace/package.json which references a git modules
3. A custom build of a node module in git, with a package.json with a scripts -> prepare key.

This was causing an extra instance of yarn to run that script, which then tried to fetch packages needed by this lib outside of the expected order. Since the lib in question is mostly dead, and barely needed in our case I was able to resolve the problem by removing the prepare script.

For everyone else having these problems I recommend removing any git packages you have referenced, and failing that to grep all the package.json files in your project for these reserved scripts since they may be the cause. This is almost certainly the case if you see this sort of duplication:

[1/4] Resolving packages...
[2/4] Fetching packages...
[1/4] Resolving packages...
[2/4] Fetching packages...

Hope this helps root cause this in a more reliable fashion.

THANK YOU! I had recently added .yarnrc with --install.mutex network to fix another issue (multiple simultaneous yarn in different repo on the same build server) and having this similar-but-shouldn't-be-possible-anymore error popped up really freaked me out. I noticed the repeated [1/4]... stuff though and turns out packages recursively calling yarn was indeed the problem!

I started by piping yarn into a file e.g. yarn --verbose 2>&1 | tee yarn-output and looking for clues around the area where I saw [1/4] repeat itself:

verbose 34.141 Performing "GET" request to "https://registry.yarnpkg.com/synchronous-promise/-/synchronous-promise-2.0.7.tgz".
verbose 34.15 Performing "GET" request to "https://registry.yarnpkg.com/toposort/-/toposort-2.0.2.tgz".
verbose 34.592 Checking for configuration file "/Users/xxxxx/Library/Caches/Yarn/v4/.tmp/e237f3006359452a9561519898e180fb.5025ebf2427d709adfb10ee07f7dc36517d03694.prepare/.npmrc".
verbose 34.592 Checking for configuration file "/Users/xxxxx/.npmrc".
verbose 34.592 Checking for configuration file "/Users/xxxxx/.nvm/versions/node/v10.15.0/etc/npmrc".
verbose 34.592 Checking for configuration file "/Users/xxxxx/Library/Caches/Yarn/v4/.tmp/e237f3006359452a9561519898e180fb.5025ebf2427d709adfb10ee07f7dc36517d03694.prepare/.npmrc".
verbose 34.592 Checking for configuration file "/Users/xxxxx/Library/Caches/Yarn/v4/.tmp/.npmrc".
verbose 34.592 Checking for configuration file "/Users/xxxxx/Library/Caches/Yarn/v4/.npmrc".
verbose 34.593 Checking for configuration file "/Users/xxxxx/Library/Caches/Yarn/.npmrc".
verbose 34.593 Checking for configuration file "/Users/xxxxx/Library/Caches/.npmrc".
verbose 34.593 Checking for configuration file "/Users/xxxxx/Library/.npmrc".
verbose 34.593 Checking for configuration file "/Users/xxxxx/.npmrc".
verbose 34.593 Checking for configuration file "/Users/.npmrc".
verbose 34.593 Checking for configuration file "/Users/xxxxx/Library/Caches/Yarn/v4/.tmp/e237f3006359452a9561519898e180fb.5025ebf2427d709adfb10ee07f7dc36517d03694.prepare/.yarnrc".
verbose 34.593 Checking for configuration file "/Users/xxxxx/.yarnrc".
verbose 34.593 Found configuration file "/Users/xxxxx/.yarnrc".
verbose 34.594 Checking for configuration file "/Users/xxxxx/.nvm/versions/node/v10.15.0/etc/yarnrc".
verbose 34.594 Checking for configuration file "/Users/xxxxx/Library/Caches/Yarn/v4/.tmp/e237f3006359452a9561519898e180fb.5025ebf2427d709adfb10ee07f7dc36517d03694.prepare/.yarnrc".
verbose 34.594 Checking for configuration file "/Users/xxxxx/Library/Caches/Yarn/v4/.tmp/.yarnrc".
verbose 34.594 Checking for configuration file "/Users/xxxxx/Library/Caches/Yarn/v4/.yarnrc".
verbose 34.594 Checking for configuration file "/Users/xxxxx/Library/Caches/Yarn/.yarnrc".
verbose 34.594 Checking for configuration file "/Users/xxxxx/Library/Caches/.yarnrc".
verbose 34.594 Checking for configuration file "/Users/xxxxx/Library/.yarnrc".
verbose 34.594 Checking for configuration file "/Users/xxxxx/.yarnrc".
verbose 34.595 Found configuration file "/Users/xxxxx/.yarnrc".
verbose 34.595 Checking for configuration file "/Users/.yarnrc".
warning package-lock.json found. Your project contains lock files generated by tools other than Yarn. It is advised not to mix package managers in order to avoid resolution inconsistencies caused by unsynchronized lock files. To clear this warning, remove package-lock.json.
[1/4] Resolving packages...
[2/4] Fetching packages...
verbose 34.869 Performing "GET" request to "https://registry.yarnpkg.com/core-js/-/core-js-2.5.7.tgz".
verbose 34.87 Performing "GET" request to "https://registry.yarnpkg.com/css-box-model/-/css-box-model-1.0.0.tgz".

luckily, I decided to check out /Users/xxxxx/Library/Caches/Yarn/v4/.tmp/e237f3006359452a9561519898e180fb.5025ebf2427d709adfb10ee07f7dc36517d03694.prepare/.yarnrc and that folder was still around - checked out the package.json and realized that it was a library that I had forked and included as my dependancy recently. I'm guessing that the prepare script isn't called when the package is included "naturally" through npm, but it is when it is included as a github repo. For the record, the package I forked was react-beautiful-dnd. Once I updated my fork to remove the prepare script, the phantom pipeline issues stopped appearing. THANK YOU!

[Yaxian]

try this way in your Dockerfile:

RUN mkdir .yarncache
RUN yarn install --cache-folder ./.yarncache
RUN rm -rf .yarncache

or in your workspace:

mkdir .yarncache
yarn install --cache-folder ./.yarncache
rm -rf .yarncache

[esutton]

• Seeing this on my CI machine but not my dev machine with yarn 1.16.0
• I have seen problem with both yarn 1.16.0 and 1.17.0

6312 suggested adding '--network-concurrency 1' which * *seems ** to work.

````

https://github.com/yarnpkg/yarn/issues/6312

Added '--network-concurrency 1'

Example Error:

error confusing-browser-globals-1.0.7.tgz:

Integrity check failed for "confusing-browser-globals" (computed integrity doesn't match our records

yarn install --network-concurrency 1
````

[TikiTDO]

@esutton Check my first comment in this thread here. You can probably get away without having to use the network-concurrency key if you find the offending lib.

[bitsal]

I'm very disappointed with this issue and had to switch to pure npm. I wonder why yarn community can't fix the issue so long.

[esutton]

• For now, I will stay with yarn install --network-concurrency 1
• Uninstalling yarn and reverting to npm will be my next step.

@bitsal yarn has caused way too much time debugging build failures. I do not recall having to do the same with npm which just seemed to work.

@TikiTDO Thanks for advice, however I am using react native which has too many packages and dependencies, to sort through which ones are problem packages. While I enjoy a good puzzle, it is far less time consuming to revert to using npm and uninstall yarn.

[bitsal]

@esutton yarn install --network-concurrency 1 doesn't fully help me
I still have errors like this:

yarn install v1.16.0

[1/4] Resolving packages...

[2/4] Fetching packages...

error An unexpected error occurred: "ENOENT: no such file or directory, open '.../.cache/yarn/v4/npm-spdx-exceptions-2.2.0-2ea450aee74f2a89bfb94519c07fcd6f41322977/node_modules/spdx-exceptions/.yarn-metadata.json'".

info If you think this is a bug, please open a bug report with the information provided in ".../yarn-error.log".

info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

script returned exit code 1

[FezVrasta]

https://github.com/yarnpkg/yarn/issues/7212#issuecomment-493720324

I can confirm the issue seems to be related to a git dependency. But a workspace is not needed in my case to repro the problem.

Also, in my case it happens on a Docker node:12.7-alpine (I tried even with 11.x) image, but on on a a standard [email protected] travis worker.

[cinderblock]

I think this is a dupe of #2629

[FezVrasta]

It looks different to me 🤔

[TikiTDO]

The --ignore-scripts suggestion may be relevant to the issue at hand, though it honestly looks like that issue is really a few problems rolled into one discussion

[dgobaud]

Update found a workaround that does work.... a git submodule and then have the location of the package be the local folder. Not ideal but it works.

Think we're having this problem too details here https://github.com/yarnpkg/yarn/issues/2629#issuecomment-522871612 seems related to having a dependency hosted on github.com

[aalexgabi]

When you have more than 1 packages that install over Git (that have prepare scripts), this error happens almost every time and is not spurious anymore.

[aalexgabi]

I confirm that this is a functional workaround:

yarn install --network-concurrency 1

[eskp]

Related https://github.com/yarnpkg/yarn/issues/2629

[dlio]

This issue appears to have resurfaced in yarn version 1.22.0 (MacOS). The issue is persistent regardless of installation method (via brew, npm, or install script). The offending package is a git repository with no scripts.

--network-concurrency and --ignore-scripts flags did not work for me.

Downgrading yarn to 1.21.1 did resolve this issue, however.
yarn policies set-version 1.21.1

[buu700]

That fixed the problem for me, thanks @dlio! (--network-concurrency 1 did not.)

Based on the errors I'm seeing in https://github.com/pierrec/node-lz4/issues/92, it seems that yarn is extracting the tgz files in an incorrect order, causing failures when they contain hard links?

[andrmoel]

Hi there.
I found a way how to reproduce this issue. The error is caused when you run a parallel yarn install at the same project. We faced this issue on our Jenkins server.

1. delete your node_modules folder
2. execute yarn cache clean
3. open 2 console tabs
4. ensure that you run yarn install parallel in both tabs

This will cause the following error.

error https://registry.yarnpkg.com/cssstyle/-/cssstyle-1.2.2.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "ENOENT: no such file or directory, chmod '/home/amoeller/.cache/yarn/v6/npm-cssstyle-1.2.2-427ea4d585b18624f6fdbf9de7a2a1a3ba713077-integrity/node_modules/cssstyle/.eslintrc.js'"

Haven't found a solution yet.

[cinderblock]

@andrmoel If you're actually running two instances on yarn concurrently, there is are a couple ways to have yarn not interfere with itself. Take a look at the --mutex options for yarn.

However, I believe most people here are not intentionally running two concurrent instances of yarn and the --mutex options do not solve the problem.

[andrmoel]

@cinderblock that works great. Thanks a lot

[shark-horse]

Provided workarounds failed for me. This change to the offending dependency in package.json worked in my case:

-    "require-css": "guybedford/require-css#<hash>",
+    "require-css": "https://github.com/guybedford/require-css#<hash>",

Other dependencies also had the shorter github references, but I did not have to change those.

I arrived at that solution through obscure hints on Github, shotgun programming, and mysticism. No clue why it works. I am a newbie all over again and question my professionalism. HTH someone, or I'll question my sanity also.

[ohana54]

I was able to consistently reproduce the error (by deleting the yarn cache after every try).
This is my minimal setup:
main package - A
dependencies:

`A` -> `B`
    -> `C` -> `B`

The dependencies are listed with http urls, not versions - both for A's package.json and C's package.json.
Example:

{
    "name": "A",
    "dependencies": {
        "C": "http://remote-cache/C/1.0.0"
    }
}

Basically what happens, is that B is being downloaded twice to the same temporary cache folder. Simplified flow:
After downloading B for the first time, it is extracted to the temp cache folder.
WHILE the extraction is ongoing, yarn tries to download B again (because it is a dependency of C).
Before yarn starts to download it, it checks the temp cache folder to see if it's already there.
It does that by looking for a specific metadata file, that is written only after an extraction is complete.
Since the first B extraction is still ongoing, the metadata file is not there, and yarn assumes B is not in the temp cache.
The problem - yarn deletes the temp cache folder (I guess to handle a corrupt state) before it starts downloading B again.
Since the deletion happens during the extraction, we get the "Extracting tar content" error.

I was able to "fix" the problem by changing yarn's code to wait for ongoing download and extractions for packages. Meaning I keep the promise from the first B download process, and return it when we try to download it again, instead of having another download process interfering with the first one.
It's a hacky fix so I don't think submitting a PR with it will help, I'm mentioning it here in case someone is familiar with the codebase and provide a better idea.

[HitkoDev]

@ohana54 Can confirm, changing https://github.com/yarnpkg/yarn/blob/0e7133ca28618513503b4e1d9063f1c18ea318e5/src/fetchers/tarball-fetcher.js#L254-L255 to cache promises by this.reference solves the problem.

[HitkoDev]

Also the same problem occurs with local tarballs (and the same fix works), as described by @ohana54

[ohana54]

Thanks @HitkoDev for the validation.
This is a major blocker for us, as we started keeping tarballs from PR changes (without the need to publish them to the real registry), and having a link to these tarballs in package.json files is currently a no-go.
I can try to create a PR with a promise cache, although I'm still not sure it's good enough.

[HitkoDev]

That would be great, that way other devs get at least a rough idea what's going on and test different possible solutions.

[ohana54]

I opened a PR which partially fixes the issue. It won't help with github urls and prepare scripts, as that area in the code was a bit more complex for me.
If any of you have a consistent way to reproduce the github case, please let me know and I can try to give it a go in my spare time.

[markjm]

Thanks @ohana54 , I wonder if it makes sense to hoist this caching to all fetchers. This way, we would expect to see the benefits in all fetchers (including git). Code example (into a fork, but am happy to bring it wherever it is welcome) is here https://github.com/VincentBailly/yarn/pull/13.

Do you see any issues with hoisting the logic like this?

[ohana54]

@markjm I'm not sure it will work, since the problem lies between the fetcher and the parent resolver - meaning while the fetcher unpacks the current resource, the parent resolver deletes the temp folder that the fetcher unpacks to.
Meaning the "guard" has to be before the folder deletion, like I did in my PR.
Does that make sense? Hopefully I'm wrong and we can have a single fix for everything :)

[IUnknown68]

I just ran into the same issue, and the explanation https://github.com/yarnpkg/yarn/issues/7212#issuecomment-493720324 seems make sense. It fits our situation.

I was able to confirm this by "manually" caching the package(s) in question, so they are not fetched again:

1) When yarn errors, remove that package from the cache.
2) Let yarn fetch the package (by adding it to a dummy project).
3) Repeat from 1) for every package that gives the error.

Now your cache should be up2date, and no concurrent fetch/extracts should happen anymore (for those packages).

Of course that's not a solution, but until the error is fixed, it might help when you get stuck.

[mscrivo]

A workaround I've found that works for CI, without having to drop network concurrency to 1 and have slow builds, is to clone the repo of the problematic package, run yarn install it, then run yarn install on app that references that package. That seems to cache it and then it doesn't need to be fetched on the main yarn install and thus no race condition.

[dmitryguesty]

@ohana54 ,
Described by you flow/error possibly happens because of network concurrency race condition.
This error happens to me running yarn install on windows with unstable internet connection.

Solved by:

adding .yarnrc file to the project with following line:

network-concurrency 1

thanks to @matanrokach for assistance

[ohana54]

@dmitryguesty yes, you can see in my PR how the race condition is solved. Unfortunately for us it's not an option limiting the concurrency to 1.

[RoXuS]

We have this error since we are on yarn (via lerna).
lerna boostrap produce this error randomly and only on our gitlab-ci:

error https://registry.yarnpkg.com/typescript/-/typescript-3.9.7.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "ENOENT: no such file or directory, stat '/usr/local/share/.cache/yarn/v6/npm-typescript-3.9.7-98d600a5ebdc38f40cb277522f12dc800e9e25fa-integrity/node_modules/typescript/lib/lib.dom.d.ts'"

Any tips to detect which package produce this error? Any option works for us (concurrency, delete yarn.lock...).

EDIT : yarn why typescript it allowed me to found the package.

[cinderblock]

@RoXuS the best stopgap solution I've found is to install the intermediate package, that also depends on TS, first. This can be to a completely different folder on the system as long as you use the same user. This will install that package in a cached area first and then subsequent installs used the cached version that doesn't cause this race condition.