Yarn: Mass concurrent fetches to npmjs.org registry may cause truncated fetches

[Jessidhia]

This is an issue with the npmjs.org registry itself, but this is probably something yarn will have to deal with internally (retries for this specific failure mode?).

This also affects pnpm and ied, other alternative npm frontends that try to do parallel fetches. Merely re-running yarn will resume the bad fetch and either complete the install, or fail when doing another fetch. Humans can do this, but shell scripts may have a hard time deciding what to do :)

Running yarn on a big repository with a bit over 1k (flattened) dependencies:

$ yarn
yarn install v0.15.1
info No lockfile found.
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
error https://registry.npmjs.org/bower-logger/-/bower-logger-0.2.2.tgz: invalid tar file
    at Extract.Parse._startEntry (/Users/kovensky/.nodenv/versions/6.7.0/lib/node_modules/yarnpkg/node_modules/tar/lib/parse.js:149:13)
    at Extract.Parse._process (/Users/kovensky/.nodenv/versions/6.7.0/lib/node_modules/yarnpkg/node_modules/tar/lib/parse.js:131:12)
    at BlockStream.<anonymous> (/Users/kovensky/.nodenv/versions/6.7.0/lib/node_modules/yarnpkg/node_modules/tar/lib/parse.js:47:8)
    at emitOne (events.js:96:13)
    at BlockStream.emit (events.js:188:7)
    at BlockStream._emitChunk (/Users/kovensky/.nodenv/versions/6.7.0/lib/node_modules/yarnpkg/node_modules/block-stream/block-stream.js:145:10)
    at BlockStream.flush (/Users/kovensky/.nodenv/versions/6.7.0/lib/node_modules/yarnpkg/node_modules/block-stream/block-stream.js:70:8)
    at BlockStream.end (/Users/kovensky/.nodenv/versions/6.7.0/lib/node_modules/yarnpkg/node_modules/block-stream/block-stream.js:66:8)
    at Extract.Parse.end (/Users/kovensky/.nodenv/versions/6.7.0/lib/node_modules/yarnpkg/node_modules/tar/lib/parse.js:86:23)
    at UnpackStream.onend (_stream_readable.js:511:10)
info Visit http://yarnpkg.com/en/docs/cli/install for documentation about this command.

$ yarn
yarn install v0.15.1
info No lockfile found.
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
warning [email protected]: The engine "rhino" appears to be invalid.
warning [email protected]: The engine "rhino" appears to be invalid.
[3/4] 🔗  Linking dependencies...
warning Unmet peer dependency "[email protected] || 0.20.x || 0.22.x".
[4/4] 📃  Building fresh packages...
success Saved lockfile.
✨  Done in 42.76s.

The error is not itself related with the bower-logger package; it just happened to be the package npmjs.org didn't feel like sending properly in the HTTP response.

Either that or the office's router is temperamental, but only with npmjs.org accesses 😕

[Daniel15]

@kittens - Shouldn't yarn's retry logic handle this case?

[petetnt]

FWIW we just run into same case:

error https://registry.yarnpkg.com/date-now/-/date-now-0.1.4.tgz: invalid tar file
    at Extract.Parse._startEntry (/root/.yarn/node_modules/tar/lib/parse.js:149:13)
    at Extract.Parse._process (/root/.yarn/node_modules/tar/lib/parse.js:131:12)
    at BlockStream.<anonymous> (/root/.yarn/node_modules/tar/lib/parse.js:47:8)
    at emitOne (events.js:96:13)
    at BlockStream.emit (events.js:188:7)
    at BlockStream._emitChunk (/root/.yarn/node_modules/block-stream/block-stream.js:145:10)
    at BlockStream.resume (/root/.yarn/node_modules/block-stream/block-stream.js:58:15)
    at Extract.Reader.resume (/root/.yarn/node_modules/fstream/lib/reader.js:249:34)
    at DirWriter.<anonymous> (/root/.yarn/node_modules/tar/lib/extract.js:64:8)
    at emitNone (events.js:86:13)
    at DirWriter.emit (events.js:185:7)

[stevie-mayhew]

I've also run into this issue this morning. After this error occurs a JSON error occurs on subsequent retries when I have no run yarn cache clean:
error SyntaxError: /Users/stevie/.yarn-cache/npm-babel-preset-es2015-6.16.0/.yarn-metadata.json: Unexpected end of JSON input

After a few yarn cache clean && yarn install attempts this error went away.

[Jessidhia]

@Daniel15 IIUC, the problem with this case is that, from the http api point of view, it _looks_ like a success; the data is just suddenly truncated. The received data should be shorter than the content-length header, though, unless the registry for some reason doesn't send that header.

[Daniel15]

the problem with this case is that, from the http api point of view, it looks like a success; the data is just suddenly truncated

Yarn could treat that as a failure case though. If the tarball doesn't extract properly, or the hash doesn't match, or the Content-Length doesn't match the actual file length, then consider it a failure and retry.

[hellivan]

Experinecing the same problem as @petetnt. Yarn fails with the "invalid tar file" message while fetching arbitray dependencies... After multiple re-runs of yarn install, the process finishes successfully.

[KidkArolis]

Hm, seems like this could be related to #1619

[mattbailey]

This is particularly frustrating in CI systems. Random & unpredictable failures, having to manually kick CI all the time really screwing up our workflow. Is the yarn team actively looking into this?

[bestander]

Afaik we fixed it, the reason was in the same tar file being downloaded and unpacked twice, unpacking would crash occasionally.
Please reopen if can reproduce after 0.25