Yarn: invalid tar file error on externally hosted package

[doctyper]

Do you want to request a feature or report a bug?

What is the current behavior?
Running yarn add appdynamics results in an invalid tar file error. Originally reported in #1619, it was suggested that I open a separate ticket for this issue.

Yarn version:
  0.17.6

Node version:
  6.9.1

Platform:
  darwin x64

npm manifest:
  {
    "name": "foo",
    "version": "1.0.0",
    "main": "index.js",
    "license": "MIT"
  }

yarn manifest:
  No manifest

Lockfile:
  No lockfile

Trace:
  Error: http://packages.appdynamics.com/nodejs/4.2.9.0/appdynamics-proxy.tgz: invalid tar file
      at Extract.Parse._startEntry (yarn/node_modules/tar/lib/parse.js:149:13)
      at Extract.Parse._process (yarn/node_modules/tar/lib/parse.js:131:12)
      at BlockStream.<anonymous> (yarn/node_modules/tar/lib/parse.js:47:8)
      at emitOne (events.js:96:13)
      at BlockStream.emit (events.js:188:7)
      at BlockStream._emitChunk (yarn/node_modules/block-stream/block-stream.js:145:10)
      at BlockStream.flush (yarn/node_modules/block-stream/block-stream.js:70:8)
      at BlockStream.end (yarn/node_modules/block-stream/block-stream.js:66:8)
      at Extract.Parse.end (yarn/node_modules/tar/lib/parse.js:86:23)
      at UnpackStream.onend (_stream_readable.js:511:10)

If the current behavior is a bug, please provide the steps to reproduce.
Run npm add appdynamics

What is the expected behavior?
Installing the package should work as npm install appdynamics does.

Please mention your node.js, yarn and operating system version.

Yarn version:
  0.17.6

Node version:
  6.9.1

Platform:
  darwin x64

[Daniel15]

Does that URL require authentication of some sort? If I hit http://packages.appdynamics.com/nodejs/4.2.9.0/appdynamics-proxy.tgz in my browser, I get an "access denied" error.

[doctyper]

I suspect you need some sort of cache key to be able to download the file.

[frank-weindel]

Also having this exact problem. npm install appdynamics works but yarn add appdynamics fails with the error shown above.

To quote @KidkArolis from #1619:

⛔️ yarn add appdynamics. This one did not work, it's different from the others. Appdynamics hosts their dependencies on their own server (e.g. http://packages.appdynamics.com/nodejs/4.2.9.0/appdynamics-proxy.tgz) which is not accessible to the public by default. My guess is they're checking request headers for some npm-cli signature. I would reopen this issue as a separate issue.

Seems like they're doing something funky, for whatever reason.

[davidxroth]

Hi, I'm the engineering manager for the appdynamics package, and wanted to shed some light onto how our package dependencies are handled.

There are some rewrite rules which will redirect the dependency based on parsing the npm user-agent string to ensure the correct binary package for the OS/node version is downloaded. For example if npm requests the dependency http://packages.appdynamics.com/nodejs/4.2.9.0/appdynamics-proxy.tgz, it would be re-written to http://packages.appdynamics.com/nodejs/4.2.9.0/appdynamics-proxy-linux-x64-v6.tgz when the user-agent string is 'npm/3.10.8 node/v6.9.1 linux x64'

There is no authentication required.

Hopefully this clarifies things, and let me know if you have any more questions.

[frank-weindel]

@davidxroth That's not a very ideal way of downloading the correct binary for a system. There are ways to detect those attributes within nodeJS so you can resolve the correct URL without resulting to redirects and user-agent scraping, which is clearly what is breaking Yarn compatibility.

Take a look at how phantomjs-prebuilt does it:
https://github.com/Medium/phantomjs/blob/master/lib/util.js#L91

[davidxroth]

@frank-weindel agreed. I never said it was optimal, just how it currently works, and what npm supports.

[vrinek]

@davidxroth is this something that the appdynamics team is working on? Should we expect it to be fixed soon or should we invest in workarounds?

[rosswarren]

It appears that yarn will send the string

User-Agent:yarn/0.18.1 npm/? node/v6.4.0 darwin x64

If instead I send the following then I receive the file rather than getting the 403 forbidden

User-Agent:npm/? node/v6.4.0 darwin x64

@davidxroth would it be possible to do something on your side just to ignore the initial yarn/xxx given that the rest of the user agent still contains the information that you need?

[rosswarren]

As a workaround I created a file called .yarnrc with the contents

user-agent "npm/? node/v6.4.0 darwin x64"

With this, the tar downloads correctly

[vrinek]

@rosswarren how did you figure this out?

I'm trying to find out the user agents used for other architectures (linux in docker specifically).

[davidxroth]

sorry all, I missed the activity on this issue.

As a quick fix, we're looking at updating our rewrite rules to handle the yarn user-agent string. I'll update once this has been deployed.

[rosswarren]

@vrinek the code used is

'user-agent': [
  `yarn/${pkg.version}`,
  'npm/?',
  `node/${process.version}`,
  process.platform,
  process.arch,
].join(' '),

You can substitute in the appropriate values. (see https://github.com/yarnpkg/yarn/blob/05bf97728cbe92f034f7d4596830f481a65fe8fa/src/registries/yarn-registry.js#L30-L36)

@davidxroth That's great! 🎉

[davidxroth]

Sorry for the delay in closing the loop on this, but I wanted to update everyone that our rewrite rules now handle yarn and well as npm, so the workarounds should no longer be necessary.

Please let me know if you see any problems.

[afenton90]

@davidxroth I am still getting this issue without the workarounds above.
I am using yarn 1.3.2.
[email protected]

I get:

"http://packages.appdynamics.com/nodejs/4.3.6.0/appdynamics-native.tgz" resulted in hash "810aff3c7da7180447d2ae3aee43fe2c6c77ee86", which did not match the requested hash "6908ed81e2e999c291c4a70b1c4fe6b888e6c312".

[JamieMason]

tl;dr: issue still present in "appdynamics": "4.4.1" and this is not an issue with Yarn or npm.

---

We just upgraded from appdynamics 4.3.6 to 4.4.1 and the following error is shown during npm installation:

npm ERR! sha1-jNtyc+yh/1rmO/8bksp2CMsPIW4= integrity checksum failed when using sha1: wanted sha1-jNtyc+yh/1rmO/8bksp2CMsPIW4= but got sha1-hQoUbY5P4QAD+r1LAlWbQi6LaC0=. (517529 bytes)

the expected sha is the one committed to the lockfile on my mac, but the one received is that derived from the tarball on linux. Because the integrity fields do not match, yarn and npm correctly mistrust the registry and reject it.

Request: If the appdynamics tarball contained each of the platform binaries and installed the appropriate one at install-time, it would keep their customers from being prevented from installing/deploying their applications in the event that http://packages.appdynamics.com is experiencing an outage.

The tarball would be bigger, but in return it would mean appdynamics could support Yarn's offline mirror, which would be a huge boost to resilience.

Thanks for your time.

[traviscrist]

@davidxroth We are seeing the same issues reported in this thread. See the 2 comments above for examples of this still occurring. Is App Dynamics looking into a further fix to resolve this issue?

[LeoK80]

I've experienced this issue with both npm as well as yarn. Issue lies with how appd derives user-agent and does it's redirecting to the binaries as indicated in answers above.
More specific: When there is a package-lock file of any kind in the project and it is pushed up to a vm (e.g. on a cloudfoundry platform) then the instance where to project is pushed to gets the error based on hash mismatch while doing an install step on the vm based on said package-lock file.

adding user agent to .npmrc or .yarnrc as @rosswarren does in answers above seems to work.

[charlespwd]

@davidxroth and others trying the workaround. I'd like to add a couple of details concerning the "fix".

1. You need to add the following to your .yarnrc
user-agent "npm/? node/v6.4.0 darwin x64"
2. You need to delete your yarn.lock and node_modules
3. You need to install your dependencies ($ yarn)
4. And only then you commit your yarn.lock, otherwise the .yarnrc fix doesn't change anything.

After that, you should be good.

In light of the comment by @joeycozza , I realize that you should put the user-agent string of your production server's. So the steps change:

1. Get the user-agent string of your deployment server
   ssh $prodserver npm config get user-agent
2. Add the user-agent string to your .yarnrc
user-agent "< insert user-agent string here >"
3. Delete your yarn.lock and node_modules
4. Install your dependencies ($ yarn)
5. Commit your new yarn.lock

[joeycozza]

Here are some more fine grained details from appdynamics about how this works, and also their warning about shrinkwrap/lockfiles

When using lockfiles with NPM and AppDynamics, one must be very careful to maintain user-agent parity between any machines sharing the lockfile. For example, a developer installs AppDynamics locally using npm install. This mutates the package-lock.json file for the project, which they then check into source control. Later, a build machine checks out the same commit, and tries to install using the lockfile. This fails with an integrity error, because the developer checked in the lockfile with AppDynamics OSX dependencies, and the build machine runs Linux. As above, the artifacts are different, and so have different integrity values. The best way to ensure no integrity conflict arises is to always ensure that your lockfiles have been built with the same NPM user-agent active.

taken from this link
https://community.appdynamics.com/t5/Knowledge-Base/Advanced-NPM-Guide-for-the-AppDynamics-Node-js-Agent/ta-p/26743