Yarn: Can't install from GitHub when always-auth is set to true

[ghost]

Do you want to request a feature or report a bug?

Bug report.

What is the current behavior?

$ yarn add --dev npm-publish-git-tag
yarn add v0.28.0-20170710.0901
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
error An unexpected error occurred: "https://codeload.github.com/bahmutov/local-or-home-npmrc/tar.gz/7b9c4e239457b0810a98b84707730618e75c9ee2: Request failed \"404 Not Found\"".

If the current behavior is a bug, please provide the steps to reproduce.

Simply run yarn add --dev npm-publish-git-tag within any directory.

You may also run yarn add --dev npm-utils, as npm-utils is the package that has the GitHub reference.

What is the expected behavior?

For yarn to successfully download, and install, the tarball provided by the URL above.

The following simple request works:

curl https://codeload.github.com/bahmutov/local-or-home-npmrc/tar.gz/7b9c4e239457b0810a98b84707730618e75c9ee2

npm install --save-dev npm-utils also works as desired.

Please mention your node.js, yarn and operating system version.

$ yarn --version
0.28.0-20170710.0901

$ node --version
v6.11.0

$ uname -a
MINGW64_NT-10.0 HBETTSPC2 2.7.0(0.307/5/3) 2017-02-17 14:20 x86_64 Msys

Windows 10

[ghost]

Steps to reproduce issue:

mkdir yarn-test
cd yarn-test
yarn add --dev [email protected]

$ yarn --version
0.28.0-20170712.1404

I have the same Yarn nightly build on Windows 7 and Windows 10.

Yarn successfully installs npm-utils on Windows 7.

I get the error mentioned above only on Windows 10.

[BYK]

Can't repro on latest master with macOS so gonna mark this as Windows-only for now. Thanks for the repro steps.

[BYK]

@destroyerofbuilds this is quite odd tho, are you sure your Windows 10 machine is not using a different DNS server or behind a proxy?

[ghost]

@BYK thank you for hinting at the differences between settings on each system.

I began looking at the differences between my .npmrc and .yarnrc files between Windows systems.

The contents of my .yarnrc file;

`# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1


registry "http://artifactory.example.com/artifactory/api/npm/npm/"
"@company:registry" "http://artifactory.example.com/artifactory/api/npm/npm-company/"
lastUpdateCheck 1499813100616

The contents of the .yarnrc differ between systems only in the value of lastUpdateCheck.

The contents of my .npmrc file:

_auth = [REDACTED]
always-auth = true

I then ran rm -rf node_modules/ && yarn cache clean && yarn add --dev [email protected] on both Windows systems, and I got the same error.

So this issue is not unique to Windows 10.

Turns out the issue is with always-auth.

If I set always-auth to false, I can successfully install npm-utils.

always-auth is/was required to publish packages to the Artifactory instance. (Since Artifactory did not have Bearer Token support, and yet we had to set _authToken to a dummy value to work with a few npm packages).

However, Artifactory now has Bearer Token support (https://www.jfrog.com/jira/browse/RTFACT-8444). So the resolution to this issue is probably upgrading the instance of Artifactory I use.

[BYK]

That's some great detective work @destroyerofbuilds, thank you very much!

Let us know if there's anything you need on our side if using bearer tokens doesn't fix it.

[ghost]

@BYK in the interest of fixing our internal CI environment to a known good version of Yarn (until we can upgrade Artifactory), I can across the fact that Yarn 0.27.3 works using the reproducer above, while Yarn 0.27.4 does not.

hbetts@HBETTSPC2 MINGW64 ~/Workspace/yarn-test
$ rm -rf node_modules/ && yarn cache clean && yarn add --dev [email protected]
yarn cache v0.27.4
warning package.json: No license field
success Cleared cache.
Done in 0.11s.
yarn add v0.27.4
warning package.json: No license field
warning No license field
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "https://codeload.github.com/bahmutov/local-or-home-npmrc/tar.gz/7b9c4e239457b0810a98b84707730618e75c9ee2: Request failed \"404 Not Found\"".
info If you think this is a bug, please open a bug report with the information provided in "C:\\Users\\hbetts\\Workspace\\yarn-test\\yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.

hbetts@HBETTSPC2 MINGW64 ~/Workspace/yarn-test
$ rm -rf node_modules/ && yarn cache clean && yarn add --dev [email protected]
yarn cache v0.27.3
warning package.json: No license field
success Cleared cache.
Done in 0.12s.
yarn add v0.27.3
warning package.json: No license field
warning No license field
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved 0 new dependencies.
warning No license field
Done in 9.20s.

This change most likely changed the behavior of Yarn - 9c5311496ce9fe7d3e736e41397a20de7f419edb

Based on the commit message, I don't know whether the change in behavior for fetching GitHub tarballs with always-auth set was intended or not.

Leaving it here in case someone else encounters the same issue.

[BYK]

@arcanis - this looks like a bug to me. What do you think?

[also]

I'm also currently seeing this issue with other packages on codeload.github.com after updating to the latest yarn. I'm using 0.27.5 with always-auth on.

[arcanis]

This commit didn't intended to change anything related to how github packages are fetched - however I'm not completely sure I know what's the right behavior Yarn should have, I'll need to check this to see what makes the most sense. If packages cannot be fetched from github with always-auth on, that seems like a problem :)

[bmv437]

I'm also seeing this issue. yarn version 0.27.5 with always-auth enabled. I need to fetch packages from a private registry, and having always-auth is the only way to do that currently.

[soda0289]

I had some time to dig into this. It seems when the patch was made in 0.27.4 (https://github.com/yarnpkg/yarn/commit/5ff69228f6cc9fa27bbaeb67412f7f66f218b9da) a check was removed isRequestToRegistry() that ensured we only added the auth token to url that matched the package url. This means if the url is github.com but the registry is npm.mycustomrepo.example.com the auth token would not be attached.

I have a fix here (https://github.com/soda0289/yarn/commit/d1faebb60452ee5ef07726c5dd6cc67fbd3821f5#diff-b053bee294c216269844e5874039b6caL19) add can submit after I add some tests. I think the same logic can be added to scoped packages as well.

@arcanis Does this make sense, was the call to isRequestToRegistry() removed intentionally?

[arcanis]

Hey @soda0289, thanks for investigating! Hm, I don't understand why this check is required. It seems redundant, since requestUrl is derived from url.resolve(registry, ...), so it probably should always return true :/

[ghost]

so it probably should always return true

Isn't url.resolve resolving the full package path, which is a GitHub URL, against the Yarn package registry URL?

My naive assumption is that it's doing the following:

url.resolve('https://registry.yarnpkg.com/', 'https://codeload.github.com/bahmutov/local-or-home-npmrc/tar.gz/7b9c4e239457b0810a98b84707730618e75c9ee2');
'https://codeload.github.com/bahmutov/local-or-home-npmrc/tar.gz/7b9c4e239457b0810a98b84707730618e75c9ee2'

[soda0289]

@arcanis

pathname is equal to the github url. When url.resolve(registry, pathname) is called it returns pathname.

This is a strange behavior, url.resolve('http://google.com', 'http://yahoo.com') === 'http:/yahoo.com' , that I did not know about. We should clean this up to make it more readable.

@destroyerofbuilds you are correct in your assumption.