Yarn: Support for private packages

[jamiebuilds]

In order to allow installing private packages Yarn will need to send a token to the headers of the request.

Private packages are @scoped/packages that were published with npm publish --access=restricted. The permissions of packages are managed through npm access and npm team which are not yet added

In the npm client, this token comes from the .npmrc and looks like this:

@nameofscope:registry=https://registry.npmjs.com/
//registry.npmjs.com/:_authToken=abc123

And it gets sent as this header:

Authorization: Bearer abc123
# alternatively:
Authorization: Basic username:password # <= base64

There's a package for retrieving the token. Although we may not want to store the token the same way npm does.

This token gets added to .npmrc on npm login. But yarn login doesn't even authenticate (it only stores username and email), so we may want to force the user to authenticate on install (in which case we need to solve scripting these installs for CI servers through some kind of environment variable).

We also need to make sure that Yarn users don't accidentally publish something publicly.

[sebmck]

We already have npm login and auth logic here. Just need to sort out the workflow.

[chicoxyzzy]

Private registry doesn't always need auth token. For example we access our private registry through corporate VPN.

[knksmith57]

^^ Agreed. Allowing for the association of a separate registry _per scope_ is sufficient for us (and I suspect many others).

[jmonster]

in which case we need to solve scripting these installs for CI servers through some kind of environment variable

• http://blog.npmjs.org/post/118393368555/deploying-with-npm-private-modules
• https://circleci.com/docs/npm-private-module-dependency/

[djMax]

When we say "we already have this logic" - I don't see any path where an Authorization header would be sent to a registry. If there was one, perhaps there'd be a temporary workaround to make this all work while something more final is sorted out. Am I missing something?

[djforth]

+1 looks like scoped packages even if they are public seem to fail.

[frederickfogerty]

To further @djforth's comment, I just installed from master, and I'm getting the same error - scoped packages are failing. It converts the / in the package name into %2f, which means the request to npm to find the package fails.

e.g. Error: https://registry.yarnpkg.com/@company%2fdata: Not found

[djMax]

That's the way it fails if auth is required. I got it to work for public scoped packages

[devongovett]

Should be fixed by #839 and #1146.

[vjpr]

@devongovett I ran into a few issues:

---

This is the check for whether auth should be used:

    if (this.token || (alwaysAuth && requestUrl.startsWith(registry))) {
      headers.authorization = this.getAuth(pathname);
    }

If registry is http://registry.npmjs.org/ then an https request will fail to have auth attached because of requestUrl.startsWith(registry).

---

I had yarn config get registry set to registry.yarnpkg.org and that was being used when trying to get my private module, instead of using @my-org:registry': 'https://registry.npmjs.org/',.

---

So the fix for me was:

//if (this.token || (alwaysAuth && requestUrl.startsWith(registry))) {
if (this.token || (alwaysAuth)) {

I was also getting an initial call to the NpmRegistry#request to @my-org%2fmodule.

[devongovett]

Yes, currently it replaces https://registry.npmjs.com/ with https://registry.yarnpkg.com/ here, which confuses the check here.

[vjpr]

EDIT: Ignore this post - it just started working for some reason.

I had to make sure to login to the scope on npm, using npm adduser --registry=http://registry.npmjs.org --scope=@foo --always-auth.

---

When I run:

npm3 adduser --registry=http://registry.npmjs.org --scope=@foo --always-auth

My npm looks like this:

_auth="xxx"
[email protected]
strict-ssl=false
//registry.npmjs.org/:_authToken=xxx
registry=http://registry.npmjs.org/
@foo:registry=http://registry.npmjs.org/
save=false
save-exact=false
save-prefix=^
always-auth=true

NpmRegistry#getAuth looks like this:

  getAuth(packageName: string): string {

    if (this.token) {
      return this.token;
    }

    for (let registry of [this.getRegistry(packageName), '', DEFAULT_REGISTRY]) {
      registry = registry.replace(/^https?:/, '');

      // Check for bearer token.
      console.log({registry})
      let auth = this.getScopedOption(registry, '_authToken');
      if (auth) {
        return `Bearer ${String(auth)}`;
      }

      // Check for basic auth token.
      auth = this.getScopedOption(registry, '_auth');
      if (auth) {
        return `Basic ${String(auth)}`;
      }

      // Check for basic username/password auth.
      const username = this.getScopedOption(registry, 'username');
      const password = this.getScopedOption(registry, '_password');
      if (username && password) {
        const pw = new Buffer(String(password), 'base64').toString();
        return 'Basic ' + new Buffer(String(username) + ':' + pw).toString('base64');
      }
    }

    return '';
  }

It ends up using the authorization header Basic xxx. It is using the _auth key.

[ikosenn]

Hey,
Has anyone managed to publish to a private npm registry created with Sinopia. I am able to do so with npm publish but yarn publish takes forever on the publishing step. I have changed the registry with yarn config set registry. Something else I noted I am not prompted for my password in the login step

[Tapppi]

Is a fix on the way for private packages? The problem @devongovett described above just bit me in CI. My current workaround is to yarn config set registry https://registry.npmjs.org/ so that yarn sets the auth token on requests for private packages.

[rovansteen]

I'm also running in the issue that yarn login doesn't ask for a password, therefore I am not able to use Gemfury (https://gemfury.com). I am not sure if it's related to this issue though. Should I create a separate issue for this?

[jmonster]

Another use case I haven't seen mentioned:

git repositories can be fetched via https or ssh. If the repo is private, you need credentials (duh). When deploying to Heroku, .netrc is the optimal way to authenticate using the .netrc buildpack

[jamiebuilds]

@rovansteen yarn login intentionally does not ask for a password. We do not want to store credentials or api tokens because that's a bad security practice

[rovansteen]

@thejameskyle ah, that makes sense. I noticed Gemfury also has a way to use an API token and that works fine with Yarn. Thanks!

[dfreeman]

We're having issues fetching the actual tarballs from our private scope-associated registry. The metadata is coming in fine, but it looks like the authorization header isn't being included in the request for the tarball.

This line in NpmRegistry#request seems to be the culprit—it calls getRegistry with the path of the tarball, when it seems to be expecting a package name instead. Because of that, it's not able to discover the scope and it falls back to the settings for the default registry.

I could imagine extending getScope to attempt to determine the scope from the URL (and I'd be happy to open a PR doing so), but that seems potentially error-prone. Maybe the associated package name needs to be plumbed through?

(Edit: Looks like this also came up in https://github.com/yarnpkg/yarn/issues/1619#issuecomment-258282647)

[gerges-zz]

I'm seeing the same issue as @dfreeman, the scope is recognized and the registry is queried. The correct tarball url and hash are retrieved, but the download doesn't contain the correct Authorization headers resulting in a download with no response body. Yarn then bails out with a hash mismatch. The error always complains with a but got da39a3ee5e6b4b0d3255bfef95601890afd80709 which is the sha of an empty file.

> touch empty
> openssl sha1 empty
SHA1(empty)= da39a3ee5e6b4b0d3255bfef95601890afd80709

EDIT: Let me know if this should be raised as a new issue

[maybeec]

Basically, I got yarn somehow authenticating against jfrog. However, publishing seems to be broken.
It is just getting stuck and I am not sure how to proceed as there is no verbose log or anything else. I could not even see any network traffic caused by an upload.

$ yarn publish --access restricted --new-version 2.0.2+1478176271464 .
yarn publish v0.16.1
[1/4] Bumping version...
info Current version: 2.0.2+SNAPSHOT
info New version: 2.0.2+1478176271464
[2/4] Logging in...
[3/4] Publishing...

[jamiebuilds]

Please use upvotes rather than commenting.

[hereandnow]

i know we should upvote and not comment, but the last comment is almost 2 months old and i'm kind of unsure if there is anything we could help with?!

[excenter]

having a .yarnrc file at the root of your user folder (on mac) containing

registry "https://npm.some-internal-site.tld"

I was able to download some internal packages. The big caveat being it's access controlled by the site rather than username/password.
hope this helps.

[shakefu]

Pretty sure this is working ... ? I've been using yarn with privately scoped packages for couple months.

@thejameskyle - Are you still having trouble with private scoped packages?

[SEAPUNK]

@shakefu The only problem I have with private packages at this point is me needing to add a registry=https://registry.npmjs.org/ to the top of my .npmrc file, because npm login just adds the line with the token, but nothing else.

[xealot]

We also use yarn with privately scoped packages and it works fine. The only thing my .npmrc has in it is a prefix=/Users/./npm and a //registry.npm.../authToken line.

Maybe make sure npm is up to date, clear file and login again?

Another person on our team found that the registry directive in .npmrc completely screwed up their ability to get privately scoped packages.

[cilindrox]

@hereandnow my .npmrc file is located on the root of the project and contains the following:

//registry.npmjs.org/:_authToken=${NPM_TOKEN}

As mentioned, privately scoped packages work the same as with npm.
I'm assuming yarn version is >= 0.19.1 and you're exporting NPM_TOKEN somewhere.

[hereandnow]

thank you guys for your help'!

[ghost]

It works only if the .npmrc with //registry.npmjs.org/:_authToken=${NPM_TOKEN} is in the same root of the project. Not if is created in %LOCALUSER% folder (Mac or Windows) with the standard procedure: npm login.

How to fix this?

[SEAPUNK]

@carmelone try modifying the .npmrc in your home dir to include registry=https://registry.npmjs.org/ at the top.

[ghost]

No, i'm using private repository. And it works except for some @scopedpackages which I download from npmjs and I still don't know how to do.

[jakubzitny]

I think that is our issue too, @carmelone. The combination of private registry and npmjs registry is the problem, right?

I can install SOME packages from private registry when I add the private registry scop to ~/.yarnrc. SOME are the ones that don't have their own dependencies from NPMJS registry.

[ghost]

@jakubzitny, yes and also when I npm login npm correctly creates in my %USERFOLDER% a file .npmrc with authentication token. I don't want to have that auth in .npmrc of the project, just in my username folder on my PC. This is the problem. yarn doesn't recognize that authentication token.

[Tapppi]

@jakubzitny @carmelone So the problem is mixing nested dependencies from private and public repository? I would bet that does not work on yarn at the moment. That sounds like you should open another issue.

[cpsubrian]

I was trying to get private scoped modules (in the npm registry) to work and I had to set the registry config in the .yarnrc (via yarn config).

$ yarn config set registry https://registry.npmjs.org/

[shakefu]

For the folks who've had to set the registry, did you try npm login --scope=@<yourscope>? That has worked for me every time.

[shishkin]

What doesn't work, at least for me, is scoped private packages from a bintray.com private registry. What happens there is the registry authentication during dependency resolution works, but the download link that bintray.com registry returns points to a different domain and thus yarn is not sending the correct authentication token to download it.

[Tapppi]

@shishkin have you tried with 'npm config set always-auth true'. This setting is off by default for security reasons. See https://docs.npmjs.com/misc/config

[shishkin]

@shishkin have you tried with 'npm config set always-auth true'. This setting is off by default for security reasons. See https://docs.npmjs.com/misc/config

@Tapppi Yes, I tried. Same error.

[vitalbone]

Thought I'd put this here, in case anyone else is still having trouble installing private packages, even if you have a newer version (>v0.16) of yarn installed on your local machine. You may be seeing a 404 error, something along the lines of:

error An unexpected error occurred: "https://registry.yarnpkg.com/@{ORG}/{PACKAGE}/-/{PACKAGE}-1.0.0.tgz: Request failed \"404 Not Found\"".
info If you think this is a bug, please open a bug report with the information provided in "/Users/{USER}/{project}/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

This worked for me, hope it helps.

npm logout
yarn logout

rm -rf ~/.npmrc
rm -rf ~/.yarnrc

npm login
yarn login

[tr4g]

So can I install a private package today by just using yarn add ? or do I need to do something else

[rchanou]

I tried @vitalbone's steps on Windows 10 with v0.22.0-20170228.1421. I still got the error and had to comment out the auth line in my .npmrc to get it to work.

[StephanBijzitter]

Logging out and in is definitely not an option for us, as that would invalidate all existing tokens.

[shakefu]

@StephanBijzitter I don't think you need to log out or in again, removing the *rc files should allow you to get new tokens while logging in without invalidating old ones. Not sure if that will work for you, though.

[StephanBijzitter]

Yeah, I confirm that generating (or storing, whatever you prefer) an .npmrc file works with Yarn for private, scoped packages.

[maximgeerinck]

@myprivaterepo:registry=https://npm.myprivaterepo/
//npm.myprivaterepo/:_authToken=$NPM_TOKEN

Doesn't seem to pick up the .npmrc as i get a forbidden error when it access this private package

[motss]

I've been struggling with this issue for so long. We internally hosted our own private NPM repositories using Sonatype but couldn't install with Yarn but NPM worked just fine.

[jonsharratt]

We managed to get our Codebox private npm project (https://github.com/craftship/codebox-npm) working using the always-auth=true (https://github.com/craftship/codebox-npm/issues/30) option in the .npmrc file.

Although you can get a hosted registry the project itself is completely open source so if your team uses GitHub (as it uses it for authentication) and you are on AWS you can deploy it pretty easily using the Serverless framework.

Just thought I would share.

[sylvesteraswin]

I am also having the same issue with Sinopia. Did anyone find a solution for this?

[BohdanTkachenko]

It does not work properly in all environments with .npmrc located at ~/.npmrc. On my local machine it works fine, but when I'm running this in Docker, it does not see ~/.npmrc when cwd is not ~. You can check it with yarn config list command.

On my local machine it outputs:

yarn config v0.23.2
info yarn config
{ 'version-tag-prefix': 'v',
  'version-git-tag': true,
  'version-git-sign': false,
  'version-git-message': 'v%s',
  'init-version': '1.0.0',
  'init-license': 'MIT',
  'save-prefix': '^',
  'ignore-scripts': false,
  'ignore-optional': false,
  registry: 'https://registry.yarnpkg.com',
  'strict-ssl': true,
  'user-agent': 'yarn/0.23.2 npm/? node/v7.9.0 darwin x64',
  lastUpdateCheck: 1492804696073 }
info npm config
{ '//npm.example.com/:_authToken': 'XXXXX-YYYYYY-ZZZZZ',
  '@example:registry': 'https://npm.example.com/' }
✨  Done in 0.05s.

While inside of Docker it outputs:

root@a1c3c4fb1fb8:/app# yarn config list
yarn config v0.23.2
info yarn config
{ 'version-tag-prefix': 'v',
  'version-git-tag': true,
  'version-git-sign': false,
  'version-git-message': 'v%s',
  'init-version': '1.0.0',
  'init-license': 'MIT',
  'save-prefix': '^',
  'ignore-scripts': false,
  'ignore-optional': false,
  registry: 'https://registry.yarnpkg.com',
  'strict-ssl': true,
  'user-agent': 'yarn/0.23.2 npm/? node/v7.9.0 linux x64',
  lastUpdateCheck: 1492856034840,
  version: '0.23.2' }
info npm config
{ version: '0.23.2',
  loglevel: 'info' }
Done in 0.03s.

So it looks like it does not execute npm config correctly.

As temporary workaround for this, in Docker I just copy ~/.npmrc to /app/.npmrc.

[stereobooster]

found here https://github.com/uber/react-map-gl

yarn start v0.23.2
$ (cd examples/custom-interactions && (path-exists node_modules || yarn) && yarn run start-local)
sh: path-exists: command not found
yarn install v0.23.2
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
error An unexpected error occurred: "https://unpm.uberinternal.com/flow-remove-types/-/flow-remove-types-1.1.2.tgz: Request failed \"401 Unauthorized\"".
info If you think this is a bug, please open a bug report with the information provided in "/react-map-gl/examples/custom-interactions/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
error Command failed with exit code 1.

[elibal]

I am also having the same issue with kendo-angular components.

C:\WorkingFolder\Projects\NG4\wck-management>yarn
yarn install v0.23.4
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
error An unexpected error occurred: "http://registry.npm.telerik.com/@progress%2
fkendo-angular-buttons/-/kendo-angular-buttons-1.0.0.tgz: Request failed \"503 S
ervice Unavailable\"".
info If you think this is a bug, please open a bug report with the information p
rovided in "C:\WorkingFolder\Projects\NG4\wck-management\yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this
command.

[bestander]

It should be fixed now.
If you have some custom example where it does not work please open a new issue.
It is important to provide steps to reproduce in this cases.

[kachkaev]

Awesome @bestander! What's the minimum yarn version where it’s expected to work?

[bestander]

There were multiple fixes across versions.
0.26 should have them all

On Tue, 23 May 2017 at 19:44, Alexander Kachkaev [e-mail]
wrote:

Awesome @bestander https://github.com/bestander! What's the minimum
yarn version where it’s expected to work?

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/yarnpkg/yarn/issues/521#issuecomment-303492335, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACBdWC90PKrW5LAg7HyY3l7c8ZoKaWtyks5r8yb9gaJpZM4KPd0N
.

[klofi]

I can confirm that private scoped packages with scope and registry defined in .npmrc started working in Yarn 0.24.6 (did not work in Yarn 0.24.5). Thank you!

[Mart112358]

yarn install
yarn install v0.24.6
info No lockfile found.
[1/4] Resolving packages...
warning cldr-data > cldr-data-downloader > [email protected]: this package has been reintegrated into npm and is now out of date with respect to npm
warning cldr-data > cldr-data-downloader > request > [email protected]: Use uuid module instead
[2/4] Fetching packages...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
error An unexpected error occurred: "http://registry.npm.telerik.com/@progress%2fkendo-angular-inputs/-/kendo-angular-inputs-1.0.3.tgz: ESOCKETTIMEDOUT".
info If you think this is a bug, please open a bug report with the information provided in "[...]\yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

[beatrizaldaz]

yarn install v0.24.6
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "http://registry.npm.telerik.com/@progress%2fkendo-angular-l10n/-/kendo-angular-l10n-1.0.0.tgz: connect ETIMEDOUT 23.253.4.114:80".
info If you think this is a bug, please open a bug report with the information p
rovided in "....\yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this
command.

[bestander]

Looks like you can't connect to telerik.com, is http proxy configured?

[beatrizaldaz]

The problem is yarn are searching by http://registry.npm.telerik.com, when I have configured yarn with strict-ssl to true:

info yarn config
{ 'version-tag-prefix': 'v',
'version-git-tag': true,
'version-git-sign': false,
'version-git-message': 'v%s',
'init-version': '1.0.0',
'init-license': 'MIT',
'save-prefix': '^',
'ignore-scripts': false,
'ignore-optional': false,
registry: 'https://registry.yarnpkg.com',
'strict-ssl': true,
'user-agent': 'yarn/0.24.6 npm/? node/v6.9.5 win32 x64',
lastUpdateCheck: 1496137030541 }
info npm config
{ 'strict-ssl': true,
'@progress:registry': 'https://registry.npm.telerik.com/',
'//registry.npm.telerik.com/:_authToken': '......' }
Done in 0.04s.

Previously I've configured the login with npm: "npm login --registry=https://registry.npm.telerik.com/ --scope=@progress"

It is neccesary that yarn searching for by "https" (https://registry.npm.telerik.com) :-)

Any idea what is the problem?

[bestander]

I don't have much experience using private registries :(
If someone sets up an account and a project package.json/yarn.lock for me
to test I could help out.

On 30 May 2017 at 10:53, beatrizaldaz [e-mail] wrote:

The problem is yarn are searching by http://registry.npm.telerik.com,
when I have configured yarn with strict-ssl to true:

info yarn config
{ 'version-tag-prefix': 'v',
'version-git-tag': true,
'version-git-sign': false,
'version-git-message': 'v%s',
'init-version': '1.0.0',
'init-license': 'MIT',
'save-prefix': '^',
'ignore-scripts': false,
'ignore-optional': false,
registry: 'https://registry.yarnpkg.com',
'strict-ssl': true,
'user-agent': 'yarn/0.24.6 npm/? node/v6.9.5 win32 x64',
lastUpdateCheck: 1496137030541 }
info npm config
{ 'strict-ssl': true,
'@progress https://github.com/progress:registry': '
https://registry.npm.telerik.com/',
'//registry.npm.telerik.com/:_authToken': '......' }
Done in 0.04s.

Previously I've configured the login with npm: "npm login --registry=
https://registry.npm.telerik.com/ --scope=@progress
https://github.com/progress"

It is neccesary that yarn looking for by "https" (
https://registry.npm.telerik.com) :-)

Any idea what is the problem?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/yarnpkg/yarn/issues/521#issuecomment-304830178, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACBdWA3GrsEnpn64ppJRL2cvJ-ayhvh5ks5r--cFgaJpZM4KPd0N
.

[beatrizaldaz]

OK, I will create a project, with a telerik trial account, to test the problem and I will send you the project link in github.

[bestander]

That would be great, @beatrizaldaz.
Can you open a new issue just for that case then?
It would be easier to track it isolated.

[dmiorandi]

About @beatrizaldaz post / Telerik. I've got same issue. In detail I've made some attemps
using following config (.npmrc). It seems almost to work but connections is made in http instead https
so is refused. Are there any temporary workaround about this (strict mode does not work)?

@progress:registry=https://registry.npm.telerik.com/
//registry.npm.telerik.com/:_authToken="YOUR_SECRET_HERE"
always-auth=true
registry="https://registry.npmjs.com/"

[balanceiskey]

So I just ran into this yesterday (yarn was at 0.24.6). I'm not sure what exactly caused it as it's been working fine for awhile. My solution was to remove both the .npm folder and .npmrc file altogether, run yarn cache clean, login again with npm login and things appeared to work fine after that. I've been jumping between versions of node and npm via nvm more lately, possible culprit? Worth noting, I also uninstalled and reinstalled yarn with brew using the --ignore-dependencies flag at some point during troubleshooting, but that by itself did not resolve the issue.

[balanceiskey]

One more note, if it's even relevant, when attempting yarn login during the course of these steps it would hang on the password prompt for some reason, which is why I did npm login.

[olalonde]

Related: https://github.com/yarnpkg/yarn/issues/2738

[demurgos]

Hi,
It's been a few months: are there any news?