Yarn: Can't install from private scoped registries

[KidkArolis]

Do you want to request a feature or report a bug?
This is a regression in 1.0.0.

Explanation
This regex const SCOPED_PKG_REGEXP = /(?:^|\/)(@[^\/?]+?)(?=%2f)/:
https://github.com/yarnpkg/yarn/blob/master/src/registries/npm-registry.js#L30

introduced in this commit:
https://github.com/yarnpkg/yarn/commit/cbb27f4fb0bb5860435ac0ae5f0dd3f0062825eb

or this PR:
https://github.com/yarnpkg/yarn/pull/4027

completely breaks support for private npm registries.

The request function is sometimes called with a package name (@my/pkg) and sometimes with a full URL (https://myregistry.dev/@my/pkg/download/@my/pkg-1.0.0.tgz).

The old regex /^@|\/@/ matched such URLs, but the new one/(?:^|\/)(@[^\/?]+?)(?=%2f)/ doesn't. This means that authorization token is not sent with any tgz requests.

cc due to involvement in the original PR @lukeggchapman @BYK

[KidkArolis]

Not sure how to best fix this yet, because I don't quite understand what the PR was trying to fix or why the regex was changed.

[BYK]

The thing is, per https://github.com/yarnpkg/yarn/pull/4027#issuecomment-318533400, https://myregistry.dev/@my/pkg/download/@my/pkg-1.0.0.tgz is not a valid scoped package URL. It should be https://myregistry.dev/@my/pkg/download/@my%2fpkg-1.0.0.tgz instead for it to be scoped. Ths is due to how NPM defined scoped packages. They still don't allow / in package names (even as a scope separator) and expects the URL-escaped version of it.

Can you give more details about your private server to help us resolve the issue?

[BYK]

You can try and see it yourself too:
https://registry.npmjs.org/@yarnpkg/lockfile gives you a 404 and https://registry.npmjs.org/@yarnpkg%2flockfile gets you the metadata for the scoped package.

[roderik]

I'm having the same issues with private scoped packages hosted on npmjs.com. Reverting back to 0.27.5 fixes this.

Example: https://www.npmjs.com/package/@settlemint/lib-ethereum

[KidkArolis]

The private registry implementation I'm using is https://github.com/cnpm/cnpmjs.org. We've been using it with yarn ever since the scoped package support was added in yarn. It broke with 1.0.0.

But even today scoped packages in both cnpm and official npm registry have slashes in the tarball urls.

Example 1 - https://registry.npmjs.org/@std%2fesm

Example 2 - http://registry.cnpmjs.org/@alibaba.ir/babel-preset-alibaba

Example 3 - https://registry.yarnpkg.com/@npm%2fdecorate

[BYK]

@KidkArolis - thanks a lot for the clarification!

I'm just very confused about this inconsistent behavior but I'll see if I can fix this clearly.

[lukeggchapman]

Yep so this is my fault, sorry everyone.
The private repo I was testing against was running verdaccio, which returns the tarball url with the escaped package name that is stored in the yarn.lock.
"tarball": "https://verdaccio.< private >.com:443/@scope%2fforms/-/forms-0.0.1.tgz"
which is inconsistent with KidkArolis's examples above and should probably change.

It seems that the url to get the package details have the package name escaped, but in the tarball url they are not.

[RadoRado]

Confirming - private packages installation is broken on 1.0.1. Reverting to 0.27.5 seems to be the solution for now.

[davidgwking]

similar experience as @RadoRado and @roderik , reverting to 0.27.5 is a short term fix

[BYK]

@lukeggchapman thanks for the updates! @KidkArolis are you using the same NPM registry?

If so, looks like that project is at fault here. Setting always-auth to true may be a workaround until the project fixes the bug and releases a new version, right?

Alternatively, yarn starts supporting both paths at the expense of being slightly incorrect. Thoughts?

[KidkArolis]

@lukeggchapman is using verdaccio, I'm using cnpmjs. If that's what you mean. But this is an issue with the official npm registry too!

I don't see this as a bug in any of the registries. As in, that's how npm official registry always worked and so I don't think yarn should break that compatibility. In fact - my guess is that the current version of yarn does not work with private npm packages in the official npm registry (not just these open source registries). I haven't verified that, because I don't have a private npm account myself, but judging by those tarball urls one would think yarn needs to correctly handle them.

I'm a little puzzled by the fact that npm no longer supports URL like https://registry.npmjs.org/@std/esm with slashes, I thought it used to support both (which isn't necessarily a good thing.

To complicate matters further, the tarball urls are generally not consistent between registries - you can see that in my screenshots in https://github.com/yarnpkg/yarn/issues/4366#issuecomment-328363479. The npm clients are probably just supposed to follow whatever they see there, but that of course makes authentication piece trickier - one has to assume some kind of structure to extract the scopes..

[runk]

I can confirm issue is with official npm registry too. See https://github.com/yarnpkg/yarn/issues/4387#issuecomment-328393602 comment with log. Same issue with both v0.28 and v1.

[lukeggchapman]

A workaround for 1.0 until there is a fix would be to always-auth: true

If this is an issue with just a scoped registry then you can set it to true just for that registry in your .npmrc file. Eg:

@testScope:registry=https://npm.private.com/
//npm.private.com/:always-auth=true

[lukeggchapman]

I'm all for supporting both, as it solves my use case.
That doesn't mean it's correct. If we're mimicking how NPM do things they do not support having the %2f in the tarball request.

Eg:

• https://registry.npmjs.org/@std/esm/-/esm-0.0.1.tgz works!
• https://registry.npmjs.org/@std%2fesm/-/esm-0.0.1.tgz gives {"error":"Not found"}

That being said, I don't see any harm in supporting both, the getScope method is now robust enough to support both.

[RadoRado]

If someone wants to revert, for us, adding npm install -g [email protected] in the CI before yarn step did the trick :+1:

[arutkowski00]

I have the same issue, after upgrading yarn from 0.27.5 to 1.0.2 i get Request failed \"404 Not Found\"". for private scoped packages that are not in my cache. The only workaround that works for me is to downgrade yarn back to 0.27.5: npm install -g [email protected]

[BYK]

@arutkowski00 this issue was closed and I think you are experiencing #4451 so I suggest following that.