Berry: [Bug] Azure artifacts authentication

[Js-Brecht]

Describe the issue

Really not sure this is a bug; more like a support question. I am having trouble getting my Azure DevOps Artifacts npm registry working with berry. It works fine in v1, using the token from .npmrc, which is generated by Azure DevOps.

The ":_password" field in the .npmrc is a base64 encoded Personal Access Token generated in Azure DevOps, and according to their documentation, the ":username" and ":email" field mean nothing. Documentation is found here

I have tried generating the auth token through their Artifact "Connect to feed" process, which generates the token for your .npmrc like this

; Treat this auth token like a password. Do not share it with anyone, including Microsoft support. This token expires on or before 10/10/2019.
; begin auth token
//pkgs.dev.azure.com/<organization>/_packaging/<artifact feed>/npm/registry/:username=<username removed.  It was the name of my organization or feed... they are the same>
//pkgs.dev.azure.com/<organization>/_packaging/<artifact feed>/npm/registry/:_password=<password removed>
//pkgs.dev.azure.com/<organization>/_packaging/<artifact feed>/npm/registry/:email=npm requires email to be set but doesn't use the value
//pkgs.dev.azure.com/<organization>/_packaging/<artifact feed>/npm/:username=<username removed>
//pkgs.dev.azure.com/<organization>/_packaging/<artifact feed>/npm/:_password=<password removed>
//pkgs.dev.azure.com/<organization>/_packaging/<artifact feed>/npm/:email=npm requires email to be set but doesn't use the value
; end auth token

I also tried generating my own PAT, and base64 encoding it, for use in that "_password" field.

I used that "_password" field as the npmAuthToken in my .yarnrc.yml, I've tried using it in the npmAuthIdent in combination with the username from above, as <username>:<token>. I've tried combining the <username>:<token> into a base64 encoded string and including that in the npmAuthToken. None of these work. I always get (401) Unauthorized.

To Reproduce

You would need a module published in an Azure DevOps Artifacts registry.

My .yarnrc.yml is below, with the sensitive information redacted.

npmRegistries:
  //pkgs.dev.azure.com/<organization>/_packaging/<azurefeed>/npm/registry:
    npmAlwaysAuth: true
    npmAuthToken: "<_password field from above>"
  //pkgs.dev.azure.com/<organization>/_packaging/<azurefeed>/npm:
    npmAlwaysAuth: true
    npmAuthToken: "<_password field from above>"

npmScopes:
  <org>:
    npmRegistryServer: https://pkgs.dev.azure.com/<organization>/_packaging/<azurefeed>/npm/registry

Environment if relevant (please complete the following information):

• OS: Linux
• Node version: v12.6.0
• Yarn version: v2.0.0

[arcanis]

Can you try using npmAuthIdent instead of a token? Given what Azure asks you to use as configuration, I think they're expecting the token to be sent as a password rather than a token, for some reason.

If it still doesn't work it'd be helpful if you could add some logs near the place that makes the http request to find out whether the server answers with the reason why the request is rejected.

[Js-Brecht]

I did try using npmAuthIdent, with the same "username" and "_password" fields from the .npmrc, in the <username>:<password> format. Still got the error. I actually tried that before I tried using npmAuthToken field. The configuration I left in my original post was the final state it was in before I gave up.

Here's the things I tried:

• Generate .npmrc configuration using Azure's "Connect To Feed"

  • Using .npmrc fields _password & username in .yarnrc.yml -> npmRegistries:

    • npmAuthToken: <_password>
    • npmAuthToken: <base64 encoded(_password)>
    • npmAuthToken: <username:_password>

    • npmAuthToken: <base64 encoded(username:_password)>

    • npmAuthIdent: <username:_password>

    • npmAuthIdent: <username:<base64 encoded(_password)>>

• Generate PAT

  • Then I did the same thing as above with the generated PAT as _password

I'm now trying this on a Windows PC. What's interesting is their vsts-npm-auth utility generates an _auth field alone in the .npmrc. It creates an .npmrc config that looks like this:

//pkgs.dev.azure.com/<organization>/_packaging/<azurefeed>/npm/registry/:_authToken=<some ridiculously long token>

That's it. It doesn't have the two different URL's (one with /registry at the end, and one without). Just the one. So I tried using that _auth token in the .yarnrc.yml configuration, as the npmAuthToken field, and was actually able to resolve packages. But still cannot fetch them. I get this:

➤ YN0001: │ HTTPError: <@scope/package-name>@npm:1.2.0: Response code 400 (Authentication information is not given in the correct format. Check the value of Authorization header.)
    at EventEmitter.emitter.on (D:\dev\source\test\yarn-test\.yarn\releases\yarn-berry.js:9730:19)
    at process._tickCallback (internal/process/next_tick.js:68:7)

I tried using that token in the various methods I described above, but the only way it worked was just that token in the npmAuthToken field. The error it would return any other way was this:

➤ BR0027: <@scope/package-name>@unknown can't be resolved to a satisfying range
➤ Errors happened when preparing the environment required to run this command.

I don't know how that vsts-npm-auth utility is generating the token. There's very little documentation on it, and absolutely nothing that details that process, plus it's not open source, so I can't just examine the source.

All I know is that Yarn v1 doesn't have any problem with the .npmrc _auth token, or the _password method either. Neither does npm, or pnpm. Just Yarn/berry. So, it's doing something different, or I have it misconfigured somehow.

Does berry have a built in logging utility, or will I need to do a tcpdump to get the logs you mentioned?

[arcanis]

Very strange ... I think the main difference is that we're not sending the email address as it's unused by all registries I'm aware of 🤔

Does berry have a built in logging utility, or will I need to do a tcpdump to get the logs you mentioned?

The best is to just add a console.log statement before the following line (it should be fairly easy to find it in the standalone bundle):

https://github.com/yarnpkg/berry/blob/master/packages/berry-core/sources/httpUtils.ts#L63

[Js-Brecht]

I put a console.log in there, but it only reported anything during the resolution stage, and those are successful (return code 200). Then it fails during the fetch stage with the 400 error (Authentication information is not given in the correct format. Check the value of Authorization header.), and the console.log doesn't get hit at all. This is on my Windows machine at work.

I will try this again on my Linux box tonight to get some logs for that 401 (Unauthorized) error.

I realized what I've been doing wrong with npmAuthIdent after replicating the process with curl. curl, by default, base64 encodes the string that you pass to the -u option. That made me realize that I've been encoding the field npmAuthIdent wrong this entire time. I have been encoding the PAT, but leaving the rest like username:<encoded PAT>, which looking at now makes no sense. I had also tried encoding the entire string, but only after I had already encoded the PAT... 🙄

So, after encoding <username>:<raw PAT> as base64, and putting that in the npmAuthIdent, I successfully authenticate with the server. So now I don't need the npmAuthToken (which would be too complicated to get for use on my Linux box). However, I am still getting the 400 (Authentication information is not given in the correct format. Check the value of Authorization header.) error. I do think it is expecting an email field, even if it doesn't mean anything.

Here's what I got from curl...

1. Hit the address that was being used as the target in the HttpUtil.
   
   
   
   • Got the JSON content for the package in question
   
   
2. Hit the tarball URL retrieved from the JSON

*   Trying 13.107.42.20...
* TCP_NODELAY set
* Connected to pkgs.dev.azure.com (13.107.42.20) port 443 (#0)
(SSL/TLS handshake omitted for brevity)
* Server auth using Basic with user '<my email address>'
> GET /<organization>/_packaging/<feed>/npm/registry/@<scope>/<package>/-/<package>-1.2.0.tgz HTTP/1.1
> Host: pkgs.dev.azure.com
> Authorization: Basic <base64 encoded <email:personal access token>
> User-Agent: curl/7.55.1
> Accept: */*
(schannel output omitted for brevity)
< HTTP/1.1 302 Found
< Cache-Control: no-cache
< Pragma: no-cache
< Expires: -1
< Location: https://1yovsblobprodeus2184.blob.core.windows.net/<server/relative/path>.blob?<package query>
< P3P: CP="CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT"
< X-TFS-ProcessId: b2b4c5ab-39f3-4afb-abb8-099a15970b28
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< ActivityId: 31953bf3-1506-4218-a267-8131454236f4
< X-TFS-Session: 31953bf3-1506-4218-a267-8131454236f4
< X-VSS-E2EID: 31953bf3-1506-4218-a267-8131454236f4
< X-VSS-UserData: <my user data>
< X-FRAME-OPTIONS: SAMEORIGIN
< Request-Context: appId=cid-v1:aafc8b1a-8b8b-40da-82f7-2e27c159556b
< Access-Control-Expose-Headers: Request-Context
< X-Content-Type-Options: nosniff
< X-MSEdge-Ref: Ref A: A41A8C44161C4BCEA1C27C42B3941EC9 Ref B: HNL01EDGE0213 Ref C: 2019-07-31T23:45:47Z
< Date: Wed, 31 Jul 2019 23:45:47 GMT
< Content-Length: 0
<
* Connection #0 to host pkgs.dev.azure.com left intact

1. Hit the file Location response, and this is what I get

*   Trying 52.179.144.64...
* TCP_NODELAY set
* Connected to 1yovsblobprodeus2184.blob.core.windows.net (52.179.144.64) port 443 (#0)
(SSL/TLS handshake omitted for brevity)
* Server auth using Basic with user '<my username (email)>'
> GET <server/relative/path>.blob?<package query> HTTP/1.1
> Host: 1yovsblobprodeus2184.blob.core.windows.net
> Authorization: Basic <base64 encoded <email:personal access token>>
> User-Agent: curl/7.55.1
> Accept: */*
>
(schannel output omitted for brevity)
< HTTP/1.1 400 Authentication information is not given in the correct format. Check the value of Authorization header.
< Content-Length: 298
< Content-Type: application/xml
< Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
< x-ms-request-id: 6079e633-601e-0064-19fa-4731fa000000
< Date: Wed, 31 Jul 2019 23:46:23 GMT
<
<?xml version="1.0" encoding="utf-8"?>
<Error><Code>InvalidAuthenticationInfo</Code><Message>Authentication information is not given in the correct format. Check the value of Authorization header.
RequestId:6079e633-601e-0064-19fa-4731fa000000
Time:2019-07-31T23:46:23.6899938Z</Message></Error>* Connection #0 to host 1yovsblobprodeus2184.blob.core.windows.net left intact

Where does an npm server usually expect to see the email address in the request? I can run that through curl, and if I am able to retrieve the tarball, we know that's the problem.

[Js-Brecht]

Nevermind, I figured it out.

I was able to retrieve the file succesfully by sending the authentication headers x-www-urlencoded

Using curl, I used -d with the www-urlencoded authentication information like this:

username=<whatever, doesn't matter>&_password=<raw PAT>

Then the --get option to force it to use the GET verb. If I just do -d, it would try to POST using Content-Type: application/x-www-urlencoded, and the server would not accept it (possible verbs are GET, HEAD, PUT, DELETE).
Using the GET verb, with the -d data formatted like above, it works.

curl -v -d "username=alsajskghaskl&_password=<PAT>" "https://1yovsblobprodeus2184.blob.core.windows.net/<server/relative/path>.blob?<crazy long package query, session id, sig, etc...>" --get --output <package name>-1.2.0.tgz

Hope this helps.

[arcanis]

But credentials are expected to be sent through the Authorization header, not the query string, I'm confused 😮

Oh wait I might know where this is coming from - in the v1, when resolving a package, we also were storing the returned archive url. In the v2, we only store the package version, and when we need to fetch the archive we combine the version to the registry according to the following convention: /<name>/-/<name>.tgz (this allows us to improve the UX when working with multiple registries, as switching from a mirror to another will properly fetch the tarballs from the new location). Maybe the tarball urls that Azure is returning don't match this convention and thus can't be accessed this way?

If that's the case we track that in #238 - it should be possible to fix it by storing non-conventional tarball urls within the lockfile, the main issue being that I cannot test it on my side since I don't have access to such environments ... so if someone else could do it it would be awesome 😅

This would however imply that the authentication credentials are somehow stored inside the lockfile when using Azure, which sounds surprising. Can you check?

[Js-Brecht]

Ah, yes, you're correct. It did seem a little odd to me that they would pass credentials that way, but I didn't even think to try it without. 🙄

The tarball naming URL's do match that convention, but that tarball URL redirects to the *.blob, which includes session IDs + various other query parameters (sv, sr, si, sig, spr, se, rscl, rscd). Most of those stay constant per each tarball, but the "rscl" appears to be a session ID, and the "sig" appears to be a type of access token, and they change on each request. I don't think storing that URL will be effective, because I doubt there's a guarantee for how long those parameters will be good.

The only thing they seem to care about when hitting that redirect is whether or not there IS an authorization header. Apparently, everything up to the redirect you get when you hit the tarball URL is authenticated, but the *.blob I get from the tarball URL is anonymous. It simply ignores those query parameters. So the 400 error about checking the Authorization header was just to say "why is there an Authorization header?" 🤣

EDIT:
It appears to be the way the redirect is being handled between versions. In v1, the authentication headers must not be sent with the redirected request, while in v2, they are.

[Js-Brecht]

Btw, I can say with certainty that the authentication credentials are not stored in the lockfile when using Azure. At least not with v1. It's just the standard naming convention:

"@<scope>/<package>@^1.2.0":                                                                                                                                                                                                                                            
   version "1.2.0"                                                                                                                                                                                                                                                                
   resolved "https://<registry url>/@<scope>/<package>/-/<package>-<version>.tgz#<hash>"                                                                                      
   integrity <sha1-hash>                                                                                                                                                                                                                                   
   dependencies:                                                                                                                                                                                                                                                       
       "@<scope>/<package>" <version>

I haven't been able to fetch the package successfully with v2 yet, so I can't say what the lockfile looks like for it.

[arcanis]

So just to be clear - the fix would be to do as we do, but if we receive a redirect not send the authentication header after handling the redirect? If so that seems acceptable. Would you be willing to make a PR? The relevant code is here:

https://github.com/yarnpkg/berry/blob/master/packages/berry-core/sources/httpUtils.ts#L63

[Js-Brecht]

Added pull request #329.

[arcanis]

Fixed by #329 🎉

[darthtrevino]

For future readers - tldr:
To connect to Azure Artifacts:

npmRegistryServer: "https://your-server"
npmAuthIdent: "base64(your-org:your-pat)"

your-pat is probably encoded as base64 in your ~/.npmrc file. You'll need to decode it first. before re-encoding the key.

[timclark97]

I was able to get passed authentication, but now I am getting a 404.
yarn add @my-scope/[email protected]
gives this error
@my-scope/my-package@npm:1.0.0: Response code 404 (Not Found)

The following returns 200
curl GET 'https://pkgs.dev.azure.com/<my-org>/_packaging/<my-feed>/npm/registry/<my-package>' --header 'Authorization: base64(my-org:my-pat)

Not sure if this is related to Yarn or Artifacts. Thought I would share this here in case somebody is having the same issue.
I have a question out on stack overflow too