yarn audit intermittent 503s on monorepos

[MarkConway94]

Do you want to request a feature or report a bug?
bug

What is the current behavior?
Running yarn audit on a monorepo is returning with a 503 error more often than not.

If the current behavior is a bug, please provide the steps to reproduce.
If you run npx [email protected] against https://github.com/newsuk/times-components in the root directory you will see:

➜  times-components git:(master) ✗ npx yarn@latest audit
npx: installed 1 in 4.242s
yarn audit v1.13.0
/Users/mark/.npm/_npx/94131/lib/node_modules/yarn/lib/cli.js:66237
            throw new (_errors || _load_errors()).ResponseError(_this3.reporter.lang('requestFailed', description), res.statusCode);
            ^

Error: Request failed "503 Service Unavailable"
    at ResponseError.ExtendableBuiltin (/Users/mark/.npm/_npx/94131/lib/node_modules/yarn/lib/cli.js:702:66)
    at new ResponseError (/Users/mark/.npm/_npx/94131/lib/node_modules/yarn/lib/cli.js:808:124)
    at Request.params.callback [as _callback] (/Users/mark/.npm/_npx/94131/lib/node_modules/yarn/lib/cli.js:66237:19)
    at Request.self.callback (/Users/mark/.npm/_npx/94131/lib/node_modules/yarn/lib/cli.js:129397:22)
    at Request.emit (events.js:159:13)
    at Request.<anonymous> (/Users/mark/.npm/_npx/94131/lib/node_modules/yarn/lib/cli.js:130369:10)
    at Request.emit (events.js:159:13)
    at IncomingMessage.<anonymous> (/Users/mark/.npm/_npx/94131/lib/node_modules/yarn/lib/cli.js:130291:12)
    at Object.onceWrapper (events.js:254:19)
    at IncomingMessage.emit (events.js:164:20)

What is the expected behavior?
Expect to consistently get back any vulnerabilities.

Please mention your node.js, yarn and operating system version.
node v9.3.0
yarn v1.13.0
macOS High Sierra 10.13.3

[jmrossy]

Related to #6929 which was closed but this is still happening.

[cyrilchampier]

same problem here, without anything reported here:
https://status.npmjs.org/

verbose 1.949 Performing "POST" request to "https://registry.yarnpkg.com/-/npm/v1/security/audits".
verbose 15.114 Request "https://registry.yarnpkg.com/-/npm/v1/security/audits" finished with status code 503.
/Users/dcyril/.nvm/versions/node/v10.15.0/lib/node_modules/yarn/lib/cli.js:66237
            throw new (_errors || _load_errors()).ResponseError(_this3.reporter.lang('requestFailed', description), res.statusCode);
            ^

Error: Request failed "503 Service Unavailable"
    at ResponseError.ExtendableBuiltin (/Users/dcyril/.nvm/versions/node/v10.15.0/lib/node_modules/yarn/lib/cli.js:702:66)
    at new ResponseError (/Users/dcyril/.nvm/versions/node/v10.15.0/lib/node_modules/yarn/lib/cli.js:808:124)
    at Request.params.callback [as _callback] (/Users/dcyril/.nvm/versions/node/v10.15.0/lib/node_modules/yarn/lib/cli.js:66237:19)
    at Request.self.callback (/Users/dcyril/.nvm/versions/node/v10.15.0/lib/node_modules/yarn/lib/cli.js:129397:22)
    at Request.emit (events.js:182:13)
    at Request.<anonymous> (/Users/dcyril/.nvm/versions/node/v10.15.0/lib/node_modules/yarn/lib/cli.js:130369:10)
    at Request.emit (events.js:182:13)
    at IncomingMessage.<anonymous> (/Users/dcyril/.nvm/versions/node/v10.15.0/lib/node_modules/yarn/lib/cli.js:130291:12)
    at Object.onceWrapper (events.js:273:13)
    at IncomingMessage.emit (events.js:187:15)

[rally25rs]

The 503 is returned from npm's API. (registry.yarnpkg.com is a redirect to npm).
Unfortunately there isn't much we can do about this since we have no control over the server.

[jmrossy]

@rally25rs Thanks for the reply. Are you certain? Why doesn't status.npmjs.org report any issues?
I've seen this error consistently for over a month.

[rally25rs]

They are pretty much the same... DNS will resolve them to the same IP range. registry.yarnpkg.com maps to yarn.npmjs.org which resolves to the same set of IPs as normal registry.npmjs.org.

I think they use a separate subdomain for yarn traffic for usage statistics purposes, but I could be wrong there.

$ nslookup registry.npmjs.org
Server:     10.0.2.3
Address:    10.0.2.3#53

Non-authoritative answer:
Name:   registry.npmjs.org
Address: 104.16.24.35
Name:   registry.npmjs.org
Address: 104.16.17.35
Name:   registry.npmjs.org
Address: 104.16.26.35
  .
  .
  .

$ nslookup registry.yarnpkg.com
Server:     10.0.2.3
Address:    10.0.2.3#53

Non-authoritative answer:
registry.yarnpkg.com    canonical name = yarn.npmjs.org.
Name:   yarn.npmjs.org
Address: 104.16.17.35
Name:   yarn.npmjs.org
Address: 104.16.18.35
Name:   yarn.npmjs.org
Address: 104.16.24.35
  .
  .
  .

[calvinli]

I've been seeing these 503s as well, it doesn't seem to have anything to do with NPM's server capacity, as it's consistent by repo. I suspect the 503 just means the backend reached a ~10 second timeout.

On one repo that was consistently failing I was able to get it to work by removing one dependency from the submitted JSON payload, but the 503s don't seem to be correlated to either payload size (in bytes) or dependency tree size, because there's another repo with a much larger dependency tree (totalDependencies > 200,000) that works just fine.

[rpellerin]

We had the same exact issue as you @calvinli . Removing a bunch of dependencies did the trick. It has to be linked to the payload size, timeout on yarn's end, of something like that.

[gumarm]

We were having this same issue with 100% failure rate when we upgraded our CI to use the latest yarn version. We have downgraded back and this issue doesn't appear anymore. It seems like it's not entirely npm's fault.

[donni106]

Same problem for me with yarn 1.12.3. In one project it is working and in the other sadly it is not.
After upgrading to yarn 1.13.0 nothing changed.

Edit: removing the package "babel-bridge" (last publish 2 years ago) made my yarn audit succeed.

[frdrks]

I have this issue as well. The audit fails consistently with a 503 error, for one specific project. While everything works fine in my other projects.

After a little investigation, it turns out that jest is the culprit. If i remove it from the dev dependencies, the audit works again. This suggests that there might be an issue with the audit command in Yarn?

[walter]

Also having consistent errors like this for an old ember project that has been recently upgraded to ember 3.8.1. My workaround was to chuck my yarn.lock file, run yarn to recreate it from scratch. yarn audit then began working.

Still evaluating if by doing this I've messed the project up, but thought I would mention it as many people seem to be experiencing this type of problem with yarn audit. I.e. that some package is problematic and causing it to consistently barf.

[andypillip]

After a little investigation, it turns out that jest is the culprit.

Not in my case.

[frdrks]

It seems to be working again. So it looks like my issue sorted itself out in the latest update of yarn, or jest.

[outime]

I consistently get 503 in this huge project. It's not related to yarn in my case at least as sending the request manually to npmjs (not even using yarn's proxy) will result in the same.

[lucastschmidt]

I consistently get 503 in this huge project. It's not related to yarn in my case at least as sending the request manually to npmjs (not even using yarn's proxy) will result in the same.

Guys ... lets think about it ... if the npm service cannot handle mono repos, due to the amount of deps ... we could simply perform multiple requests ... instead of a big one. I dont know how difficult this is to implement in yarn, never took a look at the source code.

Just a suggestion, not a critical thing right now for me. If it comes, I will try to evaluate the code.