Cli: [BUG] 429 Too Many Requests

[raphaelyancey]

What / Why

I'm getting E429 error

When

• Everytime I run npm ci (since today at least)

Where

• npm public registry

Current Behavior

• The npm ci command returns E429 error (Too Many Requests) and doesn't complete packages installation

Steps to Reproduce

• npm ci

Expected Behavior

• It should install packages

[44kia244]

Same here, but with npm -g install @vue/cli.

npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
npm ERR! code E429
npm ERR! 429 Too Many Requests - GET https://registry.npmjs.org/lowdb/-/lowdb-1.0.0.tgz

[hubertsmusz]

805 similar isseue - random 403 or 429

[MatejBalantic]

Having same issue on our pipelines. Responses vary between 403 Forbidden and 420 Too Many Requests

[kurrestahlberg]

We see this in any of our CI tasks running in AWS

Step 8/11 : RUN npm ci
 ---> Running in 87051ac87a51
npm ERR! code E429
npm ERR! 429 Too Many Requests - GET https://registry.npmjs.org/@types/xxhashjs/-/xxhashjs-0.2.1.tgz
npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2020-02-17T11_49_35_151Z-debug.log
The command '/bin/sh -c npm ci' returned a non-zero code: 1
ERROR: Job failed: exit code 1

[m33ch]

Also for me on bamboo build:

error   17-feb-2020 12:49:46    npm ERR! code E429
error   17-feb-2020 12:49:46    npm ERR! 429 Too Many Requests - GET https://registry.npmjs.org/@snyk/configstore/-/configstore-3.2.0-rc1.tgz

[quite]

Centralized infrastructure :~(
(waiting for the post-mortem, but not holding my breath...)

[raphaelyancey]

It'd be useful to have a list of (verified) public registry mirrors. I found some but I can't trust them.

[pwright08]

Same, both locally and on Circle CI

[Kellym-Kainos]

Also seeing the same using Circle CI and locally

npm ERR! code E429 npm ERR! 429 Too Many Requests - GET https://registry.npmjs.org/camelcase/-/camelcase-2.1.1.tgz

[jimhooker2002]

I'm seeing errors like..

"The owner of this website (registry.npmjs.org) has banned you temporarily from accessing this website"

and

"You are being rate limited"

I'm guessing this is all related?

[ymynem]

We are also having this issue when deploying on Heroku.

npm ERR! code E429
npm ERR! 429 Too Many Requests - GET https://registry.npmjs.org/strip-indent/-/strip-indent-1.0.1.tgz

[bitqueenbr]

Having the same errors when deploying on heroku.

[codyfy]

same here with AWS CodeBuild and npm i -g aws-cdk

> npm ERR! code E429

28 | npm ERR! 429 Too Many Requests - GET https://registry.npmjs.org/aws-cdk

general server issue?

[esrgrlk]

I also have same problem
npm ERR! code E429
npm ERR! 429 Too Many Requests - GET https://registry.npmjs.org/silent-error

[vrease]

Same here when installing packages locally.
Sweden.

```npm ERR! code E429
npm ERR! 429 Too Many Requests - GET https://registry.npmjs.org/lodash

[mattandrews]

Yep, I'm seeing this on Travis too for npm audit:

npm ERR! audit Your configured registry (https://registry.npmjs.org/) may not support audit requests, or the audit endpoint may be temporarily unavailable.

npm ERR! audit The server said:

Access denied | registry.npmjs.org used Cloudflare to restrict access
You are being rate limited
The owner of this website (registry.npmjs.org) has banned you temporarily from accessing this website.

[Franchi05]

Same thing happening over here. Getting the error when doing npm update

[wiseelf]

17-Feb-2020 11:47:48 npm ERR! code E429
17-Feb-2020 11:47:48 npm ERR! 429 Too Many Requests - GET https://registry.npmjs.org/@babel/plugin-transform-block-scoping/-/plugin-transform-block-scoping-7.8.3.tgz

Same issue here. We are using bamboo ci. Own installation.
The file is accessible from the server itself:

```$ wget https://registry.npmjs.org/@babel/plugin-transform-block-scoping/-/plugin-transform-block-scoping-7.8.3.tgz
--2020-02-17 11:59:28-- https://registry.npmjs.org/@babel/plugin-transform-block-scoping/-/plugin-transform-block-scoping-7.8.3.tgz
Resolving registry.npmjs.org (registry.npmjs.org)... 104.16.17.35, 104.16.24.35, 104.16.26.35, ...
Connecting to registry.npmjs.org (registry.npmjs.org)|104.16.17.35|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6735 (6.6K) [application/octet-stream]
Saving to: 'plugin-transform-block-scoping-7.8.3.tgz'

plugin-transform-block-scoping-7.8.3.tgz 100%[================================================================================================================================>] 6.58K --.-KB/s in 0s

2020-02-17 11:59:28 (95.5 MB/s) - 'plugin-transform-block-scoping-7.8.3.tgz' saved [6735/6735]```

[dtmzr]

Facing this issue too, is this a global thing or maybe region related? We just had something similar last year in Germany.

[dorsharon]

Same here running on Gitlab CI

[simonkarman]

Same here in The Netherlands. (AWS Codebuild from Ireland)

[serggend]

Russia to

[Volem]

Istanbul here

[NickThompson1993]

This appears to be a Cloudflare related issue to the registry.npmjs.org site.

got the following html response on update:

<!DOCTYPE html>
npm ERR! <!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
npm ERR! <!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
npm ERR! <!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
npm ERR! <!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
npm ERR! <head>
npm ERR! <title>Access denied | registry.npmjs.org used Cloudflare to restrict access</title>
npm ERR! <meta charset="UTF-8" />
npm ERR! <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
npm ERR! <meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
npm ERR! <meta name="robots" content="noindex, nofollow" />
npm ERR! <meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1" />
npm ERR! <link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" type="text/css" media="screen,projection" />
npm ERR! <!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" type="text/css" media="screen,projection" /><![endif]-->
npm ERR! <style type="text/css">body{margin:0;padding:0}</style>
npm ERR!
npm ERR!
npm ERR! <!--[if gte IE 10]><!--><script type="text/javascript" src="/cdn-cgi/scripts/zepto.min.js"></script><!--<![endif]-->
npm ERR! <!--[if gte IE 10]><!--><script type="text/javascript" src="/cdn-cgi/scripts/cf.common.js"></script><!--<![endif]-->
npm ERR!
npm ERR!
npm ERR!
npm ERR! </head>
npm ERR! <body>
npm ERR!   <div id="cf-wrapper">
npm ERR!     <div class="cf-alert cf-alert-error cf-cookie-error" id="cookie-alert" data-translate="enable_cookies">Please enable cookies.</div>
npm ERR!     <div id="cf-error-details" class="cf-error-details-wrapper">
npm ERR!       <div class="cf-wrapper cf-header cf-error-overview">
npm ERR!         <h1>
npm ERR!           <span class="cf-error-type" data-translate="error">Error</span>
npm ERR!           <span class="cf-error-code">1015</span>
npm ERR!           <small class="heading-ray-id">Ray ID: REDACTED &bull; 2020-02-17 11:26:27 UTC</small>
npm ERR!         </h1>
npm ERR!         <h2 class="cf-subheadline">You are being rate limited</h2>
npm ERR!       </div><!-- /.header -->
npm ERR!
npm ERR!       <section></section><!-- spacer -->
npm ERR!
npm ERR!       <div class="cf-section cf-wrapper">
npm ERR!         <div class="cf-columns two">
npm ERR!           <div class="cf-column">
npm ERR!             <h2 data-translate="what_happened">What happened?</h2>
npm ERR!             <p>The owner of this website (registry.npmjs.org) has banned you temporarily from accessing this website.</p>
npm ERR!           </div>
npm ERR!
npm ERR!
npm ERR!         </div>
npm ERR!       </div><!-- /.section -->
npm ERR!
npm ERR!       <div class="cf-error-footer cf-wrapper">
npm ERR!   <p>
npm ERR!     <span class="cf-footer-item">Cloudflare Ray ID: <strong>REDACTED</strong></span>
npm ERR!     <span class="cf-footer-separator">&bull;</span>
npm ERR!     <span class="cf-footer-item"><span>Your IP</span>: REDACTED</span>
npm ERR!     <span class="cf-footer-separator">&bull;</span>
npm ERR!     <span class="cf-footer-item"><span>Performance &amp; security by</span> <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=error_footer" id="brand_link" target="_blank">Cloudflare</a></span>
npm ERR!
npm ERR!   </p>
npm ERR! </div><!-- /.error-footer -->
npm ERR!
npm ERR!
npm ERR!     </div><!-- /#cf-error-details -->
npm ERR!   </div><!-- /#cf-wrapper -->
npm ERR!
npm ERR!   <script type="text/javascript">
npm ERR!   window._cf_translation = {};
npm ERR!
npm ERR!
npm ERR! </script>
npm ERR!
npm ERR! </body>
npm ERR! </html>

[CormacWhite-rsc]

Same issue happening with AWS Codebuild us-east-1. Was broken locally up til about 30 minutes ago but back working now (locally from Ireland)

[merc1er]

This appears to be a Cloudflare related issue to the registry.npmjs.org site.

Is there any mirror not using cloudflare?

[tschannik]

Same problem! Build pipelines are failing :(

[jmeade11]

Same: npm ERR! code E429

[ghost]

That's it. Internet is done. Good bye everyone.

[pedroac]

I'm going to have lunch and hope this will be fixed when I return in less than an hour.

[ahmatovicm]

We can pretty much confirm that this IS an npm issue, yet on their status page everything is listed as operational. What is then the purpose of the npm status page?

[marphi]

The same issue. AWS from us-east-1

npm ERR! code E429
npm ERR! 429 Too Many Requests - GET https://registry.npmjs.org/archy/-/archy-1.0.0.tgz

[dtmzr]

Just reached out on twitter, 🤞 that we'll have information quickly.

[roppazvan]

Same...

[andreisocaciu]

npm ERR! code E429
npm ERR! 429 Too Many Requests - GET https://registry.npmjs.org/react-scripts/-/react-scripts-3.4.0.tgz

[benchudev]

same. Different packages, but keep getting 429 too many requests doing npm install and npm ci, since earlier today

[16g]

:(

[shapeofarchitect]

We are all down since morning now . NPM is broken. Dublin here

[Wolfbyte-dev]

Works fine for me now. 5$ / package. PM me.

[bitqueenbr]

Works fine for me now. 5$ / package. PM me.

so cheap

[Hydrawisk793]

In south korea, I'm facing this issue too.

$ npm install --save-dev typescript
npm ERR! code E429
npm ERR! 429 Too Many Requests: [email protected]

Every NPM package just takes too much time to be installed.
What happend to NPM?

[AlexOlival]

Lucky we just need to sit and wait
Imagine if we were all construction workers, and suddenly all hammers stopped working around the world :thinking:

[anton-bot]

How about using the yarnpkg mirror for your builds?

[mvakula]

It's all fine http://status.npmjs.org/

[16g]

It's all fine http://status.npmjs.org/

[mvakula]

Indeed 😄

[anton-bot]

This discussion did not age well

https://github.com/yarnpkg/yarn/issues/5891

[simonkarman]

You could use: https://github.com/open-services/open-registry

# npm
npm config set registry https://npm.open-registry.dev

# yarn
yarn config set registry https://npm.open-registry.dev

[Shinchy]

Having the same issue in multiple environments (travis, local, server).

[anton-bot]

NPM: Nearly Perfect Mirror

[anton-bot]

NPM: Not Performing on Mondays

[roppazvan]

NPM: No Problem Monday

[kiknaio]

Same issue within Gitlab runners

[nandueverurs]

Same issue when tried a build in heroku . CF-error-code 1015.

The owner of this website (registry.npmjs.org) has banned you temporarily from accessing this website

[slashd0t]

I also got this error when I execute the npm install command: "Access denied | registry.npmjs.org used Cloudflare to restrict access. You are being rate limited. The owner of this website (registry.npmjs.org) has banned you temporarily from accessing this website.".
I'm from Cebu Philippines. Is this CloudFlare issue or the NPM ?

[jimhooker2002]

[muratkarsh]

The owner of this website (registry.npmjs.org) has banned you temporarily from accessing this website.

[SargoDarya]

Hey folks, as much fun as this might be please let us restrict this to actually useful stuff so people can start working again.

[gmemstr]

It looks like this issue is pretty widespread, rather than everyone posting "this is happening to me on X" how about we hang tight until we hear something from NPM? Or we can +1 a comment if affected.

[anton-bot]

NPM: Not a Package Manager

[stephnr]

We are getting outtages here in Sweden too 👌 🙆‍♂

[dorsharon]

It looks like this issue is pretty widespread, rather than everyone posting "this is happening to me on X" how about we hang tight until we hear something from NPM? Or we can +1 a comment if affected.

Nah, memes and useless comments for the win

[anaelsh]

Having the same issue on AWS build

[jacob-j]

Does this mean we can leave for second lunch?

[rmothilal]

Having the same issue on CircleCi Builds

[kapik]

Having the same issue on Github Actions !

[break-pointer]

Time to install https://github.com/verdaccio/verdaccio/

[ksemtinimahmoud]

[anant-k-singh]

Wow, Memes in comments XD

[phaberest]

Guys it's awesome to be all on the same boat and share some love while we stress out, but...could we stop telling each other "me too"?

Memes are always welcomed, btw!

[quite]

This is the consequence of over-reliance on somebody else's computer. @phaberest you're senseless and I am too :-}

[aleczratiu]

[JavierPons]

same!

[jimmycasey]

NPM just told me we can take the rest of the day off, thanks folks.

[prasadkadiyala]

Same issue here

[DennisSkoko]

[AkuSilvenius]

Does this still apply?

[segl84]

Same

[roppazvan]

Npm headquarters right now:

I assume the npmjs office right now #npm pic.twitter.com/wZy0Bo3PM8

— Nick Hammond (@thehappypenguin) February 17, 2020

[nikiamnik]

Issue is now fixed

https://status.npmjs.org/ is till green 💃

[tim-dewit]

NPM:

[anton-bot]

Rumours say NPM packages have been infected with a coronavirus

[ovr]

I think it's a good example why it's highly recommended installing own npm registry/proxy to protect similar issues when you will not able to deploy your app due NPM registry outage

[alexander-milenko]

When you are waiting while NPM will be available

[syed-alizain]

NPM wants us to take a break

[hideckies]

Writing "same" is low rating.

[RoniLeshes]

Same here on Casio FX-991ESPLUS

[anaelsh]

NPM: Now, Post Memes!

[AbnerZheng]

Same here from China.

[AnthonyGarriga]

Same issue when running npm install on our build server (Teamcity) and locally.

[dtosato]

I am starting a python course right now.

[DennisSkoko]

What I love about this is this issue isn't actually stopping me from working. However, people posting memes here is stopping me. I love it :D

[polyconvex]

I liek chocolate milk

[sbishakafenix]

Related thread.. https://github.com/nextcloud/maps/issues/300
Because i'm lazy will link my post from there.. https://github.com/nextcloud/maps/issues/300#issuecomment-586973011

[eknis]

yeah, i can npm ci

[roppazvan]

NPM: no packages mate

[turbolego]

I'm in a customer meeting right now and can't demo :[

[silberjan]

I'm in a customer meeting right now and can't demo :[

sooo. you are demoing npm install to your customers?

[pedramp]

[chlenc]

I have same problem

[vishnusankaran]

No Pipeline Monday in India too 💃

[picolino]

Oh my god, please share node_modules folder, anybody!

[kawaiiyoshiyuki]

the same here

[rennokki]

https://status.npmjs.org/

"All Systems Operational" - the biggest lie

[eknis]

Unstable but not improving?

[davidperis92]

NPM: Never Push Mocks

[quite]

@anant-k-singh yes, memes, where have you been? ;-) lots of thumb-rolling in the front-end community now, right.

...and where is that local/site-local caching npm proxy when you need it?

[mdorda]

https://status.npmjs.org/ is very usefull...

[sbishakafenix]

Someon should just upload an angularjs node_modules folder to google drive for heaven's sake

[jacob-j]

I'm in a customer meeting right now and can't demo :[

FeelsBadMan, so bad "power of demo" that the whole world is affected..

[rennokki]

They updated the status page.

[LorisBachert]

[riskyRahul]

npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
npm ERR! code E429
npm ERR! 429 Too Many Requests - GET https://registry.npmjs.org/jsonschema/-/jsonschema-1.2.5.tgz

npm ERR! A complete log of this run can be found in:
npm ERR! C:\Users\Lenovo\AppData\Roamingnpm-cache_logs\2020-02-17T12_50_04_887Z-debug.log

[clakech]

https://status.npmjs.org/incidents/5qxhhcx0vtlc?u=608t6667st9s

[SoYouDie]

[7Siete]

I'm here just for the memes

[gmemstr]

Hey all, could we keep memes and jokes to respective Slack/Discord/IRC channels rather than this GitHub issue? They are aware of this issue, and they've updated the StatusPage.

[JavierPons]

npm ERR! code E429. here!

[cyperh17]

https://status.npmjs.org/

"All Systems Operational" - the biggest lie

[casr]

jq -r '.dependencies * .devDependencies | keys[]' package.json | xargs -L 1 -I {} sh -c "echo installing {}; npm i {}; sleep 10"

It seems to work if you don't install all your packages at once... This will work if you're desperate but it's pretty slow...

[monolighter]

403 / 429 Erros for some Users

How about all ?

[nina-rafieifar]

Hi, is there any updates on this?

[rickvdbosch]

E429 here too (the Netherlands)

Building from an on-premises machine triggered by Azure DevOps.

[nikiamnik]

Finally it started working (Status page 💃)

[nagrgk]

down since 1 and half hour. hope we find the fix soon

[tyroneopinary]

[danielbater247]

Temp fix working for me, open package-lock.json
Find https://registry.npmjs.org and replace with https://registry.npmjs.com
Run npm ci

[aleczratiu]

[chlenc]

[rahulshinde19]

Good to know that I am not alone 🤣

[cyperh17]

"received by some users", they said

[benjaminnaesen]

Same here on Mars.

[CyberHippo]

all hope is lost

[yoavfranco]

[Toomavic]

What would happen to the world if npm stopped forever, that's a bit thinker

[DennisSkoko]

[guilala]

This issue made me try yarn,.. actually for the first time. Works as a charm!

[kircod]

My astrologer predicted this event.

[turbolego]

I'm in a customer meeting right now and can't demo :[

sooo. you are demoing npm install to your customers?

Did not bring my laptop, colleague of mine said he could just fetch it.
Teamviewer works tho...

[sbishakafenix]

[stephnr]

When the memes load faster than NPM

[MatGhp]

Hi npm Team

Would you please restart your computer?

Best regards

[harshv47]

Came for the issue, stayed for the memes

[anton-bot]

Please be respectful and avoid posting anything except memes in this thread. Thank you.

[chlenc]

It`s works

[hcaeryks]

npm install finally worked for me

[nikosdouvlis]

I've got an Angular9 node_modules dir for sale. DM me your offers

[Qwilan]

It's alive!

[AlonStark]

[silberjan]

@npm

[monolighter]

Ah, down for not so long, there is no memeland anymore :(

[riker09]

This is me reporting live from the npm front. Looks like it's working again!

[Greenrenge]

Cheers

[AbnerZheng]

Temp fix working for me, open package-lock.json
Find https://registry.npmjs.org and replace with https://registry.npmjs.com
Run npm ci

Thank you. It works for me.

[Shubhankardubey]

Cheers it started working

[vishnusankaran]

Oh no...my mail notifications!! 🌊

[JavierPons]

[andreaceltich]

working!

[riker09]

Okay guys, this has been hilarious. See you all next time

[MunkhdelgerMorhous]

[pedrodemargomes]

Temp fix working for me, open package-lock.json
Find https://registry.npmjs.org and replace with https://registry.npmjs.com
Run npm ci

worked for me

[rahulshinde19]

Looks like issue title will change from "Too Many Requests" to "Too Many Comments"
it's already most commented issue in the list

[riker09]

(when the github issue updates faster than the npm status page)

[kawaiiyoshiyuki]

works for me as well

[rennokki]

Have you tried rm -rf node_modules/ package-lock.json && npm install ?

waiting for downvotes

[warpech]

https://github.com/npm/cli/issues/836#issuecomment-586973004
You could use: https://github.com/open-services/open-registry

# npm
npm config set registry https://npm.open-registry.dev

# yarn
yarn config set registry https://npm.open-registry.dev

Why all the downvotes for open-registry?

---

Alternatively, try npm config set registry https://registry.npm.taobao.org

[yisehak-awm]

This has been fun

[dzungdinh94]

node_modules for sale. pm me for good offers :D

[AlexanderLevenskikh]

Have you tried rm -rf node_modules/ package-lock.json && npm install ?

429 Too many downvotes

[AleksandarDoneski]

Have fun NPMing.

[stephnr]

My reaction when NPM says to stop submitting memes

[adnanchang]

[Stonedestroyer]

Did you try restart the PC, maybe error will go away. Cause it helped me.

[8masterofpuppets8]

My build started working again! Guess they turned it off and on finally

[elfen]

Houston, it's working here.

[DimaDK02]

works now for me as well

[picolino]

node_modules for sale. pm me for good offers :D

Probably worst thing you can write in open source community 😋

[MunkhdelgerMorhous]

[athomine]

[leoskyrocker]

well seems to be fixed. works in Singapore AWS region

[GDR]

Works now!

[rishabh-arya95]

working now North Virginia AWS Region

[stephnr]

Working now - Area 51 👽

[slashd0t]

It works now. Thanks to NPM Support Team. Sana all <3

[nirmalkrishnav]

See you later at another issue, internet people

[leoskyrocker]

[kkukelka]

Gentlemen, it was a pleasure!

[AnatoleLucet]

close it, maybe?

[Dzhuneyt]

Back online in Bulgaria. Thanks, fellas!

[John-Ciuchea]

can we do this again?

[danielcarney96]

my name jeff

[sbishakafenix]

CI up and running.. now to on to work.. it's been an honour

[Bakersbakebread]

[AkuSilvenius]

This outage helped me streamline my Dockerfile, no joke.

[quite]

Thanks for holding hands.

[thanpolas]

please lock this issue, I can't get anyone back to work including myself

[joeburkey]

UK here. was encountering the 429 issue when installing next.js but react and react-dom were fine. All is ok now it seems

[AccessViolator]

Hi npm Team

Would you please restart your computer?

Best regards

They finally did it!!! 3 times...

[MaxOlsson]

Working now in Sweden 🇸🇪 !

[stepankuzmin]

See you all next time when NPM is unavailable!

[leoskyrocker]

At least github seems to be highly available

[RWOverdijk]

GitHub is trying to live-update the emoji responses and it's like emoji fireworks.

[XavierLeTohic]

[nikiamnik]

Next issue: GitHub is down :D
Cause : too many requests in NPM bug #837 836

[abeledovictor]

Success! created app at C:/XXXX

[rahulshinde19]

[roppazvan]

[arsoba]

" Monitoring - Our content delivery partner has informed us they've implemented a fix. We are monitoring. "
Cloudflare problem?

[hasokeric]

Grand Rapids, MI here

[leoskyrocker]

This is my first comment on Github with so many reactions - love you all ❤️See you soon

EDIT: so many downvotes* 😄keep it up! down I mean..

[AccessViolator]

Ralph broke the internet!

[hornta]

Just want to be a part of this. Good job NPM 👍

[Polyterative]

So we all make memes now?

[rahulshinde19]

Finally its working!!!

[jo4nymj]

(https://user-images.githubusercontent.com/57898245/74657035-b1009500-518f-11ea-9e95-290b51db7dbb.png)

[lysender]

Looks like fixed now.

[faran-tenovos]

Please lock it down, I cannot make anyone go back to work including myself

[retyui]

no

[AleksandarDoneski]

NPM issues bringing people together :)

[riker09]

Since I have all of your attention, may I interest you in my latest pyramid scheme?

[PetrKnedlik]

Everything looks fine now in Czech Republic :) Thanks NPM team

[Syed-Umair]

NPM using FORCE

[aebrs]

Ok in Iran too, funny issue!

[convenient]

[faran-tenovos]

npm headquarters be right now
Let the memes force be with you

[andrepsimoes]

🇵🇹

[NeeBondha]

lol

[gndelia]

my packages failed to install !

oh geez, I came late to the party, it's working now.

Thanks internet!

[ssuriyayuvan]

console changed by npm :100:

[caleuanhopkins]

hey, I'm here to report a bug wi...

[SamakaCD]

  ▲
▲ ▲
PSHH PSHH YOBA MI V EFIRE!!1 🚣‍♂️

[yogan]

All aboard the meme train! 🚂

[cmcarey]

Posting memes/jokes in an issue wastes the time of people who actually need to be able to read through the issue.

Stop it. Use emoji reacts if you feel the need.

[vjekofalco]

[gagiD]

We need more of these bugs, especially on Monday.

[almothafar]

@cmcarey it is never a bad thing to smile and laugh, it is was never wasting the time thing, it will improve your productivity actually, memes and jokes make us a human instead of just working machines, even Detroit become a human man!

[quite]

Hello from the same GALARETKA!!

hello from the Internet!

[porzione]

At least it may be soon become the most commented issue.

[vreemt]

Suggestion: create an entry for "CDN services" on the npm status page, as it seems the actual issue was with Cloudflare

[cmcarey]

https://github.com/npm/cli/issues/836#issuecomment-586992790

@cmcarey it is never a bad thing to smile and laugh, it is was never wasting the time thing, it will improve your productivity actually, memes and jokes make us a human instead of just working machines, even Detroit become a human man!

This is not the time or place for that. A flood of comments like we're seeing here complicates the job of sorting through to find details that may help the people that can actually fix the problem.

[marlonmleite]

Working now in Brazil :brazil:!

[andreeleuterio]

Andre from npm security here. Our content delivery partner has informed us they’ve implemented a fix. and we are continuing to monitor the situation. You can find more information in our status page: https://status.npmjs.org/.

Closing the issue but if you have any issue please contact [email protected]

[rahulshinde19]

[SargoDarya]

@aeleuterio Any chance we can get a post-mortem on this?

[cathzetjo]

Working now!
OMG. Dont do that again! Ever! )))))

[gokselb]

It's not working again. I guess, issue has not been fixed by your partners.

[danielytics]

@cmcarey it is never a bad thing to smile and laugh, it is was never wasting the time thing, it will improve your productivity actually, memes and jokes make us a human instead of just working machines, even Detroit become a human man!

Memes and jokes drown out the conversation that may be vital to fixing whatever the issue actually is, though.

[uncoooloj]

Working now 😓

[CraigMason]

Memes and jokes drown out the conversation that may be vital to fixing whatever the issue actually is, though.

Indeed, I had to scroll through about 200 memes just to see the actual status update from NPM.

[PurHur]

Just for your info: yarn was able to load the packages without triggering the rate limit :1)

yarn is great! (and safe)

[buro9]

Hello and profuse apologies from Cloudflare, a post-mortem of sorts directly in your issue comments.

I am the engineering manager for the DDoS protection team and this morning at 11:06 UTC we tweaked a rule that affected one of our signals. The signal relates to the HTTP referer header, and we have a piece of code that looks at invalid referer headers. In this case we tweaked it to include not just "obvious garbage" but "anything that does not conform to the HTTP specification"... i.e. is the referer a URI? If not then it contributes to knowledge about bad traffic.

So... why did this impact npmjs.org? It turns out that a lot of NPM traffic sends the referer as "install" which is invalid according to the HTTP specification. As NPM is also a heavily trafficked site this resulted in the DDoS systems picking this up and treating the traffic as a HTTP flood and determining that a rate-limit should be applied.

When we noticed that NPM was seeing an increase in HTTP 429s (as seen on Twitter) we contacted NPM and started an internal investigation. As soon as we identified the root cause we reverted the change, which was at 13:00 UTC.

We'll note that NPM and 1 other site use the referer for purposes outside the HTTP spec and we'll update our systems to ensure that this does not happen again. Additionally we'll improve our monitoring around changes of this nature so that we can discover impact sooner and roll back automatically.

[CraigMason]

Thanks for the explaination @buro9

Hopefully you will have some explicit tests for NPM, given the importance to the developer community.

We (and many others I'm sure) were unable to deploy a number of projects for 2 hours this morning, during EU business hours. This should also serve as a reminder for us all to have better continuity measures in place for when these events happen.

[NicolasGoeddel]

In my opinion, it would be best to make sure that the requests from the NPM installer comply with the HTTP specification.

[nc9]

In my opinion, it would be best to make sure that the requests from the NPM installer comply with the HTTP specification.

Referer should be blank, installer should be user agent

[MLDMoritz]

Thanks, I am able to download all my 5464950 dependencies every 15min for another build again.

[ahmadnassri]

@buro9 we would appreciate it if you respond to our tickets and our internal slack comms, before posting to a public issue, we still have not gotten a post-mortem report for the last two outages.

as for pointing at HTTP specifications, considering this behaviour has been in place for years, I would ask to review what change was pushed in CF today that caused this sudden "compliance with HTTP Specification" result?

I'll ask again to please follow up with our open tickets and report back to to us on the post-mortem for the last two outages, we'd rather learn about this from you directly, than seeing it in an issue on github ..

[uap-universe]

Hello and profuse apologies from Cloudflare,

I don't think you have to apologize. npm obviously messed up the referrer field and you did nothing wrong. Just because it accidentally worked out in the past doesn't mean it should stay that way. Who shall guarantee that something like this doesn't happen again in the future, because someone is respecting the spec?

[MLDMoritz]

npm obviously messed up the referrer field and you did nothing wrong. Just because it accidentally worked out in the past doesn't mean it should stay that way.

That's called BC breaks and should not occur in the same 'version'.

[uap-universe]

npm obviously messed up the referrer field and you did nothing wrong. Just because it accidentally worked out in the past doesn't mean it should stay that way.

That's called BC breaks and should not occur in the same 'version'.

Yeah I give you that point. But hopefully the decision will not be "it stays forever that way and everyone must comply".

[datenwolf]

(…)
we tweaked a rule that affected one of our signals. The signal relates to the HTTP referer header, and we have a piece of code that looks at invalid referer headers.
(…)
So... why did this impact npmjs.org? It turns out that a lot of NPM traffic sends the referer as "install" which is invalid according to the HTTP specification.

Doesn't the Referer header permit the use of relative / partial URIs? https://tools.ietf.org/html/rfc7231#section-5.5.2

5.5.2.  Referer

   The "Referer" [sic] header field allows the user agent to specify a
   URI reference for the resource from which the target URI was obtained
   (i.e., the "referrer", though the field name is misspelled).  A user
   agent MUST NOT include the fragment and userinfo components of the
   URI reference [RFC3986], if any, when generating the Referer field
   value.

     Referer = absolute-URI / partial-URI

Without probing the server for existence of the URI, how to you distinguish arbitrary, urlencoded text from an actual partial-URI?

If I follow down the deductions of the specification then an Referer header install at first glance could be perfectly valid:

https://tools.ietf.org/html/rfc7230#section-2.7

relative-part = <relative-part, see [RFC3986], Section 4.2>
partial-URI   = relative-part [ "?" query ]

https://tools.ietf.org/html/rfc3986#section-4.2

      relative-ref  = relative-part [ "?" query ] [ "#" fragment ]

      relative-part = "//" authority path-abempty
                    / path-absolute
                    / path-noscheme
                    / path-empty

https://tools.ietf.org/html/rfc3986#section-3.3

path-noscheme = segment-nz-nc *( "/" segment )
segment       = *pchar
segment-nz    = 1*pchar
segment-nz-nc = 1*( unreserved / pct-encoded / sub-delims / "@" )
                    ; non-zero-length segment without any colon ":"

pchar         = unreserved / pct-encoded / sub-delims / ":" / "@"

So if I'm not terribly misreading the specification, I have to wonder: Why the heck has that referrer header been treated as invalid? Does Cloudflare check the existence of that URI upon request (and then caches it), or is this yet again a shoot fast-and-loose "whoopsie"?

[uap-universe]

@datenwolf syntactically it should be possible to have a partial URI like install, but how about this, a few lines below your quoted text:

If the target URI was obtained from a source that does not have its
own URI [...], the user agent MUST either exclude the
Referer field or send it with a value of "about:blank".

This imho applies here.

This doesn't, however, answer your last question. But it seems that the topic is a bit more complex and splits into two parts: 1) is npm conforming to the http spec, and 2) does CF use reliable detection rules. And maybe, just maybe, the answer to both questions is: no. But I leave this to others to discuss. I just wanted to point out that prematurely apologizing for "something" may leave potential bugs unfixed, so the wording might have been a bit unfortunate towards a real "solution" to the problem - whatever it is.

[mkkhedawat]

Instead of implementing block mode the very first day, It should have been in observe mode for some time.

[datenwolf]

@datenwolf syntactically it should be possible to have a partial URI like ìnstall`, but how about this, a few lines below your quoted text:

If the target URI was obtained from a source that does not have its
own URI [...], the user agent MUST either exclude the
Referer field or send it with a value of "about:blank".

This imho applies here.

Yes, it does. Maybe I've not made it clear enough, that I'm fully aware of this. But that's entirely besides the point.

As you've already pointed out

(…) it seems that the topic is a bit more complex and splits into two parts: 1) is npm conforming to the http spec, and 2) does CF use reliable detection rules. (…)

and I already pointed out, that without an explicit check for the existence of the Referer URI, the validity of the header is impossible to decide.

Does npm violate the specification? Sure.

Can CF correctly detect that? Only by performing an explicit check of the URI. Does CF perform that check? I don't know… yet (but I might set up a testing ground, just for that).

However, without further information I just have to assume that CF again did as CF does and for ill-founded reasons broke a part of the Internet… yet again.

[isaacs]

It's worth keeping in mind in all this that there's very little that a CDN can do which will meaningfully improve their service and which does not carry the risk of "breaking a part of the internet". The nature of the service that CF provides, and their well-deserved popularity as one to provide that service, means that they are basically always playing with fire, and likely to upset quite a lot of people when they make otherwise forgivable and well-intended mistakes.

Having done this npm thing for quite some time now, I have no shortage of sympathy for their position, and I believe that it is inappropriate to heap scorn on them for this. They're doing a great job, protecting npm (and thus, the whole JS community) from a lot of bad actors and outages, and making all of our builds much faster and more reliable. We love and appreciate Cloudflare. So let's be kind here.

That being said, I do believe that npm is neither in violation of the letter nor the spirit of the relevant HTTP specifications in our use of the Referer header to indicate the command that caused a given request to be made. I hope that anyone following this discussion find the following flood of HTTP pedantry useful or at least enjoyable. If that's not your kind of thing, please go do something else, you won't have a good time reading this :)

CF deployed a change that treated unusual use of HTTP headers as a heuristic to be marked as a malicious request. When dealing primarily with traffic from web browsers, the Referer header will generally always be either missing, about:blank, or a fully-qualified URL. So a header like Referer: install is weird.

However: The specification says "URI". It does not say "URL". It certainly does not say "Fully qualified URL". Given the typically meticulous usage of "URI" vs "URL" in IETF documentation, the hair-splitting discussions that often ensue over matters like this, and the fact that Referer first appeared in rfc1945 (though it was in use long before then), _and_ that the HTTP specs have kept "URI" through multiple rounds of updates and revisions, I have to conclude that the _intent_ here is for Referer to be a URI rather than required to be a URL per se, as well as the letter of the spec.

A URI and a URL are not the same thing. Both the linked RFCs have been updated and obsoleted (in part) since their inception by subsequent RFCs, and I strongly encourage anyone still reading at this point to follow the links and learn about how Uniform Resource Location and Identification standards have changed and expanded in subtle and interesting ways over the years.

The bottom line is this: the HTTP Referer header does not need to be a Uniform Resource _Locator_, but rather a Uniform Resource _Identifier_. There is no requirement that this identifier use a well-known URI scheme, or that it be a full form rather than relative. The only limitation placed on it is that (a) it must be a URI, and (b) if the request is satisfying a direct user request that does not have any kind of identifier, such as typing a URL into an address bar, then it _must_ be omitted.

A URI explicitly does _not_ need to be locatable, fetchable, or resolveable by any given network agent, or over any given protocol.

So, install is an identifier for the thing which the user interacted with that caused the request to be made. They didn't type the url to the packument or tarball into an address bar, they typed npm install, and _it_ in turn fetched the resource via HTTPS. In order to resolve that resource, it had to make some HTTP requests. No scheme is provided, but none is required. "install" is a compact series of characters identifying a resource. It's a scheme-less (ie, not fully-qulified) URI.

As URI semantics and syntax are defined by their scheme, it is impossible, in absence of a scheme, to say that install isn't valid. For example, tel:[phone] is a valid URI (and a valid URL), but http:[phone] is not. In order to know if [phone] is a valid partial URI, you'd have to know the scheme. Its locatability would depend on the particulars of the telephone systems of northeastern Ohio, USA. If a telephone call to an automated system at that number triggered an HTTP request to be made, it would be perfectly appropriate for that HTTP request to include a Referer header of [phone]. If the server expects to get requests from such a telephone system, it can infer the scheme based on the context.

That is exactly what happens with the npm client and the npm registry. It sends a Referer header containing the command being run. (When the command contains positional arguments, anything containing / or \ is redacted, as this might be a private path, url, or git repo.) This is semantically and syntactically a correct and appropriate use of the HTTP Referer header in such a non-browser context, and it is my sincere belief that, after 30 years of revision, analysis, and review by the IETF, through multiple versions of this specification, if this was not the intent of the Referer specification, it would've said "URL" rather than "URI".

And just to reiterate, I certainly don't think Cloudflare is a bad actor here, and I'm disappointed to see how quick so many people have been to "pick sides" as if it's npm vs Cloudflare in a battle over this. We were affected by a mistake they made, but we're sometimes going to be affected by mistakes that they make, because we're their customer, and of course they're going to make mistakes from time to time, because humans and machines aren't perfect. That's just how the world is. By and large, we're very happy with the response we got, and we've all improved our monitoring and response systems in light of this hiccup.

[reschke]

FWIW, the "referer" is not defined as URI. See the spec: https://greenbytes.de/tech/webdav/rfc7231.html#rfc.section.5.5.2. It's a URI reference. "install" would be interpreted as path, relative to the request URI.

[isaacs]

@reschke Even on that reading, it's still perfectly valid.

That is: from https://registry.npmjs.org/foo, with Referer: install, the fully resolved Referer would be https://registry.npmjs.org/install, a valid URI. From https://registry.npmjs.org/foo/-/foo-1.2.3.tgz, it'd be https://registry.npmjs.org/foo/-/install, also a valid URI.

Even if I'm reading the spec more broadly than intended, it's certainly not a "violation" of the spec to use Referer in this way, and it's a (completely forgivable!) mistaken overreach to block or rate-limit requests that include Referer headers that are not fully-qualified URLs.

In light of this, however, it probably _would_ be worthwhile to put a scheme on the Referer headers that the npm cli sends. We'd have to research this to see if that makes it more or less likely that requests will be mangled by proxies. Another option, of course, is to accept that some proxies will just be overeager in filtering resulting in slightly less ideal data, but use a custom header with a more definitive meaning, like npm-command: install instead. We do this for the npm-session header to group requests together, and have found cases where we don't get this custom header, even though the user-agent implies that it is a "real" npm client (or at least, is claiming to be).