Nodejs.org: Getting random 500s on several file downloads and directory listings + downloads slow

Created on 6 Apr 2020  ·  27Comments  ·  Source: nodejs/nodejs.org

I'm getting a mix of slow / broken binary downloads, non-functional directory listings and file downloads both on my home internet connection in Germany, as well as from our CI server in AWS (don't know the region). Sometimes it works without a problem, but more than half the times, I'm getting the error.

If an error occurs, it's a 500 by nginx.

```➜ ~ curl --version
curl 7.64.1 (x86_64-apple-darwin19.0) libcurl/7.64.1 (SecureTransport) LibreSSL/2.8.3 zlib/1.2.11 nghttp2/1.39.2
Release-Date: 2019-03-27
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS GSS-API HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL UnixSockets
➜ ~ curl -v https://nodejs.org/dist/v12.13.0/SHASUMS256.txt.asc

  • Trying 2606:4700:10::6814:162e...
  • TCP_NODELAY set
  • Connected to nodejs.org (2606:4700:10::6814:162e) port 443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/cert.pem
    CApath: none
  • TLSv1.2 (OUT), TLS handshake, Client hello (1):
  • TLSv1.2 (IN), TLS handshake, Server hello (2):
  • TLSv1.2 (IN), TLS handshake, Certificate (11):
  • TLSv1.2 (IN), TLS handshake, Server key exchange (12):
  • TLSv1.2 (IN), TLS handshake, Server finished (14):
  • TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
  • TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
  • TLSv1.2 (OUT), TLS handshake, Finished (20):
  • TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
  • TLSv1.2 (IN), TLS handshake, Finished (20):
  • SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
  • ALPN, server accepted to use h2
  • Server certificate:
  • subject: OU=Domain Control Validated; OU=PositiveSSL Wildcard; CN=*.nodejs.org
  • start date: Oct 21 00:00:00 2019 GMT
  • expire date: Jan 18 23:59:59 2022 GMT
  • subjectAltName: host "nodejs.org" matched cert's "nodejs.org"
  • issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
  • SSL certificate verify ok.
  • Using HTTP2, server supports multi-use
  • Connection state changed (HTTP/2 confirmed)
  • Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  • Using Stream ID: 1 (easy handle 0x7f82ff00d600)

GET /dist/v12.13.0/SHASUMS256.txt.asc HTTP/2
Host: nodejs.org
User-Agent: curl/7.64.1
Accept: /

  • Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
    < HTTP/2 500
    < date: Mon, 06 Apr 2020 08:19:11 GMT
    < content-type: text/html
    < set-cookie: __cfduid=d491984f37389168cb113964828dd085c1586161151; expires=Wed, 06-May-20 08:19:11 GMT; path=/; domain=.nodejs.org; HttpOnly; SameSite=Lax
    < cf-cache-status: BYPASS
    < expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
    < server: cloudflare
    < cf-ray: 57fa151af99cdfd7-FRA
    <

    500 Internal Server Error

    500 Internal Server Error


    nginx


  • Connection #0 to host nodejs.org left intact
  • Closing connection 0
external
Source
picture Tobi042
👍43 👀1

Most helpful comment

reposting from the CF forum

<center><h1>500 Internal Server Error</h1></center>
<hr><center>nginx</center>

As the responses show nginx, this is most likely an issue with their server. If Cloudflare had an issue, the nginx server token would be cloudflare or nginx-cloudflare.

It would seem odd to not serve them directly from Cloudflare though…

Even with something like nodejs (which is likely on the pro/business plan), cache assets aren't stored on Cloudflare forever and will be evicted if CF's higher-paying customers need the cache. CF is a reverse proxy and, unless they decide to use Workers KV/Workers Sites, CF doesn't do any file storage.

picture judge2020 on 6 Apr 2020
👍6

All 27 comments

I have the same problem here in Austria

$ curl https://nodejs.org/dist/v11.9.0/node-v11.9.0.tar -v
*   Trying 104.20.22.46...
* TCP_NODELAY set
* Connected to nodejs.org (104.20.22.46) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: OU=Domain Control Validated; OU=PositiveSSL Wildcard; CN=*.nodejs.org
*  start date: Oct 21 00:00:00 2019 GMT
*  expire date: Jan 18 23:59:59 2022 GMT
*  subjectAltName: host "nodejs.org" matched cert's "nodejs.org"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f90a600a800)
> GET /dist/v11.9.0/node-v11.9.0.tar HTTP/2
> Host: nodejs.org
> User-Agent: curl/7.54.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 500 
< date: Mon, 06 Apr 2020 08:37:49 GMT
< content-type: text/html
< set-cookie: __cfduid=d5969ff1d463fd816402f40eddf6d280a1586162266; expires=Wed, 06-May-20 08:37:46 GMT; path=/; domain=.nodejs.org; HttpOnly; SameSite=Lax
< cf-cache-status: DYNAMIC
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< server: cloudflare
< cf-ray: 57fa30588cbacba8-VIE
< 
<html>
<head><title>500 Internal Server Error</title></head>
<body bgcolor="white">
<center><h1>500 Internal Server Error</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host nodejs.org left intact
mscharl picture mscharl on 6 Apr 2020

Same issue for https://nodejs.org/dist/v5.7.0/node-v5.7.0-linux-x64.tar.xz when trying to install with nvm install 5.7.0

bognarlaszlo picture bognarlaszlo on 6 Apr 2020

same issue https://nodejs.org/dist/v8.2.1/node-v8.2.1-linux-x64.tar.gz

nclifton picture nclifton on 6 Apr 2020

same issue: https://nodejs.org/dist/v10.16.3/node-v10.16.3-darwin-x64.tar.gz

anyone can tell why this is happening...???

rpragesh picture rpragesh on 6 Apr 2020

same issue: https://nodejs.org/dist/v10.18.1/node-v10.18.1-linux-x64.tar.gz

mustafa-turk picture mustafa-turk on 6 Apr 2020

Can confirm, this is still ongoing. Same issue: https://nodejs.org/dist/v11.15.0/node-v11.15.0-linux-x64.tar.gz

tobiasljungstrom picture tobiasljungstrom on 6 Apr 2020

Same for https://nodejs.org/dist/v10.14.1/node-v10.14.1-linux-x64.tar.gz

clpatp picture clpatp on 6 Apr 2020

Appears to be sitewide, as also impacts: https://nodejs.org/dist/v8.15.1/node-v8.15.1-linux-x64.tar.xz

(How many thousands of deployments and builds are failing right now because of this?)

dborishansky picture dborishansky on 6 Apr 2020

yet, another "same for" :)
Same for here: https://nodejs.org/dist/v12.16.1/node-v12.16.1-darwin-x64.tar.gz

BoazWarshawsky picture BoazWarshawsky on 6 Apr 2020

Does someone know if there is any web status page we can have some info about downtime?

BoazWarshawsky picture BoazWarshawsky on 6 Apr 2020

Same for https://nodejs.org/dist/v12.11.1/node-v12.11.1-linux-x64.tar.gz this time it downloads but just very slow.

lausdahl picture lausdahl on 6 Apr 2020

Seeing the same in Zurich

$ curl https://nodejs.org/dist/v10.14.0/node-v10.14.0-linux-x64.tar.gz -v
* Uses proxy env variable no_proxy == '192.168.99.100'
*   Trying 104.20.22.46:443...
* TCP_NODELAY set
* Connected to nodejs.org (104.20.22.46) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: OU=Domain Control Validated; OU=PositiveSSL Wildcard; CN=*.nodejs.org
*  start date: Oct 21 00:00:00 2019 GMT
*  expire date: Jan 18 23:59:59 2022 GMT
*  subjectAltName: host "nodejs.org" matched cert's "nodejs.org"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x26829a0)
> GET /dist/v10.14.0/node-v10.14.0-linux-x64.tar.gz HTTP/2
> Host: nodejs.org
> user-agent: curl/7.67.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 500
< date: Mon, 06 Apr 2020 12:23:46 GMT
< content-type: text/html
< set-cookie: __cfduid=d31ed357787b6880358c54a491b21667a1586175825; expires=Wed, 06-May-20 12:23:45 GMT; path=/; domain=.nodejs.org; HttpOnly; SameSite=Lax
< cf-cache-status: BYPASS
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< server: cloudflare
< cf-ray: 57fb7b5e4c183e9c-ZRH
<
<html>
<head><title>500 Internal Server Error</title></head>
<body bgcolor="white">
<center><h1>500 Internal Server Error</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host nodejs.org left intact
mtter picture mtter on 6 Apr 2020

https://nodejs.org/dist/v12.7.0/node-v12.7.0-linux-x64.tar.gz works now
but not https://nodejs.org/dist/v12.7.0/SHASUMS256.txt.asc

mogindi picture mogindi on 6 Apr 2020

Same here for https://nodejs.org/dist/v10.15.1/node-v10.15.1-win-x64.7z

DaisyT-G picture DaisyT-G on 6 Apr 2020

I've reproduced this from various networks and countries, and posted on the Cloudflare forum:
https://community.cloudflare.com/t/http-500-errors-hitting-nodejs-org-downloads/162133

It's not clear to me whether these artifacts (some of which are years old) are actually trying to be served from nodejs.org, or from Cloudflare's cache directly, so I don't know what's going on. It would seem odd to not serve them directly from Cloudflare though… 🤔

orrc picture orrc on 6 Apr 2020

Nothing in this repo controls the "downloads" directory, and the website team have no access to troubleshoot. Please do not add "same for" comments. Leaving unlocked for now

nschonni picture nschonni on 6 Apr 2020
❤4

reposting from the CF forum

<center><h1>500 Internal Server Error</h1></center>
<hr><center>nginx</center>

As the responses show nginx, this is most likely an issue with their server. If Cloudflare had an issue, the nginx server token would be cloudflare or nginx-cloudflare.

It would seem odd to not serve them directly from Cloudflare though…

Even with something like nodejs (which is likely on the pro/business plan), cache assets aren't stored on Cloudflare forever and will be evicted if CF's higher-paying customers need the cache. CF is a reverse proxy and, unless they decide to use Workers KV/Workers Sites, CF doesn't do any file storage.

judge2020 picture judge2020 on 6 Apr 2020
👍6

I am also facing the same error -
https://nodejs.org/dist/v12.13.0/node-v12.13.0-win-x64.7z

2020-04-06T14:49:11.0992666Z ##[section]Starting: Use Node 12.13.0
2020-04-06T14:49:11.1105195Z ==============================================================================
2020-04-06T14:49:11.1105544Z Task : Node.js tool installer
2020-04-06T14:49:11.1105907Z Description : Finds or downloads and caches the specified version spec of Node.js and adds it to the PATH
2020-04-06T14:49:11.1106250Z Version : 0.166.0
2020-04-06T14:49:11.1106475Z Author : Microsoft Corporation
2020-04-06T14:49:11.1106796Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/tool/node-js
2020-04-06T14:49:11.1107372Z ==============================================================================
2020-04-06T14:49:11.5418366Z Downloading: https://nodejs.org/dist/v12.13.0/node-v12.13.0-win-x64.7z
2020-04-06T14:49:12.1369360Z ##[error]Unexpected HTTP response: 500
2020-04-06T14:49:12.1444291Z ##[section]Finishing: Use Node 12.13.0

amol-wanjare-emtec picture amol-wanjare-emtec on 6 Apr 2020

@XhmikosR any update about this issue? the issue blocks our CI jobs.
thank you!

BoazWarshawsky picture BoazWarshawsky on 6 Apr 2020

/ping @nodejs/build

Trott picture Trott on 6 Apr 2020

same for https://nodejs.org/dist/v6.7.0/node-v6.7.0-linux-x64.tar.gz

michaellady picture michaellady on 6 Apr 2020

We're also experiencing timeouts across some of our build processes that rely on these distributions being available.

rbreslow picture rbreslow on 6 Apr 2020

Does someone know if there is any web status page we can have some info about downtime?

There is not, unfortunately. It's been suggested in the past, but no one ever stepped up to do it, and the project is volunteer-run. I have a feeling that if nothing else comes out of a postmortem on this incident, that will be something that finally gets done.

Trott picture Trott on 6 Apr 2020

Are there any mirros that one could use for NodeJS ? Or perhaps a company is willing to start one.

juddgaddie picture juddgaddie on 6 Apr 2020

Since @mhdawson has pointed to https://github.com/nodejs/build/issues/2263 as the single source of truth for this discussion (ref: https://github.com/nodejs/node/issues/32683#issuecomment-610014994), I'm going to lock this discussion in favor of that one. There are already further updates there ❤️

bnb picture bnb on 6 Apr 2020

Re-opening but keeping locked per @sam-github's request: https://github.com/nodejs/node/issues/32683#issuecomment-610040853

bnb picture bnb on 6 Apr 2020

This is resolved now, although root cause analysis is ongoing. Closing. See https://github.com/nodejs/build/issues/2263 for further information.

Trott picture Trott on 7 Apr 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

julianduque picture julianduque  ·  4Comments
fhemberger picture fhemberger  ·  8Comments
Aissaoui-Ahmed picture Aissaoui-Ahmed  ·  3Comments
interglobalmedia picture interglobalmedia  ·  3Comments
mognia picture mognia  ·  4Comments