Dependabot-core: Flaky auth problems with PyPI

[webknjaz]

Package manager/ecosystem

python

Manifest contents prior to update

N/A

Updated dependency

N/A

What you expected to see, versus what you actually saw

Dependabot (the old GitHub App) has created a lot of issues like https://github.com/aio-libs/aiodocker/issues/502 across aio-libs org. Restarting some of the updates helped but this one hasn't succeeded.

Logs (https://app.dependabot.com/accounts/aio-libs/update-logs/48591655) are full of HTTP 403 responses:

  proxy | 2020/09/24 09:52:24 [004] GET https://pypi.python.org:443/simple/tox-travis/
  proxy | 2020/09/24 09:52:24 [004] 403 https://pypi.python.org:443/simple/tox-travis/
  proxy | 2020/09/24 09:52:24 [006] GET https://pypi.python.org:443/simple/
  proxy | 2020/09/24 09:52:24 [006] 403 https://pypi.python.org:443/simple/

(there's this block of log lines per each dep)

I don't know anything about how dependabot is deployed but I can speculate that either Dependabot gets rate-limited (unlikely because then the HTTP code wouldn't be 403 unless there's a bug in https://github.com/pypa/warehouse) or your dependabot deployment is distributed and uses multiple API tokens in PyPI and one (some?) of them has been invalidated.

Images of the diff or a link to the PR, issue or logs

Example issue https://github.com/aio-libs/aiodocker/issues/502

[di]

Hi! PyPI admin here. This doesn't seem like an issue with our backends, these endpoints should never return a 403.

Also, as an aside, I noticed that dependabot is configured to use pypi.python.org instead of pypi.org, which likely is resulting in lots of unnecessary redirects.

[webknjaz]

Looks like all the links to this issue originate from issues created by the GitHub App version of Dependabot and not the new "native" one...

[Bystroushaak]

Wtf is this? It spammed me with 12 issues and I have no idea what why. I never configured dependabot to have anything in common with pypi.

[webknjaz]

@Bystroushaak those issues have logs, take a look at them if you are interested what Python package dependencies it detected. It is a GitHub App, Apps only have privileges to post in repos if you install them.

I promptly looked at your repos and you at least have setup.py there.

[mjpieters]

Wtf is this? It spammed me with 12 issues and I have no idea what why. I never configured dependabot to have anything in common with pypi.

Dependabot is looking at your project's requirements.txt / pyproject.toml / setup.py files to determine what dependencies you have, then looking those up on PyPI to see if there are newer versions. E.g. your tinyself project has a requirements.txt file, and GitHub parses that out into distinct dependencies, and you have configured Dependabot to go look for newer versions. Either for your whole account, or per repository.

[brrygrdn]

Hi folks, we are actively investigating this issue and will update shortly once we know more.

[tekumara]

Migrating to the native github dependabot resolves this for me.

[thepwagner]

@di 👋

Can you confirm that's true for any intermediary / DoS-mitigation tiers?

This seems to be an issue affecting all traffic from our source network to PyPi:

$ curl -v https://pypi.org:443/simple/pyflakes/
*   Trying 151.101.0.223:443...
* TCP_NODELAY set
* Connected to pypi.org (151.101.0.223) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: businessCategory=Private Organization; jurisdictionC=US; jurisdictionST=Delaware; serialNumber=3359300; C=US; ST=New Hampshire; L=Wolfeboro; O=Python Software Foundation; CN=www.python.org
*  start date: Sep 18 00:00:00 2018 GMT
*  expire date: Oct 14 12:00:00 2020 GMT
*  subjectAltName: host "pypi.org" matched cert's "pypi.org"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 Extended Validation Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55c0ea6827c0)
> GET /simple/pyflakes/ HTTP/2
> Host: pypi.org
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 403
< server: Varnish
< retry-after: 0
< accept-ranges: bytes
< date: Fri, 25 Sep 2020 11:44:54 GMT
< x-served-by: cache-wdc5520-WDC
< x-cache: MISS
< x-cache-hits: 0
< x-timer: S1601034295.772280,VS0,VE0
< strict-transport-security: max-age=31536000; includeSubDomains; preload
< x-frame-options: deny
< x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
< x-permitted-cross-domain-policies: none
< content-length: 0
<
* Connection #0 to host pypi.org left intact

The same command works elsewhere, making me think on of our source IPs:

• 3.209.160.83
• 3.93.159.41
• 3.94.32.93

May be getting blocked?

[brrygrdn]

Just following up on @thepwagner's comment - we've been in touch with the PyPI infrastructure staff and resolved an issue that was blocking our ability to update Python manifests.

The service is now operating normally and we will continue to monitor it.

[webknjaz]

@thepwagner I've just executed that command from my laptop but it succeeds. The major difference I noticed is that I'm only getting HTTP/1.1 offered (* ALPN, server accepted to use http/1.1) and your log shows switching over to HTTP/2. I must note that my curl version is a bit newer but I'm not sure if it's built with H2 support:

$ curl -v https://pypi.org:443/simple/pyflakes/
*   Trying 151.101.192.223:443...
* Connected to pypi.org (151.101.192.223) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* WARNING: failed to load NSS PEM library libnsspem.so. Using OpenSSL PEM certificates will not work.
*   CAfile: none
  CApath: none
* loaded libnssckbi.so
* ALPN, server accepted to use http/1.1
* SSL connection using TLS_AES_128_GCM_SHA256
* Server certificate:
*   subject: CN=www.python.org,O=Python Software Foundation,L=Wolfeboro,ST=New Hampshire,C=US,serialNumber=3359300,incorporationState=Delaware,incorporationCountry=US,businessCategory=Private Organization
*   start date: Sep 18 00:00:00 2018 GMT
*   expire date: Oct 14 12:00:00 2020 GMT
*   common name: www.python.org
*   issuer: CN=DigiCert SHA2 Extended Validation Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US
> GET /simple/pyflakes/ HTTP/1.1
> Host: pypi.org
> User-Agent: curl/7.69.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Connection: keep-alive
< Content-Length: 12246
< Cache-Control: max-age=600, public
< Content-Security-Policy: default-src 'none'; sandbox allow-top-navigation
< Content-Type: text/html; charset=UTF-8
< ETag: "X4Oy74MHDicSSbrM222QIg"
< Referrer-Policy: origin-when-cross-origin
< Server: nginx/1.13.9
< X-PyPI-Last-Serial: 6990660
< Accept-Ranges: bytes
< Date: Fri, 25 Sep 2020 13:11:04 GMT
< X-Served-By: cache-bwi5149-BWI, cache-fra19155-FRA
< X-Cache: HIT, HIT
< X-Cache-Hits: 1, 1
< X-Timer: S1601039465.567957,VS0,VE88
< Vary: Accept-Encoding, Accept-Encoding
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< X-Frame-Options: deny
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Permitted-Cross-Domain-Policies: none
<
[...snip...]
* Connection #0 to host pypi.org left intact
<!--SERIAL 6990660-->%

[webknjaz]

The service is now operating normally and we will continue to monitor it.

Ah, so it's resolved? Should we close this issue now?

[cetinajero]

Ah, so it's resolved? Should we close this issue now?

Fixed for me, @dependabot just closed his issue by itself.

Thanks!

[brrygrdn]

@webknjaz 👍🏻 to closing this out.

We're working on some suggested improvements from the PyPI folks to avoid this kind of issue in future.

As @cetinajero notes, the next time Dependabot runs on your affected repositories it will close these issues itself if you haven't already.

[webknjaz]

@brrygrdn I hope you add some Sentry reporting helping you to get notified about such problems before it hits many users :) Thanks for the fix!