Warehouse: files.pythonhosted.org: Painfully slow download with Deutsche Telekom (Fastly peering issue)
When downloading files from pythonhosted.org the download speeds are extremely slow in my environment. Even on a 100 MBit/s FTTH connection I hardly get more than 45 KB/s.
wget https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl
--2019-03-17 20:46:30-- https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl
Auflösen des Hostnamens »files.pythonhosted.org (files.pythonhosted.org)« … 151.101.1.63, 151.101.65.63, 151.101.129.63, ...
Verbindungsaufbau zu files.pythonhosted.org (files.pythonhosted.org)|151.101.1.63|:443 … verbunden.
HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 OK
Länge: 1921889 (1,8M) [binary/octet-stream]
Wird in »»virtualenv-16.2.0-py2.py3-none-any.whl.2«« gespeichert.
virtualenv-16.2.0-py2.py3-none-any.whl.2 16%[============> ] 308,62K 26,5KB/s eta 64s ^
curl -L --output /dev/null --silent --show-error --write-out 'lookup: %{time_namelookup}\nconnect: %{time_connect}\nappconnect: %{time_appconnect}\npretransfer: %{time_pretransfer}\nredirect: %{time_redirect}\nstarttransfer: %{time_starttransfer}\ntotal: %{time_total}\n' 'https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl'
lookup: 0,124598
connect: 0,178949
appconnect: 0,307898
pretransfer: 0,308016
redirect: 0,000000
starttransfer: 0,369662
total: 25,305663
Probably many other people from Germany are facing the same issue, as I have seen this behavior on multiple connections from Deutsche Telekom (largest ISP in Germany). Probably a Fastly/AS54113 peering issue.
My Platform
- Debian Stretch, Linux 4.9 (tried different platforms)
- Connection type:
- FTTH connection with 100 MBit/s
- IPv4 internet connection with single NAT (public IPv4 address)
- Provider: Deutsche Telekom
- Firewall: OPNsense 19.1
- Local network: 1 Gbit/s
- Location: Germany
- Misc: I can reproduce the slow download speed on a different connection (DSL, also from Deutsche Telekom) in a totally different area of Germany
DNS Resolution
dig pypi.org A
; <<>> DiG 9.10.3-P4-Debian <<>> pypi.org A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5794
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;pypi.org. IN A
;; ANSWER SECTION:
pypi.org. 54 IN A 151.101.0.223
pypi.org. 54 IN A 151.101.128.223
pypi.org. 54 IN A 151.101.192.223
pypi.org. 54 IN A 151.101.64.223
;; Query time: 54 msec
;; SERVER: 192.168.20.254#53(192.168.20.254)
;; WHEN: Sun Mar 17 20:36:49 CET 2019
;; MSG SIZE rcvd: 101
dig pypi.org AAAA
; <<>> DiG 9.10.3-P4-Debian <<>> pypi.org AAAA
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25763
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;pypi.org. IN AAAA
;; ANSWER SECTION:
pypi.org. 31 IN AAAA 2a04:4e42:600::223
pypi.org. 31 IN AAAA 2a04:4e42:200::223
pypi.org. 31 IN AAAA 2a04:4e42::223
pypi.org. 31 IN AAAA 2a04:4e42:400::223
;; Query time: 30 msec
;; SERVER: 192.168.20.254#53(192.168.20.254)
;; WHEN: Sun Mar 17 20:37:04 CET 2019
;; MSG SIZE rcvd: 149
dig files.pythonhosted.org A
; <<>> DiG 9.10.3-P4-Debian <<>> files.pythonhosted.org A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15600
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;files.pythonhosted.org. IN A
;; ANSWER SECTION:
files.pythonhosted.org. 21 IN CNAME dualstack.r.ssl.global.fastly.net.
dualstack.r.ssl.global.fastly.net. 29 IN A 151.101.1.63
dualstack.r.ssl.global.fastly.net. 29 IN A 151.101.65.63
dualstack.r.ssl.global.fastly.net. 29 IN A 151.101.129.63
dualstack.r.ssl.global.fastly.net. 29 IN A 151.101.193.63
;; Query time: 105 msec
;; SERVER: 192.168.20.254#53(192.168.20.254)
;; WHEN: Sun Mar 17 20:37:15 CET 2019
;; MSG SIZE rcvd: 162
dig files.pythonhosted.org AAAA
; <<>> DiG 9.10.3-P4-Debian <<>> files.pythonhosted.org AAAA
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42755
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;files.pythonhosted.org. IN AAAA
;; ANSWER SECTION:
files.pythonhosted.org. 9 IN CNAME dualstack.r.ssl.global.fastly.net.
dualstack.r.ssl.global.fastly.net. 29 IN AAAA 2a04:4e42::319
dualstack.r.ssl.global.fastly.net. 29 IN AAAA 2a04:4e42:200::319
dualstack.r.ssl.global.fastly.net. 29 IN AAAA 2a04:4e42:400::319
dualstack.r.ssl.global.fastly.net. 29 IN AAAA 2a04:4e42:600::319
;; Query time: 41 msec
;; SERVER: 192.168.20.254#53(192.168.20.254)
;; WHEN: Sun Mar 17 20:37:27 CET 2019
;; MSG SIZE rcvd: 210
Traceroutes
IPv4
traceroute pypi.org
traceroute to pypi.org (151.101.64.223), 30 hops max, 60 byte packets
1 gate.middleearth.lan (192.168.20.254) 0.329 ms 0.371 ms 0.396 ms
2 62.155.247.115 (62.155.247.115) 5.003 ms 5.010 ms 5.025 ms
3 hh-ea8-i.HH.DE.NET.DTAG.DE (62.154.32.218) 16.655 ms 16.678 ms 16.684 ms
4 hh-ea8-i.HH.DE.NET.DTAG.DE (62.154.32.218) 16.643 ms 16.638 ms 16.679 ms
5 80.150.168.162 (80.150.168.162) 26.662 ms 26.655 ms 26.680 ms
6 hbg-bb1-link.telia.net (213.155.135.80) 64.556 ms 62.692 ms hbg-bb4-link.telia.net (62.115.141.110) 24.717 ms
7 ffm-bb4-link.telia.net (62.115.138.172) 25.049 ms 25.778 ms 26.220 ms
8 ffm-b12-link.telia.net (62.115.142.47) 24.847 ms 24.554 ms ffm-b12-link.telia.net (62.115.142.5) 25.049 ms
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
traceroute files.pythonhosted.org
traceroute to files.pythonhosted.org (151.101.113.63), 30 hops max, 60 byte packets
1 gate.middleearth.lan (192.168.20.254) 0.806 ms 0.547 ms 0.491 ms
2 62.155.247.115 (62.155.247.115) 15.483 ms 15.484 ms 15.474 ms
3 hh-ea8-i.HH.DE.NET.DTAG.DE (62.154.32.126) 17.342 ms 17.398 ms 17.410 ms
4 hh-ea8-i.HH.DE.NET.DTAG.DE (62.154.32.126) 13.265 ms 17.316 ms 17.362 ms
5 80.150.168.162 (80.150.168.162) 28.840 ms 29.128 ms 29.025 ms
6 hbg-bb1-link.telia.net (62.115.141.112) 29.497 ms hbg-bb4-link.telia.net (213.155.135.84) 29.721 ms hbg-bb4-link.telia.net (62.115.141.110) 29.706 ms
7 ffm-bb3-link.telia.net (62.115.123.76) 26.486 ms ffm-bb4-link.telia.net (62.115.138.172) 28.574 ms ffm-bb3-link.telia.net (62.115.123.76) 27.441 ms
8 ffm-b12-link.telia.net (62.115.142.47) 27.515 ms ffm-b12-link.telia.net (62.115.142.5) 27.709 ms 28.199 ms
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
IPv4
curl -vvv -I --ipv4 https://pypi.org/pypi/pip/json
* Trying 151.101.192.223...
* TCP_NODELAY set
* Connected to pypi.org (151.101.192.223) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: businessCategory=Private Organization; jurisdictionC=US; jurisdictionST=Delaware; serialNumber=3359300; C=US; ST=New Hampshire; L=Wolfeboro; O=Python Software Foundation; CN=www.python.org
* start date: Sep 18 00:00:00 2018 GMT
* expire date: Oct 14 12:00:00 2020 GMT
* subjectAltName: host "pypi.org" matched cert's "pypi.org"
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 Extended Validation Server CA
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55fe32052e80)
> HEAD /pypi/pip/json HTTP/1.1
> Host: pypi.org
> User-Agent: curl/7.52.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
HTTP/2 200
< access-control-allow-headers: Content-Type, If-Match, If-Modified-Since, If-None-Match, If-Unmodified-Since
access-control-allow-headers: Content-Type, If-Match, If-Modified-Since, If-None-Match, If-Unmodified-Since
< access-control-allow-methods: GET
access-control-allow-methods: GET
< access-control-allow-origin: *
access-control-allow-origin: *
< access-control-expose-headers: X-PyPI-Last-Serial
access-control-expose-headers: X-PyPI-Last-Serial
< access-control-max-age: 86400
access-control-max-age: 86400
< cache-control: max-age=900, public
cache-control: max-age=900, public
< content-security-policy: base-uri 'self'; block-all-mixed-content; connect-src 'self' https://api.github.com/repos/ *.fastly-insights.com sentry.io https://2p66nmmycsj3.statuspage.io; default-src 'none'; font-src 'self' fonts.gstatic.com; form-action 'self'; frame-ancestors 'none'; frame-src 'none'; img-src 'self' https://warehouse-camo.cmh1.psfhosted.org/ www.google-analytics.com *.fastly-insights.com; script-src 'self' www.googletagmanager.com www.google-analytics.com *.fastly-insights.com https://cdn.ravenjs.com; style-src 'self' fonts.googleapis.com; worker-src *.fastly-insights.com
content-security-policy: base-uri 'self'; block-all-mixed-content; connect-src 'self' https://api.github.com/repos/ *.fastly-insights.com sentry.io https://2p66nmmycsj3.statuspage.io; default-src 'none'; font-src 'self' fonts.gstatic.com; form-action 'self'; frame-ancestors 'none'; frame-src 'none'; img-src 'self' https://warehouse-camo.cmh1.psfhosted.org/ www.google-analytics.com *.fastly-insights.com; script-src 'self' www.googletagmanager.com www.google-analytics.com *.fastly-insights.com https://cdn.ravenjs.com; style-src 'self' fonts.googleapis.com; worker-src *.fastly-insights.com
< content-type: application/json
content-type: application/json
< etag: "/Z73FUFl8uukw1tLUt+8OA"
etag: "/Z73FUFl8uukw1tLUt+8OA"
< referrer-policy: origin-when-cross-origin
referrer-policy: origin-when-cross-origin
< server: nginx/1.13.9
server: nginx/1.13.9
< x-pypi-last-serial: 4846670
x-pypi-last-serial: 4846670
< accept-ranges: bytes
accept-ranges: bytes
< date: Sun, 17 Mar 2019 19:40:46 GMT
date: Sun, 17 Mar 2019 19:40:46 GMT
< x-served-by: cache-iad2137-IAD, cache-hhn1546-HHN
x-served-by: cache-iad2137-IAD, cache-hhn1546-HHN
< x-cache: HIT, HIT
x-cache: HIT, HIT
< x-cache-hits: 1, 1
x-cache-hits: 1, 1
< x-timer: S1552851646.446026,VS0,VE1
x-timer: S1552851646.446026,VS0,VE1
< vary: Accept-Encoding, Accept-Encoding
vary: Accept-Encoding, Accept-Encoding
< strict-transport-security: max-age=31536000; includeSubDomains; preload
strict-transport-security: max-age=31536000; includeSubDomains; preload
< x-frame-options: deny
x-frame-options: deny
< x-xss-protection: 1; mode=block
x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
x-content-type-options: nosniff
< x-permitted-cross-domain-policies: none
x-permitted-cross-domain-policies: none
< content-length: 68340
content-length: 68340
<
* Curl_http_done: called premature == 0
* Connection #0 to host pypi.org left intact
curl -vvv -I --ipv4 https://files.pythonhosted.org/packages/ae/e8/2340d46ecadb1692a1e455f13f75e596d4eab3d11a57446f08259dee8f02/pip-10.0.1.tar.gz
curl -vvv -I --ipv4 https://files.pythonhosted.org/packages/ae/e8/2340d46ecadb1692a1e455f13f75e596d4eab3d11a57446f08259dee8f02/pip-10.0.1.tar.gz
* Trying 151.101.1.63...
* TCP_NODELAY set
* Connected to files.pythonhosted.org (151.101.1.63) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=California; L=San Francisco; O=Fastly, Inc; CN=r.ssl.fastly.net
* start date: Sep 4 19:42:10 2018 GMT
* expire date: Apr 14 16:28:35 2019 GMT
* subjectAltName: host "files.pythonhosted.org" matched cert's "*.pythonhosted.org"
* issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign CloudSSL CA - SHA256 - G3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5590097ede80)
> HEAD /packages/ae/e8/2340d46ecadb1692a1e455f13f75e596d4eab3d11a57446f08259dee8f02/pip-10.0.1.tar.gz HTTP/1.1
> Host: files.pythonhosted.org
> User-Agent: curl/7.52.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
HTTP/2 200
< x-amz-id-2: JtyvxRbz2pOO22Vu9KcST0ol5wXA+5vE+02gS8wdFJECeSCUGe88zBDcwi77/ZGHlb9iJbzGtFg=
x-amz-id-2: JtyvxRbz2pOO22Vu9KcST0ol5wXA+5vE+02gS8wdFJECeSCUGe88zBDcwi77/ZGHlb9iJbzGtFg=
< x-amz-request-id: 78DDFB1C0B79BAC9
x-amz-request-id: 78DDFB1C0B79BAC9
< last-modified: Thu, 19 Apr 2018 18:56:10 GMT
last-modified: Thu, 19 Apr 2018 18:56:10 GMT
< etag: "83a177756e2c801d0b3a6f7b0d4f3f7e"
etag: "83a177756e2c801d0b3a6f7b0d4f3f7e"
< x-amz-version-id: 1N.JvpbFmUQle7Hn0tnUuxQqUTsn5iEu
x-amz-version-id: 1N.JvpbFmUQle7Hn0tnUuxQqUTsn5iEu
< content-type: binary/octet-stream
content-type: binary/octet-stream
< server: AmazonS3
server: AmazonS3
< cache-control: max-age=365000000, immutable, public
cache-control: max-age=365000000, immutable, public
< accept-ranges: bytes
accept-ranges: bytes
< date: Sun, 17 Mar 2019 19:41:01 GMT
date: Sun, 17 Mar 2019 19:41:01 GMT
< age: 2333031
age: 2333031
< x-served-by: cache-sea1046-SEA, cache-fra19168-FRA
x-served-by: cache-sea1046-SEA, cache-fra19168-FRA
< x-cache: HIT, HIT
x-cache: HIT, HIT
< x-cache-hits: 7, 1
x-cache-hits: 7, 1
< x-timer: S1552851661.436121,VS0,VE15
x-timer: S1552851661.436121,VS0,VE15
< strict-transport-security: max-age=31536000; includeSubDomains; preload
strict-transport-security: max-age=31536000; includeSubDomains; preload
< x-frame-options: deny
x-frame-options: deny
< x-xss-protection: 1; mode=block
x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
x-content-type-options: nosniff
< x-permitted-cross-domain-policies: none
x-permitted-cross-domain-policies: none
< x-robots-header: noindex
x-robots-header: noindex
< content-length: 1246072
content-length: 1246072
<
* Curl_http_done: called premature == 0
* Connection #0 to host files.pythonhosted.org left intact
CDnetwork
klada
All 13 comments
Thanks for the report. Are you still experiencing this? (I see https://status.python.org/ didn't mention any incidents but it could have missed something.)
brainwane
on 18 Mar 2019
I have been experiencing these issues ever since I changed my ISP. My new ISP (Deutsche Telekom) has a different connection to Fastly and it looks like they have no direct peering agreement (which BTW is funny, because Deutsche Telekom Capital Partners invests in Fastly).
The issue also depends on the time of day. In the morning connection speeds are fine, in the evening (between 17:00 and 0:00 CEST) they are horrible. I can be 100% sure that this is not related to my actual internet connection, because when I enable VPN on my internet connection I can download at full speed from files.pythonhosted.org. I have talked to other people with Deutsche Telekom as their ISP and they can confirm that they also have the exact same issue with files.pythonhosted.org (or Fastly in general).
As you can see in the traceroute output above I am using Telia as a carrier, but Level3 is also affected (confirmed by other people using Deutsche Telekom).
From what other people tell me it sounds like IPv6 connectivity in the evening is still mostly okay (cannot test myself), so it looks like pretty much only IPv4 is affected. Please file a ticket at your CDN provider, they should make sure that their traffic can actually reach people.
Here are some more wget speed tests from different times of day:
21:03
Actually 80 KB/s is not too bad, it can get worse...
wget -4 https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl
--2019-03-18 21:03:58-- https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl
Auflösen des Hostnamens »files.pythonhosted.org (files.pythonhosted.org)« … 151.101.113.63
Verbindungsaufbau zu files.pythonhosted.org (files.pythonhosted.org)|151.101.113.63|:443 … verbunden.
HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 OK
Länge: 1921889 (1,8M) [binary/octet-stream]
Wird in »»virtualenv-16.2.0-py2.py3-none-any.whl.1«« gespeichert.
virtualenv-16.2.0-py2.py3-none-any.whl.1 100%[====================================================================================>] 1,83M 80,1KB/s in 24s
2019-03-18 21:04:23 (78,4 KB/s) - »»virtualenv-16.2.0-py2.py3-none-any.whl.1«« gespeichert [1921889/1921889]
08:31
Full speed, as this is in the morning.
wget -4 https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl
--2019-03-19 08:31:23-- https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl
Auflösen des Hostnamens »files.pythonhosted.org (files.pythonhosted.org)« … 151.101.113.63
Verbindungsaufbau zu files.pythonhosted.org (files.pythonhosted.org)|151.101.113.63|:443 … verbunden.
HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 OK
Länge: 1921889 (1,8M) [binary/octet-stream]
Wird in »»virtualenv-16.2.0-py2.py3-none-any.whl.1«« gespeichert.
virtualenv-16.2.0-py2.py3-none-any.wh 100%[=========================================================================>] 1,83M 7,55MB/s in 0,2s
2019-03-19 08:31:23 (7,55 MB/s) - »»virtualenv-16.2.0-py2.py3-none-any.whl.1«« gespeichert [1921889/1921889]
17:32
It's starting to slow down again...
wget -4 https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl
--2019-03-19 17:32:37-- https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl
Auflösen des Hostnamens »files.pythonhosted.org (files.pythonhosted.org)« … 151.101.13.63
Verbindungsaufbau zu files.pythonhosted.org (files.pythonhosted.org)|151.101.13.63|:443 … verbunden.
HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 OK
Länge: 1921889 (1,8M) [binary/octet-stream]
Wird in »»virtualenv-16.2.0-py2.py3-none-any.whl.4«« gespeichert.
virtualenv-16.2.0-py2.py3-none-any.whl.4 100%[===================================================================================================================================>] 1,83M 98,9KB/s in 18s
2019-03-19 17:32:56 (102 KB/s) - »»virtualenv-16.2.0-py2.py3-none-any.whl.4«« gespeichert [1921889/1921889]
19:43
We are at 45 KB/s again...
wget -4 https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl
--2019-03-20 19:42:52-- https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl
Auflösen des Hostnamens »files.pythonhosted.org (files.pythonhosted.org)« … 151.101.13.63
Verbindungsaufbau zu files.pythonhosted.org (files.pythonhosted.org)|151.101.13.63|:443 … verbunden.
HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 OK
Länge: 1921889 (1,8M) [binary/octet-stream]
Wird in »»virtualenv-16.2.0-py2.py3-none-any.whl«« gespeichert.
virtualenv-16.2.0-py2.py3-none-any.whl 100%[====================================================================================>] 1,83M 63,0KB/s in 41s
2019-03-20 19:43:33 (45,8 KB/s) - »»virtualenv-16.2.0-py2.py3-none-any.whl«« gespeichert [1921889/1921889]
19:45 (with VPN)
This is from the same computer, but with a VPN connection. Through the VPN I get full speed due to different routing/peering.
wget -4 https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl
--2019-03-20 19:45:44-- https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl
Auflösen des Hostnamen »files.pythonhosted.org (files.pythonhosted.org)«... 151.101.113.63
Verbindungsaufbau zu files.pythonhosted.org (files.pythonhosted.org)|151.101.113.63|:443... verbunden.
HTTP-Anforderung gesendet, warte auf Antwort... 200 OK
Länge: 1921889 (1,8M) [binary/octet-stream]
In »»virtualenv-16.2.0-py2.py3-none-any.whl«« speichern.
100%[===================================================================================================================================================================================================================>] 1.921.889 9,92MB/s in 0,2s
2019-03-20 19:45:45 (9,92 MB/s) - »»virtualenv-16.2.0-py2.py3-none-any.whl«« gespeichert [1921889/1921889]
Thank you for the detailed report. I'm submitting this information to our CDN provider to see if they have any insight.
ewdurbin
on 20 Mar 2019
@klada it's probably worth contacting someone at your ISP at the same time to attack this from both sides.
ewdurbin
on 20 Mar 2019
Our CDN provider has noted this as a known issue that they are working to resolve, but is unable to provide a timeline for resolution.
ewdurbin
on 20 Mar 2019
👍4
👎1
I'll also file a complaint at my ISP. I need some time for this though, as I have to collect more data and comparisons to VPN speed through the same line. Usually their argument with issues like these is that it's up to the content provider to make sure the traffic reaches their customers. And that's where peering agreements come into play (sadly).
Are there any mirrors which are not provided by Fastly that German people can use meanwhile? All of the mirrors I have found seem to be dead.
klada
on 21 Mar 2019
Due to the potential cost of additional bandwidth we are unable to provide an official public mirror as PyPI runs on donated services and infrastructure.
Additionally the size of a full mirror of PyPI has grown to over 4TB, so not many are able to operate a full mirror at this point.
ewdurbin
on 21 Mar 2019
I met the same problem. Hope to know a workaround.
locdoan12121997
on 27 Mar 2019
One workaround is to reuse after your initial download... @locdoan12121997 @klada
So not just pip cache dir but, considering the Wheelhouse pattern for example
- utilities - neither are >1.0 version, so it is your call whether to depend on them directly or to take them as inspiration...
- reading/blogs
- https://www.brianthicks.com/post/2014/10/16/speed-up-pip-install-with-wheel/
- https://medium.com/@amimahloof/how-to-package-a-python-project-with-all-of-its-dependencies-for-offline-install-7eb240b27418
- not any _one_ post, but if you are using docker, there are some good ways to put the wheelhouse step in the base stage(s) of a multistage build, or bake into a base image etc
some people commit their wheelhouse to their repo, some use a git submodule, some keep it out of source but do save it into their docker setup and so on. some others do a copy on s3 (see: pypicloud; you could synchronize just what you depend on, rather than full pypi)
these won't fix your problem first time around of course.
but after first download, you'll have your issue fixed as long as your dependencies don't change. then even if something is added/changed, and that hasn't been added to your wheelhouse or base image yet ... you have a small delta, so relative wait time is slower even with the internet latency issues you are experiencing
hangtwenty
on 10 Apr 2019
Fastly has solved the issue through direct peering with Deutsche Telekom. Connection speeds to files.pythonhosted.org are now fine again.
klada
on 28 Apr 2019
Fastly has solved the issue through direct peering with Deutsche Telekom. Connection speeds to files.pythonhosted.org are now fine again.
Fastly has solved nothing.
Telekom: 50 KiB/s, Digital Ocean Frankfurt: 95 MiB/s.
If I use Mullvad VPN with Telekom, I get 3 MiB/s.
traceroute to files.pythonhosted.org (151.101.37.63), 40 hops max, 60 byte packets
1 _gateway (192.168.178.1) 0.457 ms 0.552 ms 0.667 ms
2 p3e9bf683.dip0.t-ipconnect.de (62.155.246.131) 8.457 ms 8.463 ms 8.484 ms
3 62.154.15.242 (62.154.15.242) 10.871 ms 11.355 ms 11.356 ms
4 62.157.249.186 (62.157.249.186) 11.965 ms 11.967 ms 12.002 ms
5 ae-2.r21.frnkge13.de.bb.gin.ntt.net (129.250.6.41) 12.275 ms 12.321 ms 12.312 ms
6 ae-9.r25.amstnl02.nl.bb.gin.ntt.net (129.250.3.77) 22.710 ms 21.897 ms 21.910 ms
7 ae-8.r03.amstnl02.nl.bb.gin.ntt.net (129.250.2.109) 18.243 ms 18.179 ms 18.230 ms
8 81.20.67.250 (81.20.67.250) 17.893 ms 17.666 ms 17.752 ms
9 151.101.37.63 (151.101.37.63) 18.404 ms 18.498 ms 18.497 ms
DO:
traceroute to files.pythonhosted.org (151.101.113.63), 30 hops max, 60 byte packets
1 46.101.128.254 (46.101.128.254) 2.772 ms 2.789 ms 2.760 ms
2 138.197.250.180 (138.197.250.180) 1.008 ms 1.005 ms 1.401 ms
3 138.197.250.157 (138.197.250.157) 0.768 ms 0.781 ms 0.802 ms
4 fra1.decixfra.fastly.net (80.81.195.54) 0.968 ms 1.020 ms 1.042 ms
5 151.101.113.63 (151.101.113.63) 0.785 ms 0.810 ms 0.825 ms
Safari77
on 6 Jul 2020
@Safari77 what resolver are you using?
supine
on 7 Jul 2020
I was using unbound with dnscrypt-proxy, just now it resolved files.pythonhosted.org to 151.101.241.63 (50 KiB/s).
Without forwarding configured in unbound, I got 151.101.13.63 (9 MiB/s).
:nerd_face: :+1:
Safari77
on 7 Jul 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
apogoreliy
·
4Comments
nlhkabu
·
4Comments
webknjaz
·
4Comments
zt2
·
4Comments
dstufft
·
3Comments
Most helpful comment
Our CDN provider has noted this as a known issue that they are working to resolve, but is unable to provide a timeline for resolution.