Nix: HTTP error 200 (curl error: Timeout was reached) - unable to update and install

[zeecde]

Since the system build on 2019-05-30 some people get anHTTP error 200 (curl error: Timeout was reached); error on nixos unstable when a download takes longer than a few minutes.

A direct download e.g.

curl -vvv -o ghc.tar.gz https://cache.nixos.org /nar/0qwrky7iz1lgaqs03q0zzi9rj20h80966mahaz57s00gpfkdzvnl.nar.xz

works without any problems.

I tried to disable ipv6 and nix-store --realize works then:

echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6

nix-store --realize /nix/store/ks7aqyydpp0x9ygx89w71p1kaz0zy2fa-mesa-19.1.1-drivers
these paths will be fetched (77.32 MiB download, 288.49 MiB unpacked):
  /nix/store/ks7aqyydpp0x9ygx89w71p1kaz0zy2fa-mesa-19.1.1-drivers
copying path '/nix/store/ks7aqyydpp0x9ygx89w71p1kaz0zy2fa-mesa-19.1.1-drivers' from 'https://cache.nixos.or
g'...
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/ks7aqyydpp0x9ygx89w71p1kaz0zy2fa-mesa-19.1.1-drivers

but nixos-rebuild still fails:

sudo nixos-rebuild switch --upgrade
unpacking channels...
these derivations will be built:
  /nix/store/vgf431iw3ai7l45nh0zmgma53716v2y4-nixos-rebuild.drv
building '/nix/store/vgf431iw3ai7l45nh0zmgma53716v2y4-nixos-rebuild.drv'...
building Nix...
building the system configuration...
these derivations will be built:
  /nix/store/kf8a2v17f9dbc6s1d0a3fhx32ymp47zm-fish_patched-completion-generator.drv
[...]
  /nix/store/zy65hn7bii072b0jbdcr2aya55wz0k84-libkscreen-5.15.5
  /nix/store/zz40a8626rdy96kigkm7spgslrz0xvmd-SDL-1.2.15
  /nix/store/zzhdxlfmcics12kba6yxnqw9bpymf19x-xfconf-4.12.1
  /nix/store/zzjl2g84ms2kxqx9w21h30hfsrb7i92d-phonon-backend-gstreamer-qt5-4.9.0
copying path '/nix/store/q4dcn2ybm06x0xssjzz4k68gzar8ic54-linux-4.19.62' from 'https://cache.nixos.org'...
copying path '/nix/store/rrww1m72206fmskwfa33v0g3827bavl9-breeze-icons-5.58.0' from 'https://cache.nixos.or
g'...
[...]
.
copying path '/nix/store/0s35kcykjzdw3a3ffrj4y21k3jiqind8-nixos-icons-2017-03-16' from 'https://cache.nixos
.org'...
warning: unable to download 'https://cache.nixos.org/nar/1xf23nc4c1yvgm1xyszwjzm52ghj206wkdlafak7h4xq9idpz0
5i.nar.xz': HTTP error 200 (curl error: Timeout was reached); retrying in 273 ms
error 9 while decompressing xz file
copying path '/nix/store/sjl6fjz8kv5wbf2i3x92sm8br8gv4dsx-nixos-manpages' from 'https://cache.nixos.org'...
^Cerror: interrupted by the user

I switched to nixos-stable and it works again.

This was discussed on
https://discourse.nixos.org/t/nixos-cache-fetching-issue/3575/17

[aexoxea]

Nixpkgs#65015 also relates to this issue (but it does seem to be a Nix issue rather than Nixpkgs). Some of the comments there may be helpful.

[edolstra]

Should be fixed in 53247d6b116905e7233b1efd6c14845e20d27442.

[zeecde]

@edolstra Is there any way to test this now? Do I need nix from master? If yes, what exactly do I need to do?

[lopsided98]

Why are the timeouts occurring at all? It doesn't seem like the download should time out if it is making steady progress.

[zeecde]

@lopsided98 That is a good question. It should be possible to find the commit which caused problem. It must be on unstable after 2019-05-30 as this build does still not timeout on my system.

[lopsided98]

I don't think my issue has anything to with the Fastly cache, because it also occurs when downloading from my personal Hydra instance.

[lopsided98]

I run Hydra behind an nginx reverse proxy. I ran nix-channel --update (which pulls a channel from my Hydra instance) using the latest Nix master on a machine connected over a slow DSL connection. This resulted in timeout errors, but the downloads seem to have been retried correctly. I have no way of verifying that they are resuming where they left off each time. After 4 timeouts, my server started inexplicably returning 500 errors, causing the download to fail.

When the timeouts occur, I see this in my nginx logs:

5838#5838: *287626 client terminated stream 3 with status 5 while sending to client, client: 2601:18a:0:7723::a25, server: hydra.benwolsieffer.com, request: "GET /build/96378/download/1/nixexprs.tar.xz HTTP/2.0", upstream: "http://127.0.0.1:8080/build/96378/download/1/nixexprs.tar.xz", host: "hydra.benwolsieffer.com"
5838#5838: *287626 client prematurely closed connection while sending to client, client: 2601:18a:0:7723::a25, server: hydra.benwolsieffer.com, request: "GET /build/96378/download/1/nixexprs.tar.xz HTTP/2.0", upstream: "http://127.0.0.1:8080/build/96378/download/1/nixexprs.tar.xz", host: "hydra.benwolsieffer.com"

This error message is confusing, but it seems to indicate that the client (Nix) is causing the timeout, not the server.

[lopsided98]

Hydra does not seem to support ranged HTTP requests, which is why the retry functionality is not working for me. It keeps trying to download the entire file, and hits the same timeout over and over.

[lopsided98]

I believe I have found the root cause of the spurious timeouts: https://github.com/curl/curl/issues/3927

The fixed version just recently made it into nixpkgs, and I am working on verifying that it fixes this issue.

[thedavidmeister]

i'm seeing this when downloading e.g. chromium which is 700MB

[lopsided98]

@thedavidmeister What curl version are you running?

I was not able to completely verify that the curl update fixed the problem, as my machine with a slow connection locked up for some reason while testing and I won't be able to physically access it for a few more days. I was able to perform a channel update, which didn't work before.

[thedavidmeister]

@lopsided98 from the IRC conversation i had... it sounded like i risk bricking my environment if i test this before the next nix release hits nixpkgs

i commented out chromium and firefox from my configuration.nix and i'm in "wait and see" mode for now

unless i get some clear/safe debug ideas, in which case i can try it and report back

[lopsided98]

My current understanding is that the changes to Nix were just a workaround to the real problem most people are experiencing, which is that Curl times-out any download that takes longer than 5 minutes. This bug was fixed in Curl 7.65.1, which is currently included in NixOS unstable but not 19.03. Therefore, you probably don't need to upgrade Nix, just Curl.

[thedavidmeister]

this works for me in commit ec3b4dd684b396d6e09ff8ea04d709f38b51a4e2 for nixpkgs

lots of other minor bugs in that commit that i hope get cleaned up but this curl thing seems to be OK now

[ghost]

I can comfirm what @lopsided98 reported: newer curl fixes this bug.

[domenkozar]

Closing as this seems to be fixed.

[hwertz]

I have seen an error just like this; if I ran (I might have this backwards...) a "nix-shell" that was downloading larger NAR.xz items (copying ... into /nix/store/....) it'd show that xz exited with error 10; if I installed something with "nix-env -i" I got a HTTP error 200 (curl error: Timeout was reached).

Long story short, in ~/.config/nix/nix.conf I put "stalled-download-timeout = 99999999" (just an arbitrarily large number). I guess you can put this into /etc/nix/nix.conf if you prefer. PROBLEM FIXED!!!

Looks like default is 300, and it does sound fine to have a download abort if it's stalled for 5 minutes. But for whatever reason it's not... Anyway... I found the code and in essence curl "-Y" (--speed-limit) is set to "1" (byte per second), "-y" (--speed-time) is set to 300 (or whatever stalled-download-timeout is set to). Sounds fair enough. BUT, "nar.xz" are not downloaded then installed, they are downloaded into some kind of pipe, which (when it's extracting a bunch of tiny files) DOES cause the download to stall for a while. I don't think it's stalling for 5 minutes... but whatever, disabling this timeout fixes it. Cheers!