nix-build --falback does not allow you to build local packages when nixos.org is down

[vizanto]

TL;DR: nixos.org is a single point of failure.

---

Just started using nix yesterday, trying to write my own packages (on Mac OSX).
I cloned nixpkgs and started hacking.

➭ nix-build --version
nix-build (Nix) 1.3

This morning nixos.org is down. That's ok, even amazon goes offline every once in a while. However I cannot seem to get nix-build to ignore nixos.org.

Tried:

time sudo nix-build pkgs/top-level/all-packages.nix -A jzmq --fallback -vvv -K

After waiting literally for:

2,91s user 0,75s system 0% cpu 25:19,63 total

The build does not continue with the error:

*** Downloading ‘http://nixos.org/binary-cache/nar/0xl2pl3raiig8zc9d6nq374ixc3cacyf4nsf9l1vcdanbxy5j3qr.nar.bz2’ to ‘/nix/store/d83wfy078cq9x6ljr1x7ba6cp287xsg9-gnumake-3.82’...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:02:31 --:--:--     0curl: (7) couldn't connect to host

bzip2: Compressed file ends unexpectedly;
        perhaps it is corrupted?  *Possible* reason follows.
bzip2: Inappropriate ioctl for device
        Input file = (stdin), output file = (stdout)

It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

error: unexpected end-of-file
download of `http://nixos.org/binary-cache/nar/0xl2pl3raiig8zc9d6nq374ixc3cacyf4nsf9l1vcdanbxy5j3qr.nar.bz2' failed: No such file or directory
could not download ‘/nix/store/d83wfy078cq9x6ljr1x7ba6cp287xsg9-gnumake-3.82’ from any binary cache

Isn't the whole point of --fallback _not_ to require a binary cache?

Some more output follows the error:

|   |   substitution of `/nix/store/d83wfy078cq9x6ljr1x7ba6cp287xsg9-gnumake-3.82': got EOF
|   |   substitution of `/nix/store/d83wfy078cq9x6ljr1x7ba6cp287xsg9-gnumake-3.82': woken up
|   |   substitution of `/nix/store/d83wfy078cq9x6ljr1x7ba6cp287xsg9-gnumake-3.82': substitute finished
|   |   substitution of `/nix/store/3i1yircxji7aa9n9i4zrlrv5lvsh9a9l-zeromq-2.1.10': woken up
|   |   substitution of `/nix/store/jn33ykljd38iap3sk5xhyvbfh0alczxm-pkg-config-0.23': woken up
|   |   substitution of `/nix/store/1xbw8dkqy3schdq3pxx1fgbkh7vhfz4x-openjdk6-b16-24_apr_2009-r1': woken up
|   |   substitution of `/nix/store/673bzl40yrkaxsl00yg3dgld94md29z5-diffutils-3.2': woken up
|   |   substitution of `/nix/store/ad9hfdv0780w4cmb176ahpskyr3xyxnc-ed-1.5': woken up
|   |   substitution of `/nix/store/h20nwkpjdh17713swin8nkgh0087ywaf-gawk-4.0.0': woken up
|   |   substitution of `/nix/store/jgzcrf62crlzqj5n3rwyw1wxafnq0ps8-gzip-1.4': woken up
|   |   substitution of `/nix/store/q4fwa9yi7k2hygg17yl7nn5spdvmrlsz-findutils-4.4.2': woken up
|   |   substitution of `/nix/store/yq5barc33nbn7jp98qcc8ls6ziym9ljl-gnused-4.2.1': woken up
|   |   substitution of `/nix/store/jgwvnghydgimvjpfbly6q7h8anq5zfj7-pcre-8.21': woken up
|   |   substitution of `/nix/store/v47z8y0igawvy6vf3m2npw2ykj250yvj-libiconv-1.13.1': woken up
|   |   substitution of `/nix/store/8rqgqrn1a1zskmnyfgpwdlgv9n2iwpgh-expat-2.1.0': woken up
|   |   substitution of `/nix/store/yhjy4fy1w0bwlqn528pffxfyfhrq1dgg-python-2.7.3': woken up
|   |   substitution of `/nix/store/maf2xvaf7xq4krn4w54r42cppz8zg79l-libiconv-1.13.1': woken up
|   building of `/nix/store/wb8dn0x2kyslmxgjypw7bjd389hnpixg-jzmq-2.1.0.drv': goal destroyed
|   building of `/nix/store/lmkvbjyrxpis61sl3kfhwiccy6ai1kcb-git-export.drv': goal destroyed
|   building of `/nix/store/fz5wi1gmshai7m3sdxv0594gk12z6k1b-git-1.8.1.drv': goal destroyed
|   substitution of `/nix/store/k7r91gak4q4yirmnx6bimfva3imw309g-git-1.8.1': goal destroyed
|   substitution of `/nix/store/yhjy4fy1w0bwlqn528pffxfyfhrq1dgg-python-2.7.3': goal destroyed
|   substitution of `/nix/store/nhfh2qfh14gviv93my60nd1kpyn1cpgn-gettext-0.18.1.1': goal destroyed
|   substitution of `/nix/store/maf2xvaf7xq4krn4w54r42cppz8zg79l-libiconv-1.13.1': goal destroyed
|   substitution of `/nix/store/8rqgqrn1a1zskmnyfgpwdlgv9n2iwpgh-expat-2.1.0': goal destroyed
|   building of `/nix/store/3xxi6mdmzlm1n1zmcw4i717mrd4vwfh6-cacert-20120628.drv': goal destroyed
|   building of `/nix/store/5armkqw9dvx2vnhx7f2nh0d7sfl07pn5-cacert-20120628.pem.bz2.drv': goal destroyed
|   building of `/nix/store/rm3cqk01n4ivgmhiv44xbhm0q2ibxbha-mirrors-list.drv': goal destroyed
|   building of `/nix/store/cbbhkdjb526nyf709gr65hqjs5mmk82h-openjdk6-b16-24_apr_2009-r1.drv': goal destroyed
|   substitution of `/nix/store/1xbw8dkqy3schdq3pxx1fgbkh7vhfz4x-openjdk6-b16-24_apr_2009-r1': goal destroyed
|   building of `/nix/store/3c24kg1jsi1ck5wll20gqx1hgcywvic7-pkg-config-0.23.drv': goal destroyed
|   substitution of `/nix/store/jn33ykljd38iap3sk5xhyvbfh0alczxm-pkg-config-0.23': goal destroyed
|   building of `/nix/store/c3q9vhg9cm054cs7n75ka64w4m8zar6y-zeromq-2.1.10.drv': goal destroyed
|   substitution of `/nix/store/3i1yircxji7aa9n9i4zrlrv5lvsh9a9l-zeromq-2.1.10': goal destroyed
|   building of `/nix/store/clwd5v42l8m65iqvqjbxrckfi17q4am2-stdenv.drv': goal destroyed
|   substitution of `/nix/store/zyilsch6gvrfxfmr8j37bjgvh21n65yf-stdenv': goal destroyed
|   substitution of `/nix/store/yq5barc33nbn7jp98qcc8ls6ziym9ljl-gnused-4.2.1': goal destroyed
|   substitution of `/nix/store/xczvmbi8rxchb1wz01ii2iwphimgdxfv-gnugrep-2.10': goal destroyed
|   substitution of `/nix/store/v47z8y0igawvy6vf3m2npw2ykj250yvj-libiconv-1.13.1': goal destroyed
|   substitution of `/nix/store/jgwvnghydgimvjpfbly6q7h8anq5zfj7-pcre-8.21': goal destroyed
|   substitution of `/nix/store/q4fwa9yi7k2hygg17yl7nn5spdvmrlsz-findutils-4.4.2': goal destroyed
|   substitution of `/nix/store/kqbdpd16yjpangakz575wcsfrgbljcbb-patch-2.6.1': goal destroyed
|   substitution of `/nix/store/ad9hfdv0780w4cmb176ahpskyr3xyxnc-ed-1.5': goal destroyed
|   substitution of `/nix/store/jgzcrf62crlzqj5n3rwyw1wxafnq0ps8-gzip-1.4': goal destroyed
|   substitution of `/nix/store/h20nwkpjdh17713swin8nkgh0087ywaf-gawk-4.0.0': goal destroyed
|   lock released on `/nix/store/d83wfy078cq9x6ljr1x7ba6cp287xsg9-gnumake-3.82.lock'
|   substitution of `/nix/store/d83wfy078cq9x6ljr1x7ba6cp287xsg9-gnumake-3.82': goal destroyed
|   substitution of `/nix/store/673bzl40yrkaxsl00yg3dgld94md29z5-diffutils-3.2': goal destroyed
error: unexpected EOF reading a line

---

Then I tried editing

sudo nano /nix/store/4czrvlsn3a8ava9p3axl7wzy3j1l8746-nix-1.3/libexec/nix/substituters/download-from-binary-cache.pl

and changed the while loop in processRequests to:

while (0 && $activeRequests) {

to make the script exit immediately.

With this change

time sudo nix-build pkgs/top-level/all-packages.nix -A jzmq --fallback -vvv -K

finished alot faster, however again trying to download gnumake from a binary cache, giving up after 2 minutes and 32 seconds.
What does --fallback do again?

---

Then I tried (getting desperate here)

time sudo nix-build pkgs/top-level/all-packages.nix --option binary-caches '' --fallback -vvv -K -A jzmq

For some reason when I change the binary-caches option, nix-build wants to build a whole lot more packages, like curl. I'm guessing it ignores every dependency already existing in the /nix/store/?!

The build proceeds extremely slow and prints errors like these:

error checking the existence of http://nixos.org/tarballs/sha256/b7f510db60f520ba0bc8a39cccee7e913362205b4a7709e16af2cba14093099b:
curl: (7) couldn't connect to host

1. Curl fails to build.
2. Libssh2 fails to build for the same reason.
3. Eventually nixos.org is up again, and openssl starts building from source.

I hope this is fixed quickly, I would not want to use disnix in production and not be able to fix my servers when nixos.org is down! That's a serious single point of failure!
And even if --fallback did work properly, the HTTP connect timeouts are way too high.

[edolstra]

You can now set a connection timeout via $NIX_CONNECT_TIMEOUT or --option connect-timeout N. It was already possible to disable the use of the binary cache using --option use-binary-caches false.

I can't reproduce the problem with --fallback though,

[mickours]

I have exactly the same problem on my system: cache.nixos.org is down, I tried every options (-option use-binary-caches false, --option connect-timeout N, --fallback) and I always get this error:

error: unable to download ‘https://cache.nixos.org/ffkdlmsag787xjdc55v4yi4vbbc0irrc.narinfo’: Couldn't connect to server (7)

It seems that the options are taken into account.

Some details:

• I'm not on nixos (archlinux)
• I'm using nix unstable

[robinp]

For posterity: it seems this should be --option build-use-substitutes false.

The use-binary-caches option seems gone, and adding the connect-timetout option still makes the build wait a lot before starting, due to retries on name resolution error.