Nix: offline binary cache entirely breaks nixos-rebuild

[cleverca22]

[root@system76:~]# nixos-rebuild switch --fallback
building Nix...
building the system configuration...
warning: unable to download 'http://amd.localnet:5000/mz0ga4ad618mbiwk3k2029ja1anilma6.narinfo': Couldn't connect to server (7); retrying in 306 ms
...
warning: unable to download 'http://amd.localnet:5000/ychm97sxx69zj9llm76yi386a5g9p4sm.narinfo': Couldn't connect to server (7); retrying in 19324 ms
error: unexpected end-of-file

[root@system76:~]# 

the only way to rebuild the machine now, is to manually change nix.conf to exclude that binary cache

[Taeradan]

"the only way to rebuild the machine now, is to manually change nix.conf to exclude that binary cache"
How do you do that ? It's read-only, even for root, and it's getting me mad not being able to change anything on my computer anymore.....

A really blocking issue.

[Taeradan]

Note that nixos-rebuild switch --option binary-caches "" has no effect, it is still trying to use the binary caches in nix.conf.

[7c6f434c]

There is a bad way and a better way.

Bad way: it is read-only to root because of a read-only bind mount of /nix/store that can be remounted RW.

Better way: NIX_CONF_DIR=/path/to/edited/etc/nix/ nix-build '<nixpkgs/nixos>' -A system -I nixos-config=/etc/nixos/configuration.nix — after the system is built, nixos-rebuild switch will work without needing access to binary cache.

[jpierre03]

here nixos-rebuild switch --fallback --show-trace --option binary-caches https://cache.nixos.org/ forced official cache and workded

[cleverca22]

when i manually change nix.conf, i replace the symlink with a file, so the store is never modified directly
setup-etc.pl will revert it back to a symlink at the next switch

[mpickering]

@jpierre03's answer worked for me but this is still a bad issue.

[matthewbauer]

This is definitely annoying. It definitely isn't intentional behavior is it? The down host should be a warning (as the message indicates) not an error.

/cc @edolstra @shlevy

[domenkozar]

It hasn't happened yet that https://cachix.org was offline (higher chance than highly-availabile cloudfront), so this one will be a pain one day if we don't get it solved.

[dtzWill]

It hasn't happened yet that https://cachix.org was offline (higher chance than highly-availabile cloudfront), so this one will be a pain one day if we don't get it solved.

https://github.com/cachix/cachix/issues/117

Not that you need another ping on the subject, and thanks as always for cachix :grin: but as predicted this issue means any machines using cachix are unable to do anything :(

Hmm...

[chris-martin]

this one will be a pain one day

That day is today.

I am currently completely crippled by the Cachix outage right now, would appreciate any advice if there is a workaround. I'm trying to run nixos-rebuild switch --option build-use-substitutes false to remove cachix from my config, but without using the main cache it's going to take a very long time.

[chris-martin]

Workaround: This seems to be working.

nixos-rebuild switch --option substituters 'https://cache.nixos.org'

[dasJ]

My C++ isn't too good, but it looks like the problem is here (not sure but it looks legit).
If this is the problem, the exception from the downloader should be cached and if there are more substituters available, the next one should be used.

[domenkozar]

Thank you @edolstra :)

[s9gf4ult]

When merging of this feature can be expected?

[sheyll]

Just wanted to let you know that I bumped into this, too.

[NickSeagull]

Same here, currently using an alias for rebuilding :)

[hyperfekt]

Looks like this is in Nix 2.2.1 and upwards, you should be able to use nix.package = (import <nixos-unstable> {}).nix with an updated nixos-unstable channel to get this functionality until the new Nix version lands in your release channel.