Nix: Nix GC collects derivations used for IFD

[wizeman]

I have a NixOS configuration with the following contents:

{ config, pkgs
, ... }:

let

  kodiOff_src = pkgs.srcOnly {
    name = "kodiOff-src";
    src = /home/tarballs/autogen/kodi-off.tar.gz;
  };

  kodiOff_pkg = import "${kodiOff_src}/default.nix" {};

in {
# (...)
  systemd.services.kodi-off = {
#   (...)
    serviceConfig = {
      Type = "simple";
      ExecStart = "${kodiOff_pkg}/bin/kodi-off";
    };
  };
# (...)
}

Now, even though I have the following options in the /etc/nix/nix.conf of my private Hydra server:

gc-keep-outputs = true
gc-keep-derivations = true

... and even though the configuration root hasn't been gc'ed, a Nix garbage collection still appears to delete the output of the pkgs.srcOnly derivation.

The above configuration is built by my private Hydra server, but if a garbage collection runs after the configuration is built, then my laptop (which is configured to use my Hydra server as a binary cache) fails to download the package during nixops deploy, reporting a 410 Gone error instead:

building all machine configurations...
fetching path ‘/nix/store/cvz1fpbsvqqq36xwyzc33cp7jy4r7lg3-kodiOff-src’...

*** Downloading ‘http://homeserv:3000/nar/cvz1fpbsvqqq36xwyzc33cp7jy4r7lg3-kodiOff-src’ to ‘/nix/store/cvz1fpbsvqqq36xwyzc33cp7jy4r7lg3-kodiOff-src’...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (22) The requested URL returned error: 410 Gone

bzip2: Compressed file ends unexpectedly;
        perhaps it is corrupted?  *Possible* reason follows.
bzip2: Inappropriate ioctl for device
        Input file = (stdin), output file = (stdout)

It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

error: unexpected end-of-file
download of ‘http://homeserv:3000/nar/cvz1fpbsvqqq36xwyzc33cp7jy4r7lg3-kodiOff-src’ failed: No such file or directory
could not download ‘/nix/store/cvz1fpbsvqqq36xwyzc33cp7jy4r7lg3-kodiOff-src’ from any binary cache
fetching path ‘/nix/store/cvz1fpbsvqqq36xwyzc33cp7jy4r7lg3-kodiOff-src’ failed with exit code 1
error: some substitutes for the outputs of derivation ‘/nix/store/miwpgzvgqw4nn90mggvra3sjlymz10q2-kodiOff-src.drv’ failed (usually happens due to networking issues); try ‘--fallback’ to build derivation from source 
(use ‘--show-trace’ to show detailed location information)
error: unable to build all machine configurations

This happens to me all the time.

If I add the --fallback option, the nixops deploy command successfully completes (and most other packages are correctly downloaded instead of being rebuilt), but then I can't be sure that there won't be spurious rebuilds (say, because my Hydra server hasn't finished rebuilding packages).

But in any case, correct me if I'm wrong, but I think that according to the documentation, the garbage collection was not supposed to delete that derivation.

[benmos]

FWIW, I've just encountered a similar error (I'm also using nixops but am running on the public hydra).

fetching path ‘/nix/store/dwcx9w994i9kl19gfvcmfi8xz7f2nsjf-stage-2-init.sh’...

*** Downloading ‘http://hydra.nixos.org/nar/dwcx9w994i9kl19gfvcmfi8xz7f2nsjf-stage-2-init.sh’ (signed by ‘hydra.nixos.org-1’) to ‘/nix/store/dwcx9w994i9kl19gfvcmfi8xz7f2nsjf-stage-2-init.sh’...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (22) The requested URL returned error: 410 Gone

bzip2: Compressed file ends unexpectedly;
    perhaps it is corrupted?  *Possible* reason follows.
bzip2: Inappropriate ioctl for device
    Input file = (stdin), output file = (stdout)

It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

error: unexpected end-of-file
download of ‘http://hydra.nixos.org/nar/dwcx9w994i9kl19gfvcmfi8xz7f2nsjf-stage-2-init.sh’ failed: No such file or directory
could not download ‘/nix/store/dwcx9w994i9kl19gfvcmfi8xz7f2nsjf-stage-2-init.sh’ from any binary cache
fetching path ‘/nix/store/dwcx9w994i9kl19gfvcmfi8xz7f2nsjf-stage-2-init.sh’ failed with exit code 1
fetching path ‘/nix/store/ff7nk1awqi1wgwmbnbjmn7g5k4srdx67-errors-2.0.1.tar.gz’...
killing process 4145
error: some substitutes for the outputs of derivation ‘/nix/store/qhflwr9zyyilybldjbbdf3a9af0jycxi-stage-2-init.sh.drv’ failed (usually happens due to networking issues); try ‘--fallback’ to build derivation from source 
error: unable to build all machine configurations

This was after performing a local nix-collect-garbage.

I've not had a chance to investigate much yet.

[wizeman]

Apparently, this also happens to me with nixos-rebuild (i.e., even when nixops is not involved).

And even worse, the --fallback option for nixos-rebuild doesn't work:

$ nixos-rebuild switch --fallback                                                     
building Nix...
building the system configuration...
fetching path ‘/nix/store/n2rhmzj3hy7b2gg1ba4dipsrqxrxwixy-rg-src’...

*** Downloading ‘http://homeserv:3000/nar/n2rhmzj3hy7b2gg1ba4dipsrqxrxwixy-rg-src’ to ‘/nix/store/n2rhmzj3hy7b2gg1ba4dipsrqxrxwixy-rg-src’...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (22) The requested URL returned error: 410 Gone

bzip2: Compressed file ends unexpectedly;
        perhaps it is corrupted?  *Possible* reason follows.
bzip2: Inappropriate ioctl for device
        Input file = (stdin), output file = (stdout)

It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

error: unexpected end-of-file
download of ‘http://homeserv:3000/nar/n2rhmzj3hy7b2gg1ba4dipsrqxrxwixy-rg-src’ failed: No such file or directory
could not download ‘/nix/store/n2rhmzj3hy7b2gg1ba4dipsrqxrxwixy-rg-src’ from any binary cache
fetching path ‘/nix/store/n2rhmzj3hy7b2gg1ba4dipsrqxrxwixy-rg-src’ failed with exit code 1
error: some substitutes for the outputs of derivation ‘/nix/store/c7sm6c47l9s6aijimsnxnkl2sv0gk5jj-rg-src.drv’ failed (usually happens due to networking issues); try ‘-
-fallback’ to build derivation from source                                                                                                                             
(use ‘--show-trace’ to show detailed location information)

I will file a separate bug for --fallback not working.

[cleverca22]

i think the best option is to use a fetchurl call, so it can reproduce the output from any system after a GC

[wizeman]

In my case, it seems that the derivation which fails to download (because it got wrongly garbage collected) is the output of a pkgs.srcOnly derivation (the derivation with name kodiOff-src), which is itself an input to the import command in the kodifOff_pkg variable.

So if I were to guess, I would say that in my case, the problem comes from doing an import from a derivation output, which seems to be breaking the chain of references when doing the garbage collection (i.e., the GC doesn't consider the input to the import as being needed, so it deletes it, even though it is obviously needed).

[copumpkin]

Relevant to #954

[peti]

This bug prevents the entire import-from-derivation feature from working reliably in the presence of garbage collection, so IMHO this is a serious issue.

[Infinisil]

This is really a bother. I have a lot of things in my system configuration that required import-from-derivation (IFD), so whenever I run a GC and try to redeploy, it first needs to download and build all kinds of things for ~5minutes.

[petercommand]

I also use IFD a lot in my config files, and this issue is quite annoying for me. As a workaround, I added fake runtime dependencies with writeText for the derivations that are then subsequently imported, and this prevents nix-collect-garbage from gc'ing the imports

[Infinisil]

For things downloaded with the builtin fetchers (builtins.fetchTarball and builtins.fetchurl should work too) I just found this workaround: By using indirect GC roots you can gcroot the whole ~/.cache/nix/tarballs directory:

find ~/.cache/nix/tarballs -type l -execdir ln -svf ~/.cache/nix/tarballs/{} /nix/var/nix/gcroots/auto/{} \;

Or a bit simpler but not as clean (links everything (harmless) and fails if there's nothing to link):

ln -sfv ~/.cache/nix/tarballs/* /nix/var/nix/gcroots/auto

[domenkozar]

Another occurrence is:

/nix/store/vihay323q5fnimdm6gmfc50slngjdpcg-ghc-8.8.3/bin/ghc-pkg: line 12: /nix/store/vihay323q5fnimdm6gmfc50slngjdpcg-ghc-8.8.3/lib/ghc-8.8.3/bin/ghc-pkg: No such file or directory

[domenkozar]

@edolstra would it makes sense for Nix to add all derivations during build time to a temporary GC root?