Nix: Allowing derivations to verify existence of dependency subpaths before building

[Infinisil]

Is your feature request related to a problem? Please describe.

When you reference a subpath of a dependency, e.g. "${pkgs.bash}/bin/bash", it can easily happen that the given path doesn't exist. E.g. "${pkgs.neovim}/bin/vim" doesn't exist, and this would only be caught at runtime (or never).

Describe the solution you'd like

Nix could have built-in support for verifying existence of dependency subpaths. The low-level interface for this could be a special __getDerivationSubpath (or so) attribute on derivations, which is a function taking a path and returning a string to that path within the derivation:

let
  foo = derivation { ... };
  bar = derivation {
    # Existence of "bin/foo" in the foo derivation gets verified when the build starts
    buildCommand = "cp ${foo.__getDerivationSubpath "bin/foo"} $out";

    # In comparison, this would've been done previously, without verification:
    # buildCommand = "cp ${foo}/bin/foo $out";
  };
in bar

To implement this:

• The string context format will have to be changed to allow depending on specific paths, such that the __getDerivationSubpath function can return a string which can track a subpath dependency. This might look like
  nix builtins.getContext (pkgs.bash.__getDerivationSubpath "bin/bash") -> { "/nix/store/1psqjc0l1vmjsjy4ha5ywbv1l0993cka-bash-4.4-p23.drv" = { outputs = { out = [ "bin/bash" ]; }; }; }
• The .drv format will have to be changed in the same way, such that the dependency subpaths can be communicated to the builder. This might look like
  json { "inputDrvs": { "/nix/store/1psqjc0l1vmjsjy4ha5ywbv1l0993cka-bash-4.4-p23.drv": { "out" : [ "bin/bash" ] } } }
  To make this backwards compatible (and not change all hashes), Nix will probably have to support both old and new .drv formats, only opting into the new format for derivations whose path list in the string context is non-empty.
• The derivation builder will have to be changed to verify all the given subpaths of each dependency before starting the build

Describe alternatives you've considered

Also brought up was that maybe more extensive checks could be allowed, e.g. that paths are executables or directories, which would then not only describe which paths are needed, but also what for. With this it would be hard to draw the line, as you could implement all kinds of checks. This might be something to look into in the future.

Additional context

This was discussed in #nixos with @cole-h, @grahamc and @viric

[Ericson2314]

What not just have another derivation which does the check and which is also a dependency? Then we don't need to complicate the heart of Nix (store model) and merely have a builtin or even lib function to extend the string context with that dependency.

[7c6f434c]

buildCommand = "cp ${foo.__getDerivationSubpath "bin/foo"} $out";

This is more likely __getDerivationSubpath foo "bin/foo", right?

To make this backwards compatible (and not change all hashes), Nix will probably have to support both old and new .drv formats, only opting into the new format for derivations whose path list in the string context is non-empty.

Can it be a separate attribute for checks in the derivation? If it is unused, the attribute is not added.

Also brought up was that maybe more extensive checks could be allowed, e.g. that paths are executables or directories, which would then not only describe which paths are needed, but also what for. With this it would be hard to draw the line, as you could implement all kinds of checks. This might be something to look into in the future.

This is most likely to be used inside generated shell scripts.

I think it would be nice to see something on the benefits compared to in-Nixpkgs library function that generates something like $(test -d "$foo/bin/foo" && echo "$foo/bin/foo").

[grahamc]

What not just have another derivation which does the check and which is also a dependency?

I do this right now and find it can create hundreds of derivations and store paths that I don't care about but take significant amount of time to build in aggregate, that ultimately need to either dereference at use, or add another symlink hop to every execution. It is workable-ish, but a bit painful.

There is a bit of prior art here: https://github.com/nixos/nix/commit/f958bcdf1f9f66759a2512e4b7c0b0ba5647960a

[Ericson2314]

I do this right now and find it can create hundreds of derivations and store paths that I don't care about but take significant amount of time to build in aggregate

For this I rather investigate the overhead of a derivation and keep the same data model.

That ultimately need to either dereference at use, or add another symlink hop to every execution. It is workable-ish, but a bit painful.

Can the derivation just produce and empty file or something? We don't actually need to use the output, just make sure that the derivation succeeds.

[grahamc]

Can the derivation just produce and empty file or something? We don't actually need to use the output, just make sure that the derivation succeeds.

I'm not sure what this would look like -- can you provide an example?

[Ericson2314]

The main trick is to get that derivation in the string context without influencing the string. if substring doesn't strip the string context could we do something like:

let
  checkBash = check bash "/bin/bash";
in builtins.substring (builtins.stringLength checkBash) (-1) (checkBash + bash)

[grahamc]

I'll switch over to this model and give it a go, thanks!

[Ericson2314]

Hope it helps!

[Infinisil]

Actually there's lib.addContextFrom for this already. Works pretty well actually, here's a demo:

let
  pkgs = import <nixpkgs> {};
  inherit (pkgs) lib;

  derivationPath = drv: path:
    let
      check = derivation {
        name = lib.strings.sanitizeDerivationName "check-${drv.name}-${path}";
        builder = pkgs.stdenv.shell;
        args = [ "-c" ''
          requiredPath=${drv}/${lib.escapeShellArg path}
          if [[ -e "$requiredPath" ]]; then
            > $out
          else
            echo "Path $requiredPath doesn't exist!"
            exit 1
          fi
        '' ];
        system = drv.system;
      };
    in lib.addContextFrom check (drv + "/${path}");

in pkgs.writeScript "test" ''
  #!${derivationPath pkgs.bash "bin/bash"}
  echo hi
''

This can be nix-build't just fine, but if you change the bash path to e.g. "bin/hash", it will fail before building!