Nix: Add `builtins.fetchAny` to allow arbitrary fallbacks for fixed-output derivations

[Warbo]

(Spawned from some discussion on #859 )

External sources may be unreliable, e.g. HTTP URLs disappearing, git repos being deleted, P2P seeders disconnecting, etc. It would be nice if our Nix expressions could specify fallbacks to avoid breakages during a server outage, and to make 'unmaintained' derivations more future-proof (e.g. examples on blog posts, or replication instructions for published experimental results, which may be read many years after their upstreams disappear).

At the moment there are (at least) two examples of this in nixpkgs:

• pkgs.fetchurl can take a list of URLs, trying each until one succeeds.
• pkgs.fetchipfs can take a HTTP URL, which is downloaded if the given IPFS content hash isn't found (in which case the builder also tries to insert this result into the local IPFS node).

Hard-coding support for particular methods is pretty trivial to implement, since we just need to add the relevant bash commands to the builder (e.g. the fetchipfs builder calls curl in the fallback case). However, making this generic, so that another derivation is used as the fallback, seems (to me) to require specific support in Nix (or maybe a plugin? I think this is generically useful for core though).

As a concrete example, I self-host my git repos so that I retain control over their availability; yet I'm a terrible sysadmin and don't want to be a single point of failure for hosting. To avoid this I mirror everything on GitHub. Additionally, GitHub provides tarballs of each commit, which is faster than cloning the whole repo (fetchFromGitHub uses these), but might have a breaking change in the future (this seems to have happened on GitLab). I also mirror some repos on IPFS, but garbage-collect old ones.

At the moment I face a "false trichotomy": I must hard-code one of these sources into my Nix expressions, either fetchgit { ...chriswarbo.net... } (which does a full clone, and might be broken through my clumsiness) or fetchgit { ...github.com... } (which does a full clone, is outside my control, relies on GitHub being solvent, relies on my account not being suspended/deleted/censored/etc.) or fetchFromGitHub { ... } (which is fast, but might break URLs, plus all of the control issues of the previous).

My proposal is that we could instead say something like:

with import <nixpkgs> {};
with rec {
  rev    = "54d2a89";
  sha256 = "0hh56xk8z1bzv2v1j2vxmmap8bww8wkjkfqx4cf43jgigalw5miz";
  src    = builtins.fetchAny [
    # Fast, but relies on GitHub's archive service
    (fetchFromGitHub {
      inherit rev sha256;
      owner = "Warbo";
      repo  = "asv-nix";
    })
    # Slow, relies on GitHub, but is reliable as long as it still exists
    (fetchgit {
      inherit rev sha256;
      url = https://github.com/Warbo/asv-nix.git;
    })
    # May suffer outages, but is within my power to fix
    (fetchgit {
      inherit rev sha256;
      url = http://chriswarbo.net/git/asv-nix.git;
    })
  ];
};
import "${src}" {}

This seems to require specific support in Nix because evaluation/building needs to carry on even after some derivations fail to build, which AFAIK is not something we can handle in Nix via tryEval or similar. (Note that the existing, hard-coded methods don't have this problem since they only define a single derivation, whose build script does all of the necessary error handling and retrying). I don't think we can do that at the moment without recursive Nix, which is generally hacky and was specifically disabled in Nix 2.0 (e.g. see my comments and awful workarounds on #13 ;) ).

If we had recursive Nix, we could create all of the .drv files up front, and our build script would just call nix-store --realise on each until one of them worked. Without that, I imagine there would need to be some special-case handling in C++.

[grahamc]

This is a bit problematic because for evaluation to be stable, Nix would need a guarantee that every version in the fetchAny has the same sha256.

Note Nix's fetchurl already has support for a content-addressable source: https://github.com/NixOS/nixpkgs/blob/89ef631ab9c8340eef83219fd41a5419d3268849/pkgs/build-support/fetchurl/builder.sh#L61 maybe something like this could be exploited to allow a local mirror of everything.

[7c6f434c]

If fetchAny itself must be fixed-output, the problem is not much different from the situation with fetchurl alternate URLs.

[vcunat]

This doesn't seem in the correct repo. I see no reason to need to modify nix. A fixed-output derivation composed from "multiple parts" can surely do this, i.e. it would be implemented purely as (nixpkgs) expression.

[7c6f434c]

Two counterpoints:

1. If we want a multiplexor of multiple fetchers that doesn't need to know about their internals and needs to realise only one of the inputs, it seems hard to do in Nixpkgs without Nix support.

2. If there is a plan to reconsider network access of fixed-output builders, it could be useful to consider it in the context of various needs.

[Warbo]

@grahamc You're right that they all need the same hash output. I think the best option is to check this with an assertion at eval time. I did wonder if we could give a single hash argument to fetchAny, e.g. fetchAny { sources = [...]; sha256 = "..."; } but I think that's less desirable since it would complicate the fetchers (e.g. they might need to be closures accepting a hash argument, rather than plain derivations; or we might have to fiddle with their outputHash attributes which users might not expect).

[Warbo]

@vcunat It would be nice if this could be done in nixpkgs, but I don't think that's possible (at the moment). For example, let's say we have fetchAny [a b c d], where a, b, c and d are derivations. Let's say that a will fail to build, that b has a dependency that fails to build and c will succeed. The result of this fetchAny call should be a fixed-output derivation which, when built, will attempt to build a; when that fails it will attempt to build b; when that fails it will attempt to build c, and the output of that derivation will be used (d will never be built).

As far as I can tell we can't do this with pure Nix, for example we can't do this with tryEval (since that deals with evaluation errors, not build failures), and I don't think there's any equivalent for build failures:

with {
  a2 = tryEval a;
  b2 = tryEval b;
  c2 = tryEval c;
  d2 = tryEval d;
};
if a2.success then a2.value else (if b2.success then b2.value else (if c2.success then c2.value else (if d2.success then d2.value else abort "None succeeded")))

We also can't do this with a derivation, since it will depend on all of the alternatives and fail if any of them fail, e.g.:

fetchAny = alts: runCommand "fetch-any" { inherit alts; } ''
  for A in $alts
  do
    if [[ -e "$A" ]]
    then
      cp -a "$A" "$out"
      break
    fi
  done
  echo "All alternatives failed" 1>&2
  exit 1
''

Given the [a b c d] example above, this will fail becase it depends on a and b which fail.

One way to do this would be to call Nix from within the build script, but that's not well-supported by Nix at the moment (see issue #13 ). If we use my withNix hack we could do something like:

fetchAny = alts: runCommand "fetch-any" (withNix { drvs = map (x: x.drvPath) alts; }) ''
  for D in $drvs
  do
    if RESULT=$(nix-store --realise "$D")
    then
      cp -a "$RESULT" "$out"
      break
    fi
  done
  echo "All alternatives failed" 1>&2
  exit 1
''

Note that this isn't quite right, since its not fixed-output, has the wrong hash and will trigger a rebuild if we change the list of alternatives (e.g. to add an extra fallback). Still, I hope it demonstrates the idea. This approach is incredibly hacky, so I wouldn't recommend using it.

As far as I can tell, there are five ways we could implement fetchAny:

• Use a horrible, horrible hack like withNix above. Not recommended.
• Use a Nix plugin. Whilst a reasonable approach, in terms of Nix/Nixpkgs this would essentially be a "won't fix" decision that this functionality isn't desirable enough to be provided "officially". That's fine, but in that case we don't even need to write the plugin, since we can just close this off and all move on ;)
• Add specific support to Nix via a new fetchAny builtin, where the Nix C++ code takes care of skipping failed builds. I would be happy with this option, but it's not up to me ;)
• Add a different mechanism to Nix, which allows us to carry on after a build failure. fetchAny could then be implemented in nixpkgs, using this new builtin. Personally, I don't think we would want such a powerful mechanism, since it would be incredibly impure. I can only see two legitimate use cases for such a mechanism: fallbacks for fixed-output derivations (i.e. fetchAny), or using it to check if a derivation fails (but we can already do that).
• Implement recursive Nix ( #13 ), and use that to write fetchAny in nixpkgs. I would be happy with this option, but recursive Nix seems to be on perpetual hold, and getting ever more complicated (e.g. how it interacts with restricted mode, etc.).

There might be some other way to implement this. If so, it would be great to know!

[shlevy]

@Warbo We can't just naively do fetchAny in nixpkgs, but we can if we change our nixpkgs fixed-output derivations to export enough of their inner workings to allow them to be all composed into one derivation. A builtin seems like too big of a hammer for this issue.

[vcunat]

Yes, that's what I meant. Compose the shell builders into one derivation, not compose whole derivations.

[7c6f434c]

I guess, given the track record, a well-defined builtin looks like a more promising bet than trying to move again to a Nix-based composition of builders from string fragments.

For example, a builtin will probably cost less to evaluate.

[Warbo]

@shlevy @vcunat Ah I see. My only concern with that would be 'restricting' the functionality to certain 'blessed' fetchers. If there's a well-defined way to mix custom fetchers in with standard ones (say, writing some bash code that uses particular variable names), then that sounds good to me. Should I open I nixpkgs issue in that case?

[vcunat]

We have not defined that way, but it surely is possible to do in bash.

[vcunat]

From the top of my head, I can imagine a nix builtin that "combines" a list of derivations that all have _the same_ fixed output. Still, even now there's still opposition to using nix 2.0 features in 18.09, and we're not close to having nix code for this builtin yet, so maybe the builtin would be usable in one year from now for general packages in nixpkgs? (Well, yes, we could hack this case like – if your nix version is old, just take the first derivation.)