Nix: Build loops trying to substitute when output is missing

[roberth]

Describe the bug

Suppose we have a derivation with two outputs, out and doc, where out will reference doc.
out is in the cache but doc has been removed.

1. Something needs out to be valid
2. out narinfo is fetched. out depends on doc
3. doc narinfo is fetched; is missing
4. doc needs to be built. Build the drv
5. drv may be substitutable, let's fetch out. go to 2, infinitely

So the false "assumption" seems to be that a _path_ dependency implies a _derivation_ dependency, whereas it could be the same derivation.

Steps To Reproduce

$ nix copy $(nix-build channel:nixos-unstable -A nix --no-out-link) --to file://$HOME/tmp/nixcache-2020-08-26
$ nix-store -q --references $(nix-build channel:nixos-unstable -A nix --no-out-link)
...
/nix/store/wbj5y36q3si9ipwa0vnnjgvpmbggxibd-nix-2.3.7-man
...
$ rm ~/tmp/nixcache-2020-08-26/wbj5y36q3si9ipwa0vnnjgvpmbggxibd.narinfo
$ nix-store --delete $(nix-store -q --outputs $(nix-instantiate channel:nixos-unstable -A nix))
$ nix-build -vvv channel:nixos-unstable -A nix --option substituters file://$HOME/tmp/nixcache-2020-08-26 --option narinfo-cache-negative-ttl 10

-vvv is the least verbose level that prints from the loop.

Expected behavior

build.cc determines that not all outputs are substitutable and proceeds to build.

(It may still need to build input _derivations_ that by definition of input can not be the same derivation.)

nix-env --version output
nix-env (Nix) 2.3.6

Additional context

Add any other context about the problem here.

[Ericson2314]

So I've wanted to shift the division of labor between subtitution and derivation goals: I think we shoul just have substitution and building goals:

• no more wanted outputs, building goals build all outputs

• substitution goals fall back on building if cannot substitue

• building goals don't try to substitute

I beleive that would incidentally solve your problem!

[roberth]

@Ericson2314 that sounds good. So if I understand correctly, after instantiation you'd make substitution goals for all top-level outputs, which will fall back to building as necessary.
When a substitution fails, we'll still need a state for substituting the inputs. So,

building goals don't try to substitute

If I understand correctly, you mean: building goals don't try to substitute their outputs
This will be the responsibility of the code that creates the DerivationGoal.

Is that something you want to implement?

[Ericson2314]

@roberth Yes I think we're on the same page. I'd love to take a look at implementing it as soon as next week, because it is also good for building derivations that are themselves the outputs of derivations.

[domenkozar]

Most probably also the cause for #3534

[Ericson2314]

https://github.com/NixOS/nix/blob/f15651303f8596bf34c67fc8d536b1e9e7843a87/src/libstore/build.cc#L1204-L1207

Ah slight bummer, with the new plan I would still have to read in derivation in the substitution goal to decide whether to immediately fallback on building rather than trying to substitute.

[Ericson2314]

Ah, better plan: a 3rd goal type. If we have a "plain goal", where we haven't decided whether substitute or build yet, and again get rid of wantedOutputs limiting build goals to just the drv path, we solve this problem and clearly support substitutesAllowed. Yay!

[roberth]

Sounds good! So we'll start with something like RealisationGoal which then produces SubstitutionGoal or BuildGoal.
I do think it makes sense to keep wantedOutputs, so we don't fetch more than necessary from substituters, but of course we can forget wantedOutputs when switching to BuildGoal.

[Ericson2314]

I do think it makes sense to keep wantedOutputs

My idea was to have a single output per realization goal, so we aren't change the "goal of the goal" in flight.

but of course we can forget wantedOutputs when switching to BuildGoal

And indeed they are cleared already.

---

Regardless, I took a stab at this yesterday, and it's proved irksome.

For example, see retrySubstitution which currently the derivation goal restart after kicking of inputDrv subgoals to try to repair the substitution closure. I kept on imaging fancier and fancier refactors in my head, such separating derivation goals which just load a derivation from the new RealisationGoal and BuildGoal giving us 4 goal types total

I think I'd want to definitely talk more about what we want --- perhaps in real time at the hackathon? --- to try to come up with a design we all pre-agree upon. (And through in the other stuff I mention in https://github.com/NixOS/nix/pull/4114#issuecomment-705914109 and the scale of the endeavor just goes up further.)