Nix: Treat remote builders as substituters?

[lheckemann]

I can see some potential advantages to treating remote builders as substituters before trying to build any derivations — this would allow the build plan to be executed more efficiently in some cases, with less copying around of paths and potentially less overall building as well.

Consider:

• b.drv, a derivation which produces the output path /nix/store/b
• c.drv, a derivation which produces the output path /nix/store/c
• a.drv, which produces the output /nix/store/a and depends on the outputs of b.drv and c.drv. /nix/store/a has a reference to /nix/store/b but none to /nix/store/c.
• Build initiator alice, having bob and charlotte configured as remote builders, doesn't have any of the output paths available, nor can it build any of the drvs (incompatible system or something)
• Builder bob doesn't have any of the paths available, but can build them
• Builder charlotte has all the paths built.

Now running nix build /nix/store/a.drv on alice can result in something like the following:

• alice makes bob build b.drv and copies the output path over
• alice makes charlotte build c.drv (no-op) and copies the output path over
• alice copies /nix/store/c to bob and makes bob build a.drv, then copies the output path over

End result being that all three derivation's output paths have been copied to all three systems, and b.drv and a.drv have been unnecessarily rebuilt.

If alice has bob and charlotte configured as substituters additionally, alice will first query them for /nix/store/a, and upon determining that charlotte already has the path, will only copy /nix/store/a and /nix/store/b to itself, resulting in no unnecessary building and no superfluous space usage.

Are there any drawbacks to universally treating builders as (absolutely trusted, i.e. no sigs required) substituters as well?

[7c6f434c]

I think it would be an improvement even without no-signatures: it would help against unnoticed IP recycling, and you already need to enter a lot of data about remote builders anyway.

[lheckemann]

The thing with changing it to require signatures is that that would break existing setups: up to now, using remote builders hasn't required setting up signing keys. I don't see how IP recycling comes into this either.

[7c6f434c]

I would say that adding normal substituters for remote builders in parallel with the old style is the most likely way just from the testing point of view, and then addition/default/deprecation/removal-of-old have to go in this order with large enough waiting periods anyway…

[dezgeg]

Well, if you look at this step:

alice makes charlotte build c.drv (no-op) and copies the output path over

the builder charlotte was already effectively used as a substitutor as no building happened (and charlotte might even have obtained the output of c.drv via substituting from some other place, never building it itself). So security-wise there's no difference since substitution can already happen.

[lheckemann]

What this would also allow (though not imply directly) is lazy copying — so in the strongest case where only one builder is available for all the derivations that need to be realised as part of a build plan, everything is built only on that builder and only the end result and its reference closure are copied to the build initiator. Since this is a change in behaviour, I'm not sure if that's universally desirable — but I can definitely see some nice use cases for it, like nixos-rebuild on devices with small storage.

[7c6f434c]

I would say it is very annoying that it is complicated to do nix copy when the source host has all dependencies of the path you need but not the path itself, or to grab all the build dependencies of a path from a cache that can give you the path itself and its runtime dependencies instead — so a unification _and_ an option to choose which dependency-walking logic to use would be nice…

[lheckemann]

While I do agree that that would really be nice to have, I'd say that's out of scope for this issue as it is quite a bit more complex than just "treat remote builders as binary caches".

[angerman]

I use short-lived remote builders (usually ec2 spot instances). That only live for a few hours. If we would stop copying closures back from the remote builder (into my local cache), I would potentially end up rebuilding everything every time I spin up a spot instance. Costs aside, it would result in unnecessary wait time for potentially cached artifacts.

[lheckemann]

Since this use case exists, I suppose it would make sense not to universally treat remote builders as substituters, and this should be done manually by putting them in the substituters option. This could still do with some improvement though:

• Needs key management — I'm not sure it's currently possible to specify trusting specific substituters without signatures on their paths, the way remote builders are already trusted?
• “Lazy copying” (copying only runtime deps to build initiator) not possible

[Ericson2314]

Err I think it's easier to optionally require fetching closures during substitution. More generally, I think all 4 of these make sense:

• fetch
• fetch + remote build
• fetch + always get closure
• fetch + remote build + always get closure

One can few that as feature sets "generated" by combining remote builders and substitutors into a single configurable concept.

[lheckemann]

remote builders and substitutors into a single configurable concept.

Which they are in fact already, in principle, since 2.0 — they're all remote stores, just used in different ways. Very much +1 on this idea! Although the "always get closure" thing I think would be more of a per-build thing?

[Ericson2314]

Good point about 2.0. I thought this was the case but I forgot.

Yeah I suppose the closure option has no security implications, so it can be purely per-build. (The rest also make sense per-build, but for untrusted users would need to be within the daemon's whitelist.)