Nix: allowSubstitutes/preferLocalBuild are an anti-feature

[domenkozar]

I can't describe how many hours I and people I have helped have lost to save a few miliseconds of machine time.

For example preferLocalBuild ignores builders and --max-jobs per https://github.com/NixOS/nix/issues/3810 giving the user the impression that flags are being ignored.

Having such exceptions completely change the way Nix works makes it also really hard to debug binary cache issues - I've gotten lots of reports of why is binary cache not substituting.

[grahamc]

I think I would rather keep preferLocalBuild and have it respect --max-jobs instead. I don't relish the idea of uploading 10's of GB of data to a remote server over DSL to write a short text file, just to download it again.

[Ninlives]

the preferLocalBuild overwrites the --max-jobs may be an anti-feature, but allowSubstitutes/preferLocalBuild themselves are useful. I do not want to waste time on querying binary cache when I just want to write a text file.

[Ma27]

The problem I have with removing preferLocalBuild is that I regularly use it to configure on a per-derivation level whether or not I want remote builds. As @grahamc stated, this is helpful to avoid uploading tons of data to a remote server just to build e.g. a proprietary package where only an unpack and a patchelf happens. This is particularly helpful for me since I'm regularly (not during the pandemic though) at places where I have rather bad internet.

[domenkozar]

It's a great power feature that hits Nix beginners really had and given that it's used in nixpkgs there no opt-out.

Of course all expirienced Nix users will say they have no problem with it, but it makes behavior with substituters and remove builders hard to debug and reason about.

[bgamari]

I use preferLocalBuild quite heavily as it can mean the difference between waiting a few seconds and a few minutes in the case of derivations with large inputs and outputs.

[domenkozar]

I often (need to start tracking how often) get Cachix support request why is something not being substituted and I have templated an answer, but the response from Nix beginners is usually: wow

This probably can be fixed with better logging/messaging and fixing all the bugs, but I feel like that's one of those things that will never materialize.

[Ma27]

Maybe it would help to make this clear in at least the new progress bar that's currently under development: https://github.com/NixOS/nix/pull/4296

[domenkozar]

I understand that preferLocalBuild can save minutes, but allowSubstitutes saving is really marginal: as those paths still need to be uploaded, so really you only save the substitution.

For preferLocalBuild there should at least be good -v story to explain why something was built where it was.

[roberth]

allowSubstitutes can cause the download of build tools for small store paths that have already been built and uploaded. It's really not that clever, last time I checked. Mostly relevant for C/D agents and other cases where only runtime closures are necessary.

[rickynils]

From the discussion here, it seems there is a majority in favor of keeping preferLocalBuild as it works today, but do something about the max-jobs=0 situation. I'm also of the opinion that text files and similar should be built locally by default, but that it should be possible to not build them locally. The "prefer" word seems to me imply that this was intended from the start, and that the max-jobs=0 behavior is just a bug. I've opened a PR resolving that bug.