Nixpkgs: Consider changing `nixpkgs-overlays` to a list of paths

[knedlsepp]

Issue description

I recently started investigating what would be the most convenient way for an inexperienced nix user to be able to use nix to install software from a privately managed set of expressions, which I would maintain for our team.
Turns out that pointing nixpkgs-overlays in your $NIX_PATH to a git-repository tarball is a real nice way to do that:
"nixpkgs-overlays=https://github.com/knedlsepp/nixpkgs-overlays/archive/master.tar.gz".
This has the benefits of not manually having to go to ~/.config/nixpkgs/overlays and git pulling in eventual changes. It has basically the same semantics as subscribing to a channel.
To be able to take this to the next level however I would really love to be able to specify a list of overlays instead of a single path.
Since we want overlays to be stackable it seems like a no-brainer to want something like:

NIX_PATH="nixpkgs=['http://nixos.org/channels/nixos-17.03/nixexprs.tar.xz', 'http://www.github.com/overlay1/master.tar.gz', 'http://www.github.com/overlay2/master.tar.gz']:nixpkgs-16-09=[...]"

I'm not sure what would be the best syntax for that, since this might not be as easily achievable with the current colon notation, but this would certainly be a nice and tidy way to have a custom package set without the hassle of manually managing some ~/.config directory.

Maybe something along the following lines would be doable

NIX_PATH="nixpkgs='<pkgs>':'<overlay1>':'<overlay2>':nixpkgs-16-09='<pkgs-16-09>':'<overlay3>':'<overlay4>'"

[0xABAB]

I think this is another one of those examples where the benefit is questionable and the requester should first use this system for a year, talk about the great success they had with this system at a conference and then request for a specific type of integration.

NixOS is not the place to implement every random idea that ever occurred to anyone.

[7c6f434c]

@0xABAB The very fact that the feature in question is obviously not about NixOS but about NixPkgs makes your objection look more harsh than precise.

Realistically, before trying to implement such a feature, it _is_ an obviously good idea to discuss it, because if you miss any assumptions or development plans, the feature becomes too hard to keep rebased.

Also, NixPkgs has a better track record on rejecting useful-but-maintenance-costly features than on accepting big useful changes anyway… Especially if Nix itself has to be patched…

[0xABAB]

@7c6f434c I wasn't going for precise.

It appears that we are in the NixOS place https://github.com/NixOS, just because NixPkgs is a sub project doesn't make it less true.

I could have been more precise, but you assumed things for no reason and called me out on it, while you know very well that I do know the relation between NixOS and NixPkgs.

[7c6f434c]

I said «objection look[s] more harsh than precise», and you seem to agree about not going for precision — I explicitly commented on the text as written. I think harshness is more natural when a suggestion has a clear chance to be implemented and merged — and I know you know all too well that large changes don't get merged too quickly even when they are good.

[7c6f434c]

@knedlsepp as the current overlay check is listing the subdirectories inside the directory anyway. Do I understand correctly that what would be enough for your use case is just to be able to put a .nix file into the overlay directory that imports from a fetched tarball?

[knedlsepp]

Yes, putting the right .nix files into the overlays directory would probably work.
I do however feel that from a UX point of view the approach of manually curating some files in a config directory feels a little confusing. There was some talk about a tool-based approach to manage those files (Think nix-channel --add; or like the popular conda package manager conda config --add channels conda-forge). Those tools would however need to be written.

I was overjoyed when I found out the other day that one could have this private channel behaviour by simply specifying in your NixOS configuration.nix

nix.nixPath = [ "nixpkgs=https://nixos.org/channels/nixos-17.03/nixexprs.tar.xz" "nixpkgs-overlays=https://github.com/knedlsepp/nixpkgs-overlays/archive/master.tar.gz" ];

(or the equivalent $NIX_PATH on a regular distro).
It turned out however that this does not scale beyond a single overlay easily, which I think is disappointing. This brought me to think a little about the NIX_PATH and it seemed really absurd to me the way that nixpkgs-overlays is represented there. It just feels like a natural fit to want overlays to be an easily extensible list. That's probably the same reason why the initial design used the variable $NIXPKGS_OVERLAYS.
I think overlays are a great concept and should become the standard way of providing stuff on top of nixpkgs as other solutions even though they work are simply more ad-hoc and don't integrate as well as overlays.

[elitak]

I think NIX_PATH should remain as simple as possible. The key=val format is already overloading the traditional use of shell variables. If you want dynamic behavior, you can do just about anything with shell scripting, where manipulations of the environment are most apposite, anyway. For instance, you can inject overlays into existing channels using something like this, embedded in a user's bashrc:

declare -A overlays
overlays[overlay1]=https://github.com/knedlsepp/nixpkgs-overlays/archive/master.tar.gz
overlays[overlay2]=https://github.com/knedlsepp/nixpkgs-overlays/archive/master.tar.gz

injectOverlays() {
  local base="$1"; shift
  local -a overlay_names=("$@")
  nix-build --no-out-link -E "{ pkgs ? import <$base> {} }: pkgs.writeText ''hooked-$base-default.nix'' ''{ overlays ? [], ... }@args: import \${pkgs.path} (args // { overlays = [ $(for k in "${overlay_names[@]}"; do echo "(import <$k>) "; done)] ++ overlays; })''"
}

ORIG_NIX_PATH=$NIX_PATH
HOOKED_NIX_PATH=$(for k in "${!overlays[@]}"; do echo -n $k=${overlays[$k]}: ; done)
HOOKED_NIX_PATH+=nixpkgs-untouched=$(injectOverlays nixpkgs):
HOOKED_NIX_PATH+=nixpkgs=$(injectOverlays nixpkgs overlay1 overlay2):
HOOKED_NIX_PATH+=nixpkgs2=$(injectOverlays nixpkgs overlay2):
HOOKED_NIX_PATH+=$ORIG_NIX_PATH


NIX_PATH=$HOOKED_NIX_PATH
nix-build '<nixpkgs-untouched>' --no-out-link -A pythonPackages.pytest-flask # not found because overlays are absent
nix-build '<nixpkgs>' --no-out-link -A pythonPackages.pytest-flask
nix-build '<nixpkgs2>' --no-out-link -A pythonPackages.pytest-flask

(Change the overlays associative array, the compositions in HOOKED_NIX_PATHS, and the pythonPackages built at the end, to test)

[knedlsepp]

Hm. I actually just found out that setting

nix.nixPath = [ 
  "nixpkgs=https://nixos.org/channels/nixos-17.03/nixexprs.tar.xz" 
  "nixpkgs-overlays=https://github.com/knedlsepp/nixpkgs-overlays/archive/master.tar.gz" ];

Doesn't really do what I was hoping: I thought this would always download the most current version of the nixpkgs-overlays-tarball, but instead this is only done after every nix-collect-garbage

[knedlsepp]

I just found out that instead of using
nix.nixPath = [ "nixpkgs-overlays"]; on nixOS one can use nixpkgs.overlays = [...], which according to the documentation is pretty much what I was hoping to get out of changing nixpkgs-overlays to a list.

List of overlays to use with the Nix Packages collection.

So it seems instead of having nixpkgs-overlays be a list, one can set the list nixpkgs.overlays
(Am I the only one that thinks this is confusing?):

nix.nixPath = [ 
  "nixpkgs=https://nixos.org/channels/nixos-17.03/nixexprs.tar.xz"
  "nixos-config=/etc/nixos/configuration.nix"
  "knedlsepp-overlays=https://github.com/knedlsepp/nixpkgs-overlays/archive/master.tar.gz"
];
nixpkgs.overlays = [ (import <knedlsepp-overlays>) ];

Still there is the issue, that the overlay will only be updated after a nix-collect-garbage

[elitak]

This is working as intended. You're supposed to use nix-channel --update manually (you could cron it system-wide or per-user) to manage when updates occur. Trying to get dynamic (impure) behavior out of the nix expression language is difficult for a reason: you're supposed to be getting the same result every time you run a build/rebuild, not different ones. Again, anything dynamic should be happening in the shell, at runtime.

Incidentally, tarballs are cached by nix, and are only refetched when either the ETag changes or tarball-ttl (default 3600 seconds) is reached. See https://github.com/NixOS/nix/blob/master/src/libstore/download.cc#L599 Using nix-collect-garbage -d is like cluster bombing your system; I really don't advise it, as it blows away the history of every profile on the system, potentially leaving users with broken configs they can no longer rollback. In your case, you're just seeing a side-effect of nix-store --gc, where it deletes your nixexprs tarballs that haven't been marked for persistence by the garbage collector, as nix.nixPath simply sets an envar, whose contents can't be tracked properly as dependencies for the build.

[knedlsepp]

@elitak: I see. I can definitely see there are good reasons that it works the way it does. However I'm currently trying to get the nix-channel --update or the NixOS autoUpgrade feature but for overlays managed via a git repository. Do you know if there's any way to get this behavior? I somehow really dislike the current ways of doing it: cd ~/.config/nixpkgs/overlays/my-overlays; git pull.

[elitak]

Channels are generally supposed to be entire package sets. <nixpkgs-overlays> is a special case, handled here: https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/impure.nix#L43

If you want multiple overlay-channels applied to multiple package sets, you need to inject them as the overlays attr passed to the nixpkgs top-level function, either locally (as in my first comment), or remotely, by having a separate repo or subpath in a repo for each combination of overlays, e.g. "". At cursory glance, all autoUpdate does is run nix-channel --update <named-channel>; nixos-rebuild switch every interval. If you need composability, you will need to augment that nixos module or simply write your own cronjob that will do the local injection of overlays.

If you want multiple overlays applied to a single channel, this gets simpler: just set the <nixos> channel appropriately, then list the overlays you want as nixpkgs.overlays = [ (import <overlay1>) (import <overlay2>) ];.

Taking a step back from all this messy config, if all you're trying to do is curate working sets of packages for novices to use, I suggest just avoiding all the client-side complexity that overlays introduce and making a different branch of nixpkgs for each user-type, then simply setting something like nix-channel --add https://github.com/knedlsepp/nixpkgs/archive/${branchName}.tar.gz nixos and using autoUpgrade(or cronned nix-channel --update, at least). This way, the package set you're trying to express locally using overlays is instead a single remote branch on github composed of multiple merged topic branches.

[Alkeryn]

Guys you can fetchGit import

this is my .config/nixpkgs/overlay.nix
and i just have nixpkgs.overlays = import thatfile;
Edit: just noticed, sorry for the necrobump

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.