Nixpkgs: Pinning nixpkgs in configuration.nix?

Created on 7 Jun 2019 · 27Comments · Source: NixOS/nixpkgs

Issue description

I searched for information on how to pin system-wide nixpkgs in configuration.nix, but I couldn't find. I found a lot of information on how to pin nixpkgs for nix-shell or nix-build though. I'm probably just missing something obvious because I'm expecting this to be something very trivial..

The point is that I want to specify nixpkgs commit in configuration.nix because the configuration is quite tightly tied to nixpkgs version. For instance, module options might change, packages might be removed/added/renamed. Also, for easier reproducability, it'd be great if configuration.nix contained all the relevant information so that when passing the file to someone, I don't need to also specify the nixpkgs version and the receiver doesn't have to figure out how to get that nixpkgs version. They would just put the file in place and run nixos-rebuild switch, and that's it.

I tried:

  nix.nixPath = [
    "nixpkgs=https://github.com/NixOS/nixpkgs/archive/4ab1c14714fc97a27655f3a6877386da3cb237bc.tar.gz"
    "nixos-config=/etc/nixos/configuration.nix"
  ];

But this has two shortcomings:

There's no hash for verifyingthe content of the URL. I guess this could be easily solved by using builtins.fetchTarball, so no real problem here.
More importantly, when I modify nixpkgs version, I need to run nixos-rebuild switch twice for it to take effect. The first call updates my system wide nixpkgs and the second call updates my system with this new nixpkgs. As far as I understand, this is a bit problematic:
- First, what if configuration.nix isn't compatible with the nixpkgs version that is currently active in the system? Then the build fails and one cannot upgrade the system by just calling nixos-rebuild switch although configuration.nix would build just fine with the nixpkgs it specifies itself.
- Second, IIUC, nixos-rebuild boot or nixos-rebuild build wouldn't use nixpkgs specified in configuration.nix but what is currently active in the system.

So what is the correct way of specifying nixpkgs commit in configuration.nix?

nixos

Source

jluttine

Most helpful comment

Ok we're getting out of topic :)

Just to answer your question, one possibility might be to introduce a /etc/nixos/default.nix:

let
  nixpkgs = <nixpkgs>; # pin nixpkgs here if you want
  pkgs = import nixpkgs {
    config = {}; # potentially override the system config here
    overlays = []; # <- set your overlays here
  };
in
  pkgs.mkNixOS (import ./configuration.nix)

The overlays are not configurable from nixos modules anymore but that might be a bad idea anyways.

zimbatm on 25 Jul 2019

👍2

All 27 comments

You probably want to use nixpkgs.pkgs as well which sets _module.args.pkgs, so each module will use the pkgs value declared here. Or in other words whenever you have { pkgs, ... }: in a module, pkgs will be equal to the value of nixpkgs.pkgs.

On my local setup the pinning looks like this:

# ~/nixos-config/nixos.nix
{ config, ... }:
let
  nixpkgs = builtins.fetchTarball {
    url = https://github.com/NixOS/nixpkgs/archive/hash-of-the-commit-to-pin.tar.gz;
    sha256 = "checksum";
  };
in {
  # no need to inject `nixpkgs.overlays` here, this will be done by NixOS
  nixpkgs.pkgs = import "${nixpkgs}" {
    inherit (config.nixpkgs) config;
  };

  nix.nixPath = [ "nixpkgs=${nixpkgs}" ];
}

More information can be found at <nixpkgs/nixos/modules/misc/nixpkgs.nix>.

Does this help?

Ma27 on 8 Jun 2019

👍1

Thanks! That'll probably improve the solution, but as far as I understand, the latter shortcoming still exists.. That is, for instance, nixos-rebuild switch needs to be run twice for configuration.nix to use the changed nixpkgs for modules and module options. For pkgs, that should work. So I don't still see this as a "correct" solution.

jluttine on 8 Jun 2019

👍1

That is, for instance, nixos-rebuild switch needs to be run twice for configuration.nix to use the changed nixpkgs for modules and module options. For pkgs, that should work. So I don't still see this as a "correct" solution.

Yeah, forgot to mention that, sorry. The problem is that we now need some sort of option which overrides the entire module system including the code which evaluates the module system and I'm not sure if that's even possible atm.

I'm pretty sure that right now you have to (1) incorporate a new nixpkgs rev and rebuild and (2) modify the config accordingly and rebuild. In case you run into some kind of edge case where even that breaks, you could still work around with imports and disabledModules.

Ma27 on 8 Jun 2019

There might be some unsafe eval hacks but I think creating your own script that will call nixos-rebuild with the correct -I nixpkgs=… flag is cleaner.

jtojnar on 8 Jun 2019

According to man page, nixos-rebuild doesn't take -I flag.

If pinning nixpkgs in configuration.nix isn't currently possible, I think it is something that should be implemented. Is there something that makes it difficult/impossible or is it just something nobody has bothered to do yet (or is there some solution already)?

jluttine on 8 Jun 2019

It does support the flag. I guess the following line refers to that:

In addition, nixos-rebuild accepts various Nix-related flags

The issue with defining nixpkgs in configuration.nix is that by the time we evaluate the expression, we already need to know NIX_PATH to be able to use angle bracket paths. We would need to split the config parsing to two stages similarly to what we already do with virtualization options.

jtojnar on 8 Jun 2019

It does support the flag. I guess the following line refers to that:

I'm pretty sure as well. But it seems as this sentence can be missed easily, so I'll open a PR later which adds some more helpful nix flags to the man page.

There might be some unsafe eval hacks but I think creating your own script that will call nixos-rebuild with the correct -I nixpkgs=… flag is cleaner.

That's an interesting approach! But may I ask how you update your Nix path then?

The issue with defining nixpkgs in configuration.nix is that by the time we evaluate the expression, we already need to know NIX_PATH to be able to use angle bracket paths. We would need to split the config parsing to two stages similarly to what we already do with virtualization options.

That's basically what I tried to explain in https://github.com/NixOS/nixpkgs/issues/62832#issuecomment-500140031.

From my point of view it's not really worth to make the module system even more complex to achieve this, I'm using the approach with two rebuilds for a new nixpkgs for at least a year and it works pretty great here, however we should improve the documentation (either in the manual or in the wiki) before closing this issue.

Ma27 on 8 Jun 2019

That's an interesting approach! But may I ask how you update your Nix path then?

You would update it in the file as outlined in https://github.com/NixOS/nixpkgs/issues/35411#issuecomment-368172579

Or you can combine it with the hashing:

#!/bin/sh
nixpkgs=$(nix-build --no-out-link -E '(builtins.fetchTarball {
    url = https://github.com/NixOS/nixpkgs/archive/hash-of-the-commit-to-pin.tar.gz;
    sha256 = "checksum";
  })')

nixos-rebuild -I nixos-config=configuration.nix -I nixpkgs=$nixpkgs "$@"

or store the hash and url in a separate file and import it to the script through builtins.fromJSON (builtins.readFile ./nixpkgs.json).

jtojnar on 8 Jun 2019

Oh... of course... with -I the path is available as <nixpkgs> :sweat_smile:

But @jluttine is this sufficient for you? In that case I'd prepare a PR to update the docs accordingly next week.

Ma27 on 9 Jun 2019

I suppose that'll work, thanks! But I would still love to see some better built-in support for pinning the nixpkgs commit..

In that example:

do I need to specify -I nixos-config=...?
do I need to specify nix.nixPath in configuration.nix?
is that -I nixpkgs=... in nixos-rebuild sufficient for setting the nixpkgs for all subsequent calls to nix-shell, nixos-rebuild etc or do I need to set nix.nixPath or something else for that?

How about this: Could I pin nixpkgs with nix-channel somehow? Some very sloppy pseudo sketching:

nix-channel --add mynixpkgs.nix nixos

And mynixpkgs.nix would be something like:

builtins.fetchTarball {....}

Then, changing nixpkgs commit would require modiyfying mynixpkgs.nix, calling nix-channel and then I can run any nixos-rebuild or nix-shell or anything consistently. Does this make sense?

jluttine on 9 Jun 2019

do I need to specify -I nixos-config=...?

Only if you set nix.nixPath. Otherwise it defaults to /etc/nixos/configuration.nix. I would do it though, so that you can checkout the repo anywhere you want.

do I need to specify nix.nixPath in configuration.nix?

You might want that if you want other Nix commands to use that pinned repo. Probably something like the following might work.

nix.nixPath = [ "nixpkgs=${<nixpkgs>}" ];

is that -I nixpkgs=... in nixos-rebuild sufficient for setting the nixpkgs for all subsequent calls to nix-shell, nixos-rebuild etc or do I need to set nix.nixPath or something else for that?

No, -I flags only apply to the command they are passed to. They do not even apply recursively (e.g. if you run nix-build in nix-shell with -I you will want to pass it to nix-build as well). So setting nix.nixPath is probably desirable, see the previous point.

How about this: Could I pin nixpkgs with nix-channel somehow? Some very sloppy pseudo sketching:
nix-channel --add mynixpkgs.nix nixos
And mynixpkgs.nix would be something like:
builtins.fetchTarball {....}
Then, changing nixpkgs commit would require modiyfying mynixpkgs.nix, calling nix-channel and then I can run any nixos-rebuild or nix-shell or anything consistently. Does this make sense?

You could use nix-channel --add https://github.com/NixOS/nixpkgs/archive/hash-of-the-commit-to-pin.tar.gz nixos but Nix would not know the URI is immutable and would try to re-download it every once in a while. Also there would not be an hash check.

Your solution might work but I am not familiar enough with nix-channel to says so conclusively. You would need to try it or read the source code to know for sure.

jtojnar on 9 Jun 2019

👍2

~(I've only skimmed the other comments, sorry if I'm overlooking something)~

~What about this?~

{ pkgs, ... }: {
  nixpkgs.pkgs = import (builtins.fetchTarball https://nixos.org/channels/nixos-19.03/nixexprs.tar.xz);
  nix.nixPath = ["nixpkgs=${pkgs.path}"];
}

~Replacing the builtins.fetchTarball (…) with a commit-specific URL and specifying the hash for pinning purposes?~

Just realised that this has the same problem as mentioned before.

lheckemann on 12 Jun 2019

mumble, mumble, cough, https://github.com/NixOS/rfcs/pull/49

lheckemann on 16 Jul 2019

I think it was a mistake to make nixpkgs configurable from NixOS (eg: nixpkgs.overlays).

Without that option it would be possible to spawn multiple nixos configurations with a nixpkgs function: pkgs.mkNixOS ./configuration.nix.

But since we make it configurable from inside the configuration.nix, it means that nixpkgs is forced to be re-evaluated entirely for each machine. This means that deploying multiple machines becomes a O(N) operation in terms of time and memory instead of something closer to O(log(N)).

zimbatm on 23 Jul 2019

Just wondering if it would still be possible to add overlays (I mean, the package set consists of overlays with a fixpoint, so in theory it would be possible to allow something like this, however I'm not sure if that's actually a good idea and even possible as it's a while ago since I had to dig into this part of nixpkgs internals :) ). I mainly bring this up because I actually consider it a good thing to allow configurations to add e.g. patches (not all modules have a .package option) to existing packages that are needed to get a working system in the end.

But do you have any idea/plan on how it would be possible to migrate to such an approach? I'm not sure if such a change is possible atm, but I guess I'd be interested in something like this :)

But unless there's anything further to discuss, I'd propose to close this as this issue is merely a question and it seems as everything needed was said :)

Ma27 on 24 Jul 2019

Ok we're getting out of topic :)

Just to answer your question, one possibility might be to introduce a /etc/nixos/default.nix:

let
  nixpkgs = <nixpkgs>; # pin nixpkgs here if you want
  pkgs = import nixpkgs {
    config = {}; # potentially override the system config here
    overlays = []; # <- set your overlays here
  };
in
  pkgs.mkNixOS (import ./configuration.nix)

The overlays are not configurable from nixos modules anymore but that might be a bad idea anyways.

zimbatm on 25 Jul 2019

👍2

I have sort of hacked my way though this. It's more complicated than "just pinning nixpkgs in nixos-configuration.nix" but it works. https://github.com/colemickens/nixcfg I actually have my imports and overlays effectively pinned as well, along with a update.sh that updates those references. See lib.nix for how I eval nixpkgs with overlays applied, and follow default.nix to see how it all works together. Finally, nixup.sh builds the system and activates it without using nixos-rebuild.

(It looks like I still have nixPath set inside, but it's effectively not used, anywhere I do rebuilds you'll see that I unset NIX_PATH.)

colemickens on 2 Aug 2019

@zimbatm i want to build an ISO using the configuration. When i use pinning like you described, i get this error:

[davidak@ethmoid:~/code/nixos-config-greenbone/machines/targets-image]$ time nix-build -A config.system.build.isoImage
error: attribute 'mkNixOS' missing, at /home/davidak/code/nixos-config-greenbone/machines/targets-image/default.nix:8:3

Do i have to do it differently?

It would be great to have a working example documented!

davidak on 14 Aug 2019

Sorry that was a hypothetical example! Your best bet for now is to use the nixos-generators project right now.

zimbatm on 14 Aug 2019

👀1

@zimbatm how does it help with pinning? I see no option for that.

When it's just a hypothetical example, can the issue be reopened until a solution is implemented?

davidak on 14 Aug 2019

👍1

It is now possible after this is implemented: https://github.com/nix-community/nixos-generators/issues/31

Using -I nixpkgs=URL

For example i build an ISO image using an unmerged PR (branch on my fork):

nixos-generate -I nixpkgs=https://api.github.com/repos/davidak/nixpkgs/tarball/backport_nixos-containers-TimeoutStartSec -c /home/davidak/code/nixos-config-greenbone/machines/targets-image/configuration.nix -f iso

davidak on 26 Aug 2019

one possibility might be to introduce a /etc/nixos/default.nix:
let
  nixpkgs = <nixpkgs>; # pin nixpkgs here if you want
  pkgs = import nixpkgs {
    config = {}; # potentially override the system config here
    overlays = []; # <- set your overlays here
  };
in
  pkgs.mkNixOS (import ./configuration.nix)
The overlays are not configurable from nixos modules anymore but that might be a bad idea anyways.

This example is not hypothetical, since mkNixOS already exists in the form of (import <nixpkgs/nixos> {}).system. In practice it could look like this:

$PREFIX/nixos/default.nix:

let
  nixos = <nixpkgs/nixos>; # pin NixOS here if you want
  configuration = /path/to/config.nix; # no need for `NIXOS_CONFIG`
in
import nixos { inherit configuration; }

Then call nixos-rebuild -I $PREFIX to build the system using configuration modules and nixos-rebuild from the pinned version.

Note though that it will break the fallback cases when nix cannot build a newer version of itself.

For convenience I would add a wrapper to environment.systemPackages:

{ config, pkgs, ...}:
let
  nixos-rebuild = pkgs.writeScriptBin "nixos-rebuild" ''
    #!${pkgs.stdenv.shell}
    exec ${config.system.build.nixos-rebuild}/bin/nixos-rebuild -I $PREFIX $@
  '';
in
{
  config.environment.systemPackages = [ nixos-rebuild ];
}

When using autoUpgrade you have to add autoUpgrade.flags = [ "-I" "$PREFIX" ];. Yes, auto-upgrading makes sense even in this setting, for example if you want to impurely track the unstable branch.

I would like to stress for clarification that we really want to pin a nixos version, which in turn specifies the nixpkgs version the configuration modules depend on.

While the OS distribution and the package collection are two distinct things, right now they live in the same repository "for consistency" (whatever that means). So the way of specifying the dependency is a bit obfuscated, and although that is not essential, it was misleading me when working this out. In hindsight, the warning on nixpkgs.pkgs totally makes sense. Overriding that option is what we usually do not want.