Nixpkgs: Cross compilation changes have made undocumented, breaking changes to NixOS

[grahamc]

Issue description

It used to be possible to build a NixOS system on a macOS machine via:

import <nixpkgs/nixos> {
  configuration = {
    nixpkgs.system = "x86_64-linux";
  };
}

however, this now errors when evaluating pkgsi686Linux due to an assertion on line 156 of pkgs/top-level/stage.nix.

Searching the docs / changelogs for how to fix this turned up nothing, so I asked on IRC. The solution (thanks to Clever) was replace to do this instead:

import <nixpkgs/nixos> {
  configuration = {
    nixpkgs.localSystem.system = "x86_64-linux";
    imports = [ ./extra.nix ];
  };
}

This made some progress, but because I also re-imported a second nixpkgs inside of my NixOS configuration using this common idiom:

{ pkgs, config, ... }:
let
  otherpgks = import <otherpkgs> {
    # ...
    inherit (config.nixpkgs) system;
  };
in { 
 # ...
}

I now get an extensive error about not being able to coerce a set to a string.

My initial intuition was to fix it like I fixed nixpkgs:

{ pkgs, config, ... }:
let
  otherpgks = import <otherpkgs> {
    # ...
    inherit (config.nixpkgs) localSystem;
  };
in { 
 # ...
}

but the change to NixOS is not evidently paired with a change in Nixpkgs, so I got further help from Clever and settled on:

{ pkgs, config, ... }:
let
  otherpgks = import <otherpkgs> {
    # ...
    inherit (config.nixpkgs.localSystem) system;
  };
in { 
 # ...
}

This should probably get at least some release notes.

---

forceSystem is also not behaving the way it did in 18.03, which should probably also get a release note.

It used to be possible to use pkgs.(forceSystem "x86_64-linux" "x86_64") and have overlays retained, now it will do a reimport and lose overlays.

[grahamc]

cc @samueldr

[grahamc]

cc @vcunat

[samueldr]

@Ericson2314 could you provide an overview of those changes while assuming one does not understand neither cares about the host/build/target maths and stuff?

(It's not that I don't care, but if one needs to know those details to make use of what was trivial, this is a huge UX drawback with those changes.)

[grahamc]

I'm happy to do docbookification if needed :)

[Ericson2314]

None of this breakage looks intentional to me, actually. I'll investigate.

CC @matthewbauer

[Ericson2314]

OK there's a few things going on here, and I don't see any easy wins :(.

First some background:

1. The two fundamental platform-picking parameters to all of nixpkgs are localSystem and crossSystem. https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/default.nix#L20-L27. This has been the case for quite some time: https://github.com/NixOS/nixpkgs/commit/8cd4c31d6b8c6432986edf0ec5d00d2fe1c12854#diff-017a38a631b06991d33857f7681874b3

2. https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/impure.nix does a bunch of heroics to also make legacy system and platform control localSystem:

   1. https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/impure.nix#L15-L25
   2. https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/impure.nix#L78-L80
   3. https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/impure.nix#L84-L87

   It's gross but it does at least work correctly.

On the NixOS we need to do the same sort of heroics, but it doesn't work nearly as well. https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/misc/nixpkgs.nix . Fundamentally, people should be able to set either of config.system and config.localSystem and then read back the configuration from both two. But this is in fact impossible with the way the module system currently works (with ramifications for overlapping / deprecated interfaces in general. Basically, without something like the self-super distinction, there's no way to have each "update" the other without creating infinite loops.

Instead here's what's done:

1. Both options use builtins.currentSystem for their defaults:

   1. https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/misc/nixpkgs.nix#L143
   2. https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/misc/nixpkgs.nix#L182

2. https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/misc/nixpkgs.nix#L69 Only localSystem is set on the nixpkgs side (recall that impure.nix will reject if both are set because it's unclear which should take precedence). On the nixos side, localSystem does in fact override system, which is bad because it's default without builtins.currrentSystem will clobber system.

3. https://github.com/NixOS/nixpkgs/pull/38485 made all NixOS modules refer to config.localSystem.system instead to match localSystem taking precedence and work around that we cannot make config.system be updated by config.localSystem.

[Ericson2314]

So in the short term, the best I can think of doing is only have system have the builtins.currentSystem default, and make some third config.resolvedLocalSystem (without a corresponding option.*) which is both what is passed into nixpkgs and what the other packages reference.

In the long term, this duplication is a mess (even if we were to overhaul the module system to have the self-super distinction) and the only solution I see is to just provide localSystem.

Note that while I original did these changes with cross compilation in mind, their primary benefit is not that but being able to more precisely talk about the local system. For example, native compilation with musl cannot stem from any choice system because there's no way to distinguish between different ABIs/libcs for the same CPU+OS. (CC @dtzWill.)

[Ericson2314]

https://github.com/NixOS/nixpkgs/commit/4fe289860888668956b7e79e24efeb101c2f51d1 was a failed attempt to make it better. I'll try again.

[grahamc]

Does that mean the best solution is documentation?

[Ericson2314]

@grahamc Actually not.

https://github.com/NixOS/nixpkgs/issues/46320#issuecomment-419538451 So in the short term...

I can do a version of that but without the need for resolvedLocalSystem. localSystem will default to { inherit (config.nixpkgs) system; } and override system. This is the "one way" and thus cycle-less version of https://github.com/NixOS/nixpkgs/commit/4fe289860888668956b7e79e24efeb101c2f51d1, which should avoid the infinite recursion problems, and also avoid the double-default problems which you hit.

[Ericson2314]

Arg! thing are working but something keeps on clobbering localSystem's default. I thought it was nixos/lib/eval-config.nix's system paramter, but it it something else. The sooner we can get rid of all these systems, the better...

Edit it was that.

[Ericson2314]

https://github.com/NixOS/nixpkgs/pull/46341 should fix all of them.

[Ericson2314]

With that PR

{ pkgs, config, ... }:
let
  otherpgks = import <otherpkgs> {
    # ...
    inherit (config.nixpkgs) system;
  };
in { 
 # ...
}

will continue to work as long as you don't override localSystem.

{ pkgs, config, ... }:
let
  otherpgks = import <otherpkgs> {
    # ...
    inherit (config.nixpkgs) localSystem;
  };
in { 
 # ...
}

will always work, and thus is preferred.

[grahamc]

Woot! Thank you!