Nixpkgs: closure-info broken on nixos-rebuild

[NeQuissimus]

TL; DR

.attrs.sh: No such file or directory during nixos-rebuild -> for now see workaround below: https://github.com/NixOS/nixpkgs/issues/36268#issuecomment-370269274

Issue description

Cannot sudo nixos-rebuild boot --upgrade -I nixpkgs=/home/nequi/dev/upstream_nix with latest master (b84fd70d88f121de536f48c7bc08b3678fe8cabd).

The build logs ends in this:

building path(s) ‘/nix/store/7cnyjhx9vs7sr5x1fa5kbwmpvszcjw60-unit-disable-kernel-module-loading.service’
building path(s) ‘/nix/store/3za4jnyxp1z1vkaq8bdrdqn1x1b9ljpj-unit-display-manager.service’
building path(s) ‘/nix/store/7y2c37lli1r9mlzjgkvl7sbh1kmdg42g-unit-dnscrypt-proxy.service’
building path(s) ‘/nix/store/6papssyd63aihaxzm8i8a14ad9xd3jks-unit-dnsmasq.service’
building path(s) ‘/nix/store/6z1kg4w6smsza15p1m8ssd73agbfqfwn-unit-docker.service’
building path(s) ‘/nix/store/3pg0sh93cdsgkdq9lqsclg34vplbsn8v-unit-firewall.service’
building path(s) ‘/nix/store/chdqahlwa6sqd2skfwdl75wjfw8shha8-unit-getty-.service’
building path(s) ‘/nix/store/xacv6qcq7qhhg0b4zp9n1rh6havkjs0w-unit-mdadm-grow-continue-.service’
building path(s) ‘/nix/store/qky8yvibqmlmvwcwik5m496asrvgws37-closure-info’
/nix/store/aj9rz4k3xvg8q9fzqfnb0q123707zp0n-builder: line 1: .attrs.sh: No such file or directory
building path(s) ‘/nix/store/1aq46pxrg2vdqmid95vnivnsnv326vk5-emacs-all-the-icons-3.2.0’
builder for ‘/nix/store/7ycz6hmaa7rjrssq8pglbd62nqf0b1g3-closure-info.drv’ failed with exit code 1
building path(s) ‘/nix/store/sd8h3r7xp6cnrwvrw9ni4fm4j7p56av6-emacs-auto-complete-20170124.1845’
building path(s) ‘/nix/store/7w7y967gy3r3yz0w1n3lkqw1ziw5ncsa-emacs-flycheck-31’
building path(s) ‘/nix/store/j7a8lhjsn7s4gjbd8sbm6zdky2q9ns3f-emacs-projectile-20180128.655’
cannot build derivation ‘/nix/store/611p5z4v1p905mkhy52ympinz1cdmrcq-initrd.drv’: 1 dependencies couldn't be built
building path(s) ‘/nix/store/qpz6mx62s58n1fmpzrv8dsrzlrq7jyld-nixos-version’
building path(s) ‘/nix/store/83nrcqkbilc0l4qgbxq6nvk57bw9qxvg-unit-mdadm-shutdown.service’
building path(s) ‘/nix/store/2kv367hvdw4b0kyv4aspa37r7s6ml82x-unit-mdmon-.service’
building path(s) ‘/nix/store/m3jsvbx8x9b8z1rswnxmw29gdjfcqbcf-unit-network-local-commands.service’
building path(s) ‘/nix/store/d82pf5pzcwl5i24jp96wp67siv108r9y-unit-network-setup.service’
cannot build derivation ‘/nix/store/7xfncw8bsqhgr9sa46m2iygihbir93w0-nixos-system-nixus-18.03.git.b84fd70.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/7xfncw8bsqhgr9sa46m2iygihbir93w0-nixos-system-nixus-18.03.git.b84fd70.drv’ failed

I tracked this particular error to 0d0021588015105696eb4981da7a835ec6b2e45b. So I reverted it.
But then I see this error:

building path(s) ‘/nix/store/ghpzxbyfm5qin6wpajds3sqjcs68w04q-etc-os-release’
building path(s) ‘/nix/store/52xjqjz80r05i3c3fcdh9djss098fxjc-issue’
building path(s) ‘/nix/store/6z1kg4w6smsza15p1m8ssd73agbfqfwn-unit-docker.service’
building path(s) ‘/nix/store/3pg0sh93cdsgkdq9lqsclg34vplbsn8v-unit-firewall.service’
building path(s) ‘/nix/store/chdqahlwa6sqd2skfwdl75wjfw8shha8-unit-getty-.service’
building path(s) ‘/nix/store/xacv6qcq7qhhg0b4zp9n1rh6havkjs0w-unit-mdadm-grow-continue-.service’
building path(s) ‘/nix/store/83nrcqkbilc0l4qgbxq6nvk57bw9qxvg-unit-mdadm-shutdown.service’
building path(s) ‘/nix/store/2kv367hvdw4b0kyv4aspa37r7s6ml82x-unit-mdmon-.service’
building path(s) ‘/nix/store/z3cv7ivid717h0kddwl528wafdzrhy92-closure-info’
building path(s) ‘/nix/store/1aq46pxrg2vdqmid95vnivnsnv326vk5-emacs-all-the-icons-3.2.0’
/nix/store/46gz3s6zhavr1s566h9yl7gqw7fb94q2-builder: line 3: /nix/store/z3cv7ivid717h0kddwl528wafdzrhy92-closure-info: syntax error: operand expected (error token is "/nix/store/z3cv7ivid717h0kddwl528wafdzrhy92-closure-info")
/nix/store/46gz3s6zhavr1s566h9yl7gqw7fb94q2-builder: line 5: mkdir: command not found
building path(s) ‘/nix/store/sd8h3r7xp6cnrwvrw9ni4fm4j7p56av6-emacs-auto-complete-20170124.1845’
building path(s) ‘/nix/store/7w7y967gy3r3yz0w1n3lkqw1ziw5ncsa-emacs-flycheck-31’
builder for ‘/nix/store/xabyii3ka8bvk1pp0z4gwkgvfry0f09i-closure-info.drv’ failed with exit code 127
building path(s) ‘/nix/store/j7a8lhjsn7s4gjbd8sbm6zdky2q9ns3f-emacs-projectile-20180128.655’
building path(s) ‘/nix/store/bb425wwxq85ssqjghq41j8rzdpbd2xi3-emacs-undo-tree-20170706.246’
building path(s) ‘/nix/store/097gbfyml28h1kp17s6ydk8i8m75rx2j-firefox-unwrapped-58.0.2’
cannot build derivation ‘/nix/store/ifpy9k2vf6hzm8ajqpdvi4yn6alavz6d-initrd.drv’: 1 dependencies couldn't be built
building path(s) ‘/nix/store/5fw96brprz4as3b0ky1nvghq93wbaaf0-nixos-version’
building path(s) ‘/nix/store/m3jsvbx8x9b8z1rswnxmw29gdjfcqbcf-unit-network-local-commands.service’
building path(s) ‘/nix/store/d82pf5pzcwl5i24jp96wp67siv108r9y-unit-network-setup.service’
building path(s) ‘/nix/store/pzpaj3x37pmba1afg0nr2mas5hdngwmy-unit-nix-daemon.service’
building path(s) ‘/nix/store/p7ab83jq1wg3nkr972n8biyj1bjzpk7h-unit-nix-optimise.service’
cannot build derivation ‘/nix/store/vrg5r2mdpm1qzsm9xl5qirr1sj74ab5h-nixos-system-nixus-18.03.git.fdce3d6.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/vrg5r2mdpm1qzsm9xl5qirr1sj74ab5h-nixos-system-nixus-18.03.git.fdce3d6.drv’ failed

I am unable to determine what is going on with this.

Technical details

• system: "x86_64-linux"
• host os: Linux 4.15.7-hardened, NixOS, 18.03.git.a145640 (Impala)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 1.11.16
• channels(root): "nixos-18.03pre130558.7270f2139ae"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs

[vcunat]

That's changed with PR #34636. I can't really look into it properly now, but tomorrow morning I shall.

[MP2E]

I see these same errors, not sure how to proceed

[fpletz]

The reason for this failure is probably that nix-daemon is still running with Nix 1.x. .attrs.sh was added in Nix 2.0.

[vcunat]

That seems hard to work around. I believe builtins.langVersion will return version of the evaluator and not the daemon (it's a constant defined in libexpr), so the old code won't work either.

[vcunat]

~I expect it's possible to work around this by --no-build-nix when doing the switch~, but we would better have compatible builder code anyway.

[clefru]

Using --no-build-nix results in the following error:

[clemens@leto:/tmp]$ nixos-rebuild build --no-build-nix --show-trace
building the system configuration...
error: while evaluating the attribute ‘buildCommand’ of the derivation ‘nixos-system-leto-18.03.git.2069a2a’ at /nix/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:148:11:
while evaluating ‘optionalString’ at /nix/nixpkgs/lib/strings.nix:138:26, called from /nix/nixpkgs/nixos/modules/system/activation/top-level.nix:40:9:
while evaluating the attribute ‘closure’ of the derivation ‘initrd’ at /nix/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:148:11:
while evaluating anonymous function at /nix/nixpkgs/pkgs/build-support/closure-info.nix:9:1, called from /nix/nixpkgs/pkgs/build-support/kernel/make-initrd.nix:33:13:
assertion failed at /nix/nixpkgs/pkgs/build-support/closure-info.nix:11:1

[vcunat]

Yes, of course, I'm sorry :-)

[vcunat]

Well, always using the nix-1.11 code should work 17ee557f9b3ad, but we would lose some things by that (changes by @edolstra and @shlevy). Any better idea? Perhaps the builder code could detect the daemon version and decide based on that, as a compromise.

[shlevy]

@vcunat The builder could switch based on the existence of .attrs.sh

[shlevy]

Ugh no that's probably not enough, because we need to set exportReferencesGraph differently depending on the daemon version...

[avnik]

Can we revert to exportReferenceGraph until 18.04 would be released, then switch to new scheme? (so update path from random point on earlier release would be "to 18.04", then to next point on master)

[edolstra]

Reverting to the old exportReferenceGraph is not really an option because it breaks ISO generation. In fact the whole point of the new exportReferencesGraph was to get correct hashes in the Nix database of generated disk images. IIRC, Nix 2.0 is less tolerant of having incorrect hashes in the database, but I'm not sure.

Also, reverting doesn't solve the issue, just postpones it. It should be possible to upgrade from any previous NixOS version, not just the previous one. Since we do want to start using new features like structured attributes at some point, we need a way for nixos-rebuild to handle that. Currently nixos-rebuild builds (or downloads) a new Nix but uses it for evaluation only. We could also use it for building, but the reason we didn't do that is that we don't want nixos-rebuild to cause a non-revertible change to the Nix database schema. However, I don't really see another way... We could have nixos-rebuild make a dump of the database before doing a schema upgrade, or something like that.

[shlevy]

:+1: for db backup, especially since 1.11 is backwards compatible with 2.0's db...

[shlevy]

Though I can certainly see an argument for waiting a release cycle to require a 2.x daemon anyway, if there is truly a workaround (it seems plain revert isn't one)

[shlevy]

Maybe for 18.03 we can only use the new version for the ISOs (with an error message at build time if your daemon is too old) and the on master we can fix nixos-install to use a new daemon and backup the store db?

[edolstra]

Like I said, we need to address this problem properly at some point anyway so we might as well do it now.

Maybe nixos-rebuild should do something like this:

• If the running daemon version is less than 1.12preXXX (whatever the lowest required version is), then

  • If the schema version hasn't changed between the running and the new version, then use the new version to build, bypassing the daemon. This is unproblematic because there is no schema change.

  • If the schema version has changed, then print a fatal error unless the user has specified --allow-schema-upgrade. Otherwise, make a dump of the database, print some instructions on how to revert the database if necessary, and then proceed as above (i.e. build with the new version).

• If the running daemon version is recent enough, do what we currently do (i.e. evaluate but don't build with the new version).

[vcunat]

Note: the unstable channel has already advanced past the nix upgrade, so I expect many people may be hitting this now.

[shlevy]

@edolstra The only reason to wait is to not have the release blocked on getting the upgrade logic right and make it more likely that people will have a new enough daemon anyway. Either way is fine with me.

[srid]

I hit this after updating my nix channels. Now it is not even working after I did a rollback (sudo nix-channel --rollback). Could someone please provide a workaround?

/nix/store/aj9rz4k3xvg8q9fzqfnb0q123707zp0n-builder: line 1: .attrs.sh: No such file or directory

system: "x86_64-linux", 
multi-user?: yes, 
version: nix-env (Nix) 1.11.16, 
channels(srid): "", 
channels(root): "nixos-18.03pre130719.a66ce38acea, nixpkgs-unstable-18.03pre130569.7a04c2ca296", nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs

[cumber]

@srid The channel version nixos-18.03pre130719.a66ce38acea is still pointing at the unstable channel release that included nix 2.0.

I think sudo nix-channel --rollback will only undo the immediately previous update. Could you have done nixos-rebuild --switch --upgrade multiple times trying to fix it? I think that means you would have updated the channel multiple times (even though it would have fetched the same current channel version), so the immediately previous channel release is still the new one.

If that's the case, you can nix-channel --rollback <generation> to roll back to a specific channel generation though. You should be able to look in the files matching /nix/var/nix/profiles/per-user/root/channels-*-link/nixos/svn-revision to find out which is the latest generation you have that doesn't have 130719.a66ce38acea, and rollback to that one.

[cumber]

For the record, I've just been able to work around the nixos-rebuild blocker by following this procedure:

1. sudo sudo nix-channel --rollback to get my nixos channel back to something I can nixos-rebuild --switch to
2. add nix.package = pkgs.nixUnstable in my configuration.nix
3. sudo nixos-rebuild --switch, so the system nix is now a pre-release version of nix 2.0
4. remove the nix.package = pkgs.nixUnstable from configuration.nix again
5. sudo nixos-rebuild --switch --upgrade to update channel and install the stable released nix 2.0

[NeQuissimus]

I guess my last channel pull contains a bad nixUnstable if this works for you:

[nequi] ~ λ sudo nixos-rebuild switch --upgrade
unpacking channels...
error: cloning builder process: No space left on device
error: unable to start build process
error: program '/nix/store/7bfc02z2ayj9h73qwyrbnnzy14xgrazn-nix-2.0pre5968_a6c0b773/bin/nix-env' failed with exit code 1

[shlevy]

@NeQuissimus error: cloning builder process: No space left on device

[NeQuissimus]

[nequi] ~/dev/upstream_nix master λ df -H                                                                                                                                                  1 ↵
Filesystem           Size  Used Avail Use% Mounted on
devtmpfs             414M     0  414M   0% /dev
tmpfs                4.2G   57M  4.1G   2% /dev/shm
tmpfs                2.1G  4.7M  2.1G   1% /run
tmpfs                4.2G  328k  4.2G   1% /run/wrappers
nixoszfs/root/nixos  236G  4.3G  232G   2% /
tmpfs                4.2G     0  4.2G   0% /sys/fs/cgroup
nixoszfs/home        234G  2.3G  232G   1% /home
/dev/sda1            315M   67M  248M  22% /boot
tmpfs                827M   13k  827M   1% /run/user/1000

[NeQuissimus]

Oh, interesting... It's my system hardening Nix 2.0 does not like. The same setup works fine with 1.x. Let me investigate...

Edit: Looks like it's the user namespaces. (As I had suspected...)
This worked previously but no longer with 2.0:

boot.kernel.sysctl = {
  "user.max_user_namespaces" = 0;
}

[shlevy]

@NeQuissimus Nix should only need a user namespace if the sandbox is enabled.

[shlevy]

But I think maybe the nixos installer uses them now...

[NeQuissimus]

Correct but I would not want sandboxing to be disabled :)
I think I am OK for the moment, I can enable user namespaces.
I derailed this issue enough already :D

[vcunat]

Maybe nixos-rebuild should do something like this: [...]

I must say I don't much like the idea that one wouldn't be able to build 18.03 systems with nix-1.11 (default in 17.09), as that affects also machines _other_ than the one you're about to switch. We could add preferLocalBuild to your plan, but even so the approach still seems fragile. We would better ensure reliable updates without such hacks, at least from recent 17.09 to first 18.03.

Maybe it would be okay if this "incompatible" build failed with a message telling what to do? I guess we don't know any way around the ISO problem with nix-1.11, right?

[vcunat]

Well, AFAIK for NixOS updates people only need make-initrd, and that only uses $closure/store-paths. I don't know the roots of that "ISO generation problem" well; @edolstra: does it seem OK to use the old way for initrd (for now by default) and the new way elsewhere?

[coretemp]

Not being able to build 18.03 on 17.09 is an excellent way of destroying the project.

[shlevy]

I'd like to reiterate my suggestion above:

1. Revert the change to closureInfo except for the ISO generation (and VM tests?). On those builds, fail early at build-time with a message to upgrade.
2. After the branch-off, develop and test a variant of nixos-rebuild that figures out if a new daemon is needed and bypasses the daemon if so, or errors out if building without root (e.g. nixos-rebuild build)
3. When we decide enough time has passed (maybe one release cycle? Maybe a few?) allow nixpkgs master to take a hard dependency on __structuredAttrs.

@vcunat do you see any problem with this approach?

[vcunat]

That's basically what I meant, except that I intended to only "revert" it for make-initrd. Still, I'd like someone knowledgeable confirm that the ISO problem won't cause trouble with initrd.

EDIT: I'm not sure about the (2.) step, but I'm imminently concerned about the branch-off.

[vcunat]

Ah, well, Eelco just did my idea of (1.) :tada:

[vcunat]

@chisui: you need to check the commit that closed this.

[daveman1010221]

Hello, all. This morning I decided to try out NixOS for the first time. Before getting too far along in the install manual (https://nixos.org/nixos/manual/), I ran into the bug described in this thread. Since I don't understand enough about NixOS at this point to troubleshoot the problem, I'd like to know how a new user is supposed to get a working NixOS, without a ton of hassles, by simply following your guide manual, when the stable release is unstable. Thanks!

[vcunat]

@daveman1010221: do you know the exact version? Maybe the information won't be really useful, but if it's cheap for you to get it now... e.g. it's in the *.iso filename.

[shlevy]

I'm confused... This is specifically a problem with upgrading NixOS, unless we have some 18.03 prerelease ISO with nix 1.11 installed this doesn't seem likely to be the same issue...

[daveman1010221]

The problem I was seeing was a result of a bad interaction between Virtualbox's NVME controller and the kernel. Don't use the NVME controller with Virtualbox, just use a SATA controller interface.

--David

On Apr 14, 2018, at 08:39, Shea Levy [e-mail] wrote:

I'm confused... This is specifically a problem with upgrading NixOS, unless we have some 18.03 prerelease ISO with nix 1.11 installed this doesn't seem likely to be the same issue...

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[domenkozar]

This issue is not fixed.

a) NixOS 18.03 is the first release with Nix 2.0

b) There are nixos utilities using closureInfo which depends on Nix 2.0 features:

$ git grep closureInfo
nixos/lib/make-disk-image.nix:  closureInfo = pkgs.closureInfo { rootPaths = [ config.system.build.toplevel channelSources ]; };
nixos/lib/make-disk-image.nix:    nix-store --load-db < ${closureInfo}/registration
nixos/lib/make-ext4-fs.nix:  sdClosureInfo = pkgs.closureInfo { rootPaths = storePaths; };
nixos/lib/make-iso9660-image.nix:{ stdenv, perl, closureInfo, xorriso, syslinux
nixos/lib/make-iso9660-image.nix:  closureInfo = closureInfo { rootPaths = map (x: x.object) storeContents; };
nixos/lib/make-iso9660-image.sh:for i in $(< $closureInfo/store-paths); do
nixos/lib/make-iso9660-image.sh:    cp $closureInfo/registration nix-path-registration
nixos/lib/make-squashfs.nix:{ stdenv, squashfsTools, closureInfo
nixos/lib/make-squashfs.nix:      closureInfo=${closureInfo { rootPaths = storeContents; }}
nixos/lib/make-squashfs.nix:      cp $closureInfo/registration nix-path-registration
nixos/lib/make-squashfs.nix:      mksquashfs nix-path-registration $(cat $closureInfo/store-paths) $out \
nixos/modules/installer/netboot/netboot.nix:      inherit (pkgs) stdenv squashfsTools closureInfo;
nixos/modules/profiles/installation-device.nix:        jq # for closureInfo
nixos/modules/virtualisation/qemu-vm.nix:  regInfo = pkgs.closureInfo { rootPaths = config.virtualisation.pathsInNixDB; };
pkgs/build-support/kernel/make-initrd.nix:  # Note: we don't use closureInfo yet, as that won't build with nix-1.x.
pkgs/build-support/kernel/paths-from-graph.pl:# NOTE: this script is deprecated. Use closureInfo instead.
pkgs/top-level/all-packages.nix:  closureInfo = callPackage ../build-support/closure-info.nix { };

c) error message is .attrs.sh: No such file or directory

[domenkozar]

I wonder if there is a different way than builtins.langVersion to determine it won't work and at least provide a good error what to do.

[Synthetica9]

Why would you need a different way?

[dezgeg]

builtins.langVersion doesn't give the version of the nix-daemon running, which is what matters here.

[shlevy]

I think the best solution is:

1. Have the builds fail at build time with a nice message, either by the script having some logic coded in or having the needed features expressed in the drv file somehow (the latter would require a new drv format which would break not-nicely on old stores though).
2. Have nixos-rebuild be aware if some feature is needed that the current store doesn't have, and either transparently use the new nix if the new store is backwards compatible (e.g late nix 1.11 versions can read nix 2.0 stores) or present the user with the options if it's not backwards compatible.

1) is necessary to cover the nix-build -A sytem case, whereas 2) should give an as-smooth-as-possible path for most users.

[arianvp]

Given that we are a year further now, and people should be running 18.09 or 19.03 is the use of closureInfo still an issue?

[lheckemann]

Closing since nix 2.x has been used for several releases now. Objections welcome :)