nix-channel --update fails inside nixos-enter

[lukash]

I've installed NixOS into a chroot using nixos-install. Then I've entered the installation chroot with sudo `which nixos-enter` --root .... I've got one channel:

[root@nixos:/]# nix-channel --list  
nixos https://nixos.org/channels/nixos-19.09

When I try to update it, I get:

[root@nixos:/]# nix-channel --update
unpacking channels...
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
error: writing to file: Operation not permitted
error: program '/nix/store/l4zaz0z1x9w549xj0m0y9a2p8fik2w8a-nix-2.3/bin/nix-env' failed with exit code 1

Please fix the error message to contain more information (at least the file name on which the write failed).

Running the above command under strace, here's the relevant part, more or less:

[pid   240] openat(AT_FDCWD, "/proc/247/uid_map", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 16
[pid   240] write(16, "1000 30001 1", 12) = -1 EPERM (Operation not permitted)
[pid   240] close(16)                   = 0
[pid   240] kill(-247, SIGKILL <unfinished ...>

(Let me know if you need more)
I've no idea what's going on there. Not sure it matters, but I didn't have user 30001 on the host system (only 30000 as the manual said), but creating it didn't change anything.

[matthewbauer]

So it looks like it's failing to set up the sandbox... There are two ways I can think of to fix this:

• Disable sandboxing when /proc/... is not writable.
• Try to find a way to make nixos-enter allow recursive sandboxing.

[matthewbauer]

Opened https://github.com/NixOS/nix/pull/3146

[zokrezyl]

I am fighting something similar:

1. according to the documentation (see link below), you can write to the uid_map only once, likely someone wrote to it before you reached quoted line from the strace output (look in your strace for earlier lines)
2. I am not sure if it is a bug in the kernel, but apparently once you did pivot_root, you cannot create a new namespace. (I am no very familiar with the nix code, but I see that both chroot and pivot_root are used). I could imagine a situation where after a potentially ignored failing unshare, the code tries to write to the uid_map.

Was about to create a new Issue, but maybe this is a good opportunity to discuss: wouldn't it make sense to give more control over the heuristics in the code with some extra config flags?

I am about to write a tool that install nix inside a user namespace. Fro this I am using the unshare tool. The unshare tool maps the id to 0 inside the new user namespace. That would be fine, but nix tools makes a bunch of assumptions while running under uid 0 (different behaviour than under normal user). However in this usecase that is exagerated, and practically it is not possible to install nix in such a context without extra hacks. After I explicitly disabled sandbox and "build-user-group = "" in /etc/nix/nix.conf it started to work, but nix-channel --update nixpkgs fails as tar tries to do chmod.

Shall i create an extra issue for the tar xf problem?

https://manpages.debian.org/stretch/manpages/user_namespaces.7.en.html#The_/proc/%5Bpid%5D/setgroups_file

[lukash]

1. according to the documentation (see link below), you can write to the uid_map only once, likely someone wrote to it before you reached quoted line from the strace output (look in your strace for earlier lines)

No, the location was written to only once, sorry about not mentioning that. There are other cases described in the manpage that fail with EPERM... Here's a longer strace (different run than originally posted): https://pastebin.com/scTKJHkj

@matthewbauer FWIW if nested sandboxing is what is happening here, I'd like to say that there's a possibility the failure is on my side - I have a custom kernel (and there are a lot of switches added semi-recently for that) and I faintly remember having an issue with nested containers not working - I didn't investigate and there were a lot of other things that could have been wrong, but anyway.

[matthewbauer]

nixos-enter should set uid_map as well. So it looks like a real limitation of Linux that we are seeing. I have definitely used nested containers before but I think that they used pivot_root which has some different effects from chroot.

Even if this is user error, I think this is a case that’s worth handling in build.cc. NixOS currently sets sandbox-fallback to false anyway, so we’ll need to handle it in nixos-enter too.

[gilescope]

I get this even with nixos being my host OS. Is there some mode in which nixos works?

[cryptix]

I'm also running into this (channel update and rebuilding). I mangled my boot loader and am now trying to rebuild my nixos system from a chroot using nixos-enter as describe the chroot wiki article.

Is there a known workaround to this problem? I tried the _manual_ steps (bind mounting /dev, /proc, ...) but that also doesn't fix the issue.

[baracoder]

Also hit by this problem when booting remote server with a rescue image.
Sandboxing is a good hint though: after setting nix.useSandbox = false;, and rebuilding target system using nixos-install --no-root-passwd --root /mnt .. I was able to run nix commands inside nixos-enter

[BigValen]

Ran into this, from the nixos-installer, on a blank /mnt with a blank /mnt/boot. I had some config that setup user namespaces, but even after ripping that out, I still couldn't get around it.

[jamesbroadhead]

@edolstra I've just run into this chrooting into an existing nixos install with a fresh nixos liveusb. Should this be fixed in the current liveusb?

[edolstra]

@jamesbroadhead No, it's only fixed on nix master.