Nix: Nix 2.2.x fails install on CentOS 7

[joelthompson]

Nix fails to install on a fresh CentOS 7 machine. This worked properly in 2.1.3 but fails in 2.2 and 2.2.1. The error message is similar to #1521

Steps to reproduce:

1. Launch ami-4bf3d731 (the official CentOS 7 AMI) in us-east-1
2. Run sudo yum install -y bzip2
3. Run curl https://nixos.org/nix/install | sh

The result is this:

$ curl https://nixos.org/nix/install | sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2476  100  2476    0     0   4012      0 --:--:-- --:--:-- --:--:--  4012
downloading Nix 2.2.1 binary tarball for x86_64-linux from 'https://nixos.org/releases/nix/nix-2.2.1/nix-2.2.1-x86_64-linux.tar.bz2' to '/tmp/nix-binary-tarball-unpack.la1m6G1nYt'...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 22.5M  100 22.5M    0     0  2724k      0  0:00:08  0:00:08 --:--:-- 2601k
Note: a multi-user installation is possible. See https://nixos.org/nix/manual/#sect-multi-user-installation
performing a single-user installation of Nix...
copying Nix to /nix/store.................................
initialising Nix database...
installing 'nix-2.2.1'
error: cloning builder process: Invalid argument
error: unable to start build process
/tmp/nix-binary-tarball-unpack.la1m6G1nYt/unpack/nix-2.2.1-x86_64-linux/install: unable to install Nix into your default profile

I also get the same issue when trying to install using the multi-user mode (sh <(curl https://nixos.org/nix/install) --daemon) on a fresh machine. I also do a sudo yum update -y and reboot and retry the install and get the same result.

[joelthompson]

Disabling sandboxing by adding sandbox = false to /etc/nix/nix.conf seems to fix this issue.

[DougBurke]

Is there a workaround or fix for users who can not write to /etc/nix/nix.conf? Thanks.

EDITED TO ADD aha, thanks to https://github.com/NixOS/nix/issues/2404#issuecomment-457647463 I did

% mkdir -p ~/.config/nix
% echo "sandbox = false" > ~/config/nix/nix.fonf

and was able to complete the nix-2.2.1 install

[chreekat]

I think this looks like the exact same problem as #2633. Even though that issue was closed, I don't think the problem was fixed—they just found a workaround.

I'm also trying to install Nix on Centos 7, so I'm going to try to understand this unprivileged_userns_clone setting. The workaround in #2633 doesn't work for me because sysctl: cannot stat /proc/sys/kernel/unprivileged_userns_clone: No such file or directory.

[chreekat]

Well, https://lwn.net/Articles/673597/ is certainly interesting

[chreekat]

I'm just going to assume sandboxes are not supported on these hosts. I guess #2633 is the only resolution.

[AmineChikhaoui]

In case it helps anyone, turned out that CentOS/RedHat does some custom kernel setup that disables user namespaces by default. In order to enable it we did:

grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"
reboot
echo 1000 > /proc/sys/user/max_user_namespaces # 1000 is a random number, I guess anything >0 should work

after that the nix install worked fine.

[bhipple]

I've also been running nix 2.2 with sandboxes successfully for some time on RHEL7 (fully rootless, in fact). It requires the tweak @AmineChikhaoui mentioned and won't work out of the box for RHEL/CentOS, but I suspect the next major RHEL/CentOS upgrade will enable user namespaces by default and this problem will eventually go away.

[fedepell]

Just additional 2c: on CentOS 7.4 with the kernel 3.10.0-693.11.6.el7.x86_64 I could not make it work with the suggested changes (_user_namespace.enable=1_ and the _max_user_namespaces_ change).

I had to upgrade the kernel to a newer version (3.10.0-957.21.3.el7.x86_64 in my case, I would suppose anything after 7.5 was released) and enable also by boot parameter _namespace.unpriv_enable=1_ .

This seems to make sense according to what is written for example here.

[KoviRobi]

In case it helps anyone, turned out that CentOS/RedHat does some custom kernel setup that disables user namespaces by default. In order to enable it we did:

grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"
reboot
echo 1000 > /proc/sys/user/max_user_namespaces # 1000 is a random number, I guess anything >0 should work

after that the nix install worked fine.

This helped me install it onto CentOS 7, though the error I got was writing to file: Operation not permitted.

Also, I had to up the max_user_namespaces to get nix-shell to work, on my NixOS system it was
31211 so I changed it to the same

[edolstra]

I think since Nix 2.3, we no longer require user namespaces (d171090530f4a2a79efec2c385bee1a10844c706). Maybe somebody can confirm this on CentOS 7.

[fedepell]

@edolstra : I just tried Nix 2.3.3 on a vanilla Centos 1708 (which is the 7.4 for which I reported previously the problem) and I can confirm now that it works fine with no kernel update needed nor particular kernel parameters.

[edolstra]

Great, thanks! I'll close this then.

[KoviRobi]

I still get this with the latest nix, but the above fix does solve it so perhaps it can be considered solved.

[centos@... ~]$ cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

[centos@... ~]$ cat /etc/centos-release
CentOS Linux release 7.7.1908 (Core)
[centos@... ~]$ uname -srv
Linux 3.10.0-1062.18.1.el7.x86_64 #1 SMP Tue Mar 17 23:49:17 UTC 2020

This is on the Amazon FPGA Developer AMI, I have done yum update and yum upgrade (I think for yum those are just aliases).

Without user_namespace.enable=1 and max_user_namespaces of 31211 https://gist.github.com/628c1cfec2768d9fef2c1b160063a0b2

With user_namespace.enable=1 and max_user_namespaces of 31211 https://gist.github.com/a748ad1c5f6a5bd3d09db10e38644c23

(Shame I can't get the VT100 colour escapes to highlight.)

[fedepell]

I find it quite interesting that you are using a much newer kernel according to your info and seem to behave differently. In my case I tested on a vanilla 7.4 install which has:

3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017

[KoviRobi]

Well, I also get a slightly different error message of "error: writing to file: Operation not permitted" instead of "error: cloning builder process: Invalid argument", so it falls over slightly differently, but the fix of user_namespace seems to fix this too.

It's probably a newer kernel due it being CentOS 7.7, and I have also updated/upgraded it so it's not a vanilla install (it wasn't to start with, because it was whatever Amazon have done to it to make their FPGA Developer AMI)

[knedlsepp]

Did have to manually:

sudo echo "user.max_user_namespaces=15000" >> /etc/sysctl.d/01-user-namespaces.conf
sysctl --system

on a fresh install of CentOS 7.7, otherwise I'd get:

---- sudo execution ------------------------------------------------------------
I am executing:

    $ sudo HOME=/root /nix/store/4vz8sh9ngx34ivi0bw5hlycxdhvy5hvz-nix-2.3.7/bin/nix-env -i /nix/store/4vz8sh9ngx34ivi0bw5hlycxdhvy5hvz-nix-2.3.7

to installing a bootstrapping Nix in to the default Profile

installing 'nix-2.3.7'
error: writing to file: Operation not permitted