Nixpkgs: Unable to bind mount on certain hard drives

[matthewbauer]

Describe the bug
This has been reported in two discourse threads:

• https://discourse.nixos.org/t/nixos-install-unable-to-bind-mount/3786
• https://discourse.nixos.org/t/unable-to-nixos-install-from-live-cd/2533/6

The error appears when trying to install NixOS on a hard drive. It appears some hardware issue might be going on, along with maybe a Nix issue:

error: while setting up the build environment: unable to bind mount '/mnt/nix/store/85d8y2vybq07al01j7zv518yf3fa9mwq-append-initrd-secrets.drv.chroot': Permission denied

[alexeymuranov]

I am trying to install NixOS 19.03 on a USB flash drive in BIOS mode (to boot from it some machine with Windows), and I am getting

error: while setting up the build environment: unable to bind mount '/home/alexey/.mnt/nix/store/0vq9ja754cj161nwq8b8945p5b94fhnj-NetworkManager.conf.drv.chroot': Permission denied

If i disable networkmanager in configuration.nix, i get the same error for append-initrd-secrets.

Update. I have repartitionned and reformatted the flash drive, and did some other tweaks, and now the error is gone and the installation worked.

Update 2. Now I have the same problem with a different flash drive in the same situation. I still do not know what fixed it the other time, again not sure what to try.

Update 3. I tried running nixos-install from NixOS 19.09 instead of 19.03, and the installation worked.

Update 4. Again the same problem, this time under NixOS 19.09.

[EliotBerriot]

facing the same issue today. Trying to install NixOS in a separate partition on my laptop which already runs PureOS on top of Luks.

Trying a live CD install and installing from whithing PureOS itself raises:

error: while setting up the build environment: unable to bind mount '/mnt/nix/store/7gr4d45v9mhllcpf0grh73nf75br639c-Xresources-Xft.drv.chroot': Permission denied

This is the content of my NixOS configuration files:

[1] $ cat /mnt/etc/nixos/hardware-configuration.nix                                                                                                         
# Do not modify this file!  It was generated by ‘nixos-generate-config’
# and may be overwritten by future invocations.  Please make changes
# to /etc/nixos/configuration.nix instead.
{ config, lib, pkgs, ... }:

{
  imports =
    [ <nixpkgs/nixos/modules/installer/scan/not-detected.nix>
    ];

  boot.initrd.availableKernelModules = [ "xhci_pci" "ahci" "usbhid" "sd_mod" ];
  boot.initrd.kernelModules = [ ];
  boot.kernelModules = [ "kvm-intel" ];
  boot.extraModulePackages = [ ];

  fileSystems."/" =
    { device = "/dev/disk/by-uuid/43f30e2b-84ea-4afa-9ac8-3dd21e41427f";
      fsType = "ext4";
    };

  boot.initrd.luks.devices."diskcrypt".device = "/dev/disk/by-uuid/61fddfbe-e71d-4763-b3c6-171aa8afb968";

  swapDevices =
    [ { device = "/dev/disk/by-uuid/49928ad6-8344-4039-b8c0-2676fa394ddb"; }
    ];

  nix.maxJobs = lib.mkDefault 4;
  powerManagement.cpuFreqGovernor = lib.mkDefault "powersave";
}

$ cat /mnt/etc/nixos/configuration.nix         
# Edit this configuration file to define what should be installed on
# your system.  Help is available in the configuration.nix(5) man page
# and in the NixOS manual (accessible by running ‘nixos-help’).

{ config, pkgs, ... }:

{
  imports =
    [ # Include the results of the hardware scan.
      ./hardware-configuration.nix
    ];

  # Use the GRUB 2 boot loader.
  boot.loader.grub.enable = true;
  boot.loader.grub.version = 2;
  # boot.loader.grub.efiSupport = true;
  # boot.loader.grub.efiInstallAsRemovable = true;
  # boot.loader.efi.efiSysMountPoint = "/boot/efi";
  # Define on which hard drive you want to install Grub.
  boot.loader.grub.device = "/dev/sda"; # or "nodev" for efi only
  boot.loader.grub.extraEntries = ''
    menuentry "PureOS - Connie" {
      search --set=pureos --fs-uuid 22b0f2b8-e195-4dcc-9286-ca6ab3c88c82
      configfile "($pureos)/boot/grub/grub.cfg"
    }
  '';
  networking.hostName = "virginie"; # Define your hostname.
  networking.wireless.enable = true;  # Enables wireless support via wpa_supplicant.

  # The global useDHCP flag is deprecated, therefore explicitly set to false here.
  # Per-interface useDHCP will be mandatory in the future, so this generated config
  # replicates the default behaviour.
  networking.useDHCP = false;
  networking.interfaces.docker0.useDHCP = true;
  networking.interfaces.tun0.useDHCP = true;
  networking.interfaces.wlp1s0.useDHCP = true;

  # Configure network proxy if necessary
  # networking.proxy.default = "http://user:password@proxy:port/";
  # networking.proxy.noProxy = "127.0.0.1,localhost,internal.domain";

  # Select internationalisation properties.
  # i18n = {
  #   consoleFont = "Lat2-Terminus16";
  #   consoleKeyMap = "us";
  #   defaultLocale = "en_US.UTF-8";
  # };

  # Set your time zone.
  time.timeZone = "Europe/Paris";

  # List packages installed in system profile. To search, run:
  # $ nix search wget
  # environment.systemPackages = with pkgs; [
  #   wget vim
  # ];

  # Some programs need SUID wrappers, can be configured further or are
  # started in user sessions.
  # programs.mtr.enable = true;
  # programs.gnupg.agent = { enable = true; enableSSHSupport = true; };

  # List services that you want to enable:

  # Enable the OpenSSH daemon.
  # services.openssh.enable = true;

  # Open ports in the firewall.
  # networking.firewall.allowedTCPPorts = [ ... ];
  # networking.firewall.allowedUDPPorts = [ ... ];
  # Or disable the firewall altogether.
  # networking.firewall.enable = false;

  # Enable CUPS to print documents.
  services.printing.enable = true;

  # Enable sound.
  sound.enable = true;
  hardware.pulseaudio.enable = true;

  # Enable the X11 windowing system.
  services.xserver.enable = true;
  # services.xserver.layout = "us";
  # services.xserver.xkbOptions = "eurosign:e";

  # Enable touchpad support.
  services.xserver.libinput.enable = true;

  # Enable the KDE Desktop Environment.
  services.xserver.displayManager.sddm.enable = true;
  services.xserver.desktopManager.plasma5.enable = true;

  # Define a user account. Don't forget to set a password with ‘passwd’.
  users.users.eliotberriot = {
    isNormalUser = true;
    extraGroups = [ "wheel" ]; # Enable ‘sudo’ for the user.
  };

  # This value determines the NixOS release with which your system is to be
  # compatible, in order to avoid breaking some software such as database
  # servers. You should change this only after NixOS release notes say you
  # should.
  system.stateVersion = "19.09"; # Did you read the comment?

}

[EliotBerriot]

I've tried to do the install on a non-encrypted partition, with the same result (so at least we know it's not related to LUKS/disk encryption).

Edit: Stumbled upon https://discourse.nixos.org/t/nixos-install-unable-to-bind-mount/3786/6 and since the OP is using a SSD, just like myself, it may be hardware-related? My disk is a Samsung SSD 850 EVO 250GB.

[alyssais]

A couple of things you could try:

• run nixos-install under strace -f and post the output. It will likely be quite big but should compress well. (One of the discourse posts has strace output but not with -f so we can’t see what children are doing.)
• Try installing NixOS unstable and see if that changes anything.

[EliotBerriot]

Thanks @alyssais, I'll give it a go, probably tomorrow!

[EliotBerriot]

So, I've tried using the unstable channel, with the same result.

@alyssais this is the output of my latest attempt, run with:

sudo PATH="$PATH" NIX_PATH="$NIX_PATH" strace -f -o /tmp/strace_output.txt `which nixos-install` --root /mnt

strace_output.txt

EDIT: since I reproduce the issue reliably, I'm open to experimenting/posting more debug logs, feel free to ask anything :D

[TinHead]

Hi,
Getting the same on a Surface Laptop 3 running Manjaro. Trying to install NixOS on another partition.

SSD: 01:00.0 Non-Volatile memory controller: SK hynix BC501 NVMe Solid State Drive 512GB

strace_output.txt

Cheers!

[matthewbauer]

Could it be some mount flag is messing things up? Maybe try:

mount | grep /mnt

I wonder if we are hitting some inodes limit?

[TinHead]

Nothing special there.

/dev/nvme0n1p7 on /mnt type ext4 (rw,relatime)

Inodes seem to be fine:
sudo df -ih / Filesystem Inodes IUsed IFree IUse% Mounted on /dev/nvme0n1p5 3,2M 550K 2,7M 17% / LixSurface% sudo df -ih /mnt Filesystem Inodes IUsed IFree IUse% Mounted on /dev/nvme0n1p7 13M 53K 13M 1% /mnt
Let me know if you need more info.

[alexeymuranov]

I am consistently getting this error with NetworkManager.conf.drv.chroot, this time trying NixOS 20.03.

So this is not purely hardware related.

[jtrakk]

I also get it with /mnt/dev/nix/store/gispa2laayl2bh2v5dsk3pmcsp943mgc-NetworkManager.conf.drv.chroot on NixOS 20.09pre226148.0f5ce2fac0c.
The target partition is mounted LUKS, and the error occurs under both ext4 and btrfs.

Drive is "Samsung SSD 860 EVO" (SATA), similar to the "Samsung SSD 850 EVO" in @EliotBarriot's case above.

[energizah]

I can reproduce this issue with a minimal configuration.nix. I'm including some debugging info in this repo.

[energizah]

The parent directory of the installation target had insufficient permissions. chmod o+rx on that directory resolved the error.

Thanks to @euank for solving this.

[matthewbauer]

The parent directory of the installation target had insufficient permissions. chmod o+rx on that directory resolved the error.

Thanks to @euank for solving this.

Is there a change we can make to nixos-install that can make this work out of the box? I’m a little confused still that this only happens in certain cases.

[euank]

Yeah, I can explain the cases that this happens in, and also what options we have to improve things here.

First, the easiest way to reproduce this is to just run chmod 750 /mnt and try to do an install that will require building a derivation not in either your local cache or the remote cache.

That last bit is why it only sometimes happens. If nix downloads a file from a cache into a /mnt/root/nix/store with bad /mnt permissions, all is fine there. Even though /mnt forbids 'other' access, nix is running as root and can ignore those permissions entirely (root has CAP_FOWNER more specifically).

However, if nix has to build a derivation, it has to spin up a builder... And builders run in a user namespace with uids and gids mapped and significantly less permissions. That builder is what's spitting out that error.

We've specifically seen this with the NetworkManager.conf derivation, because, well, that's not really something that can be copied from the cache (requires mac address, etc) and so is getting instantiated each time.

Hopefully the above makes sense. Basically, you don't need a reasonable tree of permissions for copying stuff into the nix store, but you do for nix to spawn a builder that then operates on that store.

So, what are our options to fix this?

1. nix could do a better job of recognizing EACCES and try to present a useful error message, possibly walking up the filesystem tree itself to do so.
   Unfortunately, that seems kinda tricky to do correctly. The kernel gives us back EACCES, and it's not like we can actually derive why reliably (i.e. we don't know if it's selinux, or filesystem permissions or what).
2. nix-install could add additional validation.
   I implemented this over here: #90431
   Basically, it can walk the tree itself and verify permissions seem right.

[alexeymuranov]

Hm, in my case the mount point was ~/.mnt with permissions drwxr-xr-x. The parent directories, including /home, have the same permissions (with different ownership).

How to explain that re-partitionning a flash drive seemed to have helped once? (Or maybe not...)

[euank]

Hm, in my case the mount point was ~/.mnt with permissions drwxr-xr-x. The parent directories, including /home, have the same permissions (with different ownership).

That's unusual; normally /home/$USER will have permissions of 700, which would be consistent with my explanation.

You could try, for example, sudo -u nixbld1 /bin/sh and then ls /home/$user/.mnt/nix/store to see whether it's as straight-forward a permissions problem as I describe above, where the builder just doesn't have access to that filesystem path.

How to explain that re-partitionning a flash drive seemed to have helped once? (Or maybe not...)

It's very possible there are multiple issues here as well with the same error.
We ended up isolating this one with perf ftrace -G do_mount -- <command> to get a little more information where in the kernel it was getting EACCES, so if you have a different cause than permissions, the output of that would be helpful for understanding it better!

[TinHead]

chmod o+rx fixed it for me too!

[alexeymuranov]

@euank, sorry, i do not know how i was looking. In fact, my ~/.mnt directory had "wrong" permissions. I had ~/.mnt and ~/.mnt2, and ~/.mnt2's permissions were "right", but ~/.mnt's permissions were "wrong". When using drwxr-xr-x permissions, installation goes through fine. Sorry for the fuss.

So, maybe, when i thought that re-partitionning the flash drive fixed the issue, it just happened that i mounted it under ~/.mnt2 instead of under ~/.mnt.

[matthewbauer]

Closing with https://github.com/NixOS/nixpkgs/pull/90431 merged. Now nixos-install will give you a warning when you accidentally do this.