Nixpkgs: FUSE fs types don't work with systemd mount units

[ToxicFrog]

Issue description

I want to use systemd.mounts and systemd.automounts to create an on-demand-mounted SSHFS mountpoint. Unfortunately, when I do, mount.fuse can't find the sshfs binary even when pkgs.sshfsFuse is installed via environment.systemPackages; strace shows mount.fuse looking in /no-such-path/sshfs for it.

It looks like the root cause is that mount doesn't pass $PATH through to its children for security reasons, and relies on /bin/sh having a sensible default path compiled in. Unfortunately, /bin/sh on Nix has its default path deliberately removed and replaced with /no-such-path. (Why not /var/setuid-wrappers:/run-current-system/sw/bin or similar?)

Steps to reproduce

Write a mountpoint:

environment.systemPackages = [pkgs.sshfsFuse];
systemd.mounts = [{
  what = "user@host:path";
  where = "/wherever";
  type = "fuse.sshfs";
  options = "...";
}];

nixos-rebuild switch, and then try to systemctl start wherever.mount.

Technical details

Relevant lines from strace:

execve("/run/current-system/sw/bin/mount", ["mount", "-t", "fuse.sshfs", "user@host:path", "/wherever"], [/* 43 vars */]) = 0
[pid 14437] execve("/var/run/current-system/sw/bin/mount.fuse", ["/var/run/current-system/sw/bin/m"..., "user@host:path", "/wherever", "-o", "rw", "-t", "fuse.sshfs"], [/* 39 vars */]) = 0
[pid 14437] execve("/bin/sh", ["/bin/sh", "-c", "'sshfs' 'user@host:"...], [/* 40 vars */]) = 0
[pid 14437] stat("/no-such-path/sshfs", 0x7ffe578fdbe0) = -1 ENOENT (No such file or directory)

And here's where the default path in bash gets excised:

https://github.com/NixOS/nixpkgs/blob/master/pkgs/shells/bash/4.4.nix#L46

[aszlig]

The actual problem is not in mount(8) but in mount.fuse: https://github.com/libfuse/libfuse/blob/ac25c153691faea65cbfc1b0781b8b0dab3f4b29/util/mount.fuse.c#L228

[aszlig]

Here is a small VM test along with a workaround which illustrates this:

import <nixpkgs/nixos/tests/make-test.nix> ({ pkgs, ... }: let
  snakeoil.private = pkgs.writeText "snakeoil.key" ''
    -----BEGIN OPENSSH PRIVATE KEY-----
    b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZW
    QyNTUxOQAAACDDynhanQHXZdPYJNddayJhnDAvFAyxNJ0HG9LyEniuVgAAAJjgHFG34BxR
    twAAAAtzc2gtZWQyNTUxOQAAACDDynhanQHXZdPYJNddayJhnDAvFAyxNJ0HG9LyEniuVg
    AAAEAO59AiwfCXXvFCY1Cayg4+H/ebglpf4wBu9URPKqu0DMPKeFqdAddl09gk111rImGc
    MC8UDLE0nQcb0vISeK5WAAAAD2FzemxpZ0BtbXJubWhybQECAwQFBg==
    -----END OPENSSH PRIVATE KEY-----
  '';

  snakeoil.public = pkgs.lib.concatStrings [
    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMPKeFqdAddl09gk111rImGcMC8UDLE0n"
    "Qcb0vISeK5W"
  ];

in {
  nodes = {
    server = {
      services.openssh.enable = true;
      users.users.root.openssh.authorizedKeys.keys = [ snakeoil.public ];
    };
    client = { pkgs, lib, ... }: {
      system.fsPackages = [ pkgs.sshfs-fuse ];
      nixpkgs.config.packageOverrides = super: {
        fuse = super.fuse.overrideDerivation (drv: {
          # Very hacky workaround to make sure that mount.fuse can search PATH:
          postPatch = (drv.postPatch or "") + ''
            sed -i \
              -e '/execl/i setenv("PATH", "/run/current-system/sw/bin", 1);' \
              util/mount.fuse.c
          '';
        });
      };

      systemd.services.inject-private-ssh-key = {
        wantedBy = [ "mnt.mount" ];
        before = [ "mnt.mount" ];
        after = [ "network-online.target" ];
        serviceConfig.Type = "oneshot";
        script = ''
          install -vD -m0600 "${snakeoil.private}" /root/.ssh/id_ed25519
          # Ensure that the host key of server is in known_hosts.
          ${pkgs.openssh}/bin/ssh -o StrictHostKeyChecking=no root@server true
        '';
      };

      systemd.mounts = lib.singleton {
        wantedBy = [ "multi-user.target" ];
        what = "root@server:/test";
        where = "/mnt";
        type = "fuse.sshfs";
      };
    };
  };

  testScript = ''
    $server->waitForUnit("sshd.service");
    $server->succeed("mkdir /test", "echo foo > /test/foo");
    $client->waitForUnit("multi-user.target");
    $client->succeed('cat /mnt/foo | grep -q "^foo$"');
  '';
})

[ToxicFrog]

I think we can't just bake the path to sshfs into mount.fuse (as suggested on IRC), because it's not necessarily sshfs that it's mounting -- but it looks like mount looks for mount.fuse.<fs-type> first:

stat("/var/setuid-wrappers/mount.fuse.sshfs", 0x7ffda9c3a610) = -1 ENOENT (No such file or directory)
stat("/var/setuid-wrappers/mount.fuse", 0x7ffda9c3a610) = -1 ENOENT (No such file or directory)
stat("/var/run/current-system/sw/bin/mount.fuse.sshfs", 0x7ffda9c3a610) = -1 ENOENT (No such file or directory)
stat("/var/run/current-system/sw/bin/mount.fuse", {st_mode=S_IFREG|0555, st_size=13040, ...}) = 0

So perhaps the answer is to have each FUSE module emit a mount.fuse.foo wrapper that mount can find, and which calls mount.fuse after setting PATH to ${pkgs.foo}/sbin/?

[danbst]

Looks like solution @ToxicFrog suggested is already in master. I can successfully mount sshfs using example in topic post.

However the mount point is accessible only for root user, but this is separate issue.
EDIT: the problem solved with allow_other in mount options

[jfrankenau]

The issue has been fixed with the release of sshfs 3.3.0 as is evident from https://github.com/libfuse/sshfs/commit/82dfbeb7c5b8d316d59df2d99f22a7dac5e81605. Looks like @ToxicFrog's suggestion is the way to go.

For the other FUSE modules in Nixpkgs this leaves us with either adding the necessary symlink as done with bindfs (#14302) or reporting the issue upstream as well.

[FRidh]

It indeed works. My setup:

  systemd.mounts = [
    {
      what = "username@hostname:/";
      where = "/home/freddy/workspace";
      type = "fuse.sshfs";
      options = "identityfile=/home/freddy/.ssh/id_rsa,allow_other";
      wantedBy = [ "multi-user.target" ];
    }
  ];

```

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/mount-sshfs-command-not-found/7489/1