Zfs: Possible Workarounds for Inability to Mount as non-root under Linux

[stellarpower]

System information

Distribution Name | Ubuntu Server
Distribution Version | 20.04
Linux Kernel | 5.4.0-1012-raspi #12-Ubuntu
Architecture | aarch64
ZFS Version | 0.8.3-1ubuntu12 (dkms)
SPL Version | 0.8.3-1ubuntu12 (dkms)

Describe the problem you're observing

It seems to be a well-known problem that mounting datasets as non-root users on linux is not possible. I have seen a variety of possible options discussed online, for example, here and here.

After encountering the issue earlier in a manner that is preventing me from completing my system setup, a couple of possible workarounds has come to mind, a hackier version of one of which I have successfully implemented to an extent in Crystal (daemon running as root). If, as it seems, the kernel limitation that prevents the current OpenZFS implementation on linux from being able to mount these datasets as non-root looks like it won't be changing any time soon, I am opening this issue as a potential way to discuss any workaround options that may be feasible.

I feel this specific missing feature is a significant factor limiting my being able to use zfs on a number of my linux systems, which is a great shame as whilst not perfect, it is a huge step up from e.g. ext4. It appears to me that perhaps previous discussions naturally concentrated on the more logical solution, fixing the core of the problem at the lower level with the issues with the mount syscall in the global context. If making any required changes with this will not be possible with limitations in the linux kernel that cannot be overcome for a significant amount of time, then I am opening this issue here to open up the discussion if any less elegant but workable solutions might be considered, implemented, and integrated in some way in order to provide a 1) satisfactory, 2) secure enough and 3) workable solution to the problem, where (3) is in the sense that even if not ideal, the solution will function correctly on a linux system, and (2) implies that the fix will not compromise security in any way above the normal levels of risk - for example, if such a fix provides a new vector that could be used to compromise security (e.g. listening as root on a socket) but this vector can and should be locked off under a normal setup (appropriate permission on this socket to prevent misuse), then we could consider (2) to have been met.

Even if a workaround feature is not ideal, I believe it would be worth considering in some level of detail, as for the reasons given below in the motivation, it seems ZFS is coming towards linux more often these days, and has been adopted quite fondly within the BSD community, in any case, quite a way beyond the initial illumos/Solaris base for which it was designed.

Motivations:

Please note most of these are somewhat personal, I will add to them over time if others would like to put forward their own use cases

• ZFS is far superior in many respects to (e.g.) ext4, and I personally would like to move to root on zfs for all of my systems. Without the ability to respect mount permissions on the datasets, this would not be possible on a multi-user system, at least, not without significant compromise in the ability of unprivileged users to perform any actions with their own datasets, which is a large motivating reason for the use of ZFS in the first place.
• Canonical seems to be exploring the option of root on ZFS as an option for mainline Ubuntu going forwards, and this issue would potentially put a stop in any future integrations if not solved in some manner.
• rootless docker: issues with the overlayfs driver. ZFS seems to work out of the box whilst the default overlayfs driver is more complicated as non-root. Docker is a very popular tool in the world of containerisation and rootless docker may well gain popularity as it develops over time to fix some small features that are lacking.

Given the prevalence of docker and Ubuntu currently, and that docker's main reference support for linux lies with Ubuntu, I reckon that if both of these platforms are looking to make ZFS support more standard and tightly integrated in the future, then that would be a good indication that a number of linux projects may seek to utilise ZFS as standard.

• FUSE support: I have seen many articles online reference the FUSE implementation of zfs, however, I believe the current opinion is that it hasn't been maintained for a long time and was far from complete when it was. Without a FUSE option, it will be impossible for any unprivileged userspaced actions involving ZFS datasets requiring any of the currently disallowed operations.

Potential Workarounds

• ### Utilise a helper binary

Refactor the operations requiring root privileges into a distinct helper binary
Make use of an existing infrastructure for running of this binary as root,

• Any internal commands that require superuser privileges can call out of the binary and become root to execute the required operation via a helper binary before returning back to unprivileged execution
• So for a mounting operation, we would check the user's privileges on the dataset, bailing here if the user lacks permissions, otherwise calling e.g. sudo zfs-helper mount Pool/dataset options
  Options include
• Use the setuid/seteuid bit on this binary
  
  
  
  • in this case, the helper binary would have to do a rigorous permissions check again on the action being requested; the standard zfs command could provide a more user-friendly error message with the assumption that the helper is not to be called directly, and so will also have to perform the same checks, however as the helper will run as root, it will be necessary to ensure the operation being requested is valid before executing it lest it be co-erced into an illicit operation
  
  
• Use the sudo system
  
  
  
  • Use the sudo system to become root in order to execute the helper binary. This would require much the same as above, however allow a finer grain of control at the OS-level for disallowing some users from ever performing these operations. The correct setup of the sudoers file would come down to the end user-administrator, however including recommended setup for minimal/everyday privileges would be helpful. It should be noted that whilst prevalent not all distributions come with sudo installed at all.
  
  
• Other options
  
  
  
  • I must admit I am aware of, but don't know anything about, POSIX capabilities; same goes for what options if any SELinux and FACLs might have to do with the matter
  
  

• ### Run a mounting deamon
  Place all the functionality that is required for mounting/unmounting (and any other privileged operations) of the datasets into a separate application that runs by default as a deamon, as root.
• The zfs program will perform basic validation of commands, ensures that the calling user passes the delegated permissions, and when any operation requires something that currently the user has permissions to perform, but cannot, due to the limits in the kernel, it passes a message to the daemon, over a socket with (global) write access.
• The daemon will once again verify that the user has the correct permissions for the operation (as it might be required to listen on a socket to which any user could connect, however the first verification would potentially provide a more helpful error message), before performing the operation, inherently as root.
• systemd or equivalent may be used to start this deamon correctly at boot, before the mount targets are met but optionally after importing the required pools.
• The socket could be made globally writable, and the daemon can obtain the effective uid of the calling process to ensure that the requested operation is allowed for said user.

I open the floor up for any comments on the subject and will edit the issue text to add any options as they arise.

[clinta]

To add to this discussion, there are existing tools designed to allow non-privileged users to mount removable devices, including pmount and udisks. I'm not sure if either of those could be utilized for mounting datasets, since they're both designed for mounting devices in /media, but it might be an option.

[ivzhh]

Personally I prefer the SUID solution. Basically all the implementation requires a root/CAP_SYS_ADMIN binary (running or with suid bit). That binary must examine permission and input options carefully to prevent privilege-escalation attack.

Podman and other daemon-less, rootless approaches try to avoid a daemon. Especially if the daemon is stateful, a hotfix will be problem (e.g. dbus-daemon).

I am trying to implement it as a standalone helper binary, instead of patching mount.zfs. If it works, I can try to use it for the zfs graph driver in containers project, which currently uses syscall mount.

[stellarpower]

This is the architecture being used on ZSys, which seems to be a relatively major concentration of effort for Ubuntu's purposes regarding enhanced ZFS support. I believe the architecture was developed in communication with the OpenZFS Dev team, and might be worth looking at . I agree that the SETUID option would be quite easy to achieve, and in fact it may be possible mostly to move parts of the existing codebase and add some boilerplate in order to separate it, but equally if a more client-server model is what would be best adopted by other projects, then maybe this would be easier forming a part of the core interface, or perhaps wrapping around the C library as an alternative to facilitate interaction at a programmatic level rather than calling out to the CLI utilities. Regarding that, I believe Joyent was wrapping some core Solaris libraries in node.js to help with some of their tools.

Also, AFAIK, but I may be wrong, things like udisks require FUSE support, and it seems the zfs-fuse package is long unmaintained and has been deprecated in many places I have read, however maybe that would be a more appropriate model, or again, some modest changes that would bridge the gap a little and make maintenance of a FUSE interface easier.

[stellarpower]

@ivzhh Do you have a repo for your helper binary? Sounds quite similar to something I was trying to do for the rootless docker setup. I'm lacking time right now but would be happy to have a look at it if I did get round to it, especially as this was preventing me from using the rootless setup after quite an investment into it for my server.

[ivzhh]

@stellarpower Not yet. I am extending go-libzfs and still work-in-progress. A basic version will be ready in a week or two.

Glad to hear you are working towards the same direction.

[ivzhh]

@stellarpower Zsys is a cool project and it led me to use go-libzfs. According to Zsys,

Package authorizer deals client authorization based on a definite set of polkit actions.
The client uid and pid are obtained via the unix socket (SO_PEERCRED) information,
that are attached to the grpc request by the server.

I really like the solution too. In addition, my solution is go-based and go cannot do syscall.Setuid() due to the threading model. I need to use libcap/psx for Setuid(). So I have many reasons not to use SUID path.

However, the reason I tried to go through SUID is daemonless. For people like podman, rootless and daemonless are two important values. So that's the main reason why I chose SUID.

[clinta]

I know you're asking for a workaround, but I've been digging more into ZFS internals and I wonder if the right way to fix this would be to expose another ZFS ioctl for mounting move calling the mount syscall into the zfs module. Then delegation can all be verified the same way all other ioctl calls are.