Singularity: fakeroot: Use existing subgid/subuid mappings if present instead of config file

[jwflory]

Summary

If mappings already exist in /etc/subgid and /etc/subuid, use these instead of using config file with usernames

Background

Sometimes an environment may have hundreds, or even thousands, of users. In an environment with this many users, adding each user to a list in a config file (and maintaining the correct order) is difficult and tedious.

Other container run-times recommend allocating subuids/subgids manually. It is possible an environment using Singularity may already have allocated subuids/subgids manually to either do one or both of the following:

1. Accommodate a large number of users
2. Support another container run-time

Having Singularity manage this for users is inconvenient for these two use cases, or may be incompatible with some environments if the bits are already allocated.

Details

Singularity should use existing mappings if already present on a filesystem. This is more convenient for large-user installations and also allows compatibility with other container run-times with a rootless mode option.

This should take into consideration Network File System (NFS) and General Parallel File System / Spectrum Scale (GPFS), since these filesystems are not aware of user namespaces:

$ cd /my_gpfs/jflory/
$ unshare -r bash -c "mkdir foo; chmod 000 foo; touch foo/bar"
touch: cannot touch ‘foo/bar’: Permission denied

Outcome

1. Easier for interested system administrators to introduce Singularity in large-user environments
2. Better side-by-side compatibility with other container run-times

[jscook2345]

I believe we purposely did not make use of the subuid/subgid feature, but perhaps others can correct me here.

@cclerget ?

[DrDaveD]

I requested something similar in #3753, but I was only asking to do it when newuidmap/newgidmap was installed and singularity was not running setuid (by singularity calling out to newuidmap/newgidmap). You have a good point though that on a system with many users that are allowed to use it, the /etc/subuid and /etc/subgid format will be easier to maintain than /etc/singularity/singularity.conf. On the other hand on a system where there few users allowed to use it, the singularity.conf format is easier to configure.

[cclerget]

@jwflory @DrDaveD With sysadmin hat I don't get the point of /etc/subxid files being easier to configure in their actual state, no NSS or global management, it will probably change in a more or less near future but only on recent distribution. The current fakeroot configuration implementation was to make things easier for admins, so they doesn't need to take care of user mappings overlap and potential issues that come with, with /etc/subxid files the responsibility is left to admins.

Now based on feedbacks and expectations (#3753, #3936), there is four main points to consider :

1. singularity without setuid installation but with fakeroot, implying to install newuidmap/newgidmap binaries and use /etc/subxid files
2. singularity setuid installation and fakeroot configuration via /etc/subxid files
3. singularity setuid installation and fakeroot configuration via singularity.conf
4. singularity setuid installation and fakeroot disabled/enabled by configuration via singularity.conf because a user may be already present in /etc/subxid files

It actually doesn't make sense to keep two different fakeroot configuration based on setuid installation or not because of points 1 and 2 above, and I actually think we need to avoid those magic under the hood behaviors because all that does is to confuse people and make documentation/configuration harder.

So the idea is to get rid of singularity.conf user mappings configuration for fakeroot and defer that to /etc/subxid files, for that we could introduce a singularity config command to ease sysadmin tasks both for singularity and fakeroot configuration via /etc/subxid files with something like:

• to add a user to /etc/subuid and /etc/subgid files with a dynamic unallocated range :

sudo singularity config fakeroot add user

• to remove a user :

sudo singularity config fakeroot remove user

• to enable/disable a user in order to keep the allocated range for the user (by appending a ! to username in files) :

sudo singularity config fakeroot [enable|disable] user

Those commands will keep track of all allocated ranges via subxid files, and each time a user is added, it will check that both UIDs/GIDs in the range are not assigned to an existing user/group, for that the easiest approach is to start from the highest UID/GID possible range to minimize overlap with existing UIDs/GIDs and decrease each time a user is added.

For the point 4, allow fakeroot = yes/no would be set to no by default for Singularity setuid installation in singularity.conf and simply ignored for non privileged installation to fallback to newuidmap/newgidmap

[DrDaveD]

@cclerget That plan makes a lot of sense. You discussed something similar earlier and I didn't get it then, but now I see why it makes sense to have only one way to configure fakeroot and to make tools to make it easier for a sysadmin to manage. I'm not sure that point 4 is all that necessary with this plan, since if a system admin goes through the trouble of setting up the /etc/subxid files I can't imagine why they wouldn't want singularity to use them, but it doesn't hurt to have the option. It might just as well default to yes though I think.

[jeffcbecker]

@cclerget, our (NASA Advanced Supercomputing facility) preference is your first option, as we'd like to avoid setuid if at all possible.

singularity without setuid installation but with fakeroot, implying to install newuidmap/newgidmap binaries and use /etc/subxid files

[cclerget]

@jwflory @DrDaveD @jeffcbecker If you have time to give it a try: #3943 . singularity config stuff will be implemented in a separate PR and will be planned for the next release 3.4. With #3943 the fakeroot configuration is now entirely deferred to /etc/subxid files.

[jwflory]

@cclerget The use case I have requires two things:

1. Singularity without setuid installation
2. Convenient way to enable hundreds to thousands of unique users to use --fakeroot by default

Additionally, it is worthwhile to consider NFS and IBM Spectrum Scale/GPFS. These filesystems are not aware of user namespaces so --fakeroot may not work out of the box on those file systems either. The Podman project has some of these shortcomings documented too, I assume some of these points are also relevant to Singularity.

Unfortunately the lack of GPFS support is also a showstopper for our needs (e.g. bind-mounting in a GPFS home directory to a container).

[cclerget]

@jwflory Well regarding NFS/GPFS, the use of FUSE could help with those FS issues.

Actually, possible issues with those filesystems and fakeroot are :

• build a sandbox image with fakeroot in a directory located in this filesystem, but should work with SIF images as build process write into /tmp by default, then creates image from the /tmp location
• change identity inside container and write files with this identity, should not be common in HPC unless to run services like HTTP server or something like that.

I'm working on introducing a new command singularity config (proposal here #3944) to ease management to do things like :

for user in $(getent group hpcusers|cut -d ':' -f4|tr ',' ' '); do
    singularity config fakeroot --add $user
done

[Zohman]

Additionally, it is worthwhile to consider NFS and IBM Spectrum Scale/GPFS. These filesystems are not aware of user namespaces so --fakeroot may not work out of the box on those file systems either. The Podman project has some of these shortcomings documented too, I assume some of these points are also relevant to Singularity.

Unfortunately the lack of GPFS support is also a showstopper for our needs (e.g. bind-mounting in a GPFS home directory to a container).

@jwflory i can't seem to understand your issue and the provided document.
i work with NFS and the --fakeroot solve this issue for me completely,
the namespace transition occur from the host point of view when singularity kicks,
why GPFS/NFS need to be aware of namespaces?..
the expected end-result in my mind is to end up with the real user uid/gid on the filer when using any mounted NAS system.

in addition i can say that on-the-contrary,
when using setuid or sudo it fail as singularity try to use real-root and most of the machines
in our HPC do not have root NFS exports, so they will need to work on a local mount
where the 'root' is "trusted".

worth mentioning that singularity mounts a temporary localstatedir
where the namespace transitions done ( i guess )
is podman do it as well? if podman try to mount the container staging on the NAS system
of course it won't work with namespace and your issue start to make sense..
i misunderstood something?

@cclerget I've adapted to the --fakeroot old way and it worked pretty well,
but i'm testing subgid\subuid mode of work.

some of our systems quires users from NIS/LDAP,
hopefully managing those files won't be difficult, i'll need to check..

[Flamefire]

Convenient way to enable hundreds to thousands of unique users to use --fakeroot by default

@jwflory We have the same use case. The problem is: New users may be added at any point so a "free" UID range is hard to get: It has to be free now and in the future.

@cclerget Who is supposed to run singularity config fakeroot --add $user? The admin(s) after adding a user? That probably won't happen here :/
Additionally: As new UIDs are used, what happens when the users creates files during the build process in other locations? Those files will be owned by UIDs that don't exist afterwards and hence deletion of those files might be hard to impossible, or am I wrong here?

[DrDaveD]

@Flamefire The idea is that the user id mapping gets put permanently into /etc/subuid and /etc/subgid just like in /etc/passwd. Yes the system admin would add the user when they create the account if they want it to apply to everyone, otherwise they'd do it on request.

[jeffcbecker]

I've been testing 3.4.0 rc2, specifically the fakeroot stuff. It works but that is because I have an entry in /etc/subuid, and /etc/subgid. Is this required for fakeroot, and if so, will there be an easy way for admins to add users to these files? I believe there's been some discussion about this in the recent past, but I don't think there was any resolution.

[DrDaveD]

@jeffcbecker Yes all the discussion was in this issue. @cclerget said on July 17 that the singularity config fakeroot command would be targeted for 3.4, but I don't see it there either.

[DrDaveD]

I was wrong, there were other issues where this was discussed e.g. #3944. There's a corresponding PR #4346 that was merged on Friday, but it was too late for the 3.4 release. I am currently planning on including the patch in the EPEL 3.4.0-1.1 release anyway, because it was intended to be in 3.4.

[jeffcbecker]

Good to know. FYI, I did some fakeroot testing with 3.4 rc1, and I was able to build and modify containers using fakeroot (I have an entry in sub[u,g]id). Except for some strange warning messages at the end of doing a yum install inside a CentOS container, everything seemed to work. Thanks.

[cclerget]

Closing the issue. Feel free to re-open it if required. For discussion related to config fakeroot please use #3944. Thanks !