Lxd: Share folders with containers such that they're writeable by root inside the container

[nottrobin]

_Moved from https://github.com/lxc/lxc/issues/1024 as requested_

As I understand it the way to share a folder with a container is by creating a device. This works, but it leads to the mounted device being owned by nobody:nogroup, so root can't write to it:

[robin@xps ~ ]$ lxc launch ubuntu:16.04 my-xenial  # Create the container
[robin@xps ~ ]$ lxc config device add my-xenial share disk source=`pwd`/share path=/srv/share  # Share the current directory at `/srv/share`
[robin@xps ~ ]$ lxc exec my-xenial -- ls -ld /srv/share                                      
drwxrwxr-x 2 nobody nogroup 4096 May 13 13:22 /srv/share
[robin@xps ~ ]$ lxc exec my-xenial -- touch /srv/share/afile
touch: cannot touch '/srv/share/afile': Permission denied

Following discussion in #lxcontainers on freenode, I found that the best existing solution to this problem is to use setfacl to expand the permissions on that directory explicitly for lxd and the container's root user:

# Where 165536 is the user ID of root in my container
setfacl -Rm user:lxd:rwx,default:user:lxd:rwx,user:165536:rwx,default:user:165536:rwx share

(more expansive command-line illustration)

Would it be possible for LXC/LXD itself to setup ACLs, or do something similar, so that any shared devices are automatically set-up with write permissions for root inside the container?

Even better, would it be possible to set it up so that files created in the shared folder by root inside the container would be owned by a different non-root user (e.g. robin) on the host filesystem?

[hallyn]

Just to preserve my comment from the lxc issue (1024):

There is upstream kernel discussion going on about a vfs feature to
mount filesystems with shifted uids, which if it lands could also be
used to make this more transparent. Otherwise, yes, shifting uids by
lxd should be possible. Or lxd could have an option to mount as a
overlayfs and shift the uids in the writeable layer, with the option
to merge changes back in at unmount.

[JohnstonJ]

I think this is the relevant kernel discussion: https://lkml.org/lkml/2016/5/4/411

To me, the VFS kernel approach sounds like the right one - a generic way to shift UID/GID before a container I/O goes to disk, on any file system. It would have useful application beyond LXD. Anything else sounds like an ugly hack to me, at least for the example I propose below (messing with ACLs, using overlayfs, etc.). For some situations like #2005, maybe alternate solutions are not possible?

An option of how this might look would be a LXD disk device option that specifies whether or not to keep the namespaced UID/GID, or shift it out of the namespace completely. By default it would keep the namespaced UID/GID to preserve existing behavior. But it could also shift it out. That is, suppose the user namespace says that container UID 0 maps to hypervisor UID 100000. The existing behavior is that a file written by root in the container gets written to disk with owner being UID 100000. Proposed shifting behavior would be that the file gets written to disk with owner being UID 0. (Shifting behavior could also be introduced so that if the container is migrated to another host where lxd has different subuid range, an appropriate shift could be calculated.)

An example scenario where this is useful would be a file server like SAMBA or NFS running inside a container:

1. There is no real reason why the container has to be privileged. So, we would like it to be unprivileged.
2. But suppose the SAMBA or NFS server is integrated with some directory server like LDAP or Active Directory so that it gets UID/GID from an RFC 2307 directory.
3. An unprivileged container can't mount anything itself, so we have to have LXD provide the data storage already mounted. We would like the ACLs on the underlying volume to be stored unmodified, seeing as how we got them from LDAP. That is, if user in the LDAP directory with UID of 10000 stores a file on the containerized NFS/CIFS server, it gets stored on the underlying volume with UID of 10000 (and not, say, 110000, if the container is namespaced from 100000 to 165535). That way, the same volume could be used on a different container without the permissions getting all messed up.

As I understand it, currently, the first requirement is in conflict with the last requirement. But if we had a way to shift UIDs then we can both run unprivileged and also store the files with the proper UIDs obtained from LDAP by the container.

I think this issue is also related to #733, #2005, #1879, #850. For example in #2005 it would be useful when dealing with network resources where again, the original UID/GID needs to be preserved.

[zlanich]

@JohnstonJ I think this might be related to what I'm trying to do. Can you help me figure out if this is the case and let me know if there is any progress on this?. Here's my needs:

I'm running a VM that hosts multiple LXCs, one for each of our customers. I'd like to mount a GlusterFS volume at /data/gluster on the host, then divide that volume into subdirectories (per customer), create a Linux user for each customer, then use ACLs to restrict access per user to each of those subdirectories. I'd then like to use something like

lxc config device add some_customer_container sdb disk source=/data/gluster/someuserdir path=gluster_in_the_container

to mount a R/W volume inside the container. The container is unprivileged and therefore root in the container should only be able to R/W within the subdir the ACLs say it can.

I could be a bit off in my assumption of how I should do this, but I think you see what I'm trying to do. Can you (or anyone watching this issue) provide some insight?

[zlanich]

Turns out I finally found the user/group id map functionality in the docs and it seems that solves my problem!

[mikesteele81]

@zlanich Are you referring to userns-idmap.md?

[zlanich]

Yup.

Best Regards,

Zach Lanich
Business Owner, Entrepreneur, Creative
Owner/CTO
weCreate LLC
www.WeCreate.com

On Aug 22, 2016, at 12:45 PM, Michael Steele [email protected] wrote:

@zlanich https://github.com/zlanich Are you referring to userns-idmap.md https://github.com/lxc/lxd/blob/master/doc/userns-idmap.md?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/lxc/lxd/issues/2025#issuecomment-241474474, or mute the thread https://github.com/notifications/unsubscribe-auth/ABc76efD8kf9QmpRXAE3e-Y1jdBYeZHNks5qidIogaJpZM4IeGOF.

[stgraber]

Closing this issue as there's currently not much LXD can do to improve the situation.

We worked to make it easier to do uid/gid hole punching between host and container, making it easy to share data in the most common cases: https://stgraber.org/2017/06/15/custom-user-mappings-in-lxd-containers/

For data sharing between containers, our storage volume implementation now dynamically remaps the attached volume to match the container.

And as @hallyn mentioned a while back, we're tracking (and helping) the upstream Linux work on VFS layer mapping (shiftfs or similar) which would be the ideal solution for all our LXD needs.

When such a solution is finally agreed upon and merged upstream, we'll be sure to add LXD support for it such that on systems with the feature, data sharing between host and container and between containers will work transparently without ever having to worry about file ownership.

[broizter]

@stgraber Is there any security implications using the "Direct user/group mapping" method? Or is the only thing happening that it can read/write as a UID/GID that matches the host user, with no other implications?

My setup is that I'm using an "unprivileged" user on the host that has ownership of most of my bulk storage, and then I map that UID/GUID to a couple of containers, and pass through only the folders that the services inside my containers need.

[stgraber]

Other than filesystem access, the main other issue are ulimits. The container can now modify and deplete user limits for those uids you've passed, which could be used as a way to achieve a denial of service.

So long as you don't give access to completely untrusted users you should be fine though.

[broizter]

When you say filesystem access, do you mean the folders I added with lxc config add disk, or do you mean more than that? Sorry for the stupid questions.

[stgraber]

@broizter yeah, I just meant that users in the container can access those paths, potentially write to them and do other normal filesystem interactions (whatever ioctls are supported for unprivileged users on the particular filesystem).

[broizter]

I understand! Big thanks for taking your time to answer my question. I recently moved most of my stuff from virtual machines to LXD and I'm extremely pleased.