Rkt: Volumes in /sys masked by later mounts using stage1-coreos

[tjdett]

When using stage1-coreos the /sys mounts occur after volume mounts, unexpectedly masking them. This does not occur with stage1-fly, because it does not mount /sys.

One example of a container image that requires volume mounts under /sys is Fedora's systemd container:
https://github.com/fedora-cloud/Fedora-Dockerfiles/tree/master/systemd/systemd

What did you do?

$ sudo rkt run --interactive --insecure-options=image docker://fedora/systemd-systemd \
    --volume volume-sys-fs-cgroup,kind=host,source=/sys/fs/cgroup,readOnly=true \
    --exec /bin/bash -- -c "ls -la /sys/fs/cgroup"

What did you expect to see?

total 0
drwxr-xr-x. 13 root root 340 Jul  6 01:14 .
drwxr-xr-x.  6 root root   0 Jul  6 01:14 ..
drwxr-xr-x.  2 root root  40 Jul  6 01:14 blkio
lrwxrwxrwx.  1 root root  11 Jul  6 01:14 cpu -> cpu,cpuacct
drwxr-xr-x.  2 root root  40 Jul  6 01:14 cpu,cpuacct
lrwxrwxrwx.  1 root root  11 Jul  6 01:14 cpuacct -> cpu,cpuacct
drwxr-xr-x.  2 root root  40 Jul  6 01:14 cpuset
drwxr-xr-x.  2 root root  40 Jul  6 01:14 devices
drwxr-xr-x.  2 root root  40 Jul  6 01:14 freezer
drwxr-xr-x.  2 root root  40 Jul  6 01:14 hugetlb
drwxr-xr-x.  2 root root  40 Jul  6 01:14 memory
lrwxrwxrwx.  1 root root  16 Jul  6 01:14 net_cls -> net_cls,net_prio
drwxr-xr-x.  2 root root  40 Jul  6 01:14 net_cls,net_prio
lrwxrwxrwx.  1 root root  16 Jul  6 01:14 net_prio -> net_cls,net_prio
drwxr-xr-x.  2 root root  40 Jul  6 01:14 perf_event
drwxr-xr-x.  2 root root  40 Jul  6 01:14 pids
drwxr-xr-x.  2 root root  40 Jul  6 01:14 systemd

What did you see instead?

ls: cannot access '/sys/fs/cgroup': No such file or directory

Environment

rkt Version: 1.9.1
appc Version: 0.8.4
Go Version: go1.5.4
Go OS/Arch: linux/amd64
Features: -TPM
--
Linux 4.6.3-coreos x86_64
--
NAME=CoreOS
ID=coreos
VERSION=1097.0.0
VERSION_ID=1097.0.0
BUILD_ID=2016-07-02-0145
PRETTY_NAME="CoreOS 1097.0.0 (MoreOS)"
ANSI_COLOR="1;32"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"
--
systemd 229
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT -GNUTLS -ACL +XZ -LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD -IDN

[alban]

I think that stage1/prepare-app/prepare-app.c should check if /sys (or a subdirectory) is already mounted before mounting it.

Note: it would be consistent with systemd-nspawn/systemd that only mounts new mounts if it is not already done (idempotence), to allow the container run-time to override mounts.

[tjdett]

@alban In the example case, I'm not sure a simple "not already done" check would produce the correct behaviour.

/sys/fs/cgroup is a volume, but the mounted /sys/fs does not contain /sys/fs/cgroup. Mounting the /sys/fs shadows /sys/fs/cgroup. For example, from mount | grep ' /sys' in an affected app container:

tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (ro,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (ro,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/cpuset type cgroup (ro,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (ro,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/pids type cgroup (ro,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (ro,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/blkio type cgroup (ro,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (ro,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/devices type cgroup (ro,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/memory type cgroup (ro,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/freezer type cgroup (ro,nosuid,nodev,noexec,relatime,freezer)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime,seclabel)
tmpfs on /sys/firmware type tmpfs (ro,nosuid,nodev,context="system_u:object_r:svirt_sandbox_file_t:s0:c481,c604",mode=755)
tmpfs on /sys/fs type tmpfs (ro,nosuid,nodev,context="system_u:object_r:svirt_sandbox_file_t:s0:c481,c604",mode=755)
tmpfs on /sys/hypervisor type tmpfs (ro,nosuid,nodev,context="system_u:object_r:svirt_sandbox_file_t:s0:c481,c604",mode=755)
tmpfs on /sys/module type tmpfs (ro,nosuid,nodev,context="system_u:object_r:svirt_sandbox_file_t:s0:c481,c604",mode=755)
tmpfs on /sys/power type tmpfs (ro,nosuid,nodev,context="system_u:object_r:svirt_sandbox_file_t:s0:c481,c604",mode=755)
sysfs on /sys/block type sysfs (ro,nosuid,nodev,noexec,relatime,seclabel)
sysfs on /sys/bus type sysfs (ro,nosuid,nodev,noexec,relatime,seclabel)
sysfs on /sys/class type sysfs (ro,nosuid,nodev,noexec,relatime,seclabel)
sysfs on /sys/dev type sysfs (ro,nosuid,nodev,noexec,relatime,seclabel)
sysfs on /sys/devices type sysfs (ro,nosuid,nodev,noexec,relatime,seclabel)
sysfs on /sys/kernel type sysfs (ro,nosuid,nodev,noexec,relatime,seclabel)

Short of not performing the /sys/fs mount, I don't see how you can resolve the problem by avoiding mounts. Irritatingly, you can't simply reorder in this case either - /sys/fs is mounted read-only, so you can't create the prerequisite /sys/fs/cgroup to perform the volume mount.

[alban]

@tjdett I think that rkt should also skip the /sys mount if the user provides a volume to mount in a sub-directory of /sys. This means the usual files in /sys will not be available, but users can mount directly /sys if it is necessary.

Implemented in https://github.com/coreos/rkt/pull/2888

[iaguis]

Some files inside /sys are still masked because of https://github.com/coreos/rkt/pull/2683

We should find a way to disable those protections for some use-cases like this one.

[Quentin-M]

And we should be able to toggle this from a kubernetes manifest.

[alban]

Capturing discussion from https://github.com/coreos/rkt/issues/2158#issuecomment-233603912:

We need a way to "opt-out" of security features to fully idiomatically enable rkt-in-rkt. The current consensus is to have a --privileged[=false] flag to implement the above without specifying separate capability, seccomp, innaccessiable path settings.

This will be for next milestone.

[lucab]

Tracked via #2983.

[alban]

2888 and #2983 are merged.

This now works with --insecure-options=paths:

$ sudo rkt run --interactive --insecure-options=image,paths docker://fedora/systemd-systemd \
    --volume volume-sys-fs-cgroup,kind=host,source=/sys/fs/cgroup,readOnly=true \
    --exec /bin/bash -- -c "ls -la /sys/fs/cgroup"
image: using image from local store for image name coreos.com/rkt/stage1-coreos:1.12.0+gite3ab0df
image: using image from local store for url docker://fedora/systemd-systemd
networking: loading networks from /etc/rkt/net.d
networking: overriding "default" network with long.conf
networking: loading network default with type ptp
total 4
drwxr-xr-x. 13 root root  340 Jul  8 14:55 .
drwxr-xr-x.  3 root root 4096 Aug  9 09:32 ..
dr-xr-xr-x.  6 root root    0 Jul 19 16:52 blkio
lrwxrwxrwx.  1 root root   11 Jul  8 14:55 cpu -> cpu,cpuacct
dr-xr-xr-x.  6 root root    0 Jul 19 16:52 cpu,cpuacct
lrwxrwxrwx.  1 root root   11 Jul  8 14:55 cpuacct -> cpu,cpuacct
dr-xr-xr-x.  4 root root    0 Jul 19 16:52 cpuset
dr-xr-xr-x.  6 root root    0 Jul 19 16:52 devices
dr-xr-xr-x.  4 root root    0 Jul 20 15:16 freezer
dr-xr-xr-x.  4 root root    0 Jul 20 15:16 hugetlb
dr-xr-xr-x.  6 root root    0 Jul 19 16:52 memory
lrwxrwxrwx.  1 root root   16 Jul  8 14:55 net_cls -> net_cls,net_prio
dr-xr-xr-x.  4 root root    0 Jul 20 15:16 net_cls,net_prio
lrwxrwxrwx.  1 root root   16 Jul  8 14:55 net_prio -> net_cls,net_prio
dr-xr-xr-x.  4 root root    0 Jul 20 15:16 perf_event
dr-xr-xr-x.  6 root root    0 Jul 19 16:52 pids
dr-xr-xr-x.  6 root root    0 Jul 11 13:48 systemd