Runtime: docker:dind not working in 1.11.1 qemu-virtiofs

[skaegi]

Description of problem

We use a kata-deploy style install for using Kata with Kubernetes (containerd 1.3.4 -- overlaysfs snapshotter) and just tried updating from 1.11.0 to 1.11.1. When using the qemu-virtiofs runtime we now cannot run docker commands and get operation not permitted errors. The errors below are using the unmodified 1.11.1 release.

Expected result

(When running inside a K8 pod using kata)
docker run hello-world -- works just like it did in 1.11.0

Actual result

/ # docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
0e03bdcc26d7: Pull complete 
Digest: sha256:d58e752213a51785838f9eed2b7a498ffa1cb3aa7f946dda11af39286c3db9a9
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:415: setting cgroup config for procHooks process caused \\\"failed to write \\\\\\\"b *:* m\\\\\\\" to \\\\\\\"/sys/fs/cgroup/devices/docker/81a3c5750ffe91ea0c5992f5b532589c8aa2a64cd1cb22c6a49036bf356b32e2/devices.allow\\\\\\\": write /sys/fs/cgroup/devices/docker/81a3c5750ffe91ea0c5992f5b532589c8aa2a64cd1cb22c6a49036bf356b32e2/devices.allow: operation not permitted\\\"\"": unknown.

Just the error message:

Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:415: setting cgroup config for procHooks process caused \\"failed to write \\\\"b : m\\\\" to \\\\"/sys/fs/cgroup/devices/docker/81a3c5750ffe91ea0c5992f5b532589c8aa2a64cd1cb22c6a49036bf356b32e2/devices.allow\\\\": write /sys/fs/cgroup/devices/docker/81a3c5750ffe91ea0c5992f5b532589c8aa2a64cd1cb22c6a49036bf356b32e2/devices.allow: operation not permitted\\"\"": unknown.

Here is a simples pod spec to help illustrate...

apiVersion: v1
kind: Pod
metadata:
  name: untrusted-dind
  annotations:
    io.kubernetes.cri.untrusted-workload: "true"
spec:
  containers:
    - name: docker
      image: docker:dind
      imagePullPolicy: IfNotPresent
      command:
      - dockerd-entrypoint.sh
      - dockerd
      - --host=unix:///var/run/docker.sock
      - --host=tcp://localhost:2375
      - --mtu=1400
      securityContext:
        privileged: true
      ports:
        - containerPort: 2375

(using untrusted annotation because runtimeClass feature gate is off in my environment)

[skaegi]

It seems pretty likely that the fix for CVE-2020-2023 and CVE-2020-2026 here is causing this problem. Does something special need to be done to accommodate privileged containers?

[skaegi]

To be clear, this is a "regression" between Kata 1.11.0 and 1.11.1. I do not believe this is a problem with virtio-fs.

[Jason-ZW]

@skaegi I have the same problem. I follow the documents using kata-containers/osbuilder built the kata-containers.img.
Using the default kata-containers.img which uncompress from the kata-static-xxx.tar.gz with no error.

kata-runtime  : 1.10.4
   commit   : dc550720a31582ced60d26cd7c64d746b1449864
   OCI specs: 1.0.1-dev

createPodSandbox for pod "metrics-server-7566d596c8-j8qvr_kube-system(78d9aeb0-ec60-4f47-84a9-1d50062d0442)" failed: rpc error: code = Unknown desc = failed to create containerd task: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:415: setting cgroup config for procHooks process caused \\\"failed to write \\\\\\\"b *:* m\\\\\\\" to \\\\\\\"/sys/fs/cgroup/devices/kubepods/besteffort/pod78d9aeb0-ec60-4f47-84a9-1d50062d0442/4f8df6b8d065fd9e355e87f8fbcf20fa4a9a8b19dbb7caabb5eba4bf37b1da04/devices.allow\\\\\\\": write /sys/fs/cgroup/devices/kubepods/besteffort/pod78d9aeb0-ec60-4f47-84a9-1d50062d0442/4f8df6b8d065fd9e355e87f8fbcf20fa4a9a8b19dbb7caabb5eba4bf37b1da04/devices.allow: operation not permitted\\\"\"": unknown

[Jason-ZW]

I found the problem. I built the image with the wrong AGENT_VERSION

[amshinde]

@skaegi I have been able to reproduce your issue. I did a git bisect accross repos and saw that this commit is causing the issue:
https://github.com/kata-containers/agent/commit/a88af321f7c591404b0316320c46591fa8250992

Let me see how we can fix this.

[amshinde]

The commit https://github.com/kata-containers/agent/commit/a88af321f7c591404b0316320c46591fa8250992 restricts access of the container to the VM rootfs devices, whereas docker running in the container should write to the container's /sys/fs/cgroup.
I am not sure why the container would need access to the VM rootfs device.
@devimc @bergwolf Any idea?

[skaegi]

Awesome thanks @amshinde that makes sense. FWIW I tried 1.11.2 and have the same problem.

[amshinde]

@skaegi Changing that patch to below fixes the issue:

diff --git a/device.go b/device.go
index ae95572..01ee559 100644
--- a/device.go
+++ b/device.go
@@ -496,7 +496,7 @@ func updateDeviceCgroupForGuestRootfs(spec *pb.Spec) {
                Major:  devMajor,
                Minor:  devMinor,
                Type:   "b",
-               Access: "rwm",
+               Access: "rw",
        }

        spec.Linux.Resources.Devices = append(spec.Linux.Resources.Devices, nvdimmCg)

From the error it looks like docker needs "m" access for block devices ie ability to create device nodes for block devices.
I am not entirely sure why the nested docker container needs this, the config.json does not contain any such device cgroup entry, so not entirely sure why libcontainer tried to add the devices.allow entry for block devices.

@bergwolf @devimc Let me know if have knowledge about it.
@bergwolf By the way, the above patch does not allow read/write access for the rootfs, which means it still addresses the CVEs we had fixed previously. Let me know if the patch is good, I'll raise a PR for it in the agent repo.

[devimc]

@amshinde I'm using the latest version of kata-agent, I started a debug console to check the device cgroup of a container, the problem is that I don't see any device constraint, could you check that?

bash-4.2# ls /dev/pmem0*
/dev/pmem0  /dev/pmem0p1
bash-4.2# mount | grep pmem
/dev/pmem0p1 on / type ext4 (ro,relatime,dax,errors=remount-ro,data=ordered)
bash-4.2# pwd
/sys/fs/cgroup/devices/docker/9e3e1dc87b80daf70d573774257de495a9e6bb6a364d847c1dcb446118426ece
bash-4.2# cat devices.list 
a *:* rwm
bash-4.2# kata-agent --version
kata-agent version 1.12.0-alpha0-e01f28988780f43b1d023dd46272f334d5ec8230

[skaegi]

Thanks @amshinde I'll rebuild and try that patch.

[amshinde]

@devimc Thats strange, I do have a test that checks whether you can read/write/mknod to the rootfs and I do see the desired behaviour when I toggle the device cgroup. Let me create a debug image and check.
I am not sure about the devices.list, but I think you atleast should see entry under devices.allow.

[devimc]

@amshinde

bash-4.2# cat devices.allow 
cat: devices.allow: Invalid argument

[skaegi]

fwiw I fail the same way with 1.12.0-alpha0

[amshinde]

yes, 1.12.0-alpha0 would fail as well.
Let me know if that patch works for you. I tried it at my end and don't see the error any more.
For some reason, docker adds permissions to create device nodes for all block devices which conflicts with the approach taken by us to restrict all permission to the rootfs block device.

Will be adding an integration test for this to make sure we dont break in the future.
cc @GabyCT

[skaegi]

Thanks @amshinde -- I can confirm your patch works.
I created my own build for our environment with and without your patch on top of 1.12.0-alpha0 and it worked with your patch and failed as above without it.

I actually need a fix on top of 1.11.2 so I'll also try that. (confirmed)

[skaegi]

@amshinde this all works well for me now. I have the luxury of my own build environment so can do the patching myself but it might be a good idea to check a fix in for 1.12/master and probably for stable-1.11 too as anyone else trying this out for the first-time will fail.

[amshinde]

@skaegi Was away for a few days. I have now raised a PR to address this. Will backbort to stable as well.
https://github.com/kata-containers/agent/pull/807

[killuazhu]

We run into the same issue, about to see if the agent patch provided by @amshinde will work for us.

[jcvenegas]

@amshinde so is this not a virtiofs issue, correct ?
Can we close this issue now?

[amshinde]

@jcvenegas Yes, not related to virtiofs. Fix has been merged in master and stable branches. Closing this.

[killuazhu]

@amshinde do you know if the merge of commit to https://github.com/kata-containers/agent stable-1.11 branch would also refresh packages on http://download.opensuse.org/repositories/home:/katacontainers:/releases:/x86_64:/stable-1.11/xUbuntu_18.04/ ?