Kind: Can't run KIND on ChromeOS Linux VM

[adrianludwin]

What happened: I tried to run kind (v0.4.0) on the Linux VM in Chrome OS 75.0.3770.144:

aludwin@penguin:~$ kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.15.0) 🖼
 ✓ Preparing nodes 📦 
 ✗ Creating kubeadm config 📜 
Error: failed to create cluster: failed to get kubernetes version from node: failed to get file: exit status 1

Based on the error and a quick look at the source code, I decided to try to run kindest/node manually, and got the following result:

aludwin@penguin:~$ docker run kindest/node:v1.15.0 -it /bin/bash
mount: /sys: permission denied.

I _suspect_ (but have not tried to confirm) that this comes from the following lines in images/base/entrypoint:

fix_mount() {
  echo 'INFO: ensuring we can execute /bin/mount even with userns-remap'
  # necessary only when userns-remap is enabled on the host, but harmless
  # The binary /bin/mount should be owned by root and have the setuid bit
  chown root:root /bin/mount
  chmod -s /bin/mount

  # This is a workaround to an AUFS bug that might cause `Text file
  # busy` on `mount` command below. See more details in
  # https://github.com/moby/moby/issues/9547
  sync

  echo 'INFO: remounting /sys read-only'
  # systemd-in-a-container should have read only /sys
  # https://www.freedesktop.org/wiki/Software/systemd/ContainerInterface/
  # however, we need other things from `docker run --privileged` ...
  # and this flag also happens to make /sys rw, amongst other things
  mount -o remount,ro /sys

What you expected to happen: Cluster gets created.

How to reproduce it (as minimally and precisely as possible):

If my suspicion that the problems is in the entrypoint is correct, docker run kindest/node:v1.15.0 -it /bin/bash should be sufficient. If that's wrong, kind create cluster reliably reproduces.

Anything else we need to know?:

Environment:

• kind version: v0.4.0
• Kubernetes version: n/a
• Docker version: 19.03.1
• OS: Chrome OS 75.0.3770.144. VM is running Debian 9.

[BenTheElder]

We need to do that remount.

There is also a previous bug for this. I don't think kind will work inside the chromeOS sandbox at the moment.

[BenTheElder]

aludwin@penguin:~$ docker run kindest/node:v1.15.0 -it /bin/bash
mount: /sys: permission denied.

Permission is denied because this command line is missing flags related to that (which kind will include).

If you run kind with --loglevel=debug you can see more detail including this command.

[BenTheElder]

https://github.com/kubernetes-sigs/kind/issues/462

[adrianludwin]

Thanks for the fast response, sorry for missing the other bug. Do we know
if this is the last remaining blocker for Chrome OS? And is there a bug
assigned to Chrome OS?

Sent from my phone

On Fri, 9 Aug 2019, 11:30 am Benjamin Elder, [e-mail]
wrote:

We need to do that remount.

There is also a previous bug for this. I don't think kind will work inside
the chromeOS sandbox at the moment.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes-sigs/kind/issues/763?email_source=notifications&email_token=AE43PZDDKRT5ZXNF5SWEMPLQDWES7A5CNFSM4IKVC4MKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD37ADNI#issuecomment-519963061,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE43PZBOERWT6PVTZHUEVE3QDWES7ANCNFSM4IKVC4MA
.

[BenTheElder]

We don't know what the blocker is yet, and I don't have access to a setup suitable for testing this. The existing bug doesn't have enough details either.

This may not even be considered a bug on their end. Kubernetes / the container runtimes need relatively elevated permissions to run at the moment, I suspect the sandbox is too secure :+)

[adrianludwin]

Re #462: I did see that bug but I thought this was something different. My computer successfully got past the "preparing nodes" step but died on the next "Creating kubeadm config" step.

Can I provide any more details that would help with this? Agreed that there's a chance ChromeOS won't want to "fix" this but I'm motivated to push on it a bit :)

Here's the result of the debug logs:

aludwin@penguin:~$ kind create cluster --loglevel=debug
DEBU[12:04:09] Running: /usr/bin/docker [docker ps -q -a --no-trunc --filter label=io.k8s.sigs.kind.cluster --format {{.Names}}\t{{.Label "io.k8s.sigs.kind.cluster"}}] 
Creating cluster "kind" ...
DEBU[12:04:09] Running: /usr/bin/docker [docker inspect --type=image kindest/node:v1.15.0] 
INFO[12:04:09] Image: kindest/node:v1.15.0 present locally  
 ✓ Ensuring node image (kindest/node:v1.15.0) 🖼
DEBU[12:04:09] Running: /usr/bin/docker [docker info --format '{{json .SecurityOptions}}'] 
DEBU[12:04:09] Running: /usr/bin/docker [docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname kind-control-plane --name kind-control-plane --label io.k8s.sigs.kind.cluster=kind --label io.k8s.sigs.kind.role=control-plane --expose 43283 --publish=127.0.0.1:43283:6443 kindest/node:v1.15.0@sha256:b4d092fd2b507843dd096fe6c85d06a27a0cbd740a0b32a880fe61aba24bb478] 
 ✓ Preparing nodes 📦 
DEBU[12:04:13] Running: /usr/bin/docker [docker ps -q -a --no-trunc --filter label=io.k8s.sigs.kind.cluster --format {{.Names}}\t{{.Label "io.k8s.sigs.kind.cluster"}} --filter label=io.k8s.sigs.kind.cluster=kind] 
DEBU[12:04:13] Running: /usr/bin/docker [docker inspect -f {{index .Config.Labels "io.k8s.sigs.kind.role"}} kind-control-plane] 
DEBU[12:04:14] Running: /usr/bin/docker [docker exec --privileged kind-control-plane cat /kind/version] 
 ✗ Creating kubeadm config 📜 
DEBU[12:04:14] Running: /usr/bin/docker [docker ps -q -a --no-trunc --filter label=io.k8s.sigs.kind.cluster --format {{.Names}}\t{{.Label "io.k8s.sigs.kind.cluster"}} --filter label=io.k8s.sigs.kind.cluster=kind] 
DEBU[12:04:14] Running: /usr/bin/docker [docker rm -f -v kind-control-plane] 
Error: failed to create cluster: failed to get kubernetes version from node: failed to get file: exit status 1

I'll try to reproduce this more precisely.

[adrianludwin]

Running the docker run command on its own, without -d, has the same result I got initially:

aludwin@penguin:~$ docker run -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname kind-control-plane --name kind-control-plane --label io.k8s.sigs.kind.cluster=kind --label io.k8s.sigs.kind.role=control-plane --expose 43283 --publish=127.0.0.1:43283:6443 kindest/node:v1.15.0@sha256:b4d092fd2b507843dd096fe6c85d06a27a0cbd740a0b32a880fe61aba24bb478
mount: /sys: permission denied.

[BenTheElder]

How was docker installed?
What other security mechanisms does this system have?
Can you do mounts in a container at all?
If you run the same thing but -it --entrypoint=/bin/bash can you do mount -l to list the mounts?

[adrianludwin]

How was docker installed?

I followed the instructions for Docker-on-Pixelbook (basically sudo apt-get install docker-ce), using the Debian repo (not Ubuntu). I didn't need any of the stuff about a blacklisted syscall - it Just Worked for me when I installed it.

What other security mechanisms does this system have?

I have no idea. AFAIK it's stock ChromeOS but it's managed by my employer (Google) so they may have some additional stuff on it.

Can you do mounts in a container at all?

I can do a bind mount, yes:

$ mkdir /mnt/test
$ mkdir /home/aludwin
$ echo hi > /mnt/test/foo
$ mount --bind /mnt/test /home/aludwin
$ cat /home/aludwin/foo
hi

My VM and container don't have any block devices other than vdb and I'm not sure how to mount that (mount: /home/aludwin: special device vdb does not exist.). Is this enough to answer your question?

If you run the same thing but -it --entrypoint=/bin/bash can you do mount -l to list the mounts?

Yup:

root@kind-control-plane:/# mount -l
/dev/vdb on / type btrfs (rw,relatime,discard,space_cache,user_subvol_rm_allowed,subvolid=379,subvol=/lxd/storage-pools/default/containers/penguin/rootfs/var/lib/docker/btrfs/subvolumes/e0cd2ec9e5417357be09b8d36540d00f596f0f0d127a346faba8f67096bbb990)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755,uid=1000000,gid=1000000)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=1000005,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,mode=755,uid=1000000,gid=1000000)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,name=systemd)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k,uid=1000000,gid=1000000)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noexec,relatime,uid=1000000,gid=1000000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,uid=1000000,gid=1000000)
/dev/vdb on /etc/resolv.conf type btrfs (rw,relatime,discard,space_cache,user_subvol_rm_allowed,subvolid=265,subvol=/lxd/storage-pools/default/containers/penguin/rootfs/var/lib/docker/containers/c7cf5109d3d26d6ae53bf527f2ed790d169a8d8e85edb6adb3fd137f22ae518b/resolv.conf)
/dev/vdb on /etc/hostname type btrfs (rw,relatime,discard,space_cache,user_subvol_rm_allowed,subvolid=265,subvol=/lxd/storage-pools/default/containers/penguin/rootfs/var/lib/docker/containers/c7cf5109d3d26d6ae53bf527f2ed790d169a8d8e85edb6adb3fd137f22ae518b/hostname)
/dev/vdb on /etc/hosts type btrfs (rw,relatime,discard,space_cache,user_subvol_rm_allowed,subvolid=265,subvol=/lxd/storage-pools/default/containers/penguin/rootfs/var/lib/docker/containers/c7cf5109d3d26d6ae53bf527f2ed790d169a8d8e85edb6adb3fd137f22ae518b/hosts)
/dev/vdb on /usr/lib/modules type btrfs (ro,relatime,discard,space_cache,user_subvol_rm_allowed,subvolid=265,subvol=/lxd/storage-pools/default/containers/penguin/rootfs/lib/modules)
/dev/vdb on /var/lib/containerd type btrfs (rw,relatime,discard,space_cache,user_subvol_rm_allowed,subvolid=265,subvol=/lxd/storage-pools/default/containers/penguin/rootfs/var/lib/docker/volumes/56ea3686b15c669ecb98b632b1fc1094e1c6bd2b5679f060ce74f4e48b4c457b/_data)
devtmpfs on /dev/null type devtmpfs (rw,nosuid,noexec,relatime,size=5572304k,nr_inodes=1393076,mode=755)
devtmpfs on /dev/random type devtmpfs (rw,nosuid,noexec,relatime,size=5572304k,nr_inodes=1393076,mode=755)
devtmpfs on /dev/full type devtmpfs (rw,nosuid,noexec,relatime,size=5572304k,nr_inodes=1393076,mode=755)
devtmpfs on /dev/tty type devtmpfs (rw,nosuid,noexec,relatime,size=5572304k,nr_inodes=1393076,mode=755)
devtmpfs on /dev/zero type devtmpfs (rw,nosuid,noexec,relatime,size=5572304k,nr_inodes=1393076,mode=755)
devtmpfs on /dev/urandom type devtmpfs (rw,nosuid,noexec,relatime,size=5572304k,nr_inodes=1393076,mode=755)
devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=1000005,mode=620,ptmxmode=666)

[BenTheElder]

xref: https://bugs.chromium.org/p/chromium/issues/detail?id=878034

[rosera]

Hey, I am highly interested in getting this working on ChromeOS (I raised the original ticket reference https://bugs.chromium.org/p/chromium/issues/detail?id=878034). Thanks very much for the tips regarding debugging - very helpful.

I plan to investigate a bit more over the coming weekend. My thoughts at the moment relate to the container being restricted on the accessible ports available to Crostini. K8s ports are not on the existing list of default ports. I believe you can enable them, so it might be trial and error for this to work. Beyond that there is also the use of a privileged container - which I understood was not supported on Crostini? This is more of an issue and one which I do not think will be enabled due to security concerns.

[BenTheElder]

Hi! :wave:

K8s ports are not on the existing list of default ports

you can configure kind to use a particular port with kind create cluster --config=config.yaml and config.yaml:

apiVersion: kind.sigs.k8s.io/v1alpha3
kind: Config
networking:
  apiServerPort: 6443

Beyond that there is also the use of a privileged container - which I understood was not supported on Crostini? This is more of an issue and one which I do not think will be enabled due to security concerns.

That's a bit more of a blocker.

Each kind "node" docker container is --privileged in order to run containerd / pods on the inside, each kubelet needs its own container runtime. To run pods we need proc, cgroups, mounting ...

It _might_ be possible to hack up kind to use usermode Linux instead but that's going to be fairly involved.

If I read the logs from @adrianludwin correctly though, we actually manage to create those containers, but it fails once we try to use mount inside them. Besides the re-mounting we do*, I believe kubelet needs permission to mount in order to setup pods etc.

* (changing /sys to read only, docker forces /sys to a read-write mount if you specify --privileged),

[BenTheElder]

dupe: https://github.com/kubernetes-sigs/kind/issues/1137

I don't really have the capacity to dig into this further at the moment, and I don't think any of our other usual contributors do. AFAICT, the chromeOS sandbox is too restrictive to run Kubernetes.

This is probably generally a good thing :upside_down_face: as kind is certainly not the most secure thing to be doing, but there's not much we can do here.

At this point I'm going to recommend that we add this to known issues and close out this bug in favor of the upstream chromeOS issue.

[cbandy]

If I understand correctly, that upstream bug is targetting minikube (rather than kind specifically) and also it might be ... done/terminal/finished/closed?

Status: Verified _(Closed)_

[BenTheElder]

I don't know if the upstream bug is actually fixed, but if you read through the comments the upstream bug expands to more generally trying to run _some_ form of kubernetes-like-thing including kind, minikube, k3s...

kind is not doing anything special for operating systems otherwise, we only require privileged docker containers and whatever kernel features kubernetes requires.