Screwdriver: Evaluate Kata Containers over HyperD

[stjohnjohnson]

Looks like that team just partnered with Intel and came out with this: https://katacontainers.io

We should evaluate this and look to see if it could replace our HyperD setup.

[minz1027]

Looks very promising! Couldn't find any docs on how to set it up though. Should definitely try this out when they release the binaries and have more docs.

[tkyi]

Evaluation completed and documented in https://github.com/screwdriver-cd/screwdriver/issues/1258#issuecomment-415839125

[catto]

The website of HyperContainer used by executor-k8s-vm is gone. http://hypercontainer.io/
We should migrate the vm container runner to kata containers as soon as possible. I have started evaluating how it works with Screwdriver's executor.

[minz1027]

@catto, we have evaluated kata around September 2018, sorry the doc is in internal git...
But basically we got the the Kubernetes integration work with cri-o and kata. Here are some issues we found while testing.

• containers created with kata runtime not showing up with sudo crictl ps but if you get the container id from pod description and do sudo circtl inspect id, you will be able to see it.

• cannot install / run docker inside kata, at least no one succeeded for now: https://github.com/kata-containers/runtime/issues/358

Let me know if you guys have questions during the the setup. I may be able to help! And hope kata improves and you guys can get the experiment further! And good news they have a plan to document how to run docker in kata now: https://github.com/kata-containers/documentation/issues/414

[catto]

@minz1027 Thanks for your information. We have already confirmed we can run docker on kata containers with ContainerLinux and docker-ce. Here is an instruction that enables overlayfs support for a guest kernel though it's written in Japanese. Additional instructions may needed and it isn't documented yet.
https://github.com/suzutan/kata-overlayfs-kernel/
I think virtio-fs support is also required because disk access in a vm container is really slow. Evaluation above was in an environment with ramfs mounted on /var/lib/docker.

[minz1027]

@catto Nice job! Really happy to hear that! Have you guys also tried to integrate kata with kubernetes directly? When we were doing the experiment, we did the docker integration first to set the docker runtime to kata and hope kubernetes would work out of box. But it didn't work, sigh... So we had to install cri and change the kubernetes run time from docker to cri to make it to work. Basically what they document here: https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md

Even though the process is kinda different, but i think the guest kernel fix is the key to run docker. Hope to hear further progress from your side! 💃

[jithine]

kata 1.7.0 has been released https://github.com/kata-containers/runtime/releases we should re-evaluate docker in kata

[catto]

I have tried using docker in kata and here are the results.

Docker daemon in kata container

Docker daemon uses iptables though it won't be provided for standard containers. kata containers are VM actually, but it requires privileged container to run docker daemon in kata even if the container doesn't modify the host environment which is not recommended.
https://github.com/kata-containers/documentation/blob/master/Limitations.md#host-resource-sharing

If I run dockerd with --iptables=False option to avoid using iptables in an non-privileged kata container, it fails with this error.
Error starting daemon: Error initializing network controller: Error creating default "bridge" network: operation not permitted

Overlay fs in kata container

Kata containers run on a volume created by docker storage plugin. I believe that current standard plugin is overlay2 thus root volume of kata containers are mounted using overlay.

bash-4.2# mount
overlay on / type overlay (snip)

In this environment, dockerd also tries to use overlay2 to start the daemon and it fails because of overlay on overlay which isn't supported. Other storage plugins such as devicemapper and vfs shoudn't be used. vfs has less performance than overlay.
Note: If I specify devicemapper for dockerd in kata container, it fails with a following message.
Error starting daemon: error initializing graphdriver: driver not supported.
I think it is required to rebuild kernel with devicemapper enabled but I haven't tried yet.

One solution for this issue is mounting /tmp/foo_$uniqname to /var/lib/docker so that the volume used for container will be tmpfs. It requires large memory for this solution by the way.

[minz1027]

updates 06/19

• Security concern for privileged mode (required to run docker daemon in kata)
  Privileged mode for kata will grant access not only to the guest kernel but also a subset of devices from the base host which can be a security concern. From the open issue, ppl requested kata team to have a feature to only allow access to guest kernel, but it doesn't look like they will work on it now.
  https://github.com/kata-containers/runtime/issues/1568

• Overlay over Overlay issues
  It looks like the disk performance issue is because they use 9p filesystem for /var/lib/docker. Kata team has fixed that by adding virtio-fs support (alternative to virtio-9p) https://github.com/kata-containers/runtime/pull/1016

[jithine]

https://blog.screwdriver.cd/post/619378783194071041/kata-containers-in-screwdriver