Runtime: minikube installation with kata-fc fails

[paul-snively]

Description of problem

Following the installation guide for minikube, at least with a certain configuration, results in a test pod stuck in ContainerCreating.

psnively@oryx-pro:~|⇒  cd packaging/kata-deploy 
psnively@oryx-pro:~/packaging/kata-deploy|master 
⇒  minikube delete
! "minikube" cluster does not exist
! "minikube" profile does not exist
psnively@oryx-pro:~/packaging/kata-deploy|master 
⇒  minikube start --bootstrapper=kubeadm --container-runtime=containerd --enable-default-cni --memory 16384 --network-plugin=cni --vm-driver kvm2 --feature-gates=RuntimeClass=true --cpus 4 --disk-size 50G --kubernetes-version 1.13.7
* minikube v1.2.0 on linux (amd64)
* Creating kvm2 VM (CPUs=4, Memory=16384MB, Disk=50000MB) ...
* Configuring environment for Kubernetes v1.13.7 on containerd 1.2.5
* Pulling images ...
* Launching Kubernetes ... 
* Verifying: apiserver etcd scheduler controller
* Done! kubectl is now configured to use "minikube"
psnively@oryx-pro:~/packaging/kata-deploy|master 
⇒  kubectl apply -f kata-rbac.yaml 
serviceaccount/kata-label-node created
clusterrole.rbac.authorization.k8s.io/node-labeler created
clusterrolebinding.rbac.authorization.k8s.io/kata-label-node-rb created
psnively@oryx-pro:~/packaging/kata-deploy|master 
⇒  kubectl apply -f kata-deploy.yaml 
daemonset.apps/kata-deploy created
psnively@oryx-pro:~/packaging/kata-deploy|master 
⇒  kubectl apply -f k8s-1.13/runtimeclass-crd.yaml 
customresourcedefinition.apiextensions.k8s.io/runtimeclasses.node.k8s.io created
psnively@oryx-pro:~/packaging/kata-deploy|master 
⇒  kubectl apply -f k8s-1.13/kata-fc-runtimeClass.yaml 
runtimeclass.node.k8s.io/kata-fc created
psnively@oryx-pro:~/packaging/kata-deploy|master 
⇒  kubectl apply -f examples/test-deploy-kata-fc.yaml 
deployment.apps/php-apache-kata-fc created
service/php-apache-kata-fc created
psnively@oryx-pro:~/packaging/kata-deploy|master 
⇒  kubectl get pods
NAME                                  READY   STATUS              RESTARTS   AGE
php-apache-kata-fc-6c6f484c4b-xlv89   0/1     ContainerCreating   0          17s
psnively@oryx-pro:~/packaging/kata-deploy|master 
⇒  kubectl describe php-apache-kata-fc-6c6f484c4b-xlv89
error: the server doesn't have a resource type "php-apache-kata-fc-6c6f484c4b-xlv89"
psnively@oryx-pro:~/packaging/kata-deploy|master 
⇒  kubectl describe pod/php-apache-kata-fc-6c6f484c4b-xlv89
Name:           php-apache-kata-fc-6c6f484c4b-xlv89
Namespace:      default
Priority:       0
Node:           minikube/192.168.122.43
Start Time:     Sun, 28 Jul 2019 17:57:30 -0400
Labels:         pod-template-hash=6c6f484c4b
                run=php-apache-kata-fc
Annotations:    <none>
Status:         Pending
IP:             
Controlled By:  ReplicaSet/php-apache-kata-fc-6c6f484c4b
Containers:
  php-apache:
    Container ID:   
    Image:          k8s.gcr.io/hpa-example
    Image ID:       
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:        200m
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-g8kbn (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-g8kbn:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-g8kbn
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                  Age   From               Message
  ----     ------                  ----  ----               -------
  Normal   Scheduled               44s   default-scheduler  Successfully assigned default/php-apache-kata-fc-6c6f484c4b-xlv89 to minikube
  Warning  FailedCreatePodSandBox  23s   kubelet, minikube  Failed create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container: failed to create containerd task: rootfs (/run/kata-containers/shared/containers/9b9a1fbccac7b764c12fed12c4b13d220171d4011be902d6e7460ed41818ee10/rootfs) does not exist: unknown

Expected result

The test pod would run successfully.

Actual result

The pod is stuck per the above output.

You may be wondering, "Why Kubernetes 1.13.7 and containerd?" The answer is, that's the closest approximation to what we can get in GKE today.

I'm attaching the result of SSHing into minikube and sudo env "PATH=/opt/kata/bin:$PATH" /opt/kata/bin/kata-collect-data.sh > kata.log in case that helps.

kata.log

[jodh-intel]

Thanks for raising @paul-snively. For reference, the minikube install doc is:

• https://github.com/kata-containers/documentation/blob/master/install/minikube-installation-guide.md

@grahamwhaley - any thoughts? I really think we need to find a way to make that doc "executable" to ensure it works in all scenarios.

[grahamwhaley]

Hi @paul-snively - I have a strong suspicion this is not related to your version of k8s or containerd :-) You may note in the minikube kata guide, that kata-fc/firecracker is not mentioned - because I have not tried it, and I suspect it does not (currently) work under minikube. I just tried it locally with default minikube v1.1.1, and I can launch a kata-qemu, but a kata-fc gets stuck in ContainerCreating.

My strong suspicion is that the minikube kernel config probably has some items missing that are required by kata-fc. Much like we had on https://github.com/kubernetes/minikube/pull/4340

Let me see if I can identify any missing configs.
/cc @amshinde @devimc who have dabbled in the minikube kernel config and kata-fc kernel config areas...

[grahamwhaley]

For reference then, over on https://github.com/kata-containers/packaging/pull/657/commits/364f4251a65428c363c2ac166bd3afad534fd00f @devimc added:

# x86 specific mmio related items

# Next config are required for firecracker
CONFIG_VIRTIO_MMIO=y
CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y

but I don't see them in the minikube kernel def config over at https://github.com/kubernetes/minikube/blob/master/deploy/iso/minikube-iso/board/coreos/minikube/linux_defconfig

@devimc - minikube is running kernel v4.15 - is that new enough to support fc features do you know?

[paul-snively]

Thanks for the rapid response, gentlemen!

Another data point:

psnively@oryx-pro:~|⇒  sudo -E podman run -it --runtime=/opt/kata/bin/kata-fc alpine sh
[sudo] password for psnively:              
Error: rpc error: code = Unknown desc = rootfs (/run/kata-containers/shared/containers/b5bf05289f6b65160ade4f3770024dc7d481a73e25c474ca621f40ceda891fe0/rootfs) does not exist
: OCI runtime error

This is with kata-containers 1.8.0.

I would be comfortable bailing on minikube in favor of MicroK8s, in all honesty. The idea is to have easy local development in an environment similar to a real cloud deployment, and to be able to run in a CI/CD environment.

So far, I have to be honest: the Firecracker experience is completely underwhelming. I have yet to get it to run anywhere. With nemu nipping at its heels (including with kata-containers), I'm beginning to seriously question whether Firecracker is worth it.

Thanks for all your help regardless!

[chavafg]

I think this is because of block devices not being supported by containerd. iirc, we are waiting for containerd to have a release with the devmapper snapshotter from @ganeshmaharaj which should make this work.

/cc @ganeshmaharaj @egernst

[grahamwhaley]

@chavafg beat me to it - must have slow brain today - I just remembered we have a (slim, and not that easy to find :-( ) list of kata-fc requirements:

https://github.com/kata-containers/documentation/wiki/Initial-release-of-Kata-Containers-with-Firecracker-support#pre-requisuites

which pretty much lists virtio and block device (devicemapper) support...

[devimc]

@paul-snively I'm curious, why firecracker and not QEMU?

[grahamwhaley]

ah, microk8s - I thought that might be the cluster based on docker images (for which I believe we could not then run kata, as kata-under-docker is a bit of a no no I think) - but, no, mickok8s is a snap of all the bits of k8s you need, so an isolated k8s local cluster. I still suspect kata under microk8s might be a non-starter, as I'm not sure we can install kata (/opt/kata/*) inside that snap and then have it (qemu/kvm in particular) access the host to set up the VMs. But, @devimc would probably have more snap knowledge around kata there than anybody else. Just curious and contemplating other 'kata quick start eval platforms' beyond minikube and https://github.com/clearlinux/cloud-native-setup/tree/master/clr-k8s-examples.

[devimc]

@grahamwhaley you're right , microk8s is based on docker images, you can run kata containers but network won't work :( , @paul-snively you can integrate the kata snap with an already running instance of k8s, but currently we don't include firecracker binary in the snap

[paul-snively]

@grahamwhaley: Thanks! I'd actually found those prereq's and, I suppose naïvely, believed miniube met them.

@chavafg:

I think this is because of block devices not being supported by containerd. iirc, we are waiting for containerd to have a release with the devmapper snapshotter from @ganeshmaharaj which should make this work.

/cc @ganeshmaharaj @egernst

Good catch! From firecracker-containerd, it looks like you're correct.

@devimc:

@paul-snively I'm curious, why firecracker and not QEMU?

Startup time, footprint, security. Making OSv Run on Firecracker has some interesting details.

A related effort is NEMU. Firecracker and NEMU are both forks of QEMU—Firecracker from AWS, NEMU from Intel—that take the maturity of QEMU and strip it _way_ down to make a cloud-focused hypervisor. So far, NEMU seems a lot less finicky than Firecracker, and kata-containers already supports it, too. But poking around at the CRI ecosystem is revealing a stunning lack of maturity at this point, and has me thinking I need to allocate my innovation budget more carefully, which has me thinking about unikernel approaches possibly based on OSv, which would still be a big innovation tax bite, or Rumprun, which has the advantage of basically being a bunch of _very_ mature NetBSD drivers in userspace.

My intended app development approach is to use GraalVM's native-image to build Scala code and deploy it to Kubernetes in the fastest, lowest-overhead profile I can. This approach is explicitly supported by OSv. I thought kata-containers with Firecracker might be it, but Firecracker seems like a royal pain, especially with containerd, so I may just go the unikernel route and rely on virtlet instead.

[paul-snively]

Gentlemen, thank you all for the invaluable feedback. I'd like to reiterate https://github.com/kata-containers/runtime/issues/1915#issuecomment-515974419 before I close this ticket, in case there's anything more for me to learn there. My understanding at the moment, though, is that if I wish to use kata-containers with Firecracker in Kubernetes today, my best bet is to rely on firecracker-containerd, which may entail replacing any existing containerd in the cluster with firecracker-containerd's custom build. In addition, the storage driver must be a block driver and the host must support virtio. Does this accurately summarize the situation?

[egernst]

No, it doesn’t summarize :)

You need block based snapshotter, devmapper, which is in upstream containerd, but not yet released in containerd packages (targeting 1.3 AFAIU). So, you simply need to build containerd from source (very straight forward) and configure it to use the devmapper snapshotter.

[paul-snively]

No, it doesn’t summarize :)

You need block based snapshotter, devmapper, which is in upstream containerd, but not yet released in containerd packages (targeting 1.3 AFAIU). So, you simply need to build containerd from source (very straight forward) and configure it to use the devmapper snapshotter.

Forgive me, but it's rather clearly not that simple, along at least a few dimensions:

1. Suppose I built containerd from source as you suggest. How would I go about installing this containerd in minikube?
2. The firecracker-containerd quick start explains carefully that a container expecting to be run via firecracker-containerd must include the firecracker-containerd agent and runC. I haven't yet seen an explanation as to how to ensure kata-containers-fc containers include these necessary components.
3. The documentation does note that the snapshotter code has been contributed upstream, but gives no indication that that eventuality will subsume the instructions provided—and, given the firecracker-containerd architecture, it's far from obvious how it possibly could.

So I continue to maintain that I've identified a gap, and possibly an insurmountable one, given minikube's architecture (e.g. relying on a vendored-in containerd). By way of contrast, k3s supports a --container-runtime-endpoint argument to its server command. Given its minimalistic design, small footprint, and --rootless mode, it also becomes appealing for use in CI/CD environments, so I'll likely refocus my efforts there.

Thanks again, everyone, for the information. I believe I've gathered enough to reach closure on the issue, so I'll close the ticket.

[jodh-intel]

Hi @paul-snively - thanks for the detailed summary. I'm going to re-open as it sounds like there are still outstanding questions here.

@erick0z - I can't find any mention of devmapper or snapshotter in any of the Kata documentation. Should we move https://github.com/kata-containers/documentation/wiki/Initial-release-of-Kata-Containers-with-Firecracker-support into a formal doc to help others do you think?

@mcastelino, @grahamwhaley - any thoughts on this (including k3s)?

@paul-snively - if you get Kata working with k3s, please do let us know as we'd love to add a howto doc to our https://github.com/kata-containers/documentation repo! ;)

[jodh-intel]

/me has fun twiddling with GitHub labels :smile:

[jodh-intel]

Oops - sorry @erick0z! I meant to ping @egernst :)

[grahamwhaley]

Unless there is a supported and easy(ish) way to get a block based graph driver into minikube, then kata-fc will not be working there. We should probably note that clearly on the kata minikube install docs (sure, I'll go PR it..)

[jodh-intel]

Thanks @grahamwhaley. I do think we need a lot more doc in this area (and yes, I know it's going [to have to] change frequently, but we still need it ;)

[jodh-intel]

Adding a ref to @grahamwhaley's PR to explain this: https://github.com/kata-containers/documentation/pull/527.

[ganeshmaharaj]

@grahamwhaley @jodh-intel @paul-snively i wonder if we can use a stand-alone snapshotter (shameless plug 😁 https://github.com/ganeshmaharaj/lvm-snapshotter) would that help get past the need for a block based snapshotter with current containerd releases that are used by minikube?