Kubevirt: Support for containerized libvirtd

[gageorsburn]

Not sure if kubevirt already supports this but what would you guys think about supporting containerized libvirtd for clusters running on atomic hosts?

I'd love to get involved with implementation.

[itamarh]

@gageorsburn - KubeVirt uses libvirt, running in a container (in a pod). Currently, each VM has its own libvirt, but we are working with the libvirt team on making it cleaner to use this approach
https://www.redhat.com/archives/libvir-list/2018-May/msg01307.html
Can you elaborate on the use case you are trying to solve?

[gageorsburn]

@itamarh Currently have a production kubernetes cluster rolling on Fedora Atomic 27.

[e-minguez]

It seems the libvirtd process is executed in the virt-launcher pod https://github.com/kubevirt/kubevirt/blob/master/pkg/virt-launcher/virtwrap/util/libvirt_helper.go#L120-L123
I think @gageorsburn talks about decoupling libvirtd and being able to execute as a standalone container right?

[rmohr]

It seems the libvirtd process is executed in the virt-launcher pod

Yes, we simply can't use libvirt - as it is right now - standalone (the qemu processes need to live inside the right namespaces and cgroups and we need exact control over how and when what is started, libvirt and k8s are also overlapping in what they think that they should control and setup). With the libvirt redesign which @itamarh referenced it may be possible to support a node-wide libvirt daemon. Does that add some useful context?

[fabiand]

@gageorsburn we actually had Atomic in mind (as a representative for other stateless or read-only nodes, like ClearLinux or CoreOS Container Linux).
Today there should not be any blockers which stop you from just deploying KubeVirt on a Kubernetes/Origin cluster based on Atomic.

Did you give it a try?

[gageorsburn]

I did and had some issues with qemu starting. I assumed it was because I was on Atomic without looking into it much further but I'll give it another look and let you guys know what I find.

[fabiand]

Cool! I'm happy to help with that.
Feel free to catch me on irc or slack (k8s), or just provide some logs/errors here.

[gageorsburn]

This is actually a really interesting behavior I'm having now and I was wondering if you had any recommendations for log files I could look at / share.

After my previous attempt to deploy the demo cirrus vm other day the pods went into a completed state. I deleted the pod expecting it to be rescheduled however it was not. I attempted to do virtctl start testvm but it stated that the vm was already running. I attempted a stop and a start again but no pod gets scheduled.

Use "virtctl options" for a list of global command-line options (applies to all commands).
 gageorsburn@mbp  ~  virtctl start testvm
Error: VirtualMachine 'testvm' is already running
 ✘ gageorsburn@mbp  ~  virtctl stop testvm
 gageorsburn@mbp  ~  virtctl start testvm
 gageorsburn@mbp  ~  kubectl get po
NAME                                READY     STATUS    RESTARTS   AGE
minio-deployment-85469f6d59-vl2qb   1/1       Running   0          10d
wordpress-55cbcdd99b-qr75b          1/1       Running   1          15d
wordpress-mysql-557ffc4f69-gwk2d    1/1       Running   0          9d
 gageorsburn@mbp  ~  kubectl get vms
NAME      AGE
testvm    1d
 gageorsburn@mbp  ~  kubectl get ovms
NAME      AGE
testvm    1d
 gageorsburn@mbp  ~  kubectl get ovms
NAME      AGE
testvm    1d
 gageorsburn@mbp  ~  virtctl start testvm
Error: VirtualMachine 'testvm' is already running
 ✘ gageorsburn@mbp  ~  kubectl get po
NAME                                READY     STATUS    RESTARTS   AGE
minio-deployment-85469f6d59-vl2qb   1/1       Running   0          10d
wordpress-55cbcdd99b-qr75b          1/1       Running   1          15d
wordpress-mysql-557ffc4f69-gwk2d    1/1       Running   0          9d
 gageorsburn@mbp  ~ 

Looking at some logs it seems like one of my virtapi servers is having TLS handshake errors while the other one is not. There are also 3 virt handler pods running one of them has no logs while the other two are constantly reporting:

E0525 12:47:20.353695    9893 reflector.go:205] kubevirt.io/kubevirt/pkg/virt-handler/vm.go:228: Failed to list *v1.VirtualMachine: the server could not find the requested resource (get virtualmachines.kubevirt.io)

This seems like a bug that we should research further. I do want to attempt to redeploy to fix my use case but I'd like to know the underlying issue that occurred. So if you have any logs you'd advise me to look at besides the pod logs I would appreciate it.

[rmohr]

Use "virtctl options" for a list of global command-line options (applies to all commands).
gageorsburn@mbp  ~  virtctl start testvm
Error: VirtualMachine 'testvm' is already running

Oh, that message is misleading. This means that the vm is enabled. The container will try to make sure that the vm runs.

E0525 12:47:20.353695 9893 reflector.go:205] kubevirt.io/kubevirt/pkg/virt-handler/vm.go:228: Failed to list *v1.VirtualMachine: the server could not find the requested resource (get virtualmachines.kubevirt.io)

That message alone does not mean much if you just see it once. We deploy all manifests, including the VirtualMachine custom resource at the same time. It can happen that our controllers start before the VirtualMachine Type is properly registered in the apiserver (up to seconds). You may therefore see the message printed a few times (but it should stop then).

In order to debug this, could you look into the k8s events and share the instances?

kubectl get events
kubect get vms -o yaml
kubectl get ovms -o yaml

There you should see warnings in the event log and errors attached to the object states ...

[rmohr]

E0525 12:47:20.353695 9893 reflector.go:205] kubevirt.io/kubevirt/pkg/virt-handler/vm.go:228: Failed to list *v1.VirtualMachine: the server could not find the requested resource (get virtualmachines.kubevirt.io)

Oh you said that this is repeatedly printed there and only for parts of the components ... So that is really weird ...

[gageorsburn]

 gageorsburn@mbp  ~  kubectl get events
LAST SEEN   FIRST SEEN   COUNT     NAME                                          KIND                    SUBOBJECT                               TYPE      REASON             SOURCE                             MESSAGE
24m         24m          2         testvm.1531e354ce43cfeb                       OfflineVirtualMachine                                           Normal    SuccessfulDelete   offlinevirtualmachine-controller   Deleted virtual machine: ca0d6117-5ee6-11e8-b256-0a94ef269b70
25m         25m          1         virt-launcher-testvm-x9ssn.1531e3480e35599d   Pod                     spec.containers{volumeregistryvolume}   Normal    Killing            kubelet, shed-compute01            Killing container with id docker://volumeregistryvolume:Need to kill Pod
 gageorsburn@mbp  ~ 

 gageorsburn@mbp  ~  kubectl get vms -o yaml
apiVersion: v1
items:
- apiVersion: kubevirt.io/v1alpha1
  kind: VirtualMachine
  metadata:
    annotations:
      presets.virtualmachines.kubevirt.io/presets-applied: kubevirt.io/v1alpha1
      virtualmachinepreset.kubevirt.io/small: kubevirt.io/v1alpha1
    clusterName: ""
    creationTimestamp: 2018-05-24T00:09:56Z
    deletionGracePeriodSeconds: 0
    deletionTimestamp: 2018-05-25T12:35:02Z
    finalizers:
    - foregroundDeleteVirtualMachine
    generateName: testvm
    generation: 0
    labels:
      guest: testvm
      kubevirt.io/nodeName: shed-compute01
      kubevirt.io/size: small
    name: testvm
    namespace: default
    ownerReferences:
    - apiVersion: kubevirt.io/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: OfflineVirtualMachine
      name: testvm
      uid: 9742ef51-5ee6-11e8-ac47-0a94ef269830
    resourceVersion: "7965748"
    selfLink: /apis/kubevirt.io/v1alpha1/namespaces/default/virtualmachines/testvm
    uid: ca0d6117-5ee6-11e8-b256-0a94ef269b70
  spec:
    domain:
      devices:
        disks:
        - disk:
            bus: virtio
          name: registrydisk
          volumeName: registryvolume
        - disk:
            bus: virtio
          name: cloudinitdisk
          volumeName: cloudinitvolume
      firmware:
        uuid: 5a9fc181-957e-5c32-9e5a-2de5e9673531
      machine:
        type: ""
      resources:
        requests:
          memory: 64M
    volumes:
    - name: registryvolume
      registryDisk:
        image: kubevirt/cirros-registry-disk-demo
    - cloudInitNoCloud:
        userDataBase64: SGkuXG4=
      name: cloudinitvolume
  status:
    interfaces:
    - ipAddress: 10.233.102.99
    nodeName: shed-compute01
    phase: Scheduled
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
 gageorsburn@mbp  ~ 

 gageorsburn@mbp  ~  kubectl get ovms -o yaml
apiVersion: v1
items:
- apiVersion: kubevirt.io/v1alpha1
  kind: OfflineVirtualMachine
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"kubevirt.io/v1alpha1","kind":"OfflineVirtualMachine","metadata":{"annotations":{},"name":"testvm","namespace":"default"},"spec":{"running":false,"selector":{"matchLabels":{"guest":"testvm"}},"template":{"metadata":{"labels":{"guest":"testvm","kubevirt.io/size":"small"}},"spec":{"domain":{"devices":{"disks":[{"disk":{"bus":"virtio"},"name":"registrydisk","volumeName":"registryvolume"},{"disk":{"bus":"virtio"},"name":"cloudinitdisk","volumeName":"cloudinitvolume"}]}},"volumes":[{"name":"registryvolume","registryDisk":{"image":"kubevirt/cirros-registry-disk-demo"}},{"cloudInitNoCloud":{"userDataBase64":"SGkuXG4="},"name":"cloudinitvolume"}]}}}}
    clusterName: ""
    creationTimestamp: 2018-05-24T00:08:30Z
    generation: 0
    name: testvm
    namespace: default
    resourceVersion: "7965981"
    selfLink: /apis/kubevirt.io/v1alpha1/namespaces/default/offlinevirtualmachines/testvm
    uid: 9742ef51-5ee6-11e8-ac47-0a94ef269830
  spec:
    running: true
    template:
      metadata:
        creationTimestamp: null
        labels:
          guest: testvm
          kubevirt.io/size: small
      spec:
        domain:
          devices:
            disks:
            - disk:
                bus: virtio
              name: registrydisk
              volumeName: registryvolume
            - disk:
                bus: virtio
              name: cloudinitdisk
              volumeName: cloudinitvolume
          machine:
            type: ""
          resources: {}
        volumes:
        - name: registryvolume
          registryDisk:
            image: kubevirt/cirros-registry-disk-demo
        - cloudInitNoCloud:
            userDataBase64: SGkuXG4=
          name: cloudinitvolume
  status:
    created: true
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
 gageorsburn@mbp  ~ 

That log message mentioned above is constant by the way. Both of them are reporting it even now.

[rmohr]

Oh you said that this is repeatedly printed there and only for parts of the components ... So that is really weird ...

I guess this is also why the pod does not get recreated. Could you have a look into the virt-controller logs too? It probably gets the same error about the unknown virtual machine type and therefore can't even get that for ...

[rmohr]

That log message mentioned above is constant by the way. Both of them are reporting it even now.

If you delete the pods with the recurring message (so that the daemonset can recreate them), does the error go away?

[gageorsburn]

 gageorsburn@mbp  ~  kubectl logs virt-controller-7f9bc564b6-sl2jx -n kube-system
level=info timestamp=2018-05-25T12:35:01.970439Z pos=ovm.go:135 component=virt-controller service=http namespace=default name=testvm kind= uid=9742ef51-5ee6-11e8-ac47-0a94ef269830 msg="Started processing OVM"
level=info timestamp=2018-05-25T12:35:01.971973Z pos=ovm.go:186 component=virt-controller service=http namespace=default name=testvm kind= uid=9742ef51-5ee6-11e8-ac47-0a94ef269830 msg="Creating or the VM: false"
level=info timestamp=2018-05-25T12:35:02.032634Z pos=ovm.go:321 component=virt-controller service=http namespace=default name=testvm kind= uid=9742ef51-5ee6-11e8-ac47-0a94ef269830 msg="Dispatching delete event"
level=info timestamp=2018-05-25T12:35:02.074915Z pos=ovm.go:135 component=virt-controller service=http namespace=default name=testvm kind= uid=9742ef51-5ee6-11e8-ac47-0a94ef269830 msg="Started processing OVM"
level=info timestamp=2018-05-25T12:35:02.074982Z pos=ovm.go:186 component=virt-controller service=http namespace=default name=testvm kind= uid=9742ef51-5ee6-11e8-ac47-0a94ef269830 msg="Creating or the VM: false"
level=info timestamp=2018-05-25T12:35:02.086452Z pos=ovm.go:321 component=virt-controller service=http namespace=default name=testvm kind= uid=9742ef51-5ee6-11e8-ac47-0a94ef269830 msg="Dispatching delete event"
level=info timestamp=2018-05-25T12:35:07.549367Z pos=ovm.go:135 component=virt-controller service=http namespace=default name=testvm kind= uid=9742ef51-5ee6-11e8-ac47-0a94ef269830 msg="Started processing OVM"
level=info timestamp=2018-05-25T12:35:07.587561Z pos=ovm.go:135 component=virt-controller service=http namespace=default name=testvm kind= uid=9742ef51-5ee6-11e8-ac47-0a94ef269830 msg="Started processing OVM"
 gageorsburn@mbp  ~ 

 gageorsburn@mbp  ~  kubectl logs virt-controller-7f9bc564b6-wm8v6 -n kube-system
 gageorsburn@mbp  ~ 

[gageorsburn]

I'll give that a try and see what happens.

[gageorsburn]

Interesting that seems to have fixed all issues, and it even rescheduled the VM pod. I'm not sure what caused that previous behavior.

[rmohr]

Interesting that seems to have fixed all issues, and it even rescheduled the VM pod. I'm not sure what caused that previous behavior.

Whatever caused it, thanks for reporting it. We need to improve the virt-handler heartbeat mechanism to cope with that. In your case the heartbeat-update still worked (so fetching und patching the nodes in the wrong virt-handlers still worked), so the controllers expected the node to be fully operational and did not clean up the vm.

[fabiand]

Closing this because the issue is resolved for @gageorsburn and we have a bug thanks to @rmohr

[fabiand]

@gageorsburn does that actually mean that you have KubeVirt now running on a few Atomic Hosts?

[gageorsburn]

That's correct. I haven't had a chance to interact with the VM yet due to travel today but all of the pods are running. I don't have any logs of the initial deploy but I'm sure I'll be deploying kubevirt a couple more times. If I run into an issue with it again I'll raise another issue with more detailed logs but all is well now.

Thanks guys for your quick response on this. This definitely seems like a community I want to be a part of/contribute to!