Kubevirt: VM with Datavolume in Block mode not starting

Created on 27 Jun 2019 · 8Comments · Source: kubevirt/kubevirt

/kind bug

What happened:
Define a VM with a DataVolume in Block mode and the VM failed to start.

The virt launcher pod stays in ContainerCreating state with the following error:

Warning FailedMount 28s (x2 over 2m43s) kubelet, node01 Unable to mount volumes for pod "virt-launcher-vm-cirros-vmd4n_default(1eceea32-98b7-11e9-97da-525500d15501)": timeout expired waiting for volumes to attach or mount for pod "default"/"virt-launcher-vm-cirros-vmd4n". list of unmounted volumes=[dv-disk]. list of unattached volumes=[dv-disk infra-ready-mount virt-share-dir libvirt-runtime ephemeral-disks]

What you expected to happen:
Define a VM with a DataVolume in Block mode and the VM starts.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

KubeVirt version (use virtctl version):
Client Version: version.Info{GitVersion:"v0.17.4", GitCommit:"adfdb8c07830b99fc79d2fd1d004e862ef70979e", GitTreeState:"clean", BuildDate:"2019-06-25T11:44:21Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{GitVersion:"v0.17.4", GitCommit:"adfdb8c07830b99fc79d2fd1d004e862ef70979e", GitTreeState:"clean", BuildDate:"2019-06-25T12:49:03Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:08:12Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:00:57Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
VM or VMI specifications:
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
labels:
kubevirt.io/vm: vm-cirros
name: vm-cirros
spec:
dataVolumeTemplates:
- metadata:
  creationTimestamp: null
  name: cirros-dv
  spec:
  pvc:
  volumeMode: Block
  accessModes:
  - ReadWriteOnce
    
    resources:
    
    requests:
    
    storage: 500Mi
    
    storageClassName: csi-rbd
    
    source:
    
    http:
    
    url: "https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img"
    
    status: {}
    
    running: true
    
    template:
    
    metadata:
    
    labels:
    
    kubevirt.io/vm: vm-cirros
    
    spec:
    
    domain:
    
    devices:
    
    disks:
    - disk:
      
      bus: virtio
      
      name: dv-disk
    - disk:
      
      bus: virtio
      
      name: cloudinitdisk
      
      machine:
      
      type: ""
      
      resources:
      
      requests:
      
      memory: 256Mi
      
      terminationGracePeriodSeconds: 0
      
      volumes:
  - cloudInitNoCloud:
    
    userData: |
    
    #!/bin/sh
    
    echo 'printed from cloud-init userdata'
    
    name: cloudinitdisk
  - name: dv-disk
    
    dataVolume:
    
    name: cirros-dv
Cloud provider or hardware configuration:
OS (e.g. from /etc/os-release):
Kernel (e.g. uname -a):
Install tools:
Others:

sistorage

Source

rollandf

All 8 comments

This looks very similar to #2272 and kubernetes/kubernetes#74787

vladikr on 27 Jun 2019

This looks very similar to #2272 and kubernetes/kubernetes#74787

Not sure, that's specifically an iSCSI logout bug in K8s. I've been running rbd block mode volume VMs for quite a while now. I haven't tried using templates though. I can take a look at this and see if I can figure out what's going on.

j-griffith on 27 Jun 2019

/assign j-griffith

j-griffith on 27 Jun 2019

I see the issue if the DV is creating a block mode PVC.
Same thing if I create first the DV (block mode), then reference the DV by name in the VM yaml.

But, if I create the DV(blovk mode) first, then reference the PVC by name in the VM yaml, it will work.

rollandf on 28 Jun 2019

I see the issue if the DV is creating a block mode PVC.
Same thing if I create first the DV (block mode), then reference the DV by name in the VM yaml.

But, if I create the DV(blovk mode) first, then reference the PVC by name in the VM yaml, it will work.

@rollandf Yeah, I dropped off a while on vacation but I'm seeing the same thing, using DVs seems to be the root issue for me. I also did a sanity check on a local bock device with similar results.

j-griffith on 9 Jul 2019

Just hit this problem as well when experimenting with ceph-csi/rbd. It seems the PVC from the DV is assumed to be a VolumeMode=Filesystem, which is obviously not the case for raw block devices.

osdisk is the raw block volume, oc describe pod/virt-launcher-... shows that the PVC can not be mounted (FailedMount with timeout expired waiting for volumes to attach or mount for pod). The following mounts (including osdisk) have been configured:

    Mounts:
      /var/run/kubevirt from virt-share-dir (rw)
      /var/run/kubevirt-ephemeral-disks from ephemeral-disks (rw)
      /var/run/kubevirt-infra from infra-ready-mount (rw)
      /var/run/kubevirt-private/vmi-disks/osdisk from osdisk (rw)
      /var/run/libvirt from libvirt-runtime (rw)

Expected to see osdisk as device:

    Mounts:
      /var/run/kubevirt from virt-share-dir (rw)
      /var/run/kubevirt-ephemeral-disks from ephemeral-disks (rw)
      /var/run/kubevirt-infra from infra-ready-mount (rw)
      /var/run/libvirt from libvirt-runtime (rw)
    Devices:
      /dev/osdisk from osdisk

The workaround is to use the PVC (populated through the DV) in the VMI, instead of using the DV in the VMI.