Kubevirt: Cannot start VM with GPU attached due to permissions error in /sys

[munnerz]

Is this a BUG REPORT or FEATURE REQUEST?:

(for lack of a better label for something that could definitely be user error!)
/kind bug

What happened:

I'm trying to make use of the GPU passthrough feature in KubeVirt, in combination with https://github.com/NVIDIA/kubevirt-gpu-device-plugin/.

I am able to boot VMs just fine, however, when adding the following to my devices config:

          gpus:
          - deviceName: nvidia.com/GK106_GeForce_GTX_650_Ti
            name: gpu1

my VMs is unable to start. Checking the virt-launcher logs, I see:

{"component":"virt-launcher","level":"error","msg":"internal error: Unable to get DBus system bus connection: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory","pos":"virDBusGetSystemBus:108","subcomponent":"libvirt","thread":"37","timestamp":"2020-04-20T09:19:54.221000Z"}
{"component":"virt-launcher","level":"error","msg":"Failed to open config space file '/sys/bus/pci/devices/0000:06:00.0/config': Permission denied","pos":"virPCIDeviceConfigOpen:303","subcomponent":"libvirt","thread":"37","timestamp":"2020-04-20T09:19:54.222000Z"}
{"component":"virt-launcher","kind":"","level":"error","msg":"Starting the VirtualMachineInstance failed.","name":"windows-10","namespace":"vmlab","pos":"manager.go:1134","reason":"virError(Code=38, Domain=0, Message='Failed to open config space file '/sys/bus/pci/devices/0000:06:00.0/config': Permission denied')","timestamp":"2020-04-20T09:19:54.223410Z","uid":"80c2c1c8-d340-44eb-a16c-5a1e93d861cb"}
{"component":"virt-launcher","level":"info","msg":"kubevirt domain status: Shutoff(5):Unknown(0)","pos":"client.go:180","timestamp":"2020-04-20T09:19:54.223476Z"}
{"component":"virt-launcher","kind":"","level":"error","msg":"Failed to sync vmi","name":"windows-10","namespace":"vmlab","pos":"server.go:161","reason":"virError(Code=38, Domain=0, Message='Failed to open config space file '/sys/bus/pci/devices/0000:06:00.0/config': Permission denied')","timestamp":"2020-04-20T09:19:54.223483Z","uid":"80c2c1c8-d340-44eb-a16c-5a1e93d861cb"}
{"component":"virt-launcher","level":"info","msg":"7) OSINFO IN EVENT CALLBACK: \u003cnil\u003e","pos":"client.go:200","timestamp":"2020-04-20T09:19:54.225349Z"}
{"component":"virt-launcher","level":"info","msg":"Detected domain with UUID 46c7c5a8-3f38-5dd6-8ff8-29c71219d063","pos":"virt-launcher.go:250","timestamp":"2020-04-20T09:19:54.225519Z"}
{"component":"virt-launcher","level":"info","msg":"Monitoring loop: rate 1s start timeout 5m0s","pos":"monitor.go:237","timestamp":"2020-04-20T09:19:54.225546Z"}
{"component":"virt-launcher","level":"info","msg":"Domain name event: vmlab_windows-10","pos":"client.go:255","timestamp":"2020-04-20T09:19:54.226348Z"}
{"component":"virt-launcher","level":"error","msg":"Failed to open config space file '/sys/bus/pci/devices/0000:06:00.0/config': Permission denied","pos":"virPCIDeviceConfigOpen:303","subcomponent":"libvirt","thread":"36","timestamp":"2020-04-20T09:19:54.326000Z"}
{"component":"virt-launcher","level":"error","msg":"Failed to open config space file '/sys/bus/pci/devices/0000:06:00.1/config': Permission denied","pos":"virPCIDeviceConfigOpen:303","subcomponent":"libvirt","thread":"36","timestamp":"2020-04-20T09:19:54.326000Z"}
{"component":"virt-launcher","level":"info","msg":"DomainLifecycle event 0 with reason 1 received","pos":"client.go:280","timestamp":"2020-04-20T09:19:54.332552Z"}
{"component":"virt-launcher","level":"error","msg":"internal error: Unable to get DBus system bus connection: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory","pos":"virDBusGetSystemBus:108","subcomponent":"libvirt","thread":"36","timestamp":"2020-04-20T09:19:54.336000Z"}
{"component":"virt-launcher","level":"error","msg":"Failed to open config space file '/sys/bus/pci/devices/0000:06:00.0/config': Permission denied","pos":"virPCIDeviceConfigOpen:303","subcomponent":"libvirt","thread":"36","timestamp":"2020-04-20T09:19:54.336000Z"}
{"component":"virt-launcher","kind":"","level":"error","msg":"Starting the VirtualMachineInstance failed.","name":"windows-10","namespace":"vmlab","pos":"manager.go:1134","reason":"virError(Code=38, Domain=0, Message='Failed to open config space file '/sys/bus/pci/devices/0000:06:00.0/config': Permission denied')","timestamp":"2020-04-20T09:19:54.337612Z","uid":"80c2c1c8-d340-44eb-a16c-5a1e93d861cb"}
{"component":"virt-launcher","level":"info","msg":"kubevirt domain status: Shutoff(5):Failed(6)","pos":"client.go:180","timestamp":"2020-04-20T09:19:54.337650Z"}
{"component":"virt-launcher","level":"info","msg":"7) OSINFO IN EVENT CALLBACK: \u003cnil\u003e","pos":"client.go:200","timestamp":"2020-04-20T09:19:54.337678Z"}
{"component":"virt-launcher","kind":"","level":"error","msg":"Failed to sync vmi","name":"windows-10","namespace":"vmlab","pos":"server.go:161","reason":"virError(Code=38, Domain=0, Message='Failed to open config space file '/sys/bus/pci/devices/0000:06:00.0/config': Permission denied')","timestamp":"2020-04-20T09:19:54.337694Z","uid":"80c2c1c8-d340-44eb-a16c-5a1e93d861cb"}
{"component":"virt-launcher","level":"info","msg":"Domain name event: vmlab_windows-10","pos":"client.go:255","timestamp":"2020-04-20T09:19:54.338618Z"}
{"component":"virt-launcher","level":"error","msg":"Failed to open config space file '/sys/bus/pci/devices/0000:06:00.0/config': Permission denied","pos":"virPCIDeviceConfigOpen:303","subcomponent":"libvirt","thread":"35","timestamp":"2020-04-20T09:19:54.420000Z"}
{"component":"virt-launcher","level":"error","msg":"Failed to open config space file '/sys/bus/pci/devices/0000:06:00.1/config': Permission denied","pos":"virPCIDeviceConfigOpen:303","subcomponent":"libvirt","thread":"35","timestamp":"2020-04-20T09:19:54.420000Z"}
{"component":"virt-launcher","level":"info","msg":"DomainLifecycle event 0 with reason 1 received","pos":"client.go:280","timestamp":"2020-04-20T09:19:54.427172Z"}
{"component":"virt-launcher","level":"error","msg":"internal error: Unable to get DBus system bus connection: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory","pos":"virDBusGetSystemBus:108","subcomponent":"libvirt","thread":"35","timestamp":"2020-04-20T09:19:54.431000Z"}
{"component":"virt-launcher","level":"error","msg":"Failed to open config space file '/sys/bus/pci/devices/0000:06:00.0/config': Permission denied","pos":"virPCIDeviceConfigOpen:303","subcomponent":"libvirt","thread":"35","timestamp":"2020-04-20T09:19:54.431000Z"}
{"component":"virt-launcher","kind":"","level":"error","msg":"Starting the VirtualMachineInstance failed.","name":"windows-10","namespace":"vmlab","pos":"manager.go:1134","reason":"virError(Code=38, Domain=0, Message='Failed to open config space file '/sys/bus/pci/devices/0000:06:00.0/config': Permission denied')","timestamp":"2020-04-20T09:19:54.432304Z","uid":"80c2c1c8-d340-44eb-a16c-5a1e93d861cb"}
{"component":"virt-launcher","level":"info","msg":"kubevirt domain status: Shutoff(5):Failed(6)","pos":"client.go:180","timestamp":"2020-04-20T09:19:54.432354Z"}
{"component":"virt-launcher","level":"info","msg":"7) OSINFO IN EVENT CALLBACK: \u003cnil\u003e","pos":"client.go:200","timestamp":"2020-04-20T09:19:54.432383Z"}
{"component":"virt-launcher","kind":"","level":"error","msg":"Failed to sync vmi","name":"windows-10","namespace":"vmlab","pos":"server.go:161","reason":"virError(Code=38, Domain=0, Message='Failed to open config space file '/sys/bus/pci/devices/0000:06:00.0/config': Permission denied')","timestamp":"2020-04-20T09:19:54.432380Z","uid":"80c2c1c8-d340-44eb-a16c-5a1e93d861cb"}
{"component":"virt-launcher","level":"info","msg":"Domain name event: vmlab_windows-10","pos":"client.go:255","timestamp":"2020-04-20T09:19:54.433574Z"}

After digging into this a bit more, I can confirm that I am able to read that file from within the virt-launcher pod (as well as virt-handler, although I'm not sure if that is relevant). When attempting to write to that file from within the container however, I do get a "permission denied" error (doing the same thing from the host does not result in an error).

As a last-resort measure, I also tried chmoding the named file (and even its entire parent directory) to 777, to no avail.

What you expected to happen:

Any appropriate permissions are set correctly and the VM starts.

How to reproduce it (as minimally and precisely as possible):

Happy to try and provide this information, although given nobody else has mentioned this issue I suspect that the steps I'm using here will not be particularly helpful.

Anything else we need to know?:

I have performed passthrough using this motherboard/CPU/graphics card in the past (just not in a container, and not using kubevirt).

Environment:

• KubeVirt version (use virtctl version): v0.28.0 (also tried with v0.26.x)
• Kubernetes version (use kubectl version): v1.18.2
• VM or VMI specifications:

apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
  labels:
    app: windows-10
    kubevirt.io/os: linux
  name: windows-10
  namespace: vmlab
spec:
  running: false
  template:
    metadata:
      labels:
        app: windows-10
    spec:
      domain:
        cpu:
          cores: 4
        devices:
          disks:
          - bootOrder: 1
            disk:
              bus: virtio
            name: rootfs
          - bootOrder: 2
            cdrom:
              bus: sata
              readonly: true
            name: installer
          - cdrom:
              bus: sata
            name: virtiocontainerdisk
          gpus:
          - deviceName: nvidia.com/GK106_GeForce_GTX_650_Ti
            name: gpu1
        firmware:
          bootloader:
            efi: {}
          uuid: 46c7c5a8-3f38-5dd6-8ff8-29c71219d063
        machine:
          type: q35
        resources:
          requests:
            memory: 4096M
      volumes:
      - name: rootfs
        persistentVolumeClaim:
          claimName: windows-10-rootfs
      - name: installer
        persistentVolumeClaim:
          claimName: windows-10-installer
      - containerDisk:
          image: kubevirt/virtio-container-disk
        name: virtiocontainerdisk

• Cloud provider or hardware configuration: bare metal (AMD CPU & motherboard, AM3+)
• OS (e.g. from /etc/os-release): Debian 10 (buster)
• Kernel (e.g. uname -a): Linux kube 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1 (2020-01-26) x86_64 GNU/Linux
• Install tools: kubeadm
• Others:

[munnerz]

After rebuilding the virt-controller image to set privileged: true on the launcher pods, the issue appears to go away and the VM boots fine. Is there some missing capability/host configuration here that could be getting in the way?

[rmohr]

@munnerz do you have selinux enabled?

[munnerz]

@rmohr no, this is a Debian 10 machine. That said, when running apparmor_status I see:

root@kube:~# apparmor_status
apparmor module is loaded.
4 profiles are loaded.
4 profiles are in enforce mode.
   /usr/sbin/tcpdump
   docker-default
   nvidia_modprobe
   nvidia_modprobe//kmod
0 profiles are in complain mode.
169 processes have profiles defined.
169 processes are in enforce mode.
.... (list of every container/process + the profile in use for each one)
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

I've not configured apparmor myself, so presumably this is some Debian/Kubernetes default configuration. Are there additional steps required/a profile that needs to be defined when running KubeVirt with SELinux/Apparmor?

[rmohr]

We provide and automatically install a custom selinux policy if selinux is enabled. No one contributed anything regarding to Apparmor so far. I guess a first step would probably by to disable Apparmor to see if it is the reason why things are failing.

[surekha1723]

I am also getting same issue while trying to run GPU in kubevirt

gpus:
- deviceName: nvidia.com/GP104_GeForce_GTX_1080
name: gpu1

Error:
{"component":"virt-launcher","level":"error","msg":"Failed to open config space file '/sys/bus/pci/devices/0000:04:00.0/config': Permission denied","pos":"virPCIDeviceConfigOpen:303","subcomponent":"libvirt","thread":"63","timestamp":"2020-05-13T05:22:58.164000Z"}
{"component":"virt-launcher","level":"error","msg":"Failed to open config space file '/sys/bus/pci/devices/0000:04:00.1/config': Permission denied","pos":"virPCIDeviceConfigOpen:303","subcomponent":"libvirt","thread":"63","timestamp":"2020-05-13T05:22:58.164000Z"}

[vladikr]

@surekha1723
I'll ask the same question about SELinux. Do you have it enabled?
In case SELinux is the issue, perhaps this PR[1] will resolve it when it lands.

[1] https://github.com/kubevirt/kubevirt/pull/3399

[surekha1723]

@surekha1723
I'll ask the same question about SELinux. Do you have it enabled?
In case SELinux is the issue, perhaps this PR[1] will resolve it when it lands.

[1] #3399

sestatus
SELinux status: disabled

[kubevirt-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[kubevirt-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

[kubevirt-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

[kubevirt-bot]

@kubevirt-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.