Kubevirt: Support for GPU

[binojvr]

FEATURE REQUEST?:

enhancement

What happened:
I am planning to containerize legacy gpu intensive windows application using Nvidia Docker and kubernetes. But nvidia docker doesn't support windows as of now. Now I am planning to deploy these application inside windows VMs and launch using kubernetes and Kubevirt. is it possible to access Nvidia docker from application deployed in VM which I launched using kubevirt? If I am not able to access, how can I use GPU in application in VM Using Kubevirt?

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• KubeVirt version (use virtctl version): Latest
• Kubernetes version (use kubectl version): 1.12
• VM or VMI specifications: Windows VM
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): Ubuntu
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[volume-ji]

is there any plan to support nvidia gpu? like pci-paasthrough?

[fabiand]

Heya, yes, device passthrough, including vGPU is planned.

[kubevirt-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[kubevirt-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

[xingfeng2510]

@fabiand Hi, is GPU passthrough supported？

https://github.com/kubevirt/kubevirt/issues/953 https://github.com/kubevirt/kubevirt/pull/593 I saw related issues and pull requests were closed, but there was not docs found.

Please help me to clarify, thanks!

[cynepco3hahue]

@xingfeng2510 nope, we still do not support it, but if we will see that bunch of people interesting in GPU passthrough we can consider to add it, the most problem that we have that we do not have local and CI hosts to test it properly

[fabiand]

Today there is no support for GPU passthrough.

There is however support for SR-IOV network devices, we hope that i.e. GPU
passghrough can be built on-top of this work.

[xingfeng2510]

@cynepco3hahue If there's any forks or branches that already contain GPU PCI passthrough feature? I believe that it has already been implemented but not merged.

We need this feature and we could test it on our own hosts, thanks!

[cynepco3hahue]

@xingfeng2510 I am not aware of such fork, but you can ask under the k8s virtualization group in slack

[xingfeng2510]

@cynepco3hahue Appreciate for your reply, I will take a chance on kubevirt slack channel.

[xingfeng2510]

@fabiand Is there any hints for implementing GPU PCI passthrough based on SRIOV code? I can't access to the slack channel, then I want to try to implement a temporary version.

[Queuecumber]

/remove-lifecycle rotten

This would be huge for deploying microsoft only GPU code at scale, a lot of legacy machine learning code is windows only. I think this should be made a priority and I have a GPU machine I can test with if you need help

[cynepco3hahue]

@booxter Can you please provide metalink on SRIOV code?
In general, I expect something like this:

• pass-through GPU device to the container via device plugins - https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/
• consume the GPU under the container with the libvirt - https://wiki.gentoo.org/wiki/GPU_passthrough_with_libvirt_qemu_kvm

I believe it will require some kernel configurations like IOMMU support.

[booxter]

@cynepco3hahue @xingfeng2510 thanks for the interest in passthrough. Indeed I believe that support for it would be similar to SR-IOV network support.

Some details about SR-IOV for KubeVirt are captured here: https://github.com/kubevirt/kubevirt/blob/master/docs/sriov.md

The way it works is that:

1. Host is configured with SR-IOV VFs registered with vfio-pci kernel driver instead of regular netlink driver.
2. SR-IOV device plugin discovers and advertises the VFs to scheduler.
3. When a VMI interface is of sriov type, we fill in requests section with the corresponding resource name (found in resourceName annotation of the corresponding NetworkAttachmentDefinition CRD resource). Device plugin passes information about the PCI device IDs allocated through environment variables.
4. k8s scheduler places virt-launcher pod; launcher starts, detects SR-IOV interface type, extracts information about devices from environment variables, and configures libvirt domain to pass these devices through into qemu.

So in case of GPU, you will probably need to:

1. have a device plugin that discovers / advertises GPU devices.
2. then either via an API abstraction or directly, user would fill requests according to the name of the GPU resource the DP exposes.
3. launcher would observe that the VMI has GPU allocated (either because the API abstraction is used or by other means - e.g. device plugin carries information via envvars, or you observe pod's requests to see if a "known" GPU device was allocated) and then configure libvirt domain as per the link @cynepco3hahue listed.

[booxter]

Here is the code where we convert VMI SR-IOV attachment to pod definition: https://github.com/kubevirt/kubevirt/blob/7997dd0047f9ba8be225edc78673039e98346417/pkg/virt-controller/services/template.go#L343 (basically mount directories from the host / add capabilities for libvirt to be able to pass devices through). There is no network "attachment" code in the tree for SR-IOV because the device is not passed as a netlink device but VFIO.

[Queuecumber]

One potential workaround for this as a holdover for first class support would be to allow passing arbitrary arguments to qemu, then I could just use a node selector to make sure I'm in the right place and map the GPU in an argument. This would be useful for other things too (I also need to set kvm hidden state to on for example and I dont see that in your documentation anywhere). Is there a particular reason that there isnt generic argument support or is it just that no one has done it yet?

[booxter]

@Queuecumber there is hooks mechanism that allows to adjust XML domain as needed: https://github.com/kubevirt/kubevirt/blob/master/pkg/hooks/manager.go but of course it shouldn't be abused long term for generally useful entities like GPUs.

[Queuecumber]

Ok very nice I didn't see that documented anywhere. I agree that it isnt a sustainable solution but my POC cluster only has 1 gpu on it so fine for that

[booxter]

@Queuecumber it's probably not documented exactly because it's not a great solution and the team doesn't want it to be adopted by well meaning but short term focused community members. :) but it should do for basic prototyping.

[fabiand]

I wonder if anybody watching this issue has started to work on this already?

[KrisTC]

We are also interested in this feature.

[allanbarklie]

I see that some detailed discussions/proposals in this area can now be found on the group
https://groups.google.com/d/msg/kubevirt-dev/5G7nqXXhHM0/I-WLdDlbDwAJ

[booxter]

PR here: https://github.com/kubevirt/kubevirt/pull/2597

[rmohr]

/close

implemented in #2597. It will be possible in the next release to attach gpus via the gpu device plugin. If that does not solve the problem, feel free to reopen.

[kubevirt-bot]

@rmohr: Closing this issue.

In response to this:

/close

implemented in #2597. It will be possible in the next release to attach gpus via the gpu device plugin. If that does not solve the problem, feel free to reopen.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Just-Insane]

Does this include support for generic devices such as PCIe HBA/RAID cards or USB controllers for example?