K3d: [FEATURE] Any cuda support?

[yamajik]

Any cuda support?

[iwilltry42]

Hi there, thanks for opening this issue!
Can you please provide some more details on this?

[yamajik]

Any doc for enable nvidia in k3d with containerd?

[iwilltry42]

I have really no clue, what k3d could do here. I guess that cuda would have to be supported by Docker first of all and the passed through to the containers. Everything on from there would be up to containerd and k3s, where we don't have any influence on.
I'd be happy to implement something if you could write something up here :+1:
UPDATE: as a quick google search showed, you may even need to build your own k3s image with CUDA driver included to get this working :thinking:

[twobombs]

AFAIK K3s does not have the NV code internally to run the K8s connector for NV cards; I have tried to work through this, and failed. :) Intel GPUs through the beignet icd driver do work with OpenCL, you need to enable the privileged bit on the container.

I have worked around this issue by mounting /var/run/docker.sock into a container on k3s so that the container running on k3s has access to the underlying docker (with NV stuff installed) and can spawn nvidia-docker containers. They then run outside the cluster, on the same node.

[markrexwinkel]

I was also looking at this and got it working although my solution is a little bit rough. I basically took the following steps:

• Installed the cuda drivers on the host
• Installed nvidia docker on the host
• Created a new k3s Docker image:
  
  
  
  • Changed the base image to ubuntu:18.04
  
  
  • Installed the cuda drivers and nvidia container runtime in the container
  
  
  • Copied the deamonset for gpu in the /var/lib/rancher/k3s/server/manifests directory
  
  
  • Copied a modified config.toml.tmpl in the /var/lib/rancher/k3s/agent/etc/containerd/ directory. This is needed to configure containerd to use the nvidia container runtime
  
  
• Run the new image with the docker run --gpus all flag
• Deployed a sample pod to test the gpu.

As you can see I took the implementation from microk8s and made it work in the k3s container. For this to work the k3s container needs to be changed and k3d needs a way to specify the Docker --gpus flag.

All the files are in this gist.

[iwilltry42]

That's super cool @markrexwinkel :rocket:
I don't see a problem with adding a --gpus flag to k3d. However, I doubt that it's a good idea for us to maintain an additional image that's based on e.g. Ubuntu and includes the cuda/nvidia stuff.
I guess it would be fine though to leave this up to the users, as it's a fairly specific use case, right?
Would you be willing to write a guide for https://k3d.io (Pull-Request to the docs/ folder in this repo) based on your Gist? :)

[markrexwinkel]

@iwilltry42 sure. Give give me a couple of days, I will create a PR with a doc.

[yamajik]

@iwilltry42 sure. Give give me a couple of days, I will create a PR with a doc.

Will it work well on wsl2?

[markrexwinkel]

@yamajik It will not work on WSL2 yet. The NVIDIA Kubernetes device plugin relies on the NVIDIA Management Library (NVML) APIs which are not supported yet. See Known limitations

[iwilltry42]

Sorry for the early close. However, #395 adds the --gpus flag, that you can use to make the GPUs accessible by the nodes.
Combine this with @markrexwinkel 's guide from #392 that's going to be merged soon and you should have homegrown cuda support :+1: However, we most likely won't maintain the required special k3s image here as it would mean much overhead (and it would fit better in the k3s repo anyway.
Thanks all for your input and especially @markrexwinkel for writing up this guide!