Cluster-api-provider-azure: Add support for GPU nodes

[CecileRobertMichon]

/kind feature

Describe the solution you'd like
[A clear and concise description of what you want to happen.]
https://docs.microsoft.com/en-us/azure/virtual-machines/linux/n-series-driver-setup

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• cluster-api-provider-azure version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[CecileRobertMichon]

We should into how/if https://github.com/NVIDIA/gpu-operator can be leveraged for this. See https://docs.nvidia.com/datacenter/kubernetes/openshift-on-gpu-install-guide/index.html.

[sozercan]

/assign

[sozercan]

Looks like NVIDIA gpu-operator doesn't support containerd yet (containerd itself supports GPUs but needs runtime configuration changes to use them with device plugin).
https://github.com/NVIDIA/gpu-operator/issues/7

If we don't want to wait for containerd support, we can resolve this similar to aks-engine, install GPU driver to N series nodes and then deploy device plugin separately. Not sure if capz has similar CSE execution on specific nodes though.

[sozercan]

If anyone wants to install drivers and device plugin manually, here are instructions:

• Deploy capz with N Series SKU (includes NVIDIA GPUs).

• Once cluster is up, ssh into each agent node that has GPUs and run these:

• Install NVIDIA drivers:

sudo apt update
sudo apt install ubuntu-drivers-common -y
sudo ubuntu-drivers install
nvidia-smi # to verify gpu drivers are installed

• Install NVIDIA container runtime:

curl -s -L https://nvidia.github.io/nvidia-container-runtime/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-container-runtime/$(. /etc/os-release;echo $ID$VERSION_ID)/nvidia-container-runtime.list | sudo tee /etc/apt/sources.list.d/nvidia-container-runtime.list
sudo apt install nvidia-container-runtime -y

• Configure containerd:

sudo mkdir -p /etc/containerd
sudo vi config.toml # <- add config from: https://gist.github.com/sozercan/51a569cf173ef7e57a375978af8edf26
sudo systemctl restart containerd

• Verify containerd can access the GPUs: (you should see same output as nvidia-smi)

sudo ctr images pull docker.io/nvidia/cuda:10.0-base
sudo ctr run --rm --gpus 0 docker.io/nvidia/cuda:10.0-base nvidia-smi nvidia-smi

• Deploy NVIDIA Device plugin: (run once in Kubernetes)

kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/master/nvidia-device-plugin.yml

You should see following output in device plugin logs:

2020/05/09 04:46:53 Loading NVML
2020/05/09 04:46:53 Starting FS watcher.
2020/05/09 04:46:53 Starting OS watcher.
2020/05/09 04:46:53 Retreiving plugins.
2020/05/09 04:46:53 Starting GRPC server for 'nvidia.com/gpu'
2020/05/09 04:46:53 Starting to serve 'nvidia.com/gpu' on /var/lib/kubelet/device-plugins/nvidia.sock
2020/05/09 04:46:53 Registered device plugin for 'nvidia.com/gpu' with Kubelet

[alexeldeib]

Can we possibly pre-provide a kubeadm config template to simplify this?

[CecileRobertMichon]

@alexeldeib you mean leverage post kubeadm commands to do the install?

[CecileRobertMichon]

/unassign @sozercan

[alexeldeib]

yeah, or even just stick it in a file and have the postKubeadmCommands be bash setup.sh.

I'm warming up to the idea of using the templatized types as a way to simplify defaulting / best practices. We could have something like a default GPU kubeadm config template, so users don't need to bring their own

[CecileRobertMichon]

Yeah I like the idea of having a "reference" flavor template for GPU w/ docs using the bash script Sertac shared above for now, and then maybe open a separate issue for switching the instructions to use the nvidia operator once that works with containerd.

I'm going to mark this as help wanted.

/help

[k8s-ci-robot]

@CecileRobertMichon:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

Yeah I like the idea of having a "reference" flavor template for GPU w/ docs using the bash script Sertac shared above for now, and then maybe open a separate issue for switching the instructions to use the nvidia operator once that works with containerd.

I'm going to mark this as help wanted.

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[CecileRobertMichon]

There is also a VM extension available on Azure that might be worth looking into: https://docs.microsoft.com/en-us/azure/virtual-machines/extensions/hpccompute-gpu-linux

Not sure if it works with containerd though.

[mboersma]

/assign