K3s: Compatibility with Device Plugins

[gesslein]

Version:
k3s version v1.17.2+k3s1 (cdab19b0)

Describe the bug
Many operators that utilise the Device Plugin feature expect to use/mount /var/lib/kubelet/device-plugins/, where k3s provides the socket under /var/lib/rancher/k3s/agent/kubelet/device-plugins/.

In the kubelet code it is set to
DevicePluginPath = "/var/lib/kubelet/device-plugins/"
and while I'm not sure if this should be hardcoded like that, defacto it currently is.

Some operators can be reconfigured to use a different path, but to even find out this is your problem and you need to then forever use a modified setup is a massive hurdle. The operator I'm currently interested in, kubevirt, was not able to be reconfigured.

rancher/k3s#451 mentiones the Network Service Mesh being affected but being able to be reconfigured
rancher/k3s#443 the same for NVIDIA

To Reproduce
Start the kubevirt operator. It will mount the empty directory /var/lib/kubelet/device-plugins/ and not work.

Expected behavior
Using Device Plugins should just work out of the box.

Actual behavior
Before creating the kubevirt operator, create a symlink
ln -s /var/lib/rancher/k3s/agent/kubelet/device-plugins/ /var/lib/kubelet/device-plugins
then start the kubevirt operator and it will work.

[brandond]

I would say shame on the operator for hardcoding the path. I like that k3s tries to keep as much as possible within it's configured data-dir.

[gesslein]

I agree that hard coding this path is not ideal on the operator side of things either. If they add an option and document it the chance for an user to debug this in the first place is much higher. The proper solution would require no change on either side, for example if the path was discoverable.

Still I'd argue it is of bigger interest to k3s to "conform" to the behavior of standard kublets and run as much components as possible unmodified, than the other way around. Of course there is a reasonable debate what should count as a standard, but in this particular case at this current time, this path is a de-facto hard coded standard even in the kublet code that you vendor in k3s.

For me personally a symlink is of much much lower cost than having different code/config per environment or having to find out the hard way I need to modify operator code or configs. I might have considered not using k3s if this was the only possibility.

[ShylajaDevadiga]

Resolved in v1.17.3+k3s1. Closing issue. ls /var/lib/kubelet/device-plugins |grep kubelet.sock kubelet.sock
kubectl get vmi NAME AGE PHASE IP NODENAME testvm 3m8s Running 10.42.0.17 ip-172-31-0-88

[niusmallnan]

The milestone should be v1.17.3+k3s1?