K3s: Why is k3s using /var/lib/rancher/k3s/agent/kubelet

[deitch]

Describe the bug

Various system level utilities that depend on the location of /var/lib/kubelet/ (I came across it when working on CSI) fail, because k3s installs it at /var/lib/rancher/k3s/agent/kubelet/. In this case, it was plugins/, plugins_registry/ and pods/, but I can see others.

To Reproduce

1. Deploy k3s (which is so nice and easy)
2. Run anything that looks for it OR ls -l /var/lib/kubelet/

Expected behavior

Standard kubelet directory to exist

Additional context

Even a simple ln -s /var/lib/rancher/k3s/agent/kubelet /var/lib/kubelet would do it.

Oh yes... I love k3s.

[deitch]

Actually, it gets more complex that that. Apparently the kubelet itself is aware that its path isn't /var/lib/kubelet/ but instead /var/lib/rancher/k3s/agent/kubelet/. This creates CSI problems, as NodeStageVolume passes the mount path /var/lib/rancher/k3s/agent/kubelet/plugins/.... and Unstage passes /var/lib/rancher/k3s/agent/kubelet/pods/<pod-uuid>/volumes/kubernetes.io~csi/pvc-<pvc-uuid>/mount.

Unless you know to mount /var/lib/rancher/ into your CSI DaemonSet pod, this gets really off.

Any reason we aren't using /var/lib/kubelet/?

[ibuildthecloud]

It was just nicer this way, everything was in one directory. But then CSI came along and messed up everything. We might just have to move to /var/lib/kubelet.

I thought a symlink would fix it but you've proven it doesn't.

What if on start we create a bind mount?

[ibuildthecloud]

We'll just move to /var/lib/kubelet.

[deitch]

I actually thought about working around it, if you could have a mount reference something like this:

      volumes:
        - name: kubelet-dir
          hostPath: kubelet-path-spec

or similar where this would be resolved at runtime by the kubelet to its definition of the Kubelet path (normally /var/lib/kubelet but could be anything), but I don't know of any way to do that.

[ccll]

We'll just move to /var/lib/kubelet.

Can't wait to see this!
I'm trying to deploy rook-ceph on k3s, and the CSI plugins failed to deploy due to this issue.
Would you kindly share a rough schedule of this fix?
Thanks.

[billimek]

I'm sure you will, but please call-this out as a possible breaking change in release notes when this change ships. We have needed to work around this by hard-coding the /var/lib/rancher/k3s/agent/kubelet path in other things and will need to 'fix' this if/when the k3s default behavior changes.

[ShylajaDevadiga]

With v0.10.0-rc2, we have k3s use the default /var/lib/kubelet directory.

[deitch]

Thanks. I am going to test it out right away.

[deitch]

And done. CSI works beautifully. Thanks for the fix!