K3s: kubelet does not start even when the agent node is unable to connect to the server node

[carlosrmendes]

I have an agent node with some static pods (I've set the --kubelet-arg=pod-manifest-path=... argument) but sometimes that node goes offline and could get some reboots. The problem is when the node starts after a reboot and still is offline, the k3s doesn't start anything (i.e. containerd, kubelet, ...) and with that, don't start the static pods.

Is there any solution or workaround for this use case?

[brandond]

I tried to do the same thing - I was hoping to run etcd as a static pod, but there's a chicken-and-egg problem where k3s won't start until etcd is up, so it won't start the static pod, etc.

[carlosrmendes]

as described in https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/:
_"Static Pods are managed directly by the kubelet daemon on a specific node, without the API server observing them"_

With that, IMO, I think that kubelet on the k3s process must be started before the agent node tries to connect to the server node.

[brandond]

Yeah, having the agent run without server connectivity would be great. Bonus points if the kubelet on server instances would start up independent of the apiserver so that we could do thinks like running etcd or mysql as a static pod.

[ibuildthecloud]

The reason why this is a bit difficult is that we download the config of the kubelet from the api server. To do this properly requires a bootstrap mode for the kubelet. Basically run with the last (or no) configuration and then download the configuration and restart. The kubelet is ran embedded in the same process as the k3s agent so restart means we'd have to reexec ourselve as the kubelet can't just be restarted in memory. This all gets a bit messy.

@carlosrmendes What static pods are you running that can't be done with a daemonset? I personally haven't found any great use cases for static pods beyond bootstraping k8s itself.

[ibuildthecloud]

I do think it's a reasonable request for "agent should start without server connectivity." If the server goes down and then you restart an agent it shouldn't be blocked. Supporting static pods on the agent only nodes will be tricky, but supporting them on the server is probably feasible as this is how rke2 works.

[carlosrmendes]

my use case is run specific pods (workloads) on some agent nodes, even the nodes are offline and disconnected from the master. I don't want DaemonSets, because I want to create/schedule specific pods into specific agent nodes and despite that, to start the pods of a DaemonSet in an agent node that is offline or disconnected from the master, the agent node must have connection to the api-server (or the node must be visible as Ready by the api-server). With static pods that is not necessary, only kubelet running is needed to start static pod on an offline node.

[tdbs]

I have to agree, this is an issue. I would expect the "lightweight kubernetes" to work like kubernetes, but in this instance it does things in a way that I cannot use the kuberenetes instructions on static pods.