Nixpkgs: k3s: Service starts up before network; and doesn't shutdown properly

[ThinkChaos]

Describe the bug

1. systemd.services.k3s doesn't depend on network-online.target, which introduces a race and causes spurious k3s.service errors.

2. Stopping the service doesn't actually stop the containers, so when shutting the system down we get systemd-shutdown[1]: Waiting for: containerd-shim. This causes the shutdown time to go through the roof.

Additional context
Kind of related to #98090 because k3s-killall.sh could be useful for 2.

Notify maintainers
@euank

---

Nonetheless, thanks for maintaining this!

[freezeboy]

For number 2, would not a KillMode=control-group be enough in the service unit?

[ThinkChaos]

systemd.service.k3s was most likely copied from rancher/k3s/k3s.service and because control-group is the default (according to docs) I believe it's there for a reason.
I still gave it a shot: defined my own k3s_custom service using control-group. Instead of systemd-shutdown complaining about containerd-shim this gives us the standard systemd 1m30s timeout countdown.

I'll make a PR with the network part for 1 as that does work.

[euank]

Yup, it was copied from k3s.service.

I think the reason it's there is that often, if you just do something like systemctl restart k3s.service, it's desirable for the containers to keep running since k8s can "re-adopt" those orphaned containers.

It does sound like it interacts poorly with shutdowns, but for things like k3s upgrades, it seems desirable.
I'm not sure the right solution there.

For the networking thing, I definitely think we want an after=network-online.target. Probably a simple miss on my part.

I use the module with k3s.docker = true, which sets after=docker.service, which as a side-effect orders it after network.target, which is why I think I haven't been impacted by that yet.

One other thing that sorta falls under this issue: k3s should _also_ be ordered after firewalld.service on nixos to ensure the kube-proxy iptables rules don't race with the nixos fw rules on bootup. This is something I realized a few days ago, and hadn't gotten around to fixing.

[ThinkChaos]

It does sound like it interacts poorly with shutdowns, but for things like k3s upgrades, it seems desirable.

I agree, I noticed when I reverted back to process just now, nixos-rebuild switch waited for the containers to stop.

k3s should also be ordered after firewalld.service

I added that to the PR. I only saw your comment after submitting it.

I'm not sure the right solution there.

How about adding k3s-killall.sh and using it in ExecStopPost (ExecStop is called for restarts too)?
It's not perfect because this means one more file to keep in sync with upstream, but I don't see a better solution.