K3s: RPi4/Buster node password rejected

[marcstreeter]

Describe the bug
When installing k3s on a cluster of RPi4 (RaspberryPi 4) devices the worker nodes are unable to register.

To Reproduce
Steps to reproduce the behavior: followed these steps exactly:

• have master RPi4 and worker RPi4
• first install k3s (v0.8.1) on master
• retrieve token
• use token and local server ip for install of k3s on worker (also using script with K3S_URL and K3S_TOKEN inputs)
• checked nodes (sudo kubectl get nodes)
• only see master and journalctl shows node password mismatch

Expected behavior
expect to see master and worker nodes

Screenshots
Top image is worker and Bottom image is master

Output of journalctl for both worker (top) and master(bottom)

Additional context

outputs

Node password rejected, contents of '/var/lib/rancher/k3s/agent/node-password.txt' may not match server

I thought that maybe something specific to Raspian Buster may have to do with it (since the RPi4's only run on Buster, not Stretch) so I also applied suggested "fixes" mentioned https://github.com/rancher/k3s/issues/703 and in https://github.com/kubernetes/kubernetes/issues/71305
(still to no avail even after restart)

[erikwilson]

It is probably a good idea to run legacy iptables on the RPi4, but the issue can be resolved by removing or updating /var/lib/rancher/k3s/server/node-passwd (https://rancher.com/docs/k3s/latest/en/configuration/#node-registration)

[marcstreeter]

I don't see either file on the master(bottom) or worker(top)

[erikwilson]

Sorry, from the docs and the logs it should be /var/lib/rancher/k3s/server/cred/node-passwd

[marcstreeter]

first of all, many thanks for pointing me in some direction - I moved the file out of that directory ( as a way of "removing" but with the option to replace in case I misunderstood) and restarted all the RPi's - no effect. I also uninstalled the workers and then reinstalled them (now with that file gone) no effect. Am I misunderstanding something?

[erikwilson]

Ah, thanks for trying that out. Apologies, the issue is that both nodes have the hostname raspberrypi, if the names are unique the error should go away. The log messages could probably be made more useful in that respect.

[marcstreeter]

ahhh that was it. I had set up raspberry pi's before and somehow forgot to do that. I'll make sure I make a note to not repeat the same mistake. Thank you Erik for your patience and help!

[alexbarcelo]

Sorry to add a little noise, but I was having those exact symptoms due to a completely different problem: #802

Let me add this reference here just in case some future users end up here with this "Node password rejected" error (which was exactly what I was having). My problem was related to the credentials being stored in the server and not cleaned up because some dirty operations and errors, which is what is discussed in this other #802 issue.

I knew nothing about that credential file in the server and thus I was completely missing the point. Maybe this will happen to somebody else and they will end up here, just like me. Sorry for the noise to everyone involved in this issue. shrug