K3s: Installation on Raspberry Pi 3 Model B fails

[PRosenb]

Hardware:

• Raspberry Pi 3 Model B V1.2 (2015)
  
  
  
  • 1GB RAM
  
  
• SanDisk Extreme, 128GB

OS:

• Raspberry Pi OS (32-bit), A port of Debian with Raspberry Pi Desktop (Recommended), Released 2020-05-27
  
  
  
  • Installed with "Raspberry Pi Imager v1.4" on OSX 10.14.6
  
  

• On "boot" partition of mounted SDCard on OSX

  • Copy "wpa_supplicant.conf" from
    https://www.raspberrypi.org/forums/viewtopic.php?t=202719#p1260076
    and put Wifi SSID/Password
  • put an empty file "ssh"

• Boot Raspberry with DHCP

• Change password of user pi
  
  
  
  • passwd
  
  
• sudo apt -y update && sudo apt dist-upgrade
• enable container features
  
  
  
  • sudo nano /boot/cmdline.txt
  
  
  • add: cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
  
  
• sudo reboot
• uname -a
  -> Linux raspberrypi 5.4.51-v7+ rancher/k3s#1327 SMP Thu Jul 23 10:58:46 BST 2020 armv7l GNU/Linux
• curl -sfL https://get.k3s.io | sh -
• k3s -v
  -> k3s version v1.18.6+k3s1 (6f56fa1d)
• sudo systemctl status k3s
  -> k3s.service - Lightweight Kubernetes
  Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
  Active: activating (start) since Mon 2020-07-27 20:22:59 BST; 25s ago
  Docs: https://k3s.io
  Process: 1473 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
  Process: 1474 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
  Main PID: 1475 (k3s-server)
  Tasks: 31
  CGroup: /system.slice/k3s.service
  └─1475 /usr/local/bin/k3s server
• wait more than 1h
• sudo kubectl get node -o wide
  then it show one of the following two errors in random order
  -> Error from server (ServiceUnavailable): the server is currently unable to handle the request
  -> The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?
• top shows that it is doing some work
  -> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  9462 root 20 0 900528 227716 65512 S 206.3 24.0 0:34.83 k3s-server

I tried disabling IPv6 and reconfiguring iptables to legacy without luck.
Thanks for hits to find the issue.

[brandond]

What is the output of journalctl -u k3s --no-pager ?

[PRosenb]

Thanks for your response @brandond .

I attached the output of
journalctl -u k3s --no-pager > journalctl.log
journalctl.log

It says something about memory cgroup, but I have it in /boot/cmdline.txt..

$ cat /boot/cmdline.txt 
console=serial0,115200 console=tty1 root=PARTUUID=78c8af8b-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

[brandond]

I have heard that the most recent 32bit pi os kernels are missing support for a bunch of cgroups - you can't even enable them with kernel args. I would suggest checking the Pi forums, or use a 64bit os in the mean time.

[plsnotracking]

@PRosenb is this your issue - https://github.com/rancher/k3s/issues/2067?

[PRosenb]

Yes, it looks like the same issue.

[PRosenb]

I got it working with the 64-bit version.

$ sudo kubectl get node -o wide
NAME          STATUS     ROLES    AGE   VERSION        INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION   CONTAINER-RUNTIME
raspberrypi   NotReady   master   37h   v1.18.6+k3s1   192.168.1.17   <none>        Debian GNU/Linux 10 (buster)   5.4.51-v8+       containerd://1.3.3-k3s2

But some minute after restarting the device, the same command returns the following error in most cases.

$ sudo kubectl get node -o wide
Unable to connect to the server: net/http: TLS handshake timeout

[brandond]

@PRosenb ok, so what is in the k3s log journal now?

[plsnotracking]

@PRosenb

Can you please try sudo systemctl restart k3s post reboot and check again?

[PRosenb]

Thanks guys.

It looks like with the 64-bit OS it's just very slow, the shell is also almost not responsive. On top, CPU usage seams not too bad though..

I tried the restart command and it partly helped as you can see below. Executing the get node command still takes a very long time. I guess just shortly below/above the timeout.

$ sudo systemctl restart k3s
$ sudo kubectl get node -o wide
NAME          STATUS     ROLES    AGE     VERSION        INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION   CONTAINER-RUNTIME
raspberrypi   NotReady   master   2d10h   v1.18.6+k3s1   192.168.1.17   <none>        Debian GNU/Linux 10 (buster)   5.4.51-v8+       containerd://1.3.3-k3s2
$ sudo kubectl get node -o wide
NAME          STATUS   ROLES    AGE     VERSION        INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION   CONTAINER-RUNTIME
raspberrypi   Ready    master   2d10h   v1.18.6+k3s1   192.168.1.17   <none>        Debian GNU/Linux 10 (buster)   5.4.51-v8+       containerd://1.3.3-k3s2
$ journalctl -u k3s --no-pager > journalctl.log
$ sudo kubectl get node -o wide
Unable to connect to the server: net/http: TLS handshake timeout

Possibly the Raspberry Pi 3 Model B is too slow for the 64-bit variant or it's not (yet) optimized enough for it.

I attached the journalctl.log as requested.

[brandond]

I can attest that it runs great on 4GB Pi4B with class 10 SDHC cards. With less memory for filesystem cache you're going to be hitting the SD card pretty hard; I wouldn't be surprised if it feels pretty slow.

[PRosenb]

My SD Card is even faster, it's a V30 (SD Speed Class).
Does that imply that k3s cannot be run (anymore) on a Raspberry Pi 3?

[brandond]

You could try going back to an older 32bit OS that has working cgroup support. It sounds like maybe the arm64 releases aren't well optimized for older devices at the moment.

[jsedy7]

Thanks guys.

It looks like with the 64-bit OS it's just very slow, the shell is also almost not responsive. On top, CPU usage seams not too bad though..

I tried the restart command and it partly helped as you can see below. Executing the get node command still takes a very long time. I guess just shortly below/above the timeout.

$ sudo systemctl restart k3s
$ sudo kubectl get node -o wide
NAME          STATUS     ROLES    AGE     VERSION        INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION   CONTAINER-RUNTIME
raspberrypi   NotReady   master   2d10h   v1.18.6+k3s1   192.168.1.17   <none>        Debian GNU/Linux 10 (buster)   5.4.51-v8+       containerd://1.3.3-k3s2
$ sudo kubectl get node -o wide
NAME          STATUS   ROLES    AGE     VERSION        INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION   CONTAINER-RUNTIME
raspberrypi   Ready    master   2d10h   v1.18.6+k3s1   192.168.1.17   <none>        Debian GNU/Linux 10 (buster)   5.4.51-v8+       containerd://1.3.3-k3s2
$ journalctl -u k3s --no-pager > journalctl.log
$ sudo kubectl get node -o wide
Unable to connect to the server: net/http: TLS handshake timeout

Possibly the Raspberry Pi 3 Model B is too slow for the 64-bit variant or it's not (yet) optimized enough for it.

I attached the journalctl.log as requested.

Hi, I had a same problem. I think, that is a problem with version of your kernel.
I rolled back kernel version and everything works for now.

NAME   STATUS   ROLES    AGE   VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
w3     Ready    <none>   13h   v1.18.6+k3s1   192.168.1.122   <none>        Raspbian GNU/Linux 10 (buster)   4.19.118-v7+     containerd://1.3.3-k3s2
w2     Ready    <none>   12h   v1.18.6+k3s1   192.168.1.121   <none>        Raspbian GNU/Linux 10 (buster)   4.19.118-v7+     containerd://1.3.3-k3s2
m1     Ready    master   84d   v1.18.4+k3s1   192.168.1.110   <none>        Raspbian GNU/Linux 10 (buster)   4.19.97-v7+      containerd://1.3.3-k3s2
w4     Ready    <none>   25h   v1.18.6+k3s1   192.168.1.123   <none>        Raspbian GNU/Linux 10 (buster)   4.19.118-v7+     containerd://1.3.3-k3s2
w1     Ready    <none>   25h   v1.18.6+k3s1   192.168.1.120   <none>        Raspbian GNU/Linux 10 (buster)   4.19.118-v7+     containerd://1.3.3-k3s2

I installed all packages here: https://github.com/raspberrypi/linux/issues/3644#issuecomment-663760750

[jcleira]

Fixed today for me after doing rpi-update to this version:

uname -a
Linux k8s-master 5.4.58-v7+ #1335 SMP Thu Aug 13 22:17:06 BST 2020 armv7l GNU/Linux

The cgroup could be enabled again by the mentioned /boot/cmdline.txt` params

[gk5885]

I've been fighting with getting k3s running on some Pi 3s for a week or so and have had a bunch of the problems on this issue. I got it "working" with Ubuntu 20.04.1 LTS (64-bit), but it unbearably slow. The timeouts were so frequent it wasn't usable. So, I switched _to_ Raspberry Pi OS (32-bit).

Here's what it took to get it to work:

• You need the cgroup fixes mentioned, which are well-documented. I've been including cgroup_enable=memory cgroup_memory=1 cgroup_enable=cpuset
• I don't know if it's strictly necessary, but lots of things seem to indicate that you should turn off swap. I've been using sudo dphys-swapfile swapoff && sudo systemctl disable dphys-swapfile.service
• I went into raspi-config to set the memory split to give the GPU the minimum in order to claw back every shred of ram I could. Advanced Options -> Memory Split and set it to 16. This probably isn't actually necessary either.
• And finally, to resolve some of the weird connectivity issues, you need to use legacy iptables. My suspicion is that this is what was actually causing issues for @PRosenb originally because I got past that same error with that change. Edit: I just noticed that he actually _did_ mention trying that. 🤷

[gk5885]

Oh, and along the way sudo k3s check-config was nice to sanity check that things were looking good. I'm actually surprised at how little documentation references that helpful little command.

[PRosenb]

I can confirm that my steps above now work with the newest version of Raspberry Pi OS (32-bit) on my Raspberry Pi 3 Model B V1.2 (2015)

$ uname -a
-> Linux raspberrypi 5.4.51-v7+ #1333 SMP Mon Aug 10 16:45:19 BST 2020 armv7l GNU/Linux

Thanks for your help guys.

[Ooyixiao]

i had try your‘s every step to do using Raspberry Pi OS (32-bit).But still shows Error from server (ServiceUnavailable): the server is currently unable to handle the request.what can i do?