K3s: Failed to start ContainerManager open /proc/sys/kernel/panic: permission denied

[mar1ged]

Version:

k3s version v1.0.1 (e94a3c60)

Describe the bug

Failed to start ContainerManager open /proc/sys/kernel/panic: permission denied

To Reproduce

install k3s using the get shell script (without any special parameters)
run kubectl get nodes

Expected behavior

I would like to see at least the current host

Actual behavior

Instead The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port? gets displayed

Additional context

I checked the log by running journalctl -u k3s. There I can see this at the end:

Dez 21 22:36:49 h12345678.stratoserver.net k3s[2328]: F1221 22:36:49.814798    2328 kubelet.go:1380] Failed to start ContainerManager open /proc/sys/kernel/panic: permission denied

As you can see I run this on a Strato VPS which uses Virtuozzo. In former times Strato didn't support docker on their virtualization platform but starting from November 2019 they do: https://docs.virtuozzo.com/virtuozzo_7_users_guide/advanced-tasks/setting-up-docker-in-containers.html
I checked the server, it is able to run docker (in this case 18.09.7) without issues.
The server uses Ubuntu 18.04.3.

This is the output of k3s check-config:

 sha256sum: good
- links: good

System:
- /sbin iptables v1.6.1: older than v1.8
- swap: disabled
- routes: ok

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

modprobe: module configs not found in modules.dep
error: cannot find kernel config 
  try running this script again, specifying the kernel config:
  set CONFIG=/path/to/kernel/.config or add argument /path/to/kernel/.config

[DavidZisky]

A possible solution is to run:

sudo apt-get install linux-image-$(uname -r)

but please also provide output of:

uname -r and ls /boot/config*

[mar1ged]

uname -r returns 4.15.0, when I install for that version I get an overall download of 26GB ;-)

/boot has no config file, ls -al /boot returns this:

drwxr-xr-x  2 root root 4096 Sep 27 20:24 .
drwxr-xr-x 23 root root 4096 Dez 21 22:11 ..

[DavidZisky]

oh boy, it looks like it's nothing k3s can fix, simply because it's due to the way how Strato works. I signed up for their VPS to quickly check but it's been an hour already and it's still not provisioned. Why would you use such a service? Can't you just use normal VPS provider? :p

[mar1ged]

I have others like Hetzner, DO, scaleway, OVH vh and vultr but I get 4 vCores , 100GB SSD and 8GB RAM for 5€, that's simply a bargain ;-) On Strato I run the stuff I always need and I have Plesk + all stuff I need for my email handling.
My goal was to add some containers, but wanted to do it like on all the other providers.

[DavidZisky]

Well, ok. You just need to keep k3s somewhere else then. I'm not entirely sure how Virtuozzo works but it's not "real" VM somehow. It restricts a lot of stuff, and k3s simply can't access kernel modules which it needs. When I finally got my access I wasn't even able to see exact kernel version - virtuozzo abstracts it. That's why when trying to install kernel it tries to install all possible versions of it (therefore 26GB)

[mar1ged]

OK, I expected that this might be somehow troublesome and hoped that because Docker works now similar technologies might work too.
What I don't understand: why should I install a kernel ? There is already one available, why is this one not enough ?

[DavidZisky]

Docker requires less privileges/kernel modules than K8S/K3S.

why should I install a kernel?

I recommended it because this issue:

modprobe: module configs not found in modules.dep

often happens where there is a mismatch between installed kernel and installed headers. Then simply installing a kernel again usually fixes it. But I installed different version on that Virtuozzo VPS and I am not even able to boot it up.

[mar1ged]

Yes, I see. I ran the same statement on a regular Ubuntu. It complains about the same missing module but it finds a config file and dumps a lot of stuff.

I tried to follow advice on superuser to find more about kernel config. But there is no /lib/modules/4.15.0/build/ and thus no .config. I even checked the directory where build should symlink too, but there is no /usr/src/linux-headers-5.3.0-24-generic. To be precise there is nothing below usr/src at all.

[mar1ged]

One more note: I played some more with the VPS and installed wireguard. Wireguard somehow detected a version: Building for 4.15.0 and 4.15.0-72-generic.

[jewelt]

Could you solve your problem with Strato and Kubernetes in the meantime? We have exactly the same problem. Do you know anything new?

[mar1ged]

No, I think that we can be thankful that docker works with Strato. I'm still searching for a priceworthy solution and I think that I will head for Scaleway Kapsule

[kennedyalexander]

Thanks for doing the investigation. This helped me.
Only solution... New hosting company... Appreciate the work you have done.

[vioan]

is what I did, as well, switching from Strato to Hetzner and I am a lot happier.