K3os: Upgrade to v0.21.0-k3s1r0, get CreateContainerError on all pods

[MaximizeRr]

Version (k3OS / kernel)
k3os version v0.21.0-k3s1r0
5.4.0-72-generic #80 SMP Tue May 4 19:31:44 UTC 2021

Architecture
x86_64

Describe the bug
After updating from v0.20.6-k3s1r0 to v0.21.0-k3s1r0, i get CreateContainerError on all pod. In the k8s events i get this error message:

Error: failed to create containerd container: get apparmor_parser ver
sion: exec: "apparmor_parser": executable file not found in $PATH

To Reproduce
Upgrade a single master node from v0.20.6-k3s1r0 to v0.21.0-k3s1r0 by changing k3os-latest plan.

kubectl edit plan k3os-latest -n k3os-system
kubectl label nodes yournode k3os.io/upgrade=enabled 

Expected behavior
Pods in running status

Actual behavior
No container can be start on the cluster.
Example:

kube-system            traefik-6f9cbd9bd4-xhmmq                                          0/1     CreateContainerError        0          31m
kube-system            local-path-provisioner-5ff76fc89d-zmtnv                           0/1     CreateContainerError        0          31m
kube-system            metrics-server-86cbb8457f-sh2vg                                   0/1     CreateContainerError        0          31m
k3os-system            system-upgrade-controller-8bf4f84c4-gfmx4                         0/1     CreateContainerError        0          15h
kube-system            coredns-7448499f4d-z2hl2                                          0/1     CreateContainerError        0          29m

[brlbil]

I have the same issue with a freshly installed cluster, also created issue for k3s

[dweomer]

This is an issue arising from the fact that k3OS ships a repackaged Ubuntu kernel which has apparmor enabled by default but lacks apparmor userspace tooling combined with the fact that upstream kubelet in 1.21.x detects apparmor via kernel and attempts to enable profile(s). The solution for k3OS is to update your boot parameters, a la https://github.com/rancher/k3os/pull/695/commits/58564fea3de1a2d2d8425689216da71e6935e632, to disable apparmor.

[smshr]

I got the same error with k3s version v1.21.1+k3s1 running in debian 10.9, reverting back to the previous version v1.20.7+k3s1 fixed it.

Error: failed to create containerd container: get apparmor_parser version: exec: "apparmor_parser": executable file not found in $PATH

[brlbil]

@dweomer I made a new installation with the grub file that has apparmor=0 boot parameter. I can verify that solved the issue.

[dweomer]

I got the same error with k3s version v1.21.1+k3s1 running in debian 10.9, reverting back to the previous version v1.20.7+k3s1 fixed it.

Error: failed to create containerd container: get apparmor_parser version: exec: "apparmor_parser": executable file not found in $PATH

(for debian you should either disable apparmor at kernel boot as suggested here or install the apparmor userland)

[smshr]

(for debian you should either disable apparmor at kernel boot as suggested here or install the apparmor userland)

installing apparmor userland fixed it
apt install apparmor apparmor-utils

[dweomer]

This should be fixed in the next 0.21.x release