Microk8s: Adding "allow-privileged true" flag kills the kubelet

[avarf]

I need to run some pods as privileged and I found (https://github.com/ubuntu/microk8s/issues/216) that I have to use --allow-privileged true which I did and as suggested I restarted the required daemons but that killed my kubelet and microk8s. I even deleted everything and reinstalled microk8s but the same happened.
Please run microk8s.inspect and attach the generated tarball to this issue.

microk8s.inspect 
Inspecting services
  Service snap.microk8s.daemon-containerd is running
  Service snap.microk8s.daemon-apiserver is running
  Service snap.microk8s.daemon-proxy is running
 FAIL:  Service snap.microk8s.daemon-kubelet is not running
For more details look at: sudo journalctl -u snap.microk8s.daemon-kubelet
  Service snap.microk8s.daemon-scheduler is running
  Service snap.microk8s.daemon-controller-manager is running
  Service snap.microk8s.daemon-etcd is running
  Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system info
  Copy network configuration to the final report tarball
  Copy processes list to the final report tarball
  Copy snap list to the final report tarball
  Inspect kubernetes cluster

Building the report tarball
  Report tarball is at /var/snap/microk8s/743/inspection-report-20190806_181638.tar.gz

And

sudo journalctl -u snap.microk8s.daemon-kubelet | tail
Aug 06 18:14:51 ali-P51 microk8s.daemon-kubelet[5306]:       --vmodule moduleSpec                                                                                        comma-separated list of pattern=N settings for file-filtered logging
Aug 06 18:14:51 ali-P51 microk8s.daemon-kubelet[5306]:       --volume-plugin-dir string                                                                                  The full path of the directory in which to search for additional third party volume plugins (default "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/")
Aug 06 18:14:51 ali-P51 microk8s.daemon-kubelet[5306]:       --volume-stats-agg-period duration                                                                          Specifies interval for kubelet to calculate and cache the volume disk usage for all pods and volumes.  To disable volume calculations, set to 0. (default 1m0s) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Aug 06 18:14:51 ali-P51 microk8s.daemon-kubelet[5306]: F0806 18:14:51.854669    5306 server.go:156] unknown flag: --allow-privileged
Aug 06 18:14:52 ali-P51 systemd[1]: snap.microk8s.daemon-kubelet.service: Service hold-off time over, scheduling restart.
Aug 06 18:14:52 ali-P51 systemd[1]: snap.microk8s.daemon-kubelet.service: Scheduled restart job, restart counter is at 5.
Aug 06 18:14:52 ali-P51 systemd[1]: Stopped Service for snap application microk8s.daemon-kubelet.
Aug 06 18:14:52 ali-P51 systemd[1]: snap.microk8s.daemon-kubelet.service: Start request repeated too quickly.
Aug 06 18:14:52 ali-P51 systemd[1]: snap.microk8s.daemon-kubelet.service: Failed with result 'exit-code'.
Aug 06 18:14:52 ali-P51 systemd[1]: Failed to start Service for snap application microk8s.daemon-kubelet.

As we can see allow-priviliged is not defined for kubelet.
I also tried the --allow-privileged=true instead of --allow-privileged true but it kills my kubelet.

[ktsakalozos]

Hi @avarf

The --allow-privileged flag is removed from kubelet on the upstream v1.15 release, https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.15.md#node

I am not sure what your use case is but you may be able to use the privilege escalation from pod security policies: https://kubernetes.io/docs/concepts/policy/pod-security-policy/#privilege-escalation

[avarf]

Hi @ktsakalozos
Thanks for your answer. I want to set up an EFK stack for centralized logging and some of the components need to be run as privileged.
At the moment they are using:

securityContext:
  privileged: true

I will change it to the allowPrivilegeEscalation and inform you about the results.

[avarf]

I fetched the elasticsearch chart (https://github.com/helm/charts/tree/master/stable/elasticsearch) and changed all the "priviliged" so all the components are look like this now:

        securityContext:
          allowPrivilegeEscalation: true

And also I chnaged the "restartPolicy" to Always.

Now I can install the elasticsearch via helm:

helm install --name elasticsearch elasticsearch \
    --set master.persistence.enabled=false \
    --set data.persistence.enabled=false \
    --namespace logging

But all the elasticsearch pods are failing to launch:

logging       elasticsearch-client-6fb7844b6f-lbtq7             0/1     Init:CrashLoopBackOff   6          9m39s
logging       elasticsearch-client-6fb7844b6f-m4j6k             0/1     Init:CrashLoopBackOff   6          9m39s
logging       elasticsearch-data-0                              0/1     Init:CrashLoopBackOff   6          9m39s
logging       elasticsearch-master-0                            0/1     Init:CrashLoopBackOff   6          9m39s
logging       fluent-bit-mxgpx                                  1/1     Running                 0          27m
logging       kibana-6d9794d46f-q8vmj                           1/1     Running                 0          9m38s

And the only log that I can see is:

Error from server (BadRequest): container "elasticsearch" in pod "elasticsearch-client-6fb7844b6f-lbtq7" is waiting to start: PodInitializing

OR

Error from server (BadRequest): container "elasticsearch" in pod "elasticsearch-data-0" is waiting to start: PodInitializing

Any idea?

[avarf]

Update:
After some investigation, I found out that this is a problem with elasticsearch and pv. I will close this issue.

[davecore82]

If anyone else falls here while trying to enable privileged containers on microk8s, the way to do it is to add a PodSecurityPolicy to /var/snap/microk8s/current/args/kube-apiserver:

--enable-admission-plugins="NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,PodSecurityPolicy"

And restart the service:

sudo systemctl restart snap.microk8s.daemon-apiserver.service

EDIT: After more investigation, from what I understand to allow privileged containers you need to add the following line to /var/snap/microk8s/current/args/kube-apiserver:

--allow-privileged

And then restart the microk8s api server daemon:

sudo systemctl restart snap.microk8s.daemon-apiserver

But then it's also possible to use PodSecurityPolicy in combination with --allow-privileged if you want to allow privileged containers and restrict who is allowed to create them.

To do that, you add a PodSecurityPolicy to /var/snap/microk8s/current/args/kube-apiserver:

--enable-admission-plugins="PodSecurityPolicy"

And restart the service:

sudo systemctl restart snap.microk8s.daemon-apiserver