Origin: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"

[robdaems]

Version

[root@rhel ~]# openshift version
openshift v3.7.1+ab0f056
kubernetes v1.7.6+a08f5eeb62
etcd 3.2.8
[root@rhel ~]# uname -a
Linux rhel 3.10.0-693.17.1.el7.x86_64 #1 SMP Sun Jan 14 10:36:03 EST 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@rhel ~]# docker --version
Docker version 17.12.1-ce, build 7390fc6
[root@rhel ~]#

Steps To Reproduce

On a Clean RHEL 7.4 server (1 machine) installation I try to run openshift as a All-in-One

[root@rhel ~]# openshift start

Googling around doesn't give me a proper solution.

Current Result

W0228 16:22:37.679295    2019 cni.go:189] Unable to update cni config: No networks found in /etc/cni/net.d
I0228 16:22:37.753317    2019 docker_service.go:209] Docker cri networking managed by kubernetes.io/no-op
F0228 16:22:37.761061    2019 node.go:264] failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"

[jwforres]

@openshift/sig-pod

[aveshagarwal]

This seems just a configuration issue, change docker's cgroup driver to systemd.

[aveshagarwal]

use with docker: --exec-opt native.cgroupdriver=systemd

[robdaems]

@aveshagarwal

use with docker: --exec-opt native.cgroupdriver=systemd

Which file(s) do I've to change and which variable?
Since I found this solution also on google, but it didn't help. So I started again with a clean install.

[root@rhel ~]# docker info | grep Cgroup
Cgroup Driver: cgroupfs
[root@rhel ~]# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2018-03-01 12:31:28 CET; 1h 39min ago
     Docs: https://docs.docker.com
 Main PID: 18306 (dockerd)
   Memory: 25.9M
   CGroup: /system.slice/docker.service
           ├─18306 /usr/bin/dockerd
           └─18309 docker-containerd --config /var/run/docker/containerd/containerd.toml

Mar 01 12:31:27 rhel dockerd[18306]: time="2018-03-01T12:31:27+01:00" level=info msg=serving... address="/var/run/docker/containerd/docker-containerd.sock" module="containerd/grpc"
Mar 01 12:31:27 rhel dockerd[18306]: time="2018-03-01T12:31:27+01:00" level=info msg="containerd successfully booted in 0.007546s" module=containerd
Mar 01 12:31:27 rhel dockerd[18306]: time="2018-03-01T12:31:27.832225213+01:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Mar 01 12:31:27 rhel dockerd[18306]: time="2018-03-01T12:31:27.832986792+01:00" level=info msg="Loading containers: start."
Mar 01 12:31:28 rhel dockerd[18306]: time="2018-03-01T12:31:28.068006399+01:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Mar 01 12:31:28 rhel dockerd[18306]: time="2018-03-01T12:31:28.246296496+01:00" level=info msg="Loading containers: done."
Mar 01 12:31:28 rhel dockerd[18306]: time="2018-03-01T12:31:28.272793007+01:00" level=info msg="Docker daemon" commit=7390fc6 graphdriver(s)=overlay2 version=17.12.1-ce
Mar 01 12:31:28 rhel dockerd[18306]: time="2018-03-01T12:31:28.273026469+01:00" level=info msg="Daemon has completed initialization"
Mar 01 12:31:28 rhel dockerd[18306]: time="2018-03-01T12:31:28.298266653+01:00" level=info msg="API listen on /var/run/docker.sock"
Mar 01 12:31:28 rhel systemd[1]: Started Docker Application Container Engine.

[robdaems]

It's fixed by adapting the line in /usr/lib/systemd/system/docker.service

ExecStart=/usr/bin/dockerd \
          --exec-opt native.cgroupdriver=systemd

[krumware]

Seeing this issue in WSL.
We don't have easy control over systemd, so the above instructions are difficult to follow.

Users may also be able to add the instructions in the file /etc/default/docker.
Adding the line: DOCKER_OPTS="--exec-opt native.cgroupdriver=systemd"

After doing this, our new error is

F0627 12:04:35.582876    3520 network.go:179] error: Could not initialize Kubernetes Proxy. You must run this process as root (and if containerized, in the host network namespace as privileged) to use the service proxy: failed to initialize iptables: error appending rule: exit status 1: iptables: No chain/target/match by that name.

[harshalkwagh]

in my case i already have --exec-opt native.cgroupdriver=systemd option in /usr/lib/systemd/system/docker.service file but still my CgroupDriver is using cgroupfs

ExecStart=/usr/bin/dockerd-current \
--add-runtime docker-runc=/usr/libexec/docker/docker-runc-current \
--default-runtime=docker-runc \
--exec-opt native.cgroupdriver=systemd \
--userland-proxy-path=/usr/libexec/docker/docker-proxy-current \
--init-path=/usr/libexec/docker/docker-init-current \
--seccomp-profile=/etc/docker/seccomp.json \

Docker info:
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: 9c3c5f853ebf0ffac0d087e94daef462133b69c7 (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: fec3683b971d9c3ef73f284f176672c44b448662 (expected: 949e6facb77383876aeff8a6944dde66b3089574)

How to change cgroup to systemd

[arrahimi2]

--exec-opt native.cgroupdriver=systemd

location for docker 17.03 on ubuntu 16.04 = /lib/systemd/system/docker.service

[sirsikar-lalit]

I was able to resolve this issue for my use-case by having the same cgroup driver for docker and kubelet. In my case on CentOS 7.6 I could fix the issue by adding --exec-opt native.cgroupdriver=systemd to docker systemd process and adding --cgroup-driver=systemd to kubelet systemd process. This way both kubelet and docker are consuming the same cgroup-driver and both operate normally.

Most likely these drivers can be set with any other driver types as well but that was not a part of my testing.