Packages: docker-ce: broken, cgroup subsystem 'devices' is missing

[domomg]

Mantainer: @G-M0N3Y-2503
Environment: x86_64 on baremetal and VMWare, openwrt master

Description:
I've been having some troubles with the latest OpenWRT snapshots, regarding docker-ce.

My last two attempts have been with r-14035 three days ago, and r-14085 yesterday.

They were both compiled with default options, I only selected docker-related stuffs under Utilities.

This issue happens both inside a VMWare Fusion virtual machine, and baremetal on three different x86_64 machines.

In both cases I successfully got a working image from the build system, except when I try to run docker-ce on them, it fails because the 'device' cgroup subsystem isn't mounted, and it's also missing from /sys/fs/cgroup.

In both cases, the cgroup_root and most of the subsystems are correctly mounted.

For comparison, I fired up an old VM with r-12719 built back in March, where everything works without a hitch.

r-14085 (missing 'devices' and 'freezer' ):

r-12719:

Any attempt to use the docker CLI utility returns a "Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"

Trying to start the dockerd daemon manually ends up with failure due to the 'devices' subsystem being unmounted:

As far as I know, the last snapshot working for me is r13834, built about 3 weeks ago for a Raspberry Pi 3:

Fixes attempted so far:

• tried with the kernel boot option "cgroup_enable=devices" and, after a reboot, made the 'devices' directory inside /sys/fs/cgroup and attempted to mount it with no success, I get the following error:
  "cgroup1: Unknown subsys name 'devices' "

mkdir /sys/fs/cgroup/devices mount -t cgroup devices -o devices /sys/fs/cgroup/devices

[G-M0N3Y-2503]

I could reproduce this even with all the docker kernel options enabled.
It appears that they aren't even in /proc/cgroups so I suspect that https://github.com/openwrt/openwrt/commit/d1a8217d87bffa33fd7d4562b3ed2f797c14beaf may have introduced this error.

[G-M0N3Y-2503]

Just tested and it's definitely the change to cgroups v2 done in https://github.com/openwrt/openwrt/commit/d1a8217d87bffa33fd7d4562b3ed2f797c14beaf.
I'll look into adding the legacy options for a custom build but it looks like the snapshots will be broken until https://github.com/moby/moby/issues/40360

[peterwillcn]

https://github.com/docker/cli/issues/2104

[domomg]

Just tested and it's definitely the change to cgroups v2 done in openwrt/openwrt@d1a8217.
I'll look into adding the legacy options for a custom build but it looks like the snapshots will be broken until moby/moby#40360

First of all, thank you for pointing me to the right direction, it didn't occur to me at all to look at the kernel parameters in the config file, you saved me quite a lot of time.

I did a test rolling back to the commit immediately behind the one giving us trouble, https://github.com/openwrt/openwrt/commit/917980fd8a6589d6911797211f1871016f527f8c , compiled from there, and lo and behold, it works.

I'll stick to this version until there's a fix for this issue, I'm aware it's not a long term solution given cgroup v1 is legacy, but at least for now it works.

Thank you again for your help.

[peterwillcn]

@G-M0N3Y-2503 https://github.com/openwrt/openwrt/commit/d1a8217d87bffa33fd7d4562b3ed2f797c14beaf

You need to enable build-time options for legacy-cgroup1-only controllers (such as devices, net_prio, net_cls) in menuconfig, they are no longer included by default.
It'd be sad to encourage the use of legacy functions which prevent using the modern interfaces (which is true for cgroup1 controllers). Hence, as 19.07 came without cgroup1 per default, we will keep it like that for 20.x (and supposedly everything after). cgroup2 is enabled and supports device-firewalling using eBPF, it has been decided that the previous 'devices' cgroup will not be implemented in cgroup2 (hence Docker needs to change and make use of cgroup2 in order to work on mordern systems).

[G-M0N3Y-2503]

I'll look into making the current version of docker dependent on these legacy kernel options for now, which will mean docker won't be available in snapshots until the version that supports cgroups2 is released.

[peterwillcn]

I temporarily fixed this problem by enabling the following kernel configuration

+CONFIG_KERNEL_CGROUP_DEVICE=y
+CONFIG_KERNEL_CGROUP_FREEZER=y
+CONFIG_KERNEL_CGROUP_NET_CLASSID=y
+CONFIG_KERNEL_CGROUP_NET_PRIO=y
+CONFIG_KERNEL_NET_CLS_CGROUP=y

[najdanovicivan]

I've submitted this PR https://github.com/openwrt/openwrt/pull/3504 it should make it possible to use docker-ce on shapshot build for !SMALL_FLASH devices

[G-M0N3Y-2503]

For awareness, CGroups V2 should be supported in [Docker Engine] 2020 release