Linux: Request to enable CONFIG_CFS_BANDWIDTH for container CPU limits

[JohnStarich]

Is this the right place for my bug report?
Yes, this is a request to enable the kernel option config_cfs_bandwith. Specifically, this is to enable container CPU limits, using Docker in my case.

This is a follow-up on https://github.com/raspberrypi/linux/issues/2298. I have a PiZeroW and I'm willing to help run some performance tests to help identify any issues surrounding CPU limits.
Please advise which tests / benchmarks / guides are recommended. I'm not an experienced kernel developer but I'm happy to learn! 😄

Describe the bug
It's quite useful to set CPU limits on Docker containers to schedule them on nodes with sufficient capacity and ensure more critical workloads get enough CPU time. In my case, it would make deployments more portable between ubuntu and raspbian.

To reproduce
Easiest way to reproduce: run docker info. Any warnings are listed at the bottom.
(Another way is to deploy a Docker Swarm service with CPU limits, but this is an easier sanity-check.)

Expected behaviour
I would like to enable CPU limits, which results in no CPU limit warnings from docker info.

Actual behaviour
These warnings are shown at the bottom of docker info:

WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support

System

• Which model of Raspberry Pi? e.g. Pi3B+, PiZeroW
  
  
  
  • Seems to affect my Pi4B and PiZeroW. (Info below from the Pi4.)
  
  
• Which OS and version (cat /etc/rpi-issue)?
  
  
  
  • Raspberry Pi reference 2019-09-26 Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 80d486687ea77d31fc3fc13cf3a2f8b464e129be, stage5
  
  
• Which firmware version (vcgencmd version)?
  
  
  
  • Sep 24 2019 17:34:30 Copyright (c) 2012 Broadcom version cd3add54955f8fa065b414d8fc07c525e7ddffc8 (clean) (release) (start)
  
  
• Which kernel version (uname -a)?
  
  
  
  • Linux pitop 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux
  
  

Additional context
From previously closed issue (https://github.com/raspberrypi/linux/issues/2298):

Pea13 commented on Dec 6, 2017
Hi,

I'm heavily using Docker on raspian. My system:

raspian stretch
Linux docker1 4.14.4-v7+ #1060 SMP Tue Dec 5 16:01:44 GMT 2017 armv7l GNU/Linux
(i'm already on BRANCH next for rpi-update)
The current kernel lacks the support for config_cfs_bandwith. It's usefull with docker when you want to limit the cpu usage on your container (and since the raspberry pi is a small machine, i think it's a good idea).

% zgrep CONFIG_CFS_BANDWIDTH /proc/config.gz
# CONFIG_CFS_BANDWIDTH is not set

% /usr/share/docker-ce/contrib/check-config.sh

CONFIG_CFS_BANDWIDTH: missing

I don't know if there is a downside to enable it (impact on performance or size or both maybe).
An another solution would be to activate it on demand (i already put "cgroup_enable=memory swapaccount=1" on /boot/cmdline.txt to enable the cfs memory ).

Please let me know how I can help! Happy to learn, help run some tests, and share the results. 👍

[jmb12686]

I would also like to see cfs quota / cfs bandwidth support in the Raspbian kernel. While I understand the concern on memory and cpu overhead, now with 4 core, 4GB RAM Raspberry Pis, this needs to be addressed.

[yao2000]

cgroups cpu definitely should be compiled with. The runtime overhead is theoretically 0 because it is not turned on by default. And storage overhead is also neglectable considering the image file is already so big.

[steled]

Hi,

I'm trying to use kubernetes with crio and ran into the same error, see rasbian support in obs #4183
The guy that supports me asked me to compile a new kernel and activate the option CONFIG_CFS_BANDWIDTH
So after cloning the repo I wanted to activate the option via .config file but the option was already set to y

See below some outputs of my testing:

old kernel

pi@raspberrypi:~ $ uname -a
Linux raspberrypi 4.19.118-v7l+ #1311 SMP Mon Apr 27 14:26:42 BST 2020 armv7l GNU/Linux
pi@raspberrypi:~ $ sudo modprobe configs
pi@raspberrypi:~ $ zgrep CONFIG_CFS_BANDWIDTH /proc/config.gz
# CONFIG_CFS_BANDWIDTH is not set

new kernel

pi@raspberrypi:~ $ uname -a
Linux raspberrypi 5.4.51-v7l+ #1333 SMP Mon Aug 10 16:51:40 BST 2020 armv7l GNU/Linux
pi@raspberrypi:~ $ sudo modprobe configs
pi@raspberrypi:~ $ zgrep CONFIG_CFS_BANDWIDTH /proc/config.gz
# CONFIG_CFS_BANDWIDTH is not set

compiled kernel

pi@raspberrypi:~ $ uname -a
Linux raspberrypi 5.4.68-v7l-steled+ #1 SMP Wed Sep 30 18:24:25 BST 2020 armv7l GNU/Linux
pi@raspberrypi:~ $ sudo modprobe configs
pi@raspberrypi:~ $ zgrep CONFIG_CFS_BANDWIDTH /proc/config.gz
CONFIG_CFS_BANDWIDTH=y

Is possible that the option is implemented at a next kernel update?

[pelwell]

Commit a3e07c5dfef2664fc1d900ebb4a3f91c04ab700b, merged on September 1st, includes CONFIG_CFS_BANDWIDTH=y in all Pi defconfigs for kernel 5.4 and later. We won't be back-porting it to 4.19.

[steled]

Good to know...
Do you know when there will be a new official release where this change is included?

[pelwell]

I don't have a date for that, if you mean a Raspberry Pi OS kernel package.

[steled]

Hi,

I tested with the new kernel

Linux raspberrypi 5.4.72-v7l+ #1356 SMP Thu Oct 22 13:57:51 BST 2020 armv7l GNU/Linux

and everything seem to work now.

Thanks for your support

[popcornmix]

@JohnStarich okay to close?

[JohnStarich]

Cool! I'll try it out this weekend to see if Docker is happy with the new kernel.

[JohnStarich]

Looks like Docker is happy with CPU limits!

No more No cpu cfs ___ support warnings and my CPU limited services are deploying now. 👍

$ docker info
...
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No kernel memory TCP limit support
WARNING: No oom kill disable support

For others using Docker, I also removed the memory limit warnings by modifying /boot/cmdline.txt and adding cgroup_enable=memory to the list. After that, warnings dropped to only missing swap.

$ docker info
...
WARNING: No swap limit support

Thanks @pelwell et al! 🎉

[vhuang0501]

Can you share the OS package， @JohnStarich @pelwell

[pelwell]

All our kernels and firmware are available from the firmware repo: https://github.com/raspberrypi/firmware/commits/master

[JohnStarich]

@vhuang0501 On Raspbian, I just went through the usual OS upgrade process. i.e. apt update && apt full-upgrade