Cadvisor: Failed to start container manager: inotify_add_watch /sys/fs/cgroup/cpuacct,cpu: no such file or directory

[zhulh200868]

why I use the docker.io/google/cadvisor image will get out this question:
Failed to start container manager: inotify_add_watch /sys/fs/cgroup/cpuacct,cpu: no such file or directory

And when I use the docker.io/google/cadvisor:v0.27.4 it can work ?

I find the code and can't find something about the "/sys/fs/cgroup/cpuacct,cpu" !

[dashpole]

Please look through #1843, #1444, and #1461, and see if the solutions described there fix your issues

[zhulh200868]

@dashpole I know if I use the command like this :
mount -o remount,rw '/sys/fs/cgroup'
ln -s /sys/fs/cgroup/cpu,cpuacct /sys/fs/cgroup/cpuacct,cpu

But I don't want to use it,and when I start the cadvisor use the commad like this:
docker run \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:rw \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--volume=/sys/fs/cgroup/cpu,cpuacct:/sys/fs/cgroup/cpuacct,cpu:rw \
--publish=8080:8080 \
--name=cadvisor \
google/cadvisor:latest

It raises the warning is "System error: could not synchronise with container process ".
I want to know why the version is v0.27.4 it can work and the latest can't work ?

[maurorappa]

@zhulh200868, if you remove '--volume=/sys:/sys:ro' and the latest works fine on RHEL 7.4 BUT without container stats.

[lindhor]

I get this problem in all releases between 0.28.1 and 0.29.1 when running on CentOS 7.4.1708 (on host) with Docker version 1.12.6. Same problems with either the official google/cadvisor image based on Alpine 3.4 or my own image based on Alpine 3.5.2.

0.27.3, 0.27.4 and 0.28.0 works well (have not tried earlier).

When failing, log says:

F0303 13:38:53.371627       1 cadvisor.go:157] Failed to start container manager: inotify_add_watch /sys/fs/cgroup/cpuacct,cpu: no such file or directory

I start container with

docker run \
--privileged \
--publish=8080:8080 \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:rw \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--volume=/dev/disk/:/dev/disk:ro \
--name=cadvisor \
google/cadvisor:latest

Starting without --volume=/sys:/sys:ro works but as @maurorappa says, no containers are visible (just container images).

[dashpole]

@lindhor thanks for giving me versions that work/dont. I think this issue is because of https://github.com/google/cadvisor/pull/1807. We were experiencing some issues with fsnotify that I didn't have time to debug, and had to move back to inotify. It is depreciated, so it seems there are some problems with it.
This issue is likely linked to https://github.com/google/cadvisor/issues/1708.

[maurorappa]

@lindhor, @dashpole I used successfully this workaround;
I added a volume which basically does just the translation of the wrong path :
--volume=/sys/fs/cgroup/cpu,cpuacct:/sys/fs/cgroup/cpuacct,cpu
all containers are visible :)

[jcmcken]

Same issues here. The workaround won't work for me without doing something custom like writing a wrapper. We have some hosts that mount it one way and some that mount it the other (not sure if there's really any guarantee which way it will go). We're running cAdvisor in Mesos, so the task configuration can't be different per host. I would imagine there would be a similar problem in Kubernetes.

If there isn't, I think there needs to be some treatment of the cgroup names when cgroups are comounted. I don't think there's any guarantee of the ordering when e.g. cpu and cpuacct or net_cls and net_prio are comounted under the same hierarchy. The comounted names need to be considered identical if the members of the comount (regardless of order) are the same.

[dashpole]

Yeah, I'll prioritize getting back over to FsNotify soon. Inotify has been deprecated for a while and we should move off it.
I am closing this as a duplicate of https://github.com/google/cadvisor/issues/1708. Anyone who experiences issues can try one of the workarounds listed here, or track progress on the underlying issue there.

[dearkz]

@maurorappa thanks!! It can work for me

[agoretsky]

same issue