Cadvisor: Failed to start container manager: inotify_add_watch /sys/fs/cgroup/cpuacct,cpu: no such file or directory

[jlec]

When starting current canary on latest CentOS7, it fails with

Failed to start container manager: inotify_add_watch /sys/fs/cgroup/cpuacct,cpu: no such file or directory

It seems the naming is reversed (cpu,cpuacct vs cpuacct,cpu)

$ ll -d /sys/fs/cgroup/cpu*
drwxr-xr-x 5 root root  0 Aug 11 13:33 /sys/fs/cgroup/cpu,cpuacct
drwxr-xr-x 3 root root  0 Aug 11 13:33 /sys/fs/cgroup/cpuset
lrwxrwxrwx 1 root root 11 Aug 11 13:33 /sys/fs/cgroup/cpu -> cpu,cpuacct
lrwxrwxrwx 1 root root 11 Aug 11 13:33 /sys/fs/cgroup/cpuacct -> cpu,cpuacct

[jjduhamel]

I'm seeing this as well.

[taimir]

:+1: just stumbled upon the same problem

[vishh]

This is a docker issue and it should have been resolved in the more recent
versions cAdvisor.

On Tue, Sep 6, 2016 at 3:05 AM, Atanas Mirchev [email protected]
wrote:

👍 just stumbled upon the same problem

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/google/cadvisor/issues/1444#issuecomment-244906537,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGvIKMxRkbQvpYg4wSYJ64JczlW4vXmsks5qnTrvgaJpZM4JxdUg
.

[derekwaynecarr]

This should be fixed by https://github.com/google/cadvisor/pull/1476

[alexellis]

Also getting this issue with latest Docker image as per Github instructions page on SLES.

[soleil0-0]

Here is the solution that works https://github.com/kubernetes/kubernetes/issues/32728#issuecomment-252469277

[oszi]

This issue has not been fixed still. :( The workaround recommended by @qband did work.

REPOSITORY                  TAG                 IMAGE ID            CREATED             SIZE
docker.io/google/cadvisor   latest              75f88e3ec333        6 weeks ago         62.21 MB

[abelgana]

Experiencing the same issue. The workaround did work for me too.

[wongoo]

use the latest released version instead of the version latest.

[axiaoxin]

I run cadvisor in my centos7 is still failed
docker run --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro --volume=/dev/disk/:/dev/disk:ro --publish=8080:8080 --detach=false --name=cadvisor cadvisor:20180330 I0330 08:06:28.976330 1 storagedriver.go:50] Caching stats in memory for 2m0s I0330 08:06:28.976684 1 manager.go:151] cAdvisor running in container: "/sys/fs/cgroup/cpuacct,cpu" I0330 08:06:28.999279 1 fs.go:139] Filesystem UUIDs: map[2ecd79ee-5c25-45b5-bb2a-f7d7d4908a42:/dev/sda4 7031eb24-d2f7-497b-a0d3-119ffab6f8ef:/dev/sda1 8806a345-6dc8-4149-93b0-fd0ea6ea372f:/dev/sda3 a9724b4d-4dbb-4a1e-88fb-47fa69f0467a:/dev/sda2] I0330 08:06:28.999695 1 fs.go:140] Filesystem partitions: map[shm:{mountpoint:/dev/shm major:0 minor:101 fsType:tmpfs blockSize:0} none:{mountpoint:/ major:0 minor:100 fsType:aufs blockSize:0} tmpfs:{mountpoint:/dev major:0minor:104 fsType:tmpfs blockSize:0} /dev/root:{mountpoint:/rootfs major:8 minor:1 fsType:ext4 blockSize:0} /dev/sda3:{mountpoint:/rootfs/usr/local major:8 minor:3 fsType:ext4 blockSize:0} /dev/sda4:{mountpoint:/rootfs/data major:8 minor:4 fsType:ext4 blockSize:0}] I0330 08:06:29.006887 1 manager.go:225] Machine: {NumCores:24 CpuFrequency:2600102 MemoryCapacity:134880329728 HugePages:[{PageSize:2048 NumPages:4096}] MachineID:bf59f7060e9742e18a5153b62aee491c SystemUUID:58783500-0A81-11E5-ADD1-403EE33EA27C BootID:48492dba-828e-44c1-bbfa-0ef93a55a14e Filesystems:[{Device:/dev/sda4 DeviceMajor:8 DeviceMinor:4 Capacity:1427020939264 Type:vfs Inodes:88498176 HasInodes:true} {Device:shm DeviceMajor:0 DeviceMinor:101 Capacity:67108864 Type:vfs Inodes:16464884 HasInodes:true} {Device:none DeviceMajor:0 DeviceMinor:100 Capacity:1427020939264 Type:vfs Inodes:88498176 HasInodes:true} {Device:tmpfs DeviceMajor:0 DeviceMinor:104 Capacity:67440164864Type:vfs Inodes:16464884 HasInodes:true} {Device:/dev/root DeviceMajor:8 DeviceMinor:1 Capacity:21003628544 Type:vfs Inodes:1310720 HasInodes:true} {Device:/dev/sda3 DeviceMajor:8 DeviceMinor:3 Capacity:21003628544 Type:vfs Inodes:1310720 HasInodes:true}] DiskMap:map[43:1:{Name:nbd1 Major:43 Minor:1 Size:0 Scheduler:cfq} 43:12:{Name:nbd12 Major:43 Minor:12 Size:0 Scheduler:cfq} 43:6:{Name:nbd6 Major:43 Minor:6 Size:0 Scheduler:cfq} 43:8:{Name:nbd8 Major:43 Minor:8 Size:0 Scheduler:cfq} 8:0:{Name:sda Major:8 Minor:0 Size:1494996746240 Scheduler:cfq} 9:0:{Name:md0 Major:9 Minor:0 Size:0 Scheduler:none} 43:10:{Name:nbd10 Major:43 Minor:10 Size:0 Scheduler:cfq} 43:13:{Name:nbd13 Major:43 Minor:13 Size:0 Scheduler:cfq} 43:14:{Name:nbd14 Major:43 Minor:14 Size:0 Scheduler:cfq} 43:15:{Name:nbd15 Major:43 Minor:15 Size:0 Scheduler:cfq} 43:2:{Name:nbd2 Major:43 Minor:2 Size:0 Scheduler:cfq} 43:5:{Name:nbd5 Major:43Minor:5 Size:0 Scheduler:cfq} 43:7:{Name:nbd7 Major:43 Minor:7 Size:0 Scheduler:cfq} 43:9:{Name:nbd9 Major:43 Minor:9 Size:0 Scheduler:cfq} 43:0:{Name:nbd0 Major:43 Minor:0 Size:0 Scheduler:cfq} 43:11:{Name:nbd11 Major:43 Minor:11 Size:0 Scheduler:cfq} 43:3:{Name:nbd3 Major:43 Minor:3 Size:0 Scheduler:cfq} 43:4:{Name:nbd4 Major:43 Minor:4 Size:0 Scheduler:cfq}] NetworkDevices:[{Name:br-3f0d18e897c0 MacAddress:02:42:4e:60:5f:12 Speed:0 Mtu:1500} {Name:eth0MacAddress:7c:a2:3e:8c:f0:3c Speed:0 Mtu:1500} {Name:eth1 MacAddress:7c:a2:3e:8c:f0:3d Speed:1000 Mtu:1500} {Name:tunl0 MacAddress:00:00:00:00 Speed:0 Mtu:0}] Topology:[{Id:0 Memory:68689707008 Cores:[{Id:0 Threads:[0 12] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:1 Threads:[1 13] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:UnifiedLevel:2}]} {Id:2 Threads:[2 14] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:3 Threads:[3 15] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:4 Threads:[4 16] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:5 Threads:[5 17] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]}] Caches:[{Size:15728640Type:Unified Level:3}]} {Id:1 Memory:68719476736 Cores:[{Id:0 Threads:[6 18] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:1 Threads:[7 19] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:2 Threads:[8 20] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:3 Threads:[9 21] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:4 Threads:[10 22] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:5 Threads:[11 23] Caches:[{Size:32768 Type:Data Level:1} {Size:32768Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]}] Caches:[{Size:15728640 Type:Unified Level:3}]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} I0330 08:06:29.007899 1 manager.go:231] Version: {KernelVersion:3.10.107-1-tlinux2-0046 ContainerOsVersion:Alpine Linux v3.4 DockerVersion:1.12.6 DockerAPIVersion:1.24 CadvisorVersion:v0.28.3 CadvisorRevision:1e567c2} I0330 08:06:29.017279 1 factory.go:356] Registering Docker factory I0330 08:06:31.017629 1 factory.go:54] Registering systemd factory I0330 08:06:31.018829 1 factory.go:86] Registering Raw factory I0330 08:06:31.019825 1 manager.go:1178] Started watching for new ooms in manager W0330 08:06:31.019855 1 manager.go:313] Could not configure a source for OOM detection, disabling OOM events: open /dev/kmsg: no such file or directory I0330 08:06:31.026114 1 manager.go:329] Starting recovery of all containers I0330 08:06:31.107502 1 manager.go:334] Recovery completed F0330 08:06:31.124525 1 cadvisor.go:156] Failed to start container manager: inotify_add_watch /sys/fs/cgroup/cpuacct,cpu: no such file or directory

ll -d /sys/fs/cgroup/cpu*

lrwxrwxrwx 1 root root 11 Jan 24 10:19 /sys/fs/cgroup/cpu -> cpu,cpuacct
drwxr-xr-x 4 root root  0 Feb  7 11:38 /sys/fs/cgroup/cpu,cpuacct
lrwxrwxrwx 1 root root 11 Jan 24 10:19 /sys/fs/cgroup/cpuacct -> cpu,cpuacct
drwxr-xr-x 3 root root  0 Jan 24 10:19 /sys/fs/cgroup/cpuset

Fixed it by:

mount -o remount,rw '/sys/fs/cgroup'
ln -s /sys/fs/cgroup/cpu,cpuacct /sys/fs/cgroup/cpuacct,cpu

[kartikjena3]

@axiaoxin nice work, Working perfectly

[Cherishty]

@axiaoxin It works, but will reproduce whenever I reboot the machine, any workaround?

[tn-osimis]

@axiaoxin It works, but will reproduce whenever I reboot the machine, any workaround?

This is what we do:

$ tree cadvisor-1843-workaround/
cadvisor-1843-workaround/
├── add-symlink.service
├── add-symlink.sh
└── install.sh
$ cat cadvisor-1843-workaround/add-symlink.service 
[Unit]
Description=cAdvisor #1843 workaround

[Service]
Type=oneshot
ExecStart=/usr/local/bin/cadvisor-1843-add-symlink

[Install]
WantedBy=multi-user.target
$ cat cadvisor-1843-workaround/add-symlink.sh 
#!/usr/bin/env bash
# https://github.com/google/cadvisor/issues/1843
set -o errexit
set -o xtrace
mount --options=remount,rw /sys/fs/cgroup
ln --symbolic /sys/fs/cgroup/{"cpu,cpuacct","cpuacct,cpu"}
$ cat cadvisor-1843-workaround/install.sh 
#!/usr/bin/env bash
set -o errexit
set -o xtrace
ln add-symlink.sh /usr/local/bin/cadvisor-1843-add-symlink
ln add-symlink.service /etc/systemd/system/cadvisor-1843-add-symlink.service
systemctl enable cadvisor-1843-add-symlink
systemctl start cadvisor-1843-add-symlink
systemctl status cadvisor-1843-add-symlink