Cadvisor: permission denied when run the cadvisor in docker container.

[liuyunsh]

I just run the cadvisor by the following command:
docker run \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:rw \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--publish=9090:8080 \
--detach=true \
--name=cadvisor \
google/cadvisor:latest

Environment: centos7

error log:
I1203 14:25:54.658471 00001 storagedriver.go:89] Caching 60 recent stats in memory; using "" storage driver
log: exiting because of error: log: cannot create log: open /tmp/cadvisor.f2a70ddfacea.root.log.INFO.20141203-142554.1: permission denied

As I checked the /tmp directory, all of user on system have read and write authority.

[vmarmol]

Does the directory have space on it?

Can you also try running with --logtostderr:

docker run \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:rw \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--publish=9090:8080 \
--detach=true \
--name=cadvisor \
google/cadvisor:latest --logtostderr

[liuyunsh]

Hi vmarmol, thanks for your response.
Yes, the space on that directory is enough.
I tried the way you suggested, it seems that that container can work now, however,
The web page of http://ip:port come out a error as following:
Failed to get container "/" with error: unable to find data for container /
Any suggestion?

[vishh]

In centOS cgroups are not mounted under "/sys". So you need to pass an
additional flag "--volume=/cgroup:/cgroup". Take a look at this page
https://github.com/google/cadvisor/blob/master/docs/running.md.

On Thu, Dec 4, 2014 at 7:28 AM, liuyunsh [email protected] wrote:

Hi vmarmol, thanks for your response.
Yes, the space on that directory is enough.
I tried the way you suggested, it seems that that container can work now,
however,
The web page of http://ip:port come out a error as following:
Failed to get container "/" with error: unable to find data for container /
Any suggestion?

—
Reply to this email directly or view it on GitHub
https://github.com/google/cadvisor/issues/333#issuecomment-65526156.

[liuyunsh]

hi vishh,
Yes, centos7' cgroups located in /sys/fs/cgroup/
so I tried the following command with same error.
docker run \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:rw \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--volume=/sys/fs/cgroup/:/cgroup \
--publish=9090:8080 \
--detach=true \
--name=cadvisor \
google/cadvisor:latest --logtostderr

[scotticles]

I am getting the same thing, just wanted to try this out. -- CentOS7
sudo docker run \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:rw \
--volume=/sys:/sys:ro \
--volume=/sys/fs/cgroup/:/cgroup \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--publish=8080:8080 \
--detach=true \
--name=cadvisor \
google/cadvisor:latest --logtostderr

[vmarmol]

What is the full cAdvisor log?
On Dec 3, 2014 6:54 PM, "Scott" [email protected] wrote:

I am getting the same thing, just wanted to try this out.
sudo docker run \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:rw \
--volume=/sys:/sys:ro \
--volume=/sys/fs/cgroup/:/cgroup \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--publish=8080:8080 \
--detach=true \
--name=cadvisor \
google/cadvisor:latest --logtostderr

—
Reply to this email directly or view it on GitHub
https://github.com/google/cadvisor/issues/333#issuecomment-65530379.

[liuyunsh]

{"log":"I1204 05:06:14.456169 00001 container.go:189] Failed to update stats for container \"/system.slice\": stat /cgroup/cpu/system.slice: permission denied\n","stream":"stderr","time":"2014-12-04T05:06:14.456234515Z"}
{"log":"W1204 05:06:14.456195 00001 container.go:112] Failed to get RecentStats(\"/system.slice\") while determining the next housekeeping: unable to find data for container /system.slice\n","stream":"stderr","time":"2014-12-04T05:06:14.456234515Z"}
{"log":"I1204 05:06:14.462423 00001 container.go:189] Failed to update stats for container \"/system.slice/iscsiuio.socket\": stat /cgroup/cpu/system.slice/iscsiuio.socket: permission denied\n","stream":"stderr","time":"2014-12-04T05:06:14.462506599Z"}
{"log":"W1204 05:06:14.462450 00001 container.go:112] Failed to get RecentStats(\"/system.slice/iscsiuio.socket\") while determining the next housekeeping: unable to find data for container /system.slice/iscsiuio.socket\n","stream":"stderr","time":"2014-12-04T05:06:14.462506599Z"}
{"log":"I1204 05:06:14.468533 00001 container.go:189] Failed to update stats for container \"/system.slice/dev-mapper-centos\x2dswap.swap\": stat /cgroup/cpu/system.slice/dev-mapper-centos\x2dswap.swap: permission denied\n","stream":"stderr","time":"2014-12-04T05:06:14.468576181Z"}
{"log":"W1204 05:06:14.468563 00001 container.go:112] Failed to get RecentStats(\"/system.slice/dev-mapper-centos\x2dswap.swap\") while determining the next housekeeping: unable to find data for container /system.slice/dev-mapper-centos\x2dswap.swap\n","stream":"stderr","time":"2014-12-04T05:06:14.468613314Z"}
{"log":"I1204 05:06:14.474999 00001 container.go:189] Failed to update stats for container \"/system.slice/sshd.service\": stat /cgroup/cpu/system.slice/sshd.service: permission denied\n","stream":"stderr","time":"2014-12-04T05:06:14.475206764Z"}
{"log":"W1204 05:06:14.475040 00001 container.go:112] Failed to get RecentStats(\"/system.slice/sshd.service\") while determining the next housekeeping: unable to find data for container /system.slice/sshd.service\n","stream":"stderr","time":"2014-12-04T05:06:14.475206764Z"}
{"log":"I1204 05:06:14.481028 00001 container.go:189] Failed to update stats for container \"/system.slice/systemd-logind.service\": stat /cgroup/cpu/system.slice/systemd-logind.service: permission denied\n","stream":"stderr","time":"2014-12-04T05:06:14.481085612Z"}
{"log":"W1204 05:06:14.481068 00001 container.go:112] Failed to get RecentStats(\"/system.slice/systemd-logind.service\") while determining the next housekeeping: unable to find data for container /system.slice/systemd-logind.service\n","stream":"stderr","time":"2014-12-04T05:06:14.481129612Z"}
{"log":"I1204 05:06:14.487069 00001 container.go:189] Failed to update stats for container \"/system.slice/NetworkManager.service\": stat /cgroup/cpu/system.slice/NetworkManager.service: permission denied\n","stream":"stderr","time":"2014-12-04T05:06:14.487102662Z"}
{"log":"W1204 05:06:14.487092 00001 container.go:112] Failed to get RecentStats(\"/system.slice/NetworkManager.service\") while determining the next housekeeping: unable to find data for container /system.slice/NetworkManager.service\n","stream":"stderr","time":"2014-12-04T05:06:14.487137988Z"}
{"log":"I1204 05:06:14.493288 00001 container.go:189] Failed to update stats for container \"/system.slice/avahi-daemon.socket\": stat /cgroup/cpu/system.slice/avahi-daemon.socket: permission denied\n","stream":"stderr","time":"2014-12-04T05:06:14.493401236Z"}
{"log":"W1204 05:06:14.493310 00001 container.go:112] Failed to get RecentStats(\"/system.slice/avahi-daemon.socket\") while determining the next housekeeping: unable to find data for container /system.slice/avahi-daemon.socket\n","stream":"stderr","time":"2014-12-04T05:06:14.493401236Z"}
{"log":"I1204 05:06:14.499541 00001 container.go:189] Failed to update stats for container \"/system.slice/dev-disk-by\x2duuid-a5d93a86\x2da084\x2d4542\x2dace3\x2d6fa218904979.swap\": stat /cgroup/cpu/system.slice/dev-disk-by\x2duuid-a5d93a86\x2da084\x2d4542\x2dace3\x2d6fa218904979.swap: permission denied\n","stream":"stderr","time":"2014-12-04T05:06:14.499698731Z"}
{"log":"W1204 05:06:14.499563 00001 container.go:112] Failed to get RecentStats(\"/system.slice/dev-disk-by\x2duuid-a5d93a86\x2da084\x2d4542\x2dace3\x2d6fa218904979.swap\") while determining the next housekeeping: unable to find data for container /system.slice/dev-disk-by\x2duuid-a5d93a86\x2da084\x2d4542\x2dace3\x2d6fa218904979.swap\n","stream":"stderr","time":"2014-12-04T05:06:14.499698731Z"}

[scotticles]

Mine looks similiar I noticed at the start it said permission denied on the
docker.sock

On Wed, Dec 3, 2014, 10:10 PM liuyunsh [email protected] wrote:

{"log":"I1204 05:06:14.456169 00001 container.go:189] Failed to update
stats for container \"/system.slice\": stat /cgroup/cpu/system.slice:
permission
denied\n","stream":"stderr","time":"2014-12-04T05:06:14.456234515Z"}
{"log":"W1204 05:06:14.456195 00001 container.go:112] Failed to get
RecentStats(\"/system.slice\") while determining the next housekeeping:
unable to find data for container
/system.slice\n","stream":"stderr","time":"2014-12-04T05:06:14.456234515Z"}
{"log":"I1204 05:06:14.462423 00001 container.go:189] Failed to update
stats for container \"/system.slice/iscsiuio.socket\": stat
/cgroup/cpu/system.slice/iscsiuio.socket: permission
denied\n","stream":"stderr","time":"2014-12-04T05:06:14.462506599Z"}
{"log":"W1204 05:06:14.462450 00001 container.go:112] Failed to get
RecentStats(\"/system.slice/iscsiuio.socket\") while determining the next
housekeeping: unable to find data for container
/system.slice/iscsiuio.socket\n","stream":"stderr","time":"2014-12-04T05:06:14.462506599Z"}
{"log":"I1204 05:06:14.468533 00001 container.go:189] Failed to update
stats for container \"/system.slice/dev-mapper-centos\x2dswap.swap\": stat
/cgroup/cpu/system.slice/dev-mapper-centos\x2dswap.swap: permission
denied\n","stream":"stderr","time":"2014-12-04T05:06:14.468576181Z"}
{"log":"W1204 05:06:14.468563 00001 container.go:112] Failed to get
RecentStats(\"/system.slice/dev-mapper-centos\x2dswap.swap\") while
determining the next housekeeping: unable to find data for container
/system.slice/dev-mapper-centos\x2dswap.swap\n","stream":"stderr","time":"2014-12-04T05:06:14.468613314Z"}
{"log":"I1204 05:06:14.474999 00001 container.go:189] Failed to update
stats for container \"/system.slice/sshd.service\": stat
/cgroup/cpu/system.slice/sshd.service: permission
denied\n","stream":"stderr","time":"2014-12-04T05:06:14.475206764Z"}
{"log":"W1204 05:06:14.475040 00001 container.go:112] Failed to get
RecentStats(\"/system.slice/sshd.service\") while determining the next
housekeeping: unable to find data for container
/system.slice/sshd.service\n","stream":"stderr","time":"2014-12-04T05:06:14.475206764Z"}
{"log":"I1204 05:06:14.481028 00001 container.go:189] Failed to update
stats for container \"/system.slice/systemd-logind.service\": stat
/cgroup/cpu/system.slice/systemd-logind.service: permission
denied\n","stream":"stderr","time":"2014-12-04T05:06:14.481085612Z"}
{"log":"W1204 05:06:14.481068 00001 container.go:112] Failed to get
RecentStats(\"/system.slice/systemd-logind.service\") while determining the
next housekeeping: unable to find data for container
/system.slice/systemd-logind.service\n","stream":"stderr","time":"2014-12-04T05:06:14.481129612Z"}
{"log":"I1204 05:06:14.487069 00001 container.go:189] Failed to update
stats for container \"/system.slice/NetworkManager.service\": stat
/cgroup/cpu/system.slice/NetworkManager.service: permission
denied\n","stream":"stderr","time":"2014-12-04T05:06:14.487102662Z"}
{"log":"W1204 05:06:14.487092 00001 container.go:112] Failed to get
RecentStats(\"/system.slice/NetworkManager.service\") while determining the
next housekeeping: unable to find data for container
/system.slice/NetworkManager.service\n","stream":"stderr","time":"2014-12-04T05:06:14.487137988Z"}
{"log":"I1204 05:06:14.493288 00001 container.go:189] Failed to update
stats for container \"/system.slice/avahi-daemon.socket\": stat
/cgroup/cpu/system.slice/avahi-daemon.socket: permission
denied\n","stream":"stderr","time":"2014-12-04T05:06:14.493401236Z"}
{"log":"W1204 05:06:14.493310 00001 container.go:112] Failed to get
RecentStats(\"/system.slice/avahi-daemon.socket\") while determining the
next housekeeping: unable to find data for container
/system.slice/avahi-daemon.socket\n","stream":"stderr","time":"2014-12-04T05:06:14.493401236Z"}
{"log":"I1204 05:06:14.499541 00001 container.go:189] Failed to update
stats for container
\"/system.slice/dev-disk-by\x2duuid-a5d93a86\x2da084\x2d4542\x2dace3\x2d6fa218904979.swap\":
stat
/cgroup/cpu/system.slice/dev-disk-by\x2duuid-a5d93a86\x2da084\x2d4542\x2dace3\x2d6fa218904979.swap:
permission
denied\n","stream":"stderr","time":"2014-12-04T05:06:14.499698731Z"}
{"log":"W1204 05:06:14.499563 00001 container.go:112] Failed to get
RecentStats(\"/system.slice/dev-disk-by\x2duuid-a5d93a86\x2da084\x2d4542\x2dace3\x2d6fa218904979.swap\")
while determining the next housekeeping: unable to find data for container
/system.slice/dev-disk-by\x2duuid-a5d93a86\x2da084\x2d4542\x2dace3\x2d6fa218904979.swap\n","stream":"stderr","time":"2014-12-04T05:06:14.499698731Z"}

—
Reply to this email directly or view it on GitHub
https://github.com/google/cadvisor/issues/333#issuecomment-65538806.

[vmarmol]

Hmmm haven't seen that one before. Can you provide the output of ls -l /cgroup/cpu?
On Dec 3, 2014 9:56 PM, "Scott" [email protected] wrote:

Mine looks similiar I noticed at the start it said permission denied on
the
docker.sock

On Wed, Dec 3, 2014, 10:10 PM liuyunsh [email protected] wrote:

{"log":"I1204 05:06:14.456169 00001 container.go:189] Failed to update
stats for container \"/system.slice\": stat /cgroup/cpu/system.slice:
permission
denied\n","stream":"stderr","time":"2014-12-04T05:06:14.456234515Z"}
{"log":"W1204 05:06:14.456195 00001 container.go:112] Failed to get
RecentStats(\"/system.slice\") while determining the next housekeeping:
unable to find data for container

/system.slice\n","stream":"stderr","time":"2014-12-04T05:06:14.456234515Z"}
{"log":"I1204 05:06:14.462423 00001 container.go:189] Failed to update
stats for container \"/system.slice/iscsiuio.socket\": stat
/cgroup/cpu/system.slice/iscsiuio.socket: permission
denied\n","stream":"stderr","time":"2014-12-04T05:06:14.462506599Z"}
{"log":"W1204 05:06:14.462450 00001 container.go:112] Failed to get
RecentStats(\"/system.slice/iscsiuio.socket\") while determining the
next
housekeeping: unable to find data for container

/system.slice/iscsiuio.socket\n","stream":"stderr","time":"2014-12-04T05:06:14.462506599Z"}

{"log":"I1204 05:06:14.468533 00001 container.go:189] Failed to update
stats for container \"/system.slice/dev-mapper-centos\x2dswap.swap\":
stat
/cgroup/cpu/system.slice/dev-mapper-centos\x2dswap.swap: permission
denied\n","stream":"stderr","time":"2014-12-04T05:06:14.468576181Z"}
{"log":"W1204 05:06:14.468563 00001 container.go:112] Failed to get
RecentStats(\"/system.slice/dev-mapper-centos\x2dswap.swap\") while
determining the next housekeeping: unable to find data for container

/system.slice/dev-mapper-centos\x2dswap.swap\n","stream":"stderr","time":"2014-12-04T05:06:14.468613314Z"}

{"log":"I1204 05:06:14.474999 00001 container.go:189] Failed to update
stats for container \"/system.slice/sshd.service\": stat
/cgroup/cpu/system.slice/sshd.service: permission
denied\n","stream":"stderr","time":"2014-12-04T05:06:14.475206764Z"}
{"log":"W1204 05:06:14.475040 00001 container.go:112] Failed to get
RecentStats(\"/system.slice/sshd.service\") while determining the next
housekeeping: unable to find data for container

/system.slice/sshd.service\n","stream":"stderr","time":"2014-12-04T05:06:14.475206764Z"}

{"log":"I1204 05:06:14.481028 00001 container.go:189] Failed to update
stats for container \"/system.slice/systemd-logind.service\": stat
/cgroup/cpu/system.slice/systemd-logind.service: permission
denied\n","stream":"stderr","time":"2014-12-04T05:06:14.481085612Z"}
{"log":"W1204 05:06:14.481068 00001 container.go:112] Failed to get
RecentStats(\"/system.slice/systemd-logind.service\") while determining
the
next housekeeping: unable to find data for container

/system.slice/systemd-logind.service\n","stream":"stderr","time":"2014-12-04T05:06:14.481129612Z"}

{"log":"I1204 05:06:14.487069 00001 container.go:189] Failed to update
stats for container \"/system.slice/NetworkManager.service\": stat
/cgroup/cpu/system.slice/NetworkManager.service: permission
denied\n","stream":"stderr","time":"2014-12-04T05:06:14.487102662Z"}
{"log":"W1204 05:06:14.487092 00001 container.go:112] Failed to get
RecentStats(\"/system.slice/NetworkManager.service\") while determining
the
next housekeeping: unable to find data for container

/system.slice/NetworkManager.service\n","stream":"stderr","time":"2014-12-04T05:06:14.487137988Z"}

{"log":"I1204 05:06:14.493288 00001 container.go:189] Failed to update
stats for container \"/system.slice/avahi-daemon.socket\": stat
/cgroup/cpu/system.slice/avahi-daemon.socket: permission
denied\n","stream":"stderr","time":"2014-12-04T05:06:14.493401236Z"}
{"log":"W1204 05:06:14.493310 00001 container.go:112] Failed to get
RecentStats(\"/system.slice/avahi-daemon.socket\") while determining the
next housekeeping: unable to find data for container

/system.slice/avahi-daemon.socket\n","stream":"stderr","time":"2014-12-04T05:06:14.493401236Z"}

{"log":"I1204 05:06:14.499541 00001 container.go:189] Failed to update
stats for container

\"/system.slice/dev-disk-by\x2duuid-a5d93a86\x2da084\x2d4542\x2dace3\x2d6fa218904979.swap\":

stat

/cgroup/cpu/system.slice/dev-disk-by\x2duuid-a5d93a86\x2da084\x2d4542\x2dace3\x2d6fa218904979.swap:

permission
denied\n","stream":"stderr","time":"2014-12-04T05:06:14.499698731Z"}
{"log":"W1204 05:06:14.499563 00001 container.go:112] Failed to get

RecentStats(\"/system.slice/dev-disk-by\x2duuid-a5d93a86\x2da084\x2d4542\x2dace3\x2d6fa218904979.swap\")

while determining the next housekeeping: unable to find data for
container

/system.slice/dev-disk-by\x2duuid-a5d93a86\x2da084\x2d4542\x2dace3\x2d6fa218904979.swap\n","stream":"stderr","time":"2014-12-04T05:06:14.499698731Z"}

—
Reply to this email directly or view it on GitHub
https://github.com/google/cadvisor/issues/333#issuecomment-65538806.

—
Reply to this email directly or view it on GitHub
https://github.com/google/cadvisor/issues/333#issuecomment-65541533.

[liuyunsh]

sure,please check the following infor
[root@docker fs]# pwd
/sys/fs
[root@docker fs]# ls -l ./cgroup/cpu
lrwxrwxrwx. 1 root root 11 Dec 3 04:12 ./cgroup/cpu -> cpu,cpuacct
[root@docker fs]#

[vmarmol]

Hmmm so cAdvisor things the cgroups are mounted at /cgroup but you seem to say they're in /sys/fs/cgroup. What is the output of grep cgroup /proc/mounts. Curious why cAdvisor thinks otherwise.

[scotticles]

Here is my grep on the /proc/mounts

tmpfs /sys/fs/cgroup tmpfs rw,seclabel,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpuacct,cpu 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/net_cls cgroup rw,nosuid,nodev,noexec,relatime,net_cls 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0

[scotticles]

Here are the first few entries right after I start the cadvisor container:
I1205 03:14:31.761574 00001 storagedriver.go:89] Caching 60 recent stats in memory; using "" storage driver
I1205 03:14:31.761740 00001 manager.go:77] cAdvisor running in container: "/system.slice/docker-721c8d048f594698ba7749b41cb3d7d6439e421d0604fa840f53204a93735d1c.scope"
I1205 03:14:31.775806 00001 manager.go:91] Machine: {NumCores:1 MemoryCapacity:3903705088 Filesystems:[{Device:/dev/mapper/docker-253:1-67641253-721c8d048f594698ba7749b41cb3d7d6439e421d0604fa840f53204a93735d1c Capacity:10434662400}]}
I1205 03:14:31.779043 00001 manager.go:98] Version: {KernelVersion:3.10.0-123.9.3.el7.x86_64 ContainerOsVersion:Buildroot 2014.02 DockerVersion:Unknown CadvisorVersion:0.6.2}
E1205 03:14:31.779156 00001 cadvisor.go:62] Docker registration failed: unable to communicate with docker daemon: dial unix /var/run/docker.sock: permission denied.
I1205 03:14:31.785884 00001 factory.go:78] Registering Raw factory
I1205 03:14:31.791596 00001 manager.go:394] Added container: "/" (aliases: [], namespace: "")
I1205 03:14:31.791632 00001 manager.go:131] Starting recovery of all containers
I1205 03:14:31.797082 00001 container.go:141] Start housekeeping for container "/"
I1205 03:14:31.797233 00001 container.go:189] Failed to update stats for container "/": stat /sys/fs/cgroup/cpuacct: permission denied
W1205 03:14:31.797251 00001 container.go:112] Failed to get RecentStats("/") while determining the next housekeeping: unable to find data for container /
I1205 03:14:31.922553 00001 manager.go:394] Added container: "/system.slice" (aliases: [], namespace: "")
I1205 03:14:31.927679 00001 manager.go:394] Added container: "/system.slice/dev-disk-by\x2did-dm\x2dname\x2dcentos\x2dswap.swap" (aliases: [], namespace: "")
I1205 03:14:31.928801 00001 container.go:141] Start housekeeping for container "/system.slice"
I1205 03:14:31.929435 00001 container.go:189] Failed to update stats for container "/system.slice": stat /sys/fs/cgroup/cpu/system.slice: permission denied
W1205 03:14:31.929463 00001 container.go:112] Failed to get RecentStats("/system.slice") while determining the next housekeeping: unable to find data for container /system.slice
I1205 03:14:31.929473 00001 container.go:141] Start housekeeping for container "/system.slice/dev-disk-by\x2did-dm\x2dname\x2dcentos\x2dswap.swap"
I1205 03:14:31.929543 00001 container.go:189] Failed to update stats for container "/system.slice/dev-disk-by\x2did-dm\x2dname\x2dcentos\x2dswap.swap": stat /sys/fs/cgroup/cpu/system.slice/dev-disk-by\x2did-dm\x2dname\x2dcentos\x2dswap.swap: permission denied

[liuyunsh]

hi vmarmol
here is my result of "grep cgroup /proc/mounts"

[root@docker ~]# grep cgroup /proc/mounts
tmpfs /sys/fs/cgroup tmpfs rw,seclabel,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpuacct,cpu 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/net_cls cgroup rw,nosuid,nodev,noexec,relatime,net_cls 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0

[liuyunsh]

Hi all, It works normally now after I closed firewall,and set selinux to disable. thanks for all of your help.

[saifikhan]

In CentOS 7, cgroups are mounted under /sys/fs/cgroup/

pwd

/sys/fs/cgroup

ls -F

blkio/ cpu@ cpuacct@ cpu,cpuacct/ cpuset/ devices/ freezer/ hugetlb/ memory/ net_cls/ perf_event/ systemd/
#

[vmarmol]

@scottscreations are you still seeing the error?

[scotticles]

I turned off SELinux and it did load up, i wasn't able to see any containers listed. So I am not sure its 100% working. I've never used cadvisor to compare. I might kick up a ubuntu host and put some containers on to see.

[liuyunsh]

@scottscreations ,yeah,good luck!

[mdshuai]

If don't turn off the selinux, I also have the same issue.My os is rhel7.

[vmarmol]

@mdshuai can you provide the cAdvisor log as well as the output of /validate on cAdvisor please?

[mdshuai]

@vmarmol Just refer this info: http://fpaste.org/211068/67273142/

[vmarmol]

@mdshuai thanks! Can you run it with --logtostderr --v=2 and show the logs? It does seem like SELinux is not letting us watch the cgroups which won't let us see new containers quickly. We do manually search for new containers every minute. Can you check if after a few minutes the containers do show up?

[mdshuai]

@vmarmol More logs: http://fpaste.org/211498/91484841/

[vishh]

Can you confirm that you have indeed turned off Selinux?

On Wed, Apr 15, 2015 at 6:41 PM, DeShuai Ma [email protected]
wrote:

@vmarmol https://github.com/vmarmol More logs:
http://fpaste.org/211498/91484841/

—
Reply to this email directly or view it on GitHub
https://github.com/google/cadvisor/issues/333#issuecomment-93614221.

[vmarmol]

@vishh without selinux everything works. Trying to narrow down the failure with selinux on.

@mdshuai looking at the log its clear that we are detecting the containers every 60s as the expected fallback. We get an error when we try to delete any of our inotify watches, but I'm not sure if it is a problem with the watches in general or just the deletes (it is not clear to me that the creates are being run either). What issue are you seeing? Are there some containers that don't show up when expected?

Can you check /var/log/messages or the equivalent for SELinux messages? Trying to see if SELinux is blocking anything here. It's not clear that is the case.

Describing your setup may help a bit more as well :) Thank you for taking the time to help us debug your issue!

[liuyunsh]

@vmarmol , Could you please help to give a detail setup steps with selinux enabled?
I think this would help us a lot. thanks a lot.

[vmarmol]

@liuyunsh I'd be happy to, but I'm not sure how the setup is different from the non selinux setup :-\ I'd like to try and see if I can reproduce this problem tomorrow. Is this just a vanilla RHEL7 machine or is there any customization I also need to do?

If you could answer some of the questions above it would also really help my debugging :)

[mdshuai]

@vmarmol
1) kernel version: 3.10.0-229.el7.x86_64
2) Cadvisor version: 0.11.0;
3) selinux version
libselinux-python-2.2.2-6.el7.x86_64
selinux-policy-3.13.1-23.el7.noarch
libselinux-2.2.2-6.el7.x86_64
libselinux-utils-2.2.2-6.el7.x86_64
selinux-policy-targeted-3.13.1-23.el7.noarch
4) docker version:
docker --version
Docker version 1.4.1-dev, build d26b358/1.4.1
5) when enable the selinux, run the cadvisor and run another container ; then from the web to access the container, there is no container list.
The /var/log/message is : http://fpaste.org/213633/95841921/

Not sure this info can help you to debug the issue.

[liuyunsh]

@vmarmol ,thanks for your kindly reply. I just use centos7, I think RHEL7 also will be ok. thanks.

[tanmx]

CentOS, Fedora, and RHEL

You may need to run the container with --privileged=true and --volume=/cgroup:/cgroup:ro \ in order for cAdvisor to monitor Docker containers.

RHEL and CentOS lock down their containers a bit more. cAdvisor needs access to the Docker daemon through its socket. This requires --privileged=true in RHEL and CentOS.

On some versions of RHEL and CentOS the cgroup hierarchies are mounted in /cgroup so run cAdvisor with an additional Docker option of --volume=/cgroup:/cgroup:ro .

[andreiled]

Even with --privileged=true option added, I only got it working after replacing --volume=/sys:/sys:ro with --volume=/sys/fs/cgroup:/sys/fs/cgroup:ro.
Until then I was getting following error:

/usr/bin/docker-current: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:364: container init caused \"rootfs_linux.go:54: mounting \\\"/var/lib/docker/containers/%ID1%/secrets\\\" to rootfs \\\"/var/lib/docker/overlay2/%ID2%/merged\\\" at \\\"/var/lib/docker/overlay2/%ID2%/merged/run/secrets\\\" caused \\\"mkdir /var/lib/docker/overlay2/%ID2%/merged/run/secrets: read-only file system\\\"\"".

I am running _Docker Version 1.13.1_ on _Red Hat Enterprise Linux Server 7.5 (Maipo)_.

[czechm]

When i was running in version with --volume=/sys:/sys:ro and --volume=/sys/fs/cgroup:cgroup:ro, the container was built but it exited right after the built and cadvisor wasnt working. The errors from containter below:

docker logs cadvisor
E0531 10:16:40.317426       1 factory.go:340] devicemapper filesystem stats will not be reported: usage of thin_ls is disabled to preserve iops
W0531 10:16:40.340164       1 container.go:409] Failed to create summary reader for "/system.slice/system-getty.slice": none of the resources are being tracked.
F0531 10:16:40.346797       1 cadvisor.go:172] Failed to start container manager: inotify_add_watch /sys/fs/cgroup/cpuacct,cpu: no such file or directory

Before that when i was running with --volume=/var/run:/var/run:ro I had the same issue as andreiled. When i changed permissions to --volume=/var/run:/var/run:rw it worked.

So after all these issues the final version that worked is:
docker run --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys/fs/cgroup:/sys/fs/cgroup:ro --volume=/var/home/docker/dockerdir/:/var/home/docker/dockerdir:ro --volume=/dev/disk/:/dev/disk:ro --privileged=true --publish=8080:8080 --detach=true --name=cadvisor google/cadvisor:latest

Be aware that i changed the original directory for docker from /var/lib/docker to /var/home/docker/dockerdir.

Details:

OS: Centos 7 atomic, Kernel: 3.10.0-862.11.6.el7.x86_64
docker info
Containers: 3
Running: 3
Paused: 0
Stopped: 0
Images: 7
Server Version: 1.13.1
Storage Driver: devicemapper
Pool Name: cah-docker--pool
Pool Blocksize: 524.3 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file:
Metadata file:
Data Space Used: 11.72 GB
Data Space Total: 91.28 GB
Data Space Available: 79.56 GB
Metadata Space Used: 2.74 MB
Metadata Space Total: 251.7 MB
Metadata Space Available: 248.9 MB
Thin Pool Minimum Free Space: 9.128 GB
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.146-RHEL7 (2018-01-22)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Swarm: active
NodeID: k7mwrot41yddyzhl43hq0c0m7
Is Manager: false
Node Address: 192.168.2.229
Manager Addresses:
192.168.2.220:2377
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: /usr/libexec/docker/docker-init-current
containerd version: (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: 5eda6f6fd0c2884c2c8e78a6e7119e8d0ecedb77 (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: fec3683b971d9c3ef73f284f176672c44b448662 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
seccomp
WARNING: You're not using the default seccomp profile
Profile: /etc/docker/seccomp.json
selinux
Kernel Version: 3.10.0-862.11.6.el7.x86_64
Operating System: CentOS Linux 7.1808 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 8
Total Memory: 31.14 GiB
Name: dock10.smatlogic.com
ID: RAG7:F7QH:2FNL:VENI:VZHH:KZHH:MKIF:WJO5:UWKG:63HJ:IEYI:VDHB
Docker Root Dir: /var/home/docker/dockerdir
Debug Mode (client): false
Debug Mode (server): false
Registry: https://cart:8082/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Experimental: true
Insecure Registries:
cart.smartlogic.com:8082
cart.smartlogic.com:8083
cart:8082
cart:8083
127.0.0.0/8
Registry Mirrors:
cart:8082
Live Restore Enabled: false
Registries: cart:8082 (insecure), docker.io (secure)