Cadvisor: Task load state always be zero

Created on 8 Jun 2017  路  23Comments  路  Source: google/cadvisor

Hi team,

I started to use cadvisor, and container_tasks_state metrics which are exposed on the /metrics endpoint always show zero:

container_tasks_state{id="/docker/c3f88fe91c50d918dfeb48ee9427320f7a2d16fa32cf902faf8376b082167d06",image="docker-reg.private.co/app:0.1.10",name="app01",state="running"} 0

My cadvisor version is:

cAdvisor version: v0.25.0-17543be on port 8080

Is that a bug here? What I can see in the log is some failed msg:

W0531 03:46:31.426594       1 container.go:352] Failed to create summary reader for "/docker/215cd650a9866579d9a9e49fdbeb3a37d790b3383699a52dc6a150361c0ed040": none of the resources are being tracked.
W0531 03:51:40.641039       1 container.go:352] Failed to create summary reader for "/docker/f21e5d737833403c9a92ad425702321fb1a9c73add1c4472e31cd90be0bec87d": none of the resources are being tracked.
W0531 03:54:55.188493       1 container.go:352] Failed to create summary reader for "/docker/4a33834fc830f5467dbbbe2e1289ba796866f2795336cf2cb8e744eea42ebb63": none of the resources are being tracked.
W0531 03:55:01.072433       1 container.go:352] Failed to create summary reader for "/docker/020262216d2b2f129bba5a39dd37143b699663fa7bd3961985570914b141d566": none of the resources are being tracked.

But these seems to be not related to this issue.
Thanks.

kinsupport
Source
picture Colstuwjx
👍20

Most helpful comment

Same behaviour here, would be nice if someone could enlighten us on what might be wrong?

picture siwyd on 6 Dec 2018
👍10

All 23 comments

I have similar issue. I've launched a whole stack using example from https://github.com/stefanprodan/dockprom.

Setup: Ubuntu 16.04 via Vagrant / VirtualBox on Windows box.

compose file contains such specification of cadvisor service:

cadvisor:
    image: google/cadvisor:v0.24.1
    container_name: cadvisor
    command:
      - '-v=4'
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /var/log:/var/log:ro
    restart: unless-stopped
    expose:
      - 8080
    ports:
      - 9010:8080
    networks:
      - monitor-net
    labels:
      org.label-schema.group: "monitoring"

Note: I've added volume mapping to get OOM reporting - /var/log:/var/log:ro. But container_tasks_state was empty (always zero) both before and after adding this mapping.

After enabling verbose logging I am seeing this in cadvisor logs:

I0619 11:27:45.622192       1 factory.go:115] Factory "docker" was unable to handle container "/system.slice/var-lib-docker-aufs-mnt-f25a63e9dcd7ea8dce4158c8c16e05f3e0f124d9da48c9d6b27ca7e5d3065d5f.mount"
I0619 11:27:45.622208       1 factory.go:108] Factory "systemd" can handle container "/system.slice/var-lib-docker-aufs-mnt-f25a63e9dcd7ea8dce4158c8c16e05f3e0f124d9da48c9d6b27ca7e5d3065d5f.mount", but ignoring.
I0619 11:27:45.622220       1 manager.go:843] ignoring container "/system.slice/var-lib-docker-aufs-mnt-f25a63e9dcd7ea8dce4158c8c16e05f3e0f124d9da48c9d6b27ca7e5d3065d5f.mount"
I0619 11:27:45.622235       1 factory.go:104] Error trying to work out if we can handle /system.slice/dev-hugepages.mount: invalid container name
I0619 11:27:45.622246       1 factory.go:115] Factory "docker" was unable to handle container "/system.slice/dev-hugepages.mount"
I0619 11:27:45.622258       1 factory.go:108] Factory "systemd" can handle container "/system.slice/dev-hugepages.mount", but ignoring.
I0619 11:27:45.622268       1 manager.go:843] ignoring container "/system.slice/dev-hugepages.mount"
I0619 11:27:45.622284       1 factory.go:104] Error trying to work out if we can handle /system.slice/run-docker-netns-1\x2d5ncqrmhhap.mount: invalid container name
I0619 11:27:45.622293       1 factory.go:115] Factory "docker" was unable to handle container "/system.slice/run-docker-netns-1\\x2d5ncqrmhhap.mount"
I0619 11:27:45.622454       1 factory.go:108] Factory "systemd" can handle container "/system.slice/run-docker-netns-1\\x2d5ncqrmhhap.mount", but ignoring.
I0619 11:27:45.622475       1 manager.go:843] ignoring container "/system.slice/run-docker-netns-1\\x2d5ncqrmhhap.mount"
I0619 11:27:45.622490       1 factory.go:104] Error trying to work out if we can handle /system.slice/dev-mqueue.mount: invalid container name
I0619 11:27:45.622500       1 factory.go:115] Factory "docker" was unable to handle container "/system.slice/dev-mqueue.mount"
I0619 11:27:45.622512       1 factory.go:108] Factory "systemd" can handle container "/system.slice/dev-mqueue.mount", but ignoring.
I0619 11:27:45.622523       1 manager.go:843] ignoring container "/system.slice/dev-mqueue.mount"
I0619 11:27:45.626292       1 factory.go:104] Error trying to work out if we can handle /system.slice/var-lib-docker-aufs-mnt-458d9912e6ab495e7d2ef56dae3414ddba595d6e7edc25c77cd6361c46ed57d5.mount: error inspecting container: Error: No such container: 458d9912e6ab495e7d2ef56dae3414ddba595d6e7edc25c77cd6361c46ed57d5
I0619 11:27:45.626330       1 factory.go:115] Factory "docker" was unable to handle container "/system.slice/var-lib-docker-aufs-mnt-458d9912e6ab495e7d2ef56dae3414ddba595d6e7edc25c77cd6361c46ed57d5.mount"
I0619 11:27:45.626342       1 factory.go:108] Factory "systemd" can handle container "/system.slice/var-lib-docker-aufs-mnt-458d9912e6ab495e7d2ef56dae3414ddba595d6e7edc25c77cd6361c46ed57d5.mount", but ignoring.
I0619 11:27:45.626349       1 manager.go:843] ignoring container "/system.slice/var-lib-docker-aufs-mnt-458d9912e6ab495e7d2ef56dae3414ddba595d6e7edc25c77cd6361c46ed57d5.mount"
I0619 11:27:45.626358       1 factory.go:104] Error trying to work out if we can handle /system.slice/var-lib-lxcfs.mount: invalid container name
I0619 11:27:45.626364       1 factory.go:115] Factory "docker" was unable to handle container "/system.slice/var-lib-lxcfs.mount"
I0619 11:27:45.626368       1 factory.go:108] Factory "systemd" can handle container "/system.slice/var-lib-lxcfs.mount", but ignoring.
I0619 11:27:45.626371       1 manager.go:843] ignoring container "/system.slice/var-lib-lxcfs.mount"
I0619 11:27:45.626379       1 factory.go:104] Error trying to work out if we can handle /system.slice/run-docker-netns-cf6bf9ac3743.mount: invalid container name
I0619 11:27:45.626384       1 factory.go:115] Factory "docker" was unable to handle container "/system.slice/run-docker-netns-cf6bf9ac3743.mount"
I0619 11:27:45.626388       1 factory.go:108] Factory "systemd" can handle container "/system.slice/run-docker-netns-cf6bf9ac3743.mount", but ignoring.
I0619 11:27:45.626392       1 manager.go:843] ignoring container "/system.slice/run-docker-netns-cf6bf9ac3743.mount"
I0619 11:27:45.626421       1 factory.go:104] Error trying to work out if we can handle /system.slice/run-docker-netns-ingress_sbox.mount: invalid container name
I0619 11:27:45.626428       1 factory.go:115] Factory "docker" was unable to handle container "/system.slice/run-docker-netns-ingress_sbox.mount"
I0619 11:27:45.626432       1 factory.go:108] Factory "systemd" can handle container "/system.slice/run-docker-netns-ingress_sbox.mount", but ignoring.
I0619 11:27:45.626436       1 manager.go:843] ignoring container "/system.slice/run-docker-netns-ingress_sbox.mount"
I0619 11:27:45.626442       1 factory.go:104] Error trying to work out if we can handle /system.slice/run-user-1000.mount: invalid container name
I0619 11:27:45.626447       1 factory.go:115] Factory "docker" was unable to handle container "/system.slice/run-user-1000.mount"
I0619 11:27:45.626451       1 factory.go:108] Factory "systemd" can handle container "/system.slice/run-user-1000.mount", but ignoring.
I0619 11:27:45.626454       1 manager.go:843] ignoring container "/system.slice/run-user-1000.mount"

container_tasks_state metrics is zero for everything in cadvisor:

container_tasks_state{id="/",state="iowaiting"} 0
container_tasks_state{id="/",state="running"} 0
container_tasks_state{id="/",state="sleeping"} 0
container_tasks_state{id="/",state="stopped"} 0
container_tasks_state{id="/",state="uninterruptible"} 0
container_tasks_state{id="/docker",state="iowaiting"} 0
container_tasks_state{id="/docker",state="running"} 0
container_tasks_state{id="/docker",state="sleeping"} 0
container_tasks_state{id="/docker",state="stopped"} 0
container_tasks_state{id="/docker",state="uninterruptible"} 0
...
container_tasks_state{id="/system.slice/acpid.service",state="uninterruptible"} 0
container_tasks_state{id="/system.slice/apparmor.service",state="iowaiting"} 0
container_tasks_state{id="/system.slice/apparmor.service",state="running"} 0
container_tasks_state{id="/system.slice/apparmor.service",state="sleeping"} 0
container_tasks_state{id="/system.slice/apparmor.service",state="stopped"} 0
container_tasks_state{id="/system.slice/apparmor.service",state="uninterruptible"} 0
container_tasks_state{id="/system.slice/apport.service",state="iowaiting"} 0
container_tasks_state{id="/system.slice/apport.service",state="running"} 0
container_tasks_state{id="/system.slice/apport.service",state="sleeping"} 0
container_tasks_state{id="/system.slice/apport.service",state="stopped"} 0
container_tasks_state{id="/system.slice/apport.service",state="uninterruptible"} 0
container_tasks_state{id="/system.slice/atd.service",state="iowaiting"} 0
container_tasks_state{id="/system.slice/atd.service",state="running"} 0
container_tasks_state{id="/system.slice/atd.service",state="sleeping"} 0
container_tasks_state{id="/system.slice/atd.service",state="stopped"} 0
...
container_tasks_state{container_label_com_docker_compose_config_hash="460a694821ac8a3b069369ad933926aa407b2dc437cf6ba0dadf2e6d89efdae4",container_label_com_docker_compose_container_number="1",container_label_com_docker_compose_oneoff="False",container_label_com_docker_compose_project="dockprom",container_label_com_docker_compose_service="cadvisor",container_label_com_docker_compose_version="1.13.0",container_label_org_label_schema_group="monitoring",id="/docker/ce5a159a015df93f2d0a3b534cdcab64cd9ebcdc5c21b31f7955da81785ca91d",image="google/cadvisor:v0.24.1",name="cadvisor",state="iowaiting"} 0
container_tasks_state{container_label_com_docker_compose_config_hash="460a694821ac8a3b069369ad933926aa407b2dc437cf6ba0dadf2e6d89efdae4",container_label_com_docker_compose_container_number="1",container_label_com_docker_compose_oneoff="False",container_label_com_docker_compose_project="dockprom",container_label_com_docker_compose_service="cadvisor",container_label_com_docker_compose_version="1.13.0",container_label_org_label_schema_group="monitoring",id="/docker/ce5a159a015df93f2d0a3b534cdcab64cd9ebcdc5c21b31f7955da81785ca91d",image="google/cadvisor:v0.24.1",name="cadvisor",state="running"} 0
container_tasks_state{container_label_com_docker_compose_config_hash="460a694821ac8a3b069369ad933926aa407b2dc437cf6ba0dadf2e6d89efdae4",container_label_com_docker_compose_container_number="1",container_label_com_docker_compose_oneoff="False",container_label_com_docker_compose_project="dockprom",container_label_com_docker_compose_service="cadvisor",container_label_com_docker_compose_version="1.13.0",container_label_org_label_schema_group="monitoring",id="/docker/ce5a159a015df93f2d0a3b534cdcab64cd9ebcdc5c21b31f7955da81785ca91d",image="google/cadvisor:v0.24.1",name="cadvisor",state="sleeping"} 0
DevWojtekC picture DevWojtekC on 19 Jun 2017

Similar with cAdvisor running in Docker on a Debian GCE instance. All the container_task_states are reporting zero.

cAdvisor version: v0.25.0

OS version: Alpine Linux v3.4

Kernel version: [Supported and recommended]
    Kernel version is 3.16.0-4-amd64. Versions >= 2.6 are supported. 3.0+ are recommended.


Cgroup setup: [Supported, but not recommended]
    Cgroup memory not enabled. Available cgroups: map[cpuacct:1 memory:0 devices:1 freezer:1 blkio:1 net_prio:1 cpu:1 net_cls:1 perf_event:1 cpuset:1]
    Following cgroups are required: [cpu cpuacct]
    Following other cgroups are recommended: [memory blkio cpuset devices freezer]


Cgroup mount setup: [Supported and recommended]
    Cgroups are mounted at /sys/fs/cgroup.
    Cgroup mount directories: blkio cpu cpu,cpuacct cpuacct cpuset devices freezer net_cls net_cls,net_prio net_prio perf_event systemd
    Any cgroup mount point that is detectible and accessible is supported. /sys/fs/cgroup is recommended as a standard location.
    Cgroup mounts:
    cgroup /sys/fs/cgroup/systemd cgroup ro,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
    cgroup /sys/fs/cgroup/cpuset cgroup ro,nosuid,nodev,noexec,relatime,cpuset 0 0
    cgroup /sys/fs/cgroup/cpu,cpuacct cgroup ro,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
    cgroup /sys/fs/cgroup/devices cgroup ro,nosuid,nodev,noexec,relatime,devices 0 0
    cgroup /sys/fs/cgroup/freezer cgroup ro,nosuid,nodev,noexec,relatime,freezer 0 0
    cgroup /sys/fs/cgroup/net_cls,net_prio cgroup ro,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
    cgroup /sys/fs/cgroup/blkio cgroup ro,nosuid,nodev,noexec,relatime,blkio 0 0
    cgroup /sys/fs/cgroup/perf_event cgroup ro,nosuid,nodev,noexec,relatime,perf_event 0 0
    cgroup /rootfs/sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
    cgroup /rootfs/sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
    cgroup /rootfs/sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
    cgroup /rootfs/sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
    cgroup /rootfs/sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
    cgroup /rootfs/sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
    cgroup /rootfs/sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
    cgroup /rootfs/sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
    cgroup /rootfs/var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/systemd cgroup ro,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
    cgroup /rootfs/var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/cpuset cgroup ro,nosuid,nodev,noexec,relatime,cpuset 0 0
    cgroup /rootfs/var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/cpu,cpuacct cgroup ro,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
    cgroup /rootfs/var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/devices cgroup ro,nosuid,nodev,noexec,relatime,devices 0 0
    cgroup /rootfs/var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/freezer cgroup ro,nosuid,nodev,noexec,relatime,freezer 0 0
    cgroup /rootfs/var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/net_cls,net_prio cgroup ro,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
    cgroup /rootfs/var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/blkio cgroup ro,nosuid,nodev,noexec,relatime,blkio 0 0
    cgroup /rootfs/var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/perf_event cgroup ro,nosuid,nodev,noexec,relatime,perf_event 0 0
    cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
    cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
    cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
    cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
    cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
    cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
    cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
    cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/systemd cgroup ro,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/cpuset cgroup ro,nosuid,nodev,noexec,relatime,cpuset 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/cpu,cpuacct cgroup ro,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/devices cgroup ro,nosuid,nodev,noexec,relatime,devices 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/freezer cgroup ro,nosuid,nodev,noexec,relatime,freezer 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/net_cls,net_prio cgroup ro,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/blkio cgroup ro,nosuid,nodev,noexec,relatime,blkio 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/perf_event cgroup ro,nosuid,nodev,noexec,relatime,perf_event 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/rootfs/sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/rootfs/sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/rootfs/sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/rootfs/sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/rootfs/sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/rootfs/sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/rootfs/sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/rootfs/sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/rootfs/var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/systemd cgroup ro,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/rootfs/var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/cpuset cgroup ro,nosuid,nodev,noexec,relatime,cpuset 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/rootfs/var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/cpu,cpuacct cgroup ro,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/rootfs/var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/devices cgroup ro,nosuid,nodev,noexec,relatime,devices 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/rootfs/var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/freezer cgroup ro,nosuid,nodev,noexec,relatime,freezer 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/rootfs/var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/net_cls,net_prio cgroup ro,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/rootfs/var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/blkio cgroup ro,nosuid,nodev,noexec,relatime,blkio 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/rootfs/var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/perf_event cgroup ro,nosuid,nodev,noexec,relatime,perf_event 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
    cgroup /var/lib/docker/aufs/mnt/a1c0713e359b223b5cd05e33d53818017568d17e00e28932b65fd8f7aa180c3b/sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0


Docker version: [Supported and recommended]
    Docker version is 17.03.1-ce. Versions >= 1.0 are supported. 1.2+ are recommended.


Docker driver setup: [Supported and recommended]
    Docker exec driver is . Storage driver is aufs.


Block device setup: [Supported, but not recommended]
    None of the devices support 'cfq' I/O scheduler. No disk stats can be reported.
     Disk "sdb" Scheduler type "noop".
     Disk "sda" Scheduler type "noop".


Inotify watches:
nshttpd picture nshttpd on 26 Jun 2017

Just updated to Starting cAdvisor version: v0.26.1-d19cc94 on port 8080 .. and same thing.

nshttpd picture nshttpd on 26 Jun 2017

+1

caiohasouza picture caiohasouza on 23 Aug 2017

Have you tried enabling load reader -enable_load_reader?

ghost picture ghost on 24 Nov 2017

Hi,
I have similar issue. container_tasks_state metrics which are exposed on the /metrics endpoint always returns zero.
My configuration is:
cadvisor_version_info{cadvisorRevision="aaaa65d",cadvisorVersion="v0.29.0",dockerVersion="17.12.0-ce",kernelVersion="4.15.0-1.el7.elrepo.x86_64",osVersion="Alpine Linux v3.4"}
My docker compose file contains such specification of cadvisor service:

 cadvisor:
    deploy:
      mode: global
      resources:
        limits:
          memory: 128M
        reservations:
          memory: 64M
    image: google/cadvisor:v0.29.0
    hostname: '{{.Node.Hostname}}'
    networks:
      - default
    ports:
      - 9098:8080
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /:/rootfs:ro
      - /var/run:/var/run
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    command:
      - '-docker_only'
      - '-housekeeping_interval=5s'
      - `'-disable_metrics=disk,tcp,udp'

There are no errors in the log.

igorkatz picture igorkatz on 1 Mar 2018

Make sure you are setting the flag --enable_load_reader=true, otherwise you wont see any metrics.

dashpole picture dashpole on 15 Mar 2018
👍1

I have a similar issue too.
I added the -enable_load_reader=true, but it's not working.
here is my compose file:

 cadvisor:
    image: google/cadvisor:latest
    stdin_open: true
    volumes:
    - /:/rootfs:ro
    - /var/run:/var/run:rw
    - /sys:/sys:ro
    - /var/lib/docker/:/var/lib/docker:ro
    - /dev/disk/:/dev/disk/:ro
    tty: true
    ports:
    - 1080:8080/tcp
    command:
    - -enable_load_reader=true
    labels:
      io.rancher.container.pull_image: always
      io.rancher.host.agent: cadvisor-pr
lzpsqzr picture lzpsqzr on 13 Sep 2018

Have the same issue at the moment, also after adding the row below to my cAdvisor docker compose:

--enable_load_reader=true

JBS5 picture JBS5 on 9 Nov 2018

Same here, adding--enable_load_reader=true will not work. This is my docker-compose:

cadvisor:
  image: google/cadvisor
  command: "--enable_load_reader=true"
  privileged: true
  volumes:
    - /:/rootfs:ro
    - /var/run:/var/run:rw
    - /sys:/sys:ro
    - /var/lib/docker/:/var/lib/docker:ro
    - /cgroup:/cgroup:ro
  network_mode: "host"
  restart: always
BirkhoffLee picture BirkhoffLee on 4 Dec 2018
👍3

Same behaviour here, would be nice if someone could enlighten us on what might be wrong?

siwyd picture siwyd on 6 Dec 2018
👍10

same problem here, can anyone solve it?

fabtrompet picture fabtrompet on 23 Jan 2019

@dashpole still encounter this issue running the latest v0.32.0. Adding --enable_load_reader=true doesn't seem to help. Would you be able to provide some guidance for us? Let me know if you need any help to reproduce this or there is something I can do to assist.

mapshen picture mapshen on 6 Feb 2019

I remember looking into this before... cAdvisor needs to be on the host network, but i'm not sure how to accomplish that with a docker compose. See https://github.com/google/cadvisor/issues/2051 for more details, and a kustomize patch to do this in kubernetes.

dashpole picture dashpole on 6 Feb 2019

Thanks for prompt response!

Did a quick test and it looks like to be working. This is the compose file I use to deploy in swarm mode (thanks to moby/issues/25873 as network_mode is ignored):

version: "3.7"

networks:
  host_network:
    external: true
    name: host

services:
  cadvisor:
    image: google/cadvisor:v0.32.0
      command: "--enable_load_reader=true"
    networks:
      - host_network
    deploy:
      mode: global
mapshen picture mapshen on 7 Feb 2019

Prometheus and cadvisory must be in one docker network.

burtsevyg picture burtsevyg on 16 May 2019

I am using the host network with "enable_load_reader=true". Also had to set a different port for cadvisor since I'm using the default one - 8080:

networks:
  outside:
    external:
      name: "host"
  cadvisor:
    image: google/cadvisor
    networks:
      - outside
    command: --enable_load_reader=true -port=8384
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /:/rootfs:ro
      - /var/run:/var/run
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    deploy:
      mode: global

With this I've chnaged the prometheus.yml config file to set the new targets like this:

  - job_name: 'cadvisor'
    static_configs:
      - targets: ['xx.xx.xx.xx:8384']
      - targets: ['xx.xx.xx.xx:8384']
      - targets: ['xx.xx.xx.xx:8384']

Now I'm getting the following results when I run a "container_tasks_state{container_label_com_docker_swarm_service_name="portainer"}" query:

Element Value
container_tasks_state{....state="iowaiting"}    0
container_tasks_state{....state="running"}  0
container_tasks_state{....state="sleeping"} 42
container_tasks_state{....state="stopped"}  0
container_tasks_state{....state="uninterruptible"}  0

Now all the containers are in state sleeping with a weird value for it.
Am I missing something?

sofronic picture sofronic on 11 Jul 2019

@sofronic is it possible your container tasks are sleeping? Try running a container with the image gcr.io/google-containers/stress:v1 with args --cpus=1, from https://github.com/vishh/stress. That should continuously use 1 cpu core, and shouldn't sleep.

dashpole picture dashpole on 11 Jul 2019

@dashpole I've ran stress:v1 as a swarm service and it is comming with a state "running" == 1 and "sleeping" == 4. I've noticed that some other containers are poping up as running from tme to time. But I've got 100+ of them in 87 swarm services. So only containers using 1 cpu or more will come up as runing?

sofronic picture sofronic on 16 Jul 2019

@sofronic if your application isn't actively doing processing, such as listening on a port, or time.Sleep()ing , it will be in a sleeping state.

dashpole picture dashpole on 16 Jul 2019

I use kubelet api: http://10.142.113.20:10255/metrics/cadvisor. It has the same problem.
My k8s version is 1.13.2

MIBc picture MIBc on 26 Jul 2019

I'm facing the same problem .So I enabled load reader and network_mode: "host".
It is giving values only for state=sleeping value as 15.
other states are always 0.
I'm using docker service and replica are around 2K .There are high chances of containers going in stopped state or iowait too.
Any way to solve this?

isabelnoronha61 picture isabelnoronha61 on 25 Jun 2020

I'm facing the same problem .So I enabled load reader and network_mode: "host".
It is giving values only for state=sleeping value as 15.
other states are always 0.
I'm using docker service and replica are around 2K .There are high chances of containers going in stopped state or iowait too.
Any way to solve this?

same issue. do not know the reason. But I find the if the container is killed or stopped. All the metrics will disappear also rather than increase the number of the stopped statue.

AlphaWong picture AlphaWong on 21 Aug 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

jlec picture jlec  路  5Comments
Thermi picture Thermi  路  6Comments
ephracis picture ephracis  路  4Comments
sylr picture sylr  路  7Comments
jmirc picture jmirc  路  6Comments