Node_exporter: node exporter is not ignoring a mount point

[Dlorite]

We are running a node exporter in a docker container v0.15.1 with the following docker-compose.

volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command: 
      - '--path.procfs=/host/proc' 
      - '--path.sysfs=/host/sys'
      - --collector.filesystem.ignored-mount-points
      - "^/(sys|proc|dev|host|etc|rootfs/var/lib/docker/containers|rootfs/var/lib/docker/overlay2|rootfs/run/docker/netns|rootfs/var/lib/docker/aufs|rootfs/var/lib/docker/devicemapper)($$|/)"

But even the log with debug level is saying

time="2018-04-18T13:51:09Z" level=debug msg="Ignoring mount point: /rootfs/opt/docker/devicemapper/mnt/5edaab07eea690d26b99f6258322f170441af87111d6ca0312fac519fc0e1a38/rootfs/sys/fs/cgroup/freezer" source="filesystem_linux.go:42"

In /var/log/messages are coming errors like this,

journal: time="2018-04-15T01:28:08Z" level=error msg="Error on statfs() system call for \"/rootfs/opt/docker/devicemapper/mnt/5edaab07eea690d26b99f6258322f170441af87111d6ca0312fac519fc0e1a38/rootfs\": permission denied" source="filesystem_linux.go:57"#015

There is any solution for this?

[discordianfish]

The regexp just doesn't match the mountpoint. There is no opt/docker in the regex.

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

[RuchikaChugh1]

My regexp still doesn't work.
am running node exporter on a host with a certain user which does not have rights to see /docker/container or /docker/overlay2 folder. This host contains docker and container as well and Node exporter is not running on the containers. I am trying to run node exporter on this host and getting the following error in /var/log/messages.

2018-06-29T05:23:26.964166-04:00 sps-me-comsvc-1 SPS.info: PID = 23648 EXIT_STATUS = 0
2018-06-29T05:23:29.122301-04:00 sps-me-comsvc-1 node_exporter: time="2018-06-29T05:23:29-04:00" level=error msg="Error on statfs() system call for "/docker/overlay2/be4fee185971bb13a48226d8d610e02076d6fffe066042f38384c2f0917db552/merged": permission denied" source="filesystem_linux.go:57"
2018-06-29T05:23:29.133638-04:00 sps-me-comsvc-1 node_exporter: time="2018-06-29T05:23:29-04:00" level=error msg="Error on statfs() system call for "/docker/containers/278f9489922b0406e09c135a0cc7d635665f9fb4abe23d057a26125c13eee70f/shm": permission denied" source="filesystem_linux.go:57"
2018-06-29T05:23:29.133826-04:00 sps-me-comsvc-1 node_exporter: time="2018-06-29T05:23:29-04:00" level=error msg="Error on statfs() system call for "/docker/overlay2/136637097fd4f98d8a81f30c09d74d80133bc83d6d23e9e70f0c0f1962f825d7/merged": permission denied" source="filesystem_linux.go:57"
2018-06-29T05:23:29.134036-04:00 sps-me-comsvc-1 node_exporter: time="2018-06-29T05:23:29-04:00" level=error msg="Error on statfs() system call for "/docker/containers/e7484c108dbf460d46960693bad4941f69d37126a0f7814ce817870ed8279174/shm": permission denied" source="filesystem_linux.go:57"
2018-06-29T05:23:29.134243-04:00 sps-me-comsvc-1 node_exporter: time="2018-06-29T05:23:29-04:00" level=error msg="Error on statfs() system call for "/docker/overlay2/2c8f8c6c6e060e5a4436965e5660ebeb89492ebc8f03211c2d606022eb66b8fc/merged": permission denied" source="filesystem_linux.go:57"
2018-06-29T05:23:29.134435-04:00 sps-me-comsvc-1 node_exporter: time="2018-06-29T05:23:29-04:00" level=error msg="Error on statfs() system call for "/docker/containers/b86bee44bfe08e0a04634ba2e1c26e0fb1bd24c16267f3575f3ab46ad70fa65d/shm": permission denied" source="filesystem_linux.go:57"

I have put the following in in the execution command

ExecStart=-/opt/tpa/node_exporter/node_exporter
--collector.filesystem.ignored-mount-points="^/docker/(overlay2|containers).*/(shm|merged)"

I even tried with

ExecStart=-/opt/tpa/node_exporter/node_exporter
--collector.filesystem.ignored-mount-points="^/docker/.*)"

Or

ExecStart=-/opt/tpa/node_exporter/node_exporter
--collector.filesystem.ignored-mount-points="^/docker.*"

But none of these work.

drwx------. 5 root root 4096 Jun 11 02:37 containers
drwx------. 2 root root 4096 Jun 10 16:44 volumes

#

[Unit]
Description=Prometheus Node Exporter

[Service]
User=tpaadmin
Group=tpa
WorkingDirectory=/opt/tpa/node_exporter

[nodakai]

@RuchikaChugh1

Maybe you have already resolved your issue, but just for a record...

 --collector.filesystem.ignored-mount-points="^/docker/(overlay2|containers).*/(shm|merged)"

Double quotes have to be removed; _i.e._ options in the ExecStart line have to look like

 --collector.filesystem.ignored-mount-points=^/docker/(overlay2|containers).*/(shm|merged)

The original notation was telling node_exporter to ignore paths beginning with a double quote "/docker/cointainers/... which don't exist.

https://www.freedesktop.org/software/systemd/man/systemd.service.html#Command%20lines