Node_exporter: is there a way to include all mounting points

[oviis]

Hi there,
can i include all disk mounting points that can be scraped?
At the moment i can only see following mounting points:

• "/"
• "/etc/hosts"
• "/etc/resolv.conf"
• "/etc/hostname"
• "/var/run/secrets/kubernetes.io/serviceaccount"

I am interested specifically in mounting points on AWS made by Kubernetes, this locations looks like:

Filesystem Size Used Avail Use% Mounted on
/dev/xvdbe 148G 561M 140G 1% /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/vol-d10f1f60
/dev/xvdbd 976M 1.3M 908M 1% /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/vol-01f0edb0

[SuperQ]

You can adjust the flag -collector.diskstats.ignored-devices that excludes them by default.

The default is:

^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\\d+n\\d+p)\\d+$

You probably want:

^(ram|loop|fd|(h|s|v)d[a-z]|nvme\\d+n\\d+p)\\d+$

[oviis]

@SuperQ thanks for the very very fast reply 👍

i have tried your change, i have start the docker container with the given parameter like "--collector.diskstats.ignored-devices '^(ram|loop|fd|(h|s|v)d[a-z]|nvmed+nd+p)d+$', have not helped.
should i need to mount some volumes at start point?

also i have the root partition device start with xvda1, like
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 99G 5.5G 89G 6% /
which have been excluded by the default regex, but the root device can be scraped

[SuperQ]

It is not recommended to run the node_exporter in docker. It needs to have non-containerized access to the node in order to work properly.

[oviis]

thanks 👍 , i have managed to scrap this information by mounting this volumes with parameter "-v /var/lib/kubelet/plugins/kubernetes.io:/var/lib/kubelet/plugins/kubernetes.io"

P.S. i need to run inside container, because orchestration tools or OS like CoreOS only run containers

[SuperQ]

If you use rkt instead of docker, you can use a "fly" container. :smile:

[rdtr]

I ran into the same problem using kubernetes and your advices would work!

        command: ["/bin/node_exporter"]
        args:
        - '--collector.diskstats.ignored-devices=^(ram|loop|fd|(h|s|v)d[a-z]|nvme\\d+n\\d+p)\\d+$'
        volumeMounts:
        - name: k8s-plugin 
          mountPath: /var/lib/kubelet/plugins/kubernetes.io
          readOnly: true
      volumes:
      - name: k8s-plugin
        hostPath:
          path: /var/lib/kubelet/plugins/kubernetes.io

The setting above showed an additionally provisioned persistent volume (which is AWS EBS).
Thank you for your information.

[bjoernhaeuser]

I tried this and it works, unfortunately only when the volume is mounted before the node_exporter is started. New mounts are not picked up properly. Is this known?

[lucasmior]

@bjoernhaeuser seems like there is an issue when you create the mounted volume after starts the node_exporter. Take a look on https://github.com/prometheus/node_exporter/issues/474 and https://github.com/kubernetes/kubernetes/issues/44301