Node_exporter: node_filesystem_device_error for ZFS datasets in v0.17.0-rc.0 yet v0.16.0 works

Created on 18 Nov 2018  路  8Comments  路  Source: prometheus/node_exporter

Host operating system: output of uname -a

Linux nerv 4.19.1-arch1-1-ARCH #1 SMP PREEMPT Sun Nov 4 16:49:26 UTC 2018 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 0.16.0 (branch: HEAD, revision: d42bd70f4363dced6b77d8fc311ea57b63387e4f)
  build user:       root@a67a9bc13a69
  build date:       20180515-15:52:42
  go version:       go1.9.6

node_exporter command line flags

 --path.procfs=/host/proc'
 --path.sysfs=/host/sys'
 --collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)'

Are you running node_exporter in Docker?


Yes, configured like such:

  nodeexporter:
    image: prom/node-exporter:v0.17.0-rc.0
    container_name: nodeexporter
    user: root
    privileged: true
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)'
    restart: unless-stopped
    expose:
      - 9100
    network_mode: host

What did you do that produced an error?

Trying to access exported disk metrics for ZFS filesystems/datasets however all devices except the ZFS rootfs have node_filesystem_device_error set to 1:

node_filesystem_device_error{device="nerv/ROOT/arch",fstype="zfs",mountpoint="/"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/025d3bc629326ba93a460bb2a5a746d0f47c2e2cc29c2e1c79c567731dd8d8eb",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/025d3bc629326ba93a460bb2a5a746d0f47c2e2cc29c2e1c79c567731dd8d8eb"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/044728e064ead4ba35a6cd99d2f98526caf86c11fdab405182b3c0f2b4738b88",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/044728e064ead4ba35a6cd99d2f98526caf86c11fdab405182b3c0f2b4738b88"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/14dd29cc1f932c375736e858d1f4ad29e5342994e9e0d715a158771f9fec5d69",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/14dd29cc1f932c375736e858d1f4ad29e5342994e9e0d715a158771f9fec5d69"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/16b88f38cdb0fcd09944d5836ba7979dd02909e1cc62ba67f3ce58e492395bc0",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/16b88f38cdb0fcd09944d5836ba7979dd02909e1cc62ba67f3ce58e492395bc0"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/1767b4f5fc6339f86543383851c7a38add0e5c845ebb45c580fcd262cddc1bb4",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/1767b4f5fc6339f86543383851c7a38add0e5c845ebb45c580fcd262cddc1bb4"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/1bd49c24843bda0cb29dcad170126ee83c5b4b1656114bfabd0a35fe857c86cb",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/1bd49c24843bda0cb29dcad170126ee83c5b4b1656114bfabd0a35fe857c86cb"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/1e68c12bd9db792ffbc51eb57d5ed84ac35e7ecd3c09fe5cf9198835066a43aa",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/1e68c12bd9db792ffbc51eb57d5ed84ac35e7ecd3c09fe5cf9198835066a43aa"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/20d2b01c38e80aa5252d6131b8633812e833cc586879db9f59171b4708301adf",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/20d2b01c38e80aa5252d6131b8633812e833cc586879db9f59171b4708301adf"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/211c1111033060e8c7d8e839f9b4259fb5379f499a38a3312bec266828b56552",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/211c1111033060e8c7d8e839f9b4259fb5379f499a38a3312bec266828b56552"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/23409af703856228901c4eb55a33b37e80d8ad1531c8ab4ec7423afd727b152d",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/23409af703856228901c4eb55a33b37e80d8ad1531c8ab4ec7423afd727b152d"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/245413ee2ca8ef56e5fe470b58840955358d7f29c7510dda2159830d2783765f",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/245413ee2ca8ef56e5fe470b58840955358d7f29c7510dda2159830d2783765f"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/2f6f46c6965649636377bd8f4557882135f01e6480b9c12c9eba5284d79bf344",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/2f6f46c6965649636377bd8f4557882135f01e6480b9c12c9eba5284d79bf344"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/3438e6e608bd28bcd296c6c85cbedce38e20602e4f7bf91db60b6fc2be1431d1",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/3438e6e608bd28bcd296c6c85cbedce38e20602e4f7bf91db60b6fc2be1431d1"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/3ae758036f3981be81cee49c0af5a4fdd7eff37448db3f1b153cf2af5d715db6",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/3ae758036f3981be81cee49c0af5a4fdd7eff37448db3f1b153cf2af5d715db6"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/3bb61cdbb7cdd5435c257f8d0980ae22c5ef93c216f79298fc55c20f56f34e97",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/3bb61cdbb7cdd5435c257f8d0980ae22c5ef93c216f79298fc55c20f56f34e97"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/3e4186e88233fcda7b2e1f34d50ac0a7713847d4d17f10ea184b7f1d220dd9e6",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/3e4186e88233fcda7b2e1f34d50ac0a7713847d4d17f10ea184b7f1d220dd9e6"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/3e6ef2ee79250c998e3aaa08c2e57ff566dfca8fbe873d97f2e2ea39bc9accef",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/3e6ef2ee79250c998e3aaa08c2e57ff566dfca8fbe873d97f2e2ea39bc9accef"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/47df832fb42fc4ba7c56c5a844917d7445f3c3e3ee9a4a6536c0d3d661ff757a",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/47df832fb42fc4ba7c56c5a844917d7445f3c3e3ee9a4a6536c0d3d661ff757a"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/4a5dc71bade3c2f0e8fc24105e089a9744e15cd95cc0b12a258fce0df2dc053d",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/4a5dc71bade3c2f0e8fc24105e089a9744e15cd95cc0b12a258fce0df2dc053d"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/4d5e0801ae7672f55b710018364999c8af0843de0ed8ec186ef6206a978e8ca7",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/4d5e0801ae7672f55b710018364999c8af0843de0ed8ec186ef6206a978e8ca7"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/4d6dc20f68be86dd64ca0e9b04fc65febd9d9fc95bd9d510625eb2eac959e402",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/4d6dc20f68be86dd64ca0e9b04fc65febd9d9fc95bd9d510625eb2eac959e402"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/4e0ed437ea1b3c2893834ba48f264de61f837832ef77037c30038df89df1faea",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/4e0ed437ea1b3c2893834ba48f264de61f837832ef77037c30038df89df1faea"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/52e85ffadb63a27f0ccccfadec9ab708a67e9c2708e34c7995855db3401a965c",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/52e85ffadb63a27f0ccccfadec9ab708a67e9c2708e34c7995855db3401a965c"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/5d4280853d4d73206585c03b4fa2ce2846bea79020c040e4666eaa0e815076a5",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/5d4280853d4d73206585c03b4fa2ce2846bea79020c040e4666eaa0e815076a5"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/6675b2a1f0e0c0352f50ea2890901b6eff3a1a42affb06c3af6eb0ba5f51ee06",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/6675b2a1f0e0c0352f50ea2890901b6eff3a1a42affb06c3af6eb0ba5f51ee06"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/7069e4e73d44eaf17b871c4079e9f1c9c53876c864f6f67f97406bd3e687180e",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/7069e4e73d44eaf17b871c4079e9f1c9c53876c864f6f67f97406bd3e687180e"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/7486219fa5c3ed1bf6c00344892df09064e8f3b9b78ee6639b74c1d92871e639",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/7486219fa5c3ed1bf6c00344892df09064e8f3b9b78ee6639b74c1d92871e639"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/7b5b3548c0fea4c877d5f71f749884a64d064c7d6bfc28a9ed97a63498d002cb",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/7b5b3548c0fea4c877d5f71f749884a64d064c7d6bfc28a9ed97a63498d002cb"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/7d34719fb66bf00ad53dcdc56c90f6a0667c9fc564c7bd7f330bd8f41900657a",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/7d34719fb66bf00ad53dcdc56c90f6a0667c9fc564c7bd7f330bd8f41900657a"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/8070c60e22fabea5ef32babf7af2b0004da63c8bce1b2e2650bd3d8644b97aea",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/8070c60e22fabea5ef32babf7af2b0004da63c8bce1b2e2650bd3d8644b97aea"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/87df4621d2956a0cf865381ac39e9628cf4852e0c3f0fcb4bd4dc64292ac72a0",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/87df4621d2956a0cf865381ac39e9628cf4852e0c3f0fcb4bd4dc64292ac72a0"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/8a881a4ba6a645c86ebaf5cd98e5d0cf83c836ff998e549cad847e7adffe8eaf",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/8a881a4ba6a645c86ebaf5cd98e5d0cf83c836ff998e549cad847e7adffe8eaf"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/8ce5b366258b1f6544d6e8a828d339c12e854ceff322c061bffc7aad525732d5",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/8ce5b366258b1f6544d6e8a828d339c12e854ceff322c061bffc7aad525732d5"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/96b96c7dd8970baeed1ae3950d268c240c71659d066624f0d41c6daa80141adc",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/96b96c7dd8970baeed1ae3950d268c240c71659d066624f0d41c6daa80141adc"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/9c45e614b9f652347b4e284d1d7249e5f912e0bb174084921a8ca08fca9a31a8",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/9c45e614b9f652347b4e284d1d7249e5f912e0bb174084921a8ca08fca9a31a8"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/a41e08cac2aed4afa3fcf7edf3ac4dfe9932467ac975a48f0e1d4add2a989cb5",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/a41e08cac2aed4afa3fcf7edf3ac4dfe9932467ac975a48f0e1d4add2a989cb5"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/abb4d533c9f3ae38dbbf8aaa3912a78b39f844398d7b34c796a4e6d9bdf72fb2",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/abb4d533c9f3ae38dbbf8aaa3912a78b39f844398d7b34c796a4e6d9bdf72fb2"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/be0014a2dc284bd19f91c90cd8c54a1190b4be5870cd49899a4c77a58d6fa956",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/be0014a2dc284bd19f91c90cd8c54a1190b4be5870cd49899a4c77a58d6fa956"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/c0162969ee82bc52e2a08c4b5d72119ab9e9e626392af73ebbec8d16d4cdd1fb",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/c0162969ee82bc52e2a08c4b5d72119ab9e9e626392af73ebbec8d16d4cdd1fb"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/c4cf8cb218e6a1cc8fd14b347d908a0d04db10f32adb35a4fd75951868ad0143",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/c4cf8cb218e6a1cc8fd14b347d908a0d04db10f32adb35a4fd75951868ad0143"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/c697fd3aa88f8142da8669eeadb1f7134805acc0c2e385b1e66e8dca3016f3ba",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/c697fd3aa88f8142da8669eeadb1f7134805acc0c2e385b1e66e8dca3016f3ba"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/c72f347df89d17d79caebe0156de46dd735e7401f1d5ca986613ad1dd25ea98a",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/c72f347df89d17d79caebe0156de46dd735e7401f1d5ca986613ad1dd25ea98a"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/ca3162748f5a0b9c1a58e4e9d133b7ab31a1ea143980ab642e80dd0d2d1d2752",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/ca3162748f5a0b9c1a58e4e9d133b7ab31a1ea143980ab642e80dd0d2d1d2752"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/cfec369943539a1d09073bf2cfb547ef8d3815bc6533b54ade2635725a6084bc",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/cfec369943539a1d09073bf2cfb547ef8d3815bc6533b54ade2635725a6084bc"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/d1eddf5584a2dfba56786816b064ef6601eb087bd1bf437aa694e43c64bd40c3",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/d1eddf5584a2dfba56786816b064ef6601eb087bd1bf437aa694e43c64bd40c3"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/d26fc7a3b00155a8356aa27c7cdf039fa85b7d754aa8414b334910b86a4f8484",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/d26fc7a3b00155a8356aa27c7cdf039fa85b7d754aa8414b334910b86a4f8484"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/d5261af6da1706e3a425d0497b2ac41fdcfb783ed2ad6213263f3e888e77c7d7",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/d5261af6da1706e3a425d0497b2ac41fdcfb783ed2ad6213263f3e888e77c7d7"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/d77a77268ee6805cbea659cd16dd7f333aa62df6de01b97dc9a040d5ec3217cd",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/d77a77268ee6805cbea659cd16dd7f333aa62df6de01b97dc9a040d5ec3217cd"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/d79fa0ddd3cb478c2ce38c0a9b1b56976fe365b38c03a0724b242551c14a75d4",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/d79fa0ddd3cb478c2ce38c0a9b1b56976fe365b38c03a0724b242551c14a75d4"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/e03c56885678507177b685912b092f4c17692cd0c97f39bfcd61a83dc8b281fe",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/e03c56885678507177b685912b092f4c17692cd0c97f39bfcd61a83dc8b281fe"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/e05aa62a7a3a338a265d34e296d7ca5878b7e926c9b91d13084c758f06d9d003",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/e05aa62a7a3a338a265d34e296d7ca5878b7e926c9b91d13084c758f06d9d003"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/e7984544cad8e735f371ff90e49ac95a8deb832ef95eee3489f883f81b71b77a",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/e7984544cad8e735f371ff90e49ac95a8deb832ef95eee3489f883f81b71b77a"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/f43eaa0c6da1692abd73cfb9d177445d0b0af01fc2229be5c4a47d654ba1dd59",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/f43eaa0c6da1692abd73cfb9d177445d0b0af01fc2229be5c4a47d654ba1dd59"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/f796184c58af3c4c8299275488a705e14db20bd4694e5cbe786e4edc551da307",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/f796184c58af3c4c8299275488a705e14db20bd4694e5cbe786e4edc551da307"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/fa71c435e4015ea23c2f03e6563c3198086627a38013cf47aa6b299d8e843039",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/fa71c435e4015ea23c2f03e6563c3198086627a38013cf47aa6b299d8e843039"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/fc378e0d26b7620549809359a6eb56360850621e71b01a85260bfd6fdabfce6d",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/fc378e0d26b7620549809359a6eb56360850621e71b01a85260bfd6fdabfce6d"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/fc5132f48f173f722520e2a412baa601bb4b8dcb7c3d729f6b1ea7325941573b",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/fc5132f48f173f722520e2a412baa601bb4b8dcb7c3d729f6b1ea7325941573b"} 1
node_filesystem_device_error{device="nerv/ROOT/arch/fe2b21ad538bb656249699a2691b9a0e7e016bdfdabb4ee97cc2ca1b9f158408",fstype="zfs",mountpoint="/var/lib/docker/zfs/graph/fe2b21ad538bb656249699a2691b9a0e7e016bdfdabb4ee97cc2ca1b9f158408"} 1
node_filesystem_device_error{device="nerv01",fstype="zfs",mountpoint="/mnt/nerv01"} 1
node_filesystem_device_error{device="nerv01/.Backups",fstype="zfs",mountpoint="/mnt/nerv01/.Backups"} 1
node_filesystem_device_error{device="nerv01/.Databases",fstype="zfs",mountpoint="/mnt/nerv01/.Databases"} 1
node_filesystem_device_error{device="nerv01/.TMP",fstype="zfs",mountpoint="/mnt/nerv01/.TMP"} 1
node_filesystem_device_error{device="nerv01/.config",fstype="zfs",mountpoint="/mnt/nerv01/.config"} 1

What did you expect to see?

When using v0.16.0, I can see the filesystem metrics for all the devices/mountpoints and the node_filesystem_device_error metric returns 0 (seen below). Yet the new version v0.17.0-rc.0 does not export the same metrics.
I understand that v0.17.0-rc.0 is a pre-release, however, I am using this because I am running kernel 4.19 and the issue with with diskstats on this version of the kernel https://github.com/prometheus/node_exporter/pull/1109

node_filesystem_device_error{device="nerv/ROOT/arch",fstype="zfs",mountpoint="/rootfs"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/025d3bc629326ba93a460bb2a5a746d0f47c2e2cc29c2e1c79c567731dd8d8eb",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/025d3bc629326ba93a460bb2a5a746d0f47c2e2cc29c2e1c79c567731dd8d8eb"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/044728e064ead4ba35a6cd99d2f98526caf86c11fdab405182b3c0f2b4738b88",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/044728e064ead4ba35a6cd99d2f98526caf86c11fdab405182b3c0f2b4738b88"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/14dd29cc1f932c375736e858d1f4ad29e5342994e9e0d715a158771f9fec5d69",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/14dd29cc1f932c375736e858d1f4ad29e5342994e9e0d715a158771f9fec5d69"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/16b88f38cdb0fcd09944d5836ba7979dd02909e1cc62ba67f3ce58e492395bc0",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/16b88f38cdb0fcd09944d5836ba7979dd02909e1cc62ba67f3ce58e492395bc0"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/1767b4f5fc6339f86543383851c7a38add0e5c845ebb45c580fcd262cddc1bb4",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/1767b4f5fc6339f86543383851c7a38add0e5c845ebb45c580fcd262cddc1bb4"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/1bd49c24843bda0cb29dcad170126ee83c5b4b1656114bfabd0a35fe857c86cb",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/1bd49c24843bda0cb29dcad170126ee83c5b4b1656114bfabd0a35fe857c86cb"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/1e68c12bd9db792ffbc51eb57d5ed84ac35e7ecd3c09fe5cf9198835066a43aa",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/1e68c12bd9db792ffbc51eb57d5ed84ac35e7ecd3c09fe5cf9198835066a43aa"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/20d2b01c38e80aa5252d6131b8633812e833cc586879db9f59171b4708301adf",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/20d2b01c38e80aa5252d6131b8633812e833cc586879db9f59171b4708301adf"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/211c1111033060e8c7d8e839f9b4259fb5379f499a38a3312bec266828b56552",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/211c1111033060e8c7d8e839f9b4259fb5379f499a38a3312bec266828b56552"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/23409af703856228901c4eb55a33b37e80d8ad1531c8ab4ec7423afd727b152d",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/23409af703856228901c4eb55a33b37e80d8ad1531c8ab4ec7423afd727b152d"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/245413ee2ca8ef56e5fe470b58840955358d7f29c7510dda2159830d2783765f",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/245413ee2ca8ef56e5fe470b58840955358d7f29c7510dda2159830d2783765f"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/2f6f46c6965649636377bd8f4557882135f01e6480b9c12c9eba5284d79bf344",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/2f6f46c6965649636377bd8f4557882135f01e6480b9c12c9eba5284d79bf344"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/3438e6e608bd28bcd296c6c85cbedce38e20602e4f7bf91db60b6fc2be1431d1",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/3438e6e608bd28bcd296c6c85cbedce38e20602e4f7bf91db60b6fc2be1431d1"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/3adc218f7f9706d4dceed23bd1da0dec9e445921f73a684234c533e2bab1e156",fstype="zfs",mountpoint="/"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/3adc218f7f9706d4dceed23bd1da0dec9e445921f73a684234c533e2bab1e156",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/3adc218f7f9706d4dceed23bd1da0dec9e445921f73a684234c533e2bab1e156"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/3ae758036f3981be81cee49c0af5a4fdd7eff37448db3f1b153cf2af5d715db6",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/3ae758036f3981be81cee49c0af5a4fdd7eff37448db3f1b153cf2af5d715db6"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/3bb61cdbb7cdd5435c257f8d0980ae22c5ef93c216f79298fc55c20f56f34e97",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/3bb61cdbb7cdd5435c257f8d0980ae22c5ef93c216f79298fc55c20f56f34e97"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/3e4186e88233fcda7b2e1f34d50ac0a7713847d4d17f10ea184b7f1d220dd9e6",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/3e4186e88233fcda7b2e1f34d50ac0a7713847d4d17f10ea184b7f1d220dd9e6"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/3e6ef2ee79250c998e3aaa08c2e57ff566dfca8fbe873d97f2e2ea39bc9accef",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/3e6ef2ee79250c998e3aaa08c2e57ff566dfca8fbe873d97f2e2ea39bc9accef"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/47df832fb42fc4ba7c56c5a844917d7445f3c3e3ee9a4a6536c0d3d661ff757a",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/47df832fb42fc4ba7c56c5a844917d7445f3c3e3ee9a4a6536c0d3d661ff757a"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/4a5dc71bade3c2f0e8fc24105e089a9744e15cd95cc0b12a258fce0df2dc053d",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/4a5dc71bade3c2f0e8fc24105e089a9744e15cd95cc0b12a258fce0df2dc053d"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/4d5e0801ae7672f55b710018364999c8af0843de0ed8ec186ef6206a978e8ca7",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/4d5e0801ae7672f55b710018364999c8af0843de0ed8ec186ef6206a978e8ca7"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/4d6dc20f68be86dd64ca0e9b04fc65febd9d9fc95bd9d510625eb2eac959e402",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/4d6dc20f68be86dd64ca0e9b04fc65febd9d9fc95bd9d510625eb2eac959e402"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/4e0ed437ea1b3c2893834ba48f264de61f837832ef77037c30038df89df1faea",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/4e0ed437ea1b3c2893834ba48f264de61f837832ef77037c30038df89df1faea"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/52e85ffadb63a27f0ccccfadec9ab708a67e9c2708e34c7995855db3401a965c",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/52e85ffadb63a27f0ccccfadec9ab708a67e9c2708e34c7995855db3401a965c"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/6675b2a1f0e0c0352f50ea2890901b6eff3a1a42affb06c3af6eb0ba5f51ee06",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/6675b2a1f0e0c0352f50ea2890901b6eff3a1a42affb06c3af6eb0ba5f51ee06"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/7069e4e73d44eaf17b871c4079e9f1c9c53876c864f6f67f97406bd3e687180e",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/7069e4e73d44eaf17b871c4079e9f1c9c53876c864f6f67f97406bd3e687180e"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/7486219fa5c3ed1bf6c00344892df09064e8f3b9b78ee6639b74c1d92871e639",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/7486219fa5c3ed1bf6c00344892df09064e8f3b9b78ee6639b74c1d92871e639"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/7b5b3548c0fea4c877d5f71f749884a64d064c7d6bfc28a9ed97a63498d002cb",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/7b5b3548c0fea4c877d5f71f749884a64d064c7d6bfc28a9ed97a63498d002cb"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/7d34719fb66bf00ad53dcdc56c90f6a0667c9fc564c7bd7f330bd8f41900657a",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/7d34719fb66bf00ad53dcdc56c90f6a0667c9fc564c7bd7f330bd8f41900657a"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/8070c60e22fabea5ef32babf7af2b0004da63c8bce1b2e2650bd3d8644b97aea",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/8070c60e22fabea5ef32babf7af2b0004da63c8bce1b2e2650bd3d8644b97aea"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/87df4621d2956a0cf865381ac39e9628cf4852e0c3f0fcb4bd4dc64292ac72a0",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/87df4621d2956a0cf865381ac39e9628cf4852e0c3f0fcb4bd4dc64292ac72a0"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/8a881a4ba6a645c86ebaf5cd98e5d0cf83c836ff998e549cad847e7adffe8eaf",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/8a881a4ba6a645c86ebaf5cd98e5d0cf83c836ff998e549cad847e7adffe8eaf"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/8ce5b366258b1f6544d6e8a828d339c12e854ceff322c061bffc7aad525732d5",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/8ce5b366258b1f6544d6e8a828d339c12e854ceff322c061bffc7aad525732d5"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/96b96c7dd8970baeed1ae3950d268c240c71659d066624f0d41c6daa80141adc",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/96b96c7dd8970baeed1ae3950d268c240c71659d066624f0d41c6daa80141adc"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/9c45e614b9f652347b4e284d1d7249e5f912e0bb174084921a8ca08fca9a31a8",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/9c45e614b9f652347b4e284d1d7249e5f912e0bb174084921a8ca08fca9a31a8"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/a41e08cac2aed4afa3fcf7edf3ac4dfe9932467ac975a48f0e1d4add2a989cb5",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/a41e08cac2aed4afa3fcf7edf3ac4dfe9932467ac975a48f0e1d4add2a989cb5"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/abb4d533c9f3ae38dbbf8aaa3912a78b39f844398d7b34c796a4e6d9bdf72fb2",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/abb4d533c9f3ae38dbbf8aaa3912a78b39f844398d7b34c796a4e6d9bdf72fb2"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/be0014a2dc284bd19f91c90cd8c54a1190b4be5870cd49899a4c77a58d6fa956",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/be0014a2dc284bd19f91c90cd8c54a1190b4be5870cd49899a4c77a58d6fa956"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/c0162969ee82bc52e2a08c4b5d72119ab9e9e626392af73ebbec8d16d4cdd1fb",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/c0162969ee82bc52e2a08c4b5d72119ab9e9e626392af73ebbec8d16d4cdd1fb"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/c4cf8cb218e6a1cc8fd14b347d908a0d04db10f32adb35a4fd75951868ad0143",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/c4cf8cb218e6a1cc8fd14b347d908a0d04db10f32adb35a4fd75951868ad0143"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/c697fd3aa88f8142da8669eeadb1f7134805acc0c2e385b1e66e8dca3016f3ba",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/c697fd3aa88f8142da8669eeadb1f7134805acc0c2e385b1e66e8dca3016f3ba"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/c72f347df89d17d79caebe0156de46dd735e7401f1d5ca986613ad1dd25ea98a",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/c72f347df89d17d79caebe0156de46dd735e7401f1d5ca986613ad1dd25ea98a"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/ca3162748f5a0b9c1a58e4e9d133b7ab31a1ea143980ab642e80dd0d2d1d2752",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/ca3162748f5a0b9c1a58e4e9d133b7ab31a1ea143980ab642e80dd0d2d1d2752"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/cfec369943539a1d09073bf2cfb547ef8d3815bc6533b54ade2635725a6084bc",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/cfec369943539a1d09073bf2cfb547ef8d3815bc6533b54ade2635725a6084bc"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/d1eddf5584a2dfba56786816b064ef6601eb087bd1bf437aa694e43c64bd40c3",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/d1eddf5584a2dfba56786816b064ef6601eb087bd1bf437aa694e43c64bd40c3"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/d26fc7a3b00155a8356aa27c7cdf039fa85b7d754aa8414b334910b86a4f8484",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/d26fc7a3b00155a8356aa27c7cdf039fa85b7d754aa8414b334910b86a4f8484"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/d5261af6da1706e3a425d0497b2ac41fdcfb783ed2ad6213263f3e888e77c7d7",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/d5261af6da1706e3a425d0497b2ac41fdcfb783ed2ad6213263f3e888e77c7d7"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/d77a77268ee6805cbea659cd16dd7f333aa62df6de01b97dc9a040d5ec3217cd",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/d77a77268ee6805cbea659cd16dd7f333aa62df6de01b97dc9a040d5ec3217cd"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/d79fa0ddd3cb478c2ce38c0a9b1b56976fe365b38c03a0724b242551c14a75d4",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/d79fa0ddd3cb478c2ce38c0a9b1b56976fe365b38c03a0724b242551c14a75d4"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/e03c56885678507177b685912b092f4c17692cd0c97f39bfcd61a83dc8b281fe",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/e03c56885678507177b685912b092f4c17692cd0c97f39bfcd61a83dc8b281fe"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/e05aa62a7a3a338a265d34e296d7ca5878b7e926c9b91d13084c758f06d9d003",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/e05aa62a7a3a338a265d34e296d7ca5878b7e926c9b91d13084c758f06d9d003"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/e7984544cad8e735f371ff90e49ac95a8deb832ef95eee3489f883f81b71b77a",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/e7984544cad8e735f371ff90e49ac95a8deb832ef95eee3489f883f81b71b77a"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/f43eaa0c6da1692abd73cfb9d177445d0b0af01fc2229be5c4a47d654ba1dd59",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/f43eaa0c6da1692abd73cfb9d177445d0b0af01fc2229be5c4a47d654ba1dd59"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/f796184c58af3c4c8299275488a705e14db20bd4694e5cbe786e4edc551da307",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/f796184c58af3c4c8299275488a705e14db20bd4694e5cbe786e4edc551da307"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/fa71c435e4015ea23c2f03e6563c3198086627a38013cf47aa6b299d8e843039",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/fa71c435e4015ea23c2f03e6563c3198086627a38013cf47aa6b299d8e843039"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/fc378e0d26b7620549809359a6eb56360850621e71b01a85260bfd6fdabfce6d",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/fc378e0d26b7620549809359a6eb56360850621e71b01a85260bfd6fdabfce6d"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/fc5132f48f173f722520e2a412baa601bb4b8dcb7c3d729f6b1ea7325941573b",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/fc5132f48f173f722520e2a412baa601bb4b8dcb7c3d729f6b1ea7325941573b"} 0
node_filesystem_device_error{device="nerv/ROOT/arch/fe2b21ad538bb656249699a2691b9a0e7e016bdfdabb4ee97cc2ca1b9f158408",fstype="zfs",mountpoint="/rootfs/var/lib/docker/zfs/graph/fe2b21ad538bb656249699a2691b9a0e7e016bdfdabb4ee97cc2ca1b9f158408"} 0
node_filesystem_device_error{device="nerv01",fstype="zfs",mountpoint="/rootfs/mnt/nerv01"} 0
node_filesystem_device_error{device="nerv01/.Backups",fstype="zfs",mountpoint="/rootfs/mnt/nerv01/.Backups"} 0
node_filesystem_device_error{device="nerv01/.Databases",fstype="zfs",mountpoint="/rootfs/mnt/nerv01/.Databases"} 0
node_filesystem_device_error{device="nerv01/.TMP",fstype="zfs",mountpoint="/rootfs/mnt/nerv01/.TMP"} 0
node_filesystem_device_error{device="nerv01/.config",fstype="zfs",mountpoint="/rootfs/mnt/nerv01/.config"} 0

What did you see instead?

A metric implying that node-exporter encountered an error while collecting statistics for devices.

No errors were observed in the node-exporter logs either:

time="2018-11-18T07:31:23Z" level=info msg="Starting node_exporter (version=0.17.0-rc.0, branch=HEAD, revision=751996761903af4cffe18cf2d980b8ae9a202204)" source="node_exporter.go:82"
time="2018-11-18T07:31:23Z" level=info msg="Build context (go=go1.11.1, user=root@c0c3850b6d2b, date=20181020-06:27:39)" source="node_exporter.go:83"
time="2018-11-18T07:31:23Z" level=info msg="Enabled collectors:" source="node_exporter.go:90"
time="2018-11-18T07:31:23Z" level=info msg=" - arp" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - bcache" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - bonding" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - conntrack" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - cpu" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - diskstats" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - edac" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - entropy" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - filefd" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - filesystem" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - hwmon" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - infiniband" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - ipvs" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - loadavg" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - mdadm" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - meminfo" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - netclass" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - netdev" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - netstat" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - nfs" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - nfsd" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - sockstat" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - stat" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - textfile" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - time" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - timex" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - uname" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - vmstat" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - xfs" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg=" - zfs" source="node_exporter.go:97"
time="2018-11-18T07:31:23Z" level=info msg="Listening on :9100" source="node_exporter.go:111"
accepted bug
Source
picture nightah

Most helpful comment

@nightah I think this issue may be due to the changes in 0f9842f20a036d2db8f29102e6bc0cd7b1672865 which introduced the path.rootfs option in 0.17. If you set this to /rootfs in your docker container does it fix the issue for you?

picture pgier on 18 Jan 2019
👍3

All 8 comments

This seems to persist and carries through to v0.17.0 too.

nightah picture nightah on 12 Dec 2018

Can you try turning on the debug output with --log.level="debug"? Hopefully that will provide some more info about why the failures are occurring.

pgier picture pgier on 12 Dec 2018

Seeing a number of errors on startup with debug logging turned on:

time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/mnt/nerv01\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/mnt/nerv01/.Backups\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/mnt/nerv01/.Databases\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/mnt/nerv01/.TMP\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/mnt/nerv01/.config\": no such file or directory" source="filesystem_linux.go:89"

time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/zfs/graph/7d157978869b578fb4e06a8da0acac063261072129b7b561621a81190e4a09b7\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/zfs/graph/c4cf8cb218e6a1cc8fd14b347d908a0d04db10f32adb35a4fd75951868ad0143\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/zfs/graph/8dba88f90aaaeebe05bcf859140f7bd1a5ed7910671dff7ecac743571d5d2ee9\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/zfs/graph/e7984544cad8e735f371ff90e49ac95a8deb832ef95eee3489f883f81b71b77a\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/zfs/graph/7486219fa5c3ed1bf6c00344892df09064e8f3b9b78ee6639b74c1d92871e639\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/zfs/graph/fe2b21ad538bb656249699a2691b9a0e7e016bdfdabb4ee97cc2ca1b9f158408\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/zfs/graph/e184827b3ea9fd36205709899930ba7ad0e394c4b43cb76dbd34429da95104f0\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/zfs/graph/d0732d006a0187d104e3ae6a3cd733640959fc524008fe1deb780886ed977b65\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/zfs/graph/9cfe97fb3905f3a36a66b83861ea738e1191512909bf777057d4650cd25c8d05\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/zfs/graph/211c1111033060e8c7d8e839f9b4259fb5379f499a38a3312bec266828b56552\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/zfs/graph/3a67f69e2859e55c74a36796b646a3fb8ba50a78c090bead6af38fcbad00e32d\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/zfs/graph/abb4d533c9f3ae38dbbf8aaa3912a78b39f844398d7b34c796a4e6d9bdf72fb2\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/zfs/graph/e35901c26cf3e4cb551f8fa382dad8f7dc71a5661a26410221c97467d5e66554\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/zfs/graph/14dd29cc1f932c375736e858d1f4ad29e5342994e9e0d715a158771f9fec5d69\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/zfs/graph/d79fa0ddd3cb478c2ce38c0a9b1b56976fe365b38c03a0724b242551c14a75d4\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/zfs/graph/36773063797fd902fd49f98f6bc2018b67ffe4f9423b6ff37708f0b2a4323ecc\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/zfs/graph/6702f04da02d3bc66a9ab3fd901d037737716ea14bb1588296a3f4be658b94f6\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/zfs/graph/82b5cc74b04dc53df81ffd4072820ae55d938b63df0531e8965963a6df35a889\": no such file or directory" source="filesystem_linux.go:89"

time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/containers/03a575d70950494b9c96158fcb1e53e8aa91d0d3aecb51fbee82fee7242eb016/mounts/shm\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/containers/94318fd83423c6cf633f1beb62e1a2882792ed558132c240767be17d2e997450/mounts/shm\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/containers/9a68276563fdc06b03aa9d24f37eaae1fae0dfd9a885220062dc5f95f1c7f8e4/mounts/shm\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/containers/a593c9c1fa781d2a1322ed71a2fbae58d2aa8ea47fa13e9543f65b7ca2ba1ed0/mounts/shm\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/containers/5a1d5c145ddb64cbe32513402f2b4e19971b3011f649da71ba11eb1400ce31f5/mounts/shm\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/containers/c33f89e1c0e4f303e33002acdb7a8315c2185d469918107c8ce85b49dc932ed1/mounts/shm\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/containers/c83e4b8f57335c48f249ca20771e84a4ca15d2b87397041f48c62cbea4fb40bb/mounts/shm\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/containers/c67911f72eb2c7233f00faa9b559c4df5b8ae59770a76dcf46dbe4a2ebd2c436/mounts/shm\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/containers/dc8579a278bc8ce575212b102519af719aee446d9bfc9967accc1b5a0abc672f/mounts/shm\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/containers/ce9d3ec7c95a8f7e0e4a0232f7507a942e9cbeac6b19daa2bee77b863b5f3b0f/mounts/shm\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/containers/e09d60b8990b6784bb62af4e52c660106fb698f5020d9379a5eefd14e57574b7/mounts/shm\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/containers/f3239e0591f0f72449f7084044b25bc7e0f147fc8206b5cc24cee7cd83422fc5/mounts/shm\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/containers/f52898c9101b75199109fe00e7d6ece2e6a8ab4191b13579a606ee1927f570e0/mounts/shm\": no such file or directory" source="filesystem_linux.go:89"
time="2018-12-12T22:55:39Z" level=debug msg="Error on statfs() system call for \"/var/lib/docker/containers/5152724d0c30175a38e5cebf169fe4f6c78f61a257a5f7c3fc25220bbb666ca4/mounts/shm\": no such file or directory" source="filesystem_linux.go:89"

Looks like it has something to do with the escaping and double quotations.
If I run a stat -f \"/mnt/nerv01/.Backups\" I get the following returned: stat: cannot read file system information for '"/mnt/nerv01"': No such file or directory

However I attempt to run stat -f /mnt/nerv01/.Backups the following is returned:

  File: "/mnt/nerv01/.Backups"
    ID: 6e3db6e3008bc999 Namelen: 255     Type: zfs
Block size: 131072     Fundamental block size: 131072
Blocks: Total: 62249691   Free: 59661183   Available: 59661183
Inodes: Total: 15273419023 Free: 15273262923

EDIT: Okay looks to be more specifically the escaping of the double quotation, if I try stat -f "/mnt/nerv01/.Backups" this also returns positive results:

  File: "/mnt/nerv01/.Backups"
    ID: 6e3db6e3008bc999 Namelen: 255     Type: zfs
Block size: 131072     Fundamental block size: 131072
Blocks: Total: 62249691   Free: 59661183   Available: 59661183
Inodes: Total: 15273419023 Free: 15273262923
nightah picture nightah on 12 Dec 2018

Hi all,
same issue here with a glusterfs mountpoint:

Error on statfs() system call for \"/mnt/share\": no such file or directory

Greetings,
floek

floek picture floek on 15 Jan 2019

@nightah I think this issue may be due to the changes in 0f9842f20a036d2db8f29102e6bc0cd7b1672865 which introduced the path.rootfs option in 0.17. If you set this to /rootfs in your docker container does it fix the issue for you?

pgier picture pgier on 18 Jan 2019
👍3

@pgier: Yes this resolves the errors, looks like the paths have slightly changed despite setting path.rootfs back to /rootfs but I can work around that.

nightah picture nightah on 18 Jan 2019

Looks like this is solved.

SuperQ picture SuperQ on 18 Jan 2019

node_exporter v0.17

  1. set path.rootfs to /rootfs
  2. ignore docker or kubelet device mountpoint 
            - --collector.filesystem.ignored-mount-points
            - '^/(dev|proc|sys|var/lib/docker/.+|opt/data/docker/.+|opt/data/kubelet/.+)($|/)'
            (change it with you docker or kubelet workdir)
gaopeiliang picture gaopeiliang on 9 Oct 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

vielmetti picture vielmetti  路  3Comments
dorkmatt picture dorkmatt  路  3Comments
mInrOz picture mInrOz  路  5Comments
tmegow picture tmegow  路  5Comments
lesovsky picture lesovsky  路  3Comments