Node_exporter: ERROR: mdadm collector failed

[gregorygtseng]

after 0.000073s: error parsing mdstatus: error parsing mdstat: too few matches found in statusline: 4186624 blocks super 1.2 256k chunks" source="node_exporter.go:92"

This is what my /proc/mdstat looks like:

$ cat /proc/mdstat
Personalities : [raid0]
md0 : active raid0 xvdb[0]
      4186624 blocks super 1.2 256k chunks

unused devices: <none>

System:

$ uname -a
Linux hostname 3.10.0-123.8.1.el7.x86_64 #1 SMP Mon Sep 22 19:06:58 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

I'm running node_exporter v0.12.0rc3 compiled with go1.5.3
Anyone have any ideas how to fix this?

[juliusv]

@gregorygtseng Thanks for the report. This is the format that Node Exporter currently expects /proc/mdstat to have: https://github.com/prometheus/node_exporter/blob/master/collector/fixtures/proc/mdstat

Before fixing this, I'm curious, what kernel version are you running?

[juliusv]

Ah, it seems that the different format is due to raid0 being displayed differently than the existing RAID types. See also: https://www.safaribooksonline.com/library/view/managing-raid-on/9780596802035/ch04s03.html

[gregorygtseng]

Ah, yes. Thanks for the quick response. My kernel information is in the first post. Well, I'm glad I could provide an example of raid0 :)

[kkvilekval]

Here's another example:

6/6/2016 1:08:11 PMERRO[0542] ERROR: mdadm collector failed after 0.000101s: error parsing mdstatus: error parsing mdstat: too few matches found in statusline:       585922432 blocks 64k chunks  source=node_exporter.go:91
6/6/2016 1:08:26 PMERRO[0557] ERROR: mdadm collector failed after 0.000096s: error 
parsing mdstatus: error parsing mdstat: too few matches found in statusline:       585922432 blocks 64k chunks  source=node_exporter.go:91
6/6/2016 1:08:41 PMERRO[0572] ERROR: mdadm collector failed after 0.003142s: error parsing mdstatus: error parsing mdstat: too few matches found in statusline:       585922432 blocks 64k chunks  source=node_exporter.go:91

core@b0003 ~ $ uname -a
Linux b0003 4.3.6-coreos #2 SMP Tue May 3 21:48:31 UTC 2016 x86_64 Intel(R) Xeon(R) CPU 5140 @ 2.33GHz GenuineIntel GNU/Linux

And the mdstat

core@b0003 ~ $ cat /proc/mdstat 
Personalities : [raid0] 
md127 : active raid0 sda1[0] sdb1[1]
      585922432 blocks 64k chunks

[SuperQ]

Seems simple enough to fix the parser, I'll see what I can do.

[SuperQ]

253 is merged, please let me know if this fixes the issue for you.

[mcdan]

Just testing with the merged commit and I still see collector errors, here is my mdstat info, kernel /linux version details and the error i see from the collector:

➜  node_exporter git:(master) ✗ cat /proc/mdstat                    
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md126 : active raid5 sda[2] sdb[1] sdc[0]
      1953519616 blocks super external:/md127/0 level 5, 64k chunk, algorithm 0 [3/3] [UUU]

md127 : inactive sdb[2](S) sdc[1](S) sda[0](S)
      7932 blocks super external:imsm

unused devices: <none>

OS details:
➜ node_exporter git:(master) ✗ uname -a Linux radiohead 3.19.0-65-generic #73~14.04.1-Ubuntu SMP Wed Jun 29 21:05:22 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Error from collector:
ERRO[0044] ERROR: mdadm collector failed after 0.000094s: error parsing mdstatus: error parsing mdstat: too few matches found in statusline: 7932 blocks super external:imsm source=node_exporter.go:91

Compiled the collector at b55019f10b3 with go 1.4.2

[filippog]

I'm running into a similar parsing error with node_exporter from git 1549f308 on linux 4.4.2:

$ cat /proc/mdstat 
Personalities : [raid1] 
md2 : active raid1 sda3[0] sdb3[1]
      438449152 blocks super 1.2 [2/2] [UU]
      bitmap: 0/4 pages [0KB], 65536KB chunk

md1 : active (auto-read-only) raid1 sda2[0] sdb2[1]
      976320 blocks super 1.2 [2/2] [UU]
        resync=PENDING

md0 : active raid1 sda1[0] sdb1[1]
      48794624 blocks super 1.2 [2/2] [UU]

unused devices: <none>

and the error prometheus-node-exporter[96676]: time="2016-09-14T08:37:42Z" level=error msg="ERROR: mdadm collector failed after 0.000289s: error parsing mdstatus: error parsing mdstat: too few matches found in buildline: \tresync=PENDING" source="node_exporter.go:91"

[SuperQ]

@mcdan @filippog It looks like those are different issues. One of them is covered by https://github.com/prometheus/node_exporter/issues/308.

[mcdan]

Yea, resync=pending is #308
I think mine is a machine with an odd ball raid config. I'm working on a patch for it now but it might make the existing parsing overly complex. Will submit PR when I have something.

[mcdan]

PR for fix: https://github.com/prometheus/node_exporter/pull/312

[samzhang111]

I'm experiencing a similar issue on Centos 7 on an Amazon EC2 m3.medium machine. Specifically, the ScyllaDB 1.3 AMI (ami-4bda0d2b). Is this the same issue?

[centos@ip-172-12-122-12~]$ cat /proc/mdstat
Personalities : [raid0]
md0 : active raid0 xvdb[0]
      4186624 blocks super 1.2 256k chunks

unused devices: <none>

[centos@ip-172-12-122-12 ~]$ uname -a
Linux ip-172-12-122-12 3.10.0-327.4.5.el7.x86_64 #1 SMP Mon Jan 25 22:07:14 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[centos@ip-172-12-122-12 ~]$ ./node_exporter
INFO[0000] Starting node_exporter (version=0.13.0-rc.1, branch=master, revision=abe8e297a6161bc341dab101c8dcbbf199334356)  source=node_
exporter.go:135
...
INFO[0000] Listening on :9100                            source=node_exporter.go:176
ERRO[0001] ERROR: mdadm collector failed after 0.000094s: error parsing mdstatus: error parsing mdstat: invalid raid0 status line:
  4186624 blocks super 1.2 256k chunks  source=node_exporter.go:91
ERRO[0016] ERROR: mdadm collector failed after 0.000060s: error parsing mdstatus: error parsing mdstat: invalid raid0 status line:
  4186624 blocks super 1.2 256k chunks  source=node_exporter.go:91

[mcdan]

This doesn't look like the same issue because it seems to be throwing a different error. My fix does not help with that. I added your text to the test suite and it fails with the same error as you say above. I'm happy to fix it too but I haven't gotten this PR in yet and I'm wondering if this issue is going to become "mdstat" doesn't parse things right, bucket of problems.

[samzhang111]

It would be greatly appreciated if you fix this in your PR :). I don't see too big a harm in collecting mdam related issues here, though I can open a new issue if needed.

[mcdan]

Nah that's fine with me too, I just didn't want to gum up your process. I will try and get this fixed tomorrow/this weekend and add it to the existing PR.

[mcdan]

Sorry took me a while to remember how to setup go, still very much a beginner. The PR should now also fix @samzhang111 's issue as well.