Node_exporter: Feature request: IPC metrics

[tzz]

After looking at http://www.brendangregg.com/blog/2017-05-09/cpu-utilization-is-wrong.html I wanted to calculate the Instructions Per Cycle (IPC) using node_exporter. Currently neither the instructions nor the cycles counters are available through node_exporter.

Currently I can get the data with perf stat -D 1000 -a -d -x, -c -- sleep 10 which waits 1 second, then samples for 10 seconds. I can then look for cycles and instructions in the output. I don't know what the equivalent kernel APIs are, and if they are available to node_exporter through procfs or other means.

[tzz]

From the command line, the counters can be retrieved without root by using echo LEVEL > /proc/sys/kernel/perf_event_paranoid

You may not have permission to collect stats.
Consider tweaking /proc/sys/kernel/perf_event_paranoid:
 -1 - Not paranoid at all
  0 - Disallow raw tracepoint access for unpriv
  1 - Disallow cpu events for unpriv
  2 - Disallow kernel profiling for unpriv

I don't see a way to get to the perf events API through procfs or sysfs right now.

[discordianfish]

Interesting.. Just saw that there is this already: https://github.com/tcolgate/perfexporter

But certainly open to include that in the node-exporter, pending the implementation details. Looks like there is better support in golang in the making: https://github.com/golang/go/issues/24918

[discordianfish]

But looks like that needs root, which would be a problem for including it in the node-exporter.

[tzz]

But looks like that needs root, which would be a problem for including it in the node-exporter.

OK, I see, but see the above about /proc/sys/kernel/perf_event_paranoid? That will apply to any perf client, right? So maybe it makes the root requirement less of a problem?

[discordianfish]

Ah, yeah I think if a user can run the node-exporter unprivileged by setting perf_event_paranoid, that would be a good compromise. Still need probably makes sense to wait for golang/go#24918 to land in a release.

[hodgesds]

I've been working on a little go library that would allow you to do that, the API is not finalized but I've been thinking of implementing this feature.

[hodgesds]

I think I pretty much have this implemented, if you want to test it out and have any opinions on configuration that would be helpful as well.

[tzz]

@hodgesds @discordianfish I think #1274 definitely closes the feature request here. Let me know if you agree.

[discordianfish]

Good point, thanks!

[tzz]

I wanted to thank you both @hodgesds and @discordianfish for working on this feature. It's really useful and well implemented.

[hodgesds]

No problem, glad it's useful to others!