Node_exporter: Turn on/off go and http metrics

[nsauro]

It would be nice to have an option to turn on/off go and http_request related metrics.

[SuperQ]

You can drop metrics with a relabel config by matching on __name__.

[nsauro]

Sure. But I was hoping to be able to treat those metrics like any other collector and be able to enable/disable via the collectors.enabled flag.

[mdlayher]

I believe they are built directly into the Go client (not node_exporter). I don't know if it supports the ability to enable and disable them individually.

[grobie]

It's possible. The client library provides a collector for go runtime
metrics and can instrument the http handler. There is an instrumented
handler and one can active these things manually, depending on a flag.

In general I'm always skeptical when people request to deactivate this
metrics, but I can see that depending on the number of node_exporter they
add quite a bit of additional load on the server:

curl -s http://localhost:9100/metrics | grep '^(http\|go)_' | wc -l

=> 48

I'd be fine adding such an option.

On Fri, Apr 14, 2017 at 4:57 PM Matt Layher [e-mail]
wrote:

I believe they are built directly into the Go client (not node_exporter).
I don't know if it supports the ability to enable and disable them
individually.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/prometheus/node_exporter/issues/556#issuecomment-294224621,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAANaOaaQfShsdjxJ229gMiPEjFqwtoqks5rv8-egaJpZM4M9_qF
.

[brian-brazil]

In general I'm always skeptical when people request to deactivate this
metrics

As am I. May I ask why you want to disable these?

[grobie]

As per our IRC discussion, Brian and I agreed to disable the golang runtime metrics by default as these are usually only needed by application developers e.g. us as the maintainers. The new client_golang will reduce the http_ metrics by ~60% which should make the overhead of having them neglectable.

[brian-brazil]

Just the go_memstats metrics I believe. GC and goroutine stats are cheap, and more generally applicable.

[grobie]

That makes it difficult, as there is only one collector exposing all
runtime metrics.

On Fri, Apr 14, 2017 at 5:52 PM Brian Brazil [e-mail]
wrote:

Just the go_memstats metrics I believe. GC and goroutine stats are cheap,
and more generally applicable.

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/prometheus/node_exporter/issues/556#issuecomment-294234475,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAANaK2ks3KMjyUfv3EjtcCuyr5eNIYSks5rv9yIgaJpZM4M9_qF
.

[brian-brazil]

We'd have to split it.

[beorn7]

go_memstats can have a dramatic impact with high-frequency scraping (stop the world involved, with possible improvement in Go 1.9). It would totally make sense to split the go collector into a "cheap" and an "expensive" part in client_golang. We should, however, double-check what collections are actually expensive or cheap, respectively.

[nsauro]

Thank you for the responses. As per @brian-brazil 's question of "As am I. May I ask why you want to disable these?"

Pretty much which the reasons that were mentioned in this thread: not collecting/storing data which is of no consequence to my current circumstances. If in the future, that changes, then I can just switch them back on.

Anyways, thanks for the rapid correspondence! I've been using Prometheus for a bit now and am very impressed with it, it has helped us quite a bit.

[SuperQ]

One quick note about "not relevant to me right now". One philosophy of how we recommend Prometheus use is to record everything, even things that seem a little useless right now, because it may be useful in the future, especially in a post-mortem debugging session. I can't count the number of times where I've said "If only I had that metric before the outage".

Of course, there are metrics that are really useless, and we should remove those. We don't want exporters to be providing huge amounts of meta-telemetry.

👍 This has been a useful discussion.

[alsm]

Late to the conversation, but I'm also looking to do this. While there aren't that many stats in there (for go, process and http) I know I don't need them and I want to minimise the storage footprint per system scraped across 1000s of systems (which is also why I don't want to use relabeling to drop them on prometheus)

[SuperQ]

@alsm, Dropping them is just fine as a workaround. There is minimal cost to scrape the data, and the drop method saves the storage space space. Even with 1000s of systems.

[discordianfish]

Did we make up our mind about this? I don't really care either way. This issue is 'accepted', so do we (still?) agree we want a flag to disable these stats?

[SuperQ]

@discordianfish I know @beorn7 was working on ways to make this easier in client_golang.

We've cut down the metrics output to ~45 node_exporter (go|process|promhttp)_... metrics.

It should also be simpler to add flags to adjust the handler() function now to drop these metrics.

I think we still want support this feature, but I'm not personally going to put any effort into it.

[beorn7]

It's very easy to implement with the current client_golang.
I can send a PR at my next convenience if there is consensus we want this feature.

[discordianfish]

Great thanks! So yeah, it looks like we have consensus.

[beorn7]

Hmmmm… re-reading the whole issue, I'm not so sure anymore.

Much further up, there was the concern that the Go memory metrics are too expensive (for the node exporter itself) to collect. However, with Go 1.9+, that should not be problem anymore.
And to be clear: There is no mechanism in client_golang (nor any plans) to micromanage collectors so that they would only collect a subset of the metrics they usually collect.

With that being off the table, the remaining part here is to avoid saving metrics about the node exporter itself (in cases where those are not needed but the resource footprint of saving them on the Prometheus server is relevant). One side says this should simply done by configuring Prometheus to drop the unwanted metrics. The other side says we should provide a command line flag for the node exporter to not expose those in the first place.

The latter is very easy to do. I would suggest to "switch off" all the go_*, process_* and promhttp_* metrics via that flag, but keep the build_info metric. I think switching off just a subset of those metrics would be both confusing to the user and messy to implement.

Is there consensus to do it in this way?

[beorn7]

See #1148.

[discordianfish]

Fixed by #1148