Node_exporter: [Feature Suggestion]

[therealbill]

Name: Node Labels
Purpose: To provide a way to add labels which would be present on every metric exported.

I can see two ways of doing this, and I think both are valid use cases. First, command line.
--labels=dc:dc1,ff:bar where parsing the flags would result in a mapping of string -> string which would be added as, in the above example: dc=dc1,ff=bar

The second way would be to have a --labels-file option which reads in the file, which would be expected to be in labelname=value lines - one label per line.

This would be quite useful for adding custom labels from the node_exporter side for things you won't get generically or computationally. For example you could have a labels file of:

owner=bob
environment=staging
priority=2

Which would result in the labels "owner=bob,environment=staging,priority=2" being added and thus Bob could filter for systems he owns, or a response team could filter on priority. Of course, alerting also could filter on these and/or use them in their alerts.

[brian-brazil]

You should use target labels for this, see https://prometheus.io/docs/operating/configuration/#

It is an anti-pattern for an exporter to do what you suggest.

[therealbill]

That doc doesn't seem say anything about how to do this. Any specific examples, particularly those using discovery as opposed to manually writing jobs for hundreds of nodes?

[brian-brazil]

There's an example near the end of http://www.robustperception.io/automatically-monitoring-ec2-instances/

[matthiasr]

Node exporter used to have this feature. We removed it in favour of two things:

1. target labels and/or relabelling

If you have hundreds of targets, you are not manually typing them out anyway. In many contexts, it is possible to infer labels through the same mechanism that is used to generate the target list, whatever that is. For example, in Kubernetes, the service discovery returns hidden labels that can then be turned into actual labels on each of the metrics. In DNS SD, we often pass in multiple DNS names (which, in our case, have a lot of structure anyway), then use parts of them to also create additional labels for the concrete use case. This all happens on the Prometheus server side, because the server should know what it is scraping.

1. "Metadata" time-series that are exported client side

Using the textfile collector, you can write any and all metrics you want. For us, this includes roles and contacts, with a constant value of 1. These are stored very efficiently server side. See this article for the general approach. Prometheus 0.19 has syntax changes to make this even more flexible. That this requires an explicit merge in the query is deliberate. It avoids a discontinuity in _all_ time series that is inevitable when the label set changes.

I hope this makes sense!

[srikanth108]

Hi Team,

We want to monitor our whitebox application process (process name-iqs) and some specific ports (Eg:- port number 5262, 5263) using Prometheus monitoring tool and currently we are unable to monitor them since the Node Exporter does not collect and provide such metrics.

Since we have limited knowledge in Node Exporter and Prometheus monitoring tool, please assist here.

Thanks,
Srikanth

[mdlayher]

Please reach out on the prometheus-users mailing list.

[srikanth108]

Thank you for response :)