Node_exporter: Feature: allow node exporter to push metrics to pushgateway

[echoAlexey]

Is there a way to make node exporter to push metrics to push gateway?

Our infra have nodes dynamically created, these nodes are inside vApp and can't be accessible from outside. Having push gateway + Prometheus as a static environment with pre-configured dynamic nodes with node exporter would work for us.

[SuperQ]

No, you will need to provide access to the nodes from your Prometheus
server. Prometheus offers a wide variety of service discovery methods that
dynamically adjust the list of targets based on the changing environment.

[echoAlexey]

Thank you,

We will investigate this option.

[RRAlex]

The same situation arises if you run a swarm and want to run one node-exporter per swarm host, within docker, and have them behind a reverse proxy for authentication.

A node-exporter -> pushgateway would allow to pull a whole cluster from the main entry point.

[Marqin]

Is there any progress on this?

[brian-brazil]

There are no plans to ever implement this. Prometheus is a pull system, and the pushgateway is not a way around that. See https://prometheus.io/docs/practices/pushing/

[RRAlex]

Federation actually seems like a solution: scrape from within with a short retention time, then pull from root prometheus node (or use remote write if the firewall isn't going to let any traffic in)

[brian-brazil]

That's not recommended, federation is not intended as a way to dump full Prometheus servers. Rather it's meant for pulling aggregated stats. Doing as you suggest would lead to artifacts.

[Silex]

If one absolutely has to do it:

curl -s http://localhost:9100/metrics | curl --data-binary @- http://pushgateway.example.org:9091/metrics/job/some_job/instance/some_instance

In some cronjob.

[alkuzad]

Working push directly from node, grepping-out go_info and other conflicting with pushgateway:

curl -s http://localhost:9256/metrics | grep -v "\(\(^\| \)go_\|http_request\|http_requests\|http_response\|process_\)" | curl --data-binary @- http://pushgateway.example.com:9091/metrics/job/node/instance/<INSTANCE_NAME>

[aditya-konarde]

While I do understand that prometheus doesn't ever intend to support such a configuration, it would be awesome if we could all contribute to a doc on how we're getting metrics out from a set of nodes inside a VPN.

I think the workaround solution needs to be somewhat standard and documented, as opposed to a hack, in a cronjob.

I'm trying to implement a similar setup right now, will see if the setup is PR-worthy to get the doc started.

[discordianfish]

The curl | curl examples above are straight forward ways to push to the pushgw. I don't think we need any changes to the node-exporter. Either way, this is a general Prometheus topic so you might rather open an issue in the docs repo if you feel like more docs on this should be provided. Going to lock this now.