Victoriametrics: Is it possible for VictoriaMetrics to substitue Prometheus+Pushgateway?

[ozn0417]

Im currently using Prometheus with a pushgateway where I have some batch jobs and other scripts that push stats/data to push gateway, which gets propagated to prometheus (prometheus pulls data from here). Im exploring VictoriaMetrics as it provides additional promQL operations that pure prometheus doesn't and my setup is following:
script/batchjob -> pushgateway <- prometheus -> VictoriaMetrics

Im wondering if its possible to just get rid of both pushgateway and prometheus and have my batchjob/script directly interface with VictoriaMetrics?

[valyala]

There are plans for creating vmagent service, which would accept data in various protocols, including Pushgateway protocol, and then send it to VictoriaMetrics in the most efficient way. The vmagent service will support the following ingestion protocols:

• Prometheus remote_write protocol
• Influx line protocol
• Graphite plaintext protocol
• OpenTSDB put protocol (both telnet and http)
• StatsD protocol with widely used extensions such as DogStatsD.
• Pushgateway protocol

Additionally, the service would accept Prometheus-compatible scrape configs for scraping data from the configured targets.

As for now, the following options exist for your use case additionally to the scriipt -> pushgateway <- prometheus -> VictoriaMetrics case:

• To push data from your batch jobs and other scripts to statsd, which then would push the data to VictoriaMetrics via Graphite protocol. See these docs for details.
• To push data directly to VictoriaMetrics via supported ingestion protocols.

[ozn0417]

Thanks, these help! I'll utilize them and post any questions/comments I have.

[valyala]

Let's keep this issue open until vmagent is implemented.

[valyala]

The first version of vmagent has been released - see https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md

[ledmonster]

@valyala
Thank you for providing an exciting module. Does vmagent support Pushgateway protocol now? Or do you have a plan to support it in near future?

If vmagent doesn't support pushgateway protocol, I'm considering that convert pushgateway protocol to Prometheus remote_write protocol and push it to vmagent. Do you think does it feasible? Are there any library to do so?

Also, is it okay to push metrics to VictoriaMetrics directly without vmagent?

Sorry for asking many questions, thanks.

[ledmonster]

I found scrapework in lib. I'll try following parser logic.

• https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/lib/promscrape/scrapework_test.go#L314

[hagen1778]

Hi @ledmonster!

Does vmagent support Pushgateway protocol now? Or do you have a plan to support it in near future?

Yes! Pls see for details https://github.com/VictoriaMetrics/VictoriaMetrics#how-to-import-data-in-prometheus-exposition-format

I'm considering that convert pushgateway protocol to Prometheus remote_write protocol and push it to vmagent. Do you think does it feasible? Are there any library to do so?

I'm sure that some users do so. However, Prometheus remote_write protocol is more complicated than influx protocol, for example. So it is totally up to you which one is more convenient to use.

Also, is it okay to push metrics to VictoriaMetrics directly without vmagent?

It is totally OK - vmagent and VM are using the same libraries for parsing metrics. So if your services can reach VM directly - you can configure them to write into VM. The use cases of vmagent are following:

• run vmagent for services in isolated environment when it is hard to allow access to the external VM installation for each of services;
• when you want to replicate metrics into multiple VM installations - vmagent can be configured with multiple remote-endpoints for pushing metrics;
• when you have unstable network connection - vmagent can buffer metrics which it can't send on disk, so when network recovers it can reread the buffers and send them without loss.
  Pls see more here https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmagent#use-cases

[ledmonster]

@hagen1778 Hi!! I was not aware that pushgateway protocol is defined as prometheus text-based format.

In my case, my services can reach VM directly, so I adopted direct access approach. I succeeded to convert text protocol to remote_write one. Also, I added additional labels ("instance" and "job"), because I want to make data pushgateway compatible. It would be nice if /api/v1/import/prometheus supports additional labels. Thank you!!

[hagen1778]

Nice job @ledmonster!

It would be nice if /api/v1/import/prometheus supports additional labels

Prometheus import format was designed to ingest data in the same format as prometheus exporters print it in plain/text. So all the labels presented in the imported data are respected. But this format is not for the Pushgateway substitution. The latter one was implemented to deal somehow with inability of Prometheus to accept writes, which is not the case for VM. However, this sounds like a feature request. Care to open one with more detailed description?

[ledmonster]

For me, remote_write based direct access to VM is working well, now. I had to fill timestamps to data after parsing prometheus text-based format, though.(Now, I want a kind of VM client library. 😄 )

Anyway, I'll create a feature request to support additional labels on api/v1/import/prometheus. Since pushgateway is supporting additional labels, user can easily switch to VM with this feature.

So, this issue can be closed, isn't it?

[valyala]

FYI, v1.40.1 release adds ability to add extra labels when importing data via /api/v1/import* handlers. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/719 for details.