Vector: Add support for Prometheus remote_write sink

[valyala]

It would be great if vector cound route metrics to Prometheus-compatible remote storage over remote_write API. This will automatically enable data writing to any remote storage from the following list - https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage .

[langecode]

Guess the opposite could also be interesting? Having a source able to consume data form Prometheus remote_write.

[langecode]

Just one more thought on the current Prometheus support. The current support for Prometheus metrics is based on sampling. The source mimics the Prometheus scrape and the sink exposes a /metrics endpoint. This inevitably introduces the risk of loss of precision along the way. Supporting the remote write protocol would allow for treating the metrics events as events with no aggregation taking place in Vector.

Thus we could run Vector inside of a Kubernetes cluster, receive the metrics from a Prometheus instance in-cluster as endpoints are scraped, use Vector to Vector as a transport pipe (along with our logs) and pass the metric events to some TSDB capable of receiving data through the remote write protocol, e.g., VictoriaMetrics for further processing, analytics, and long time storage.

[MOZGIII]

The remote_write is on my radar, something that I wanted to look into, alongside the integrations with Thanos and Cortex - maybe we could write directly to them via the remote_write API.

[langecode]

Indeed as @valyala mention there is a number of TSDB backends supporting remote_write so you would get integrations to those "for free". There is a protobuf definition for the data format. Without at all being expert on rust but having looked a bit at the Vector source code, my guess is that it would be pretty similar to the vector source/sink pairs.