Victoriametrics: Roadmap

[tenmozes]

• [x] Prometheus-compatible scraper. See vmagent
• [x] Prometheus-compatible relabeling proxy. See vmagent
• [x] Data replication and filtering proxy. See vmagent
• [ ] Support of Object Storages (GCS, S3, Azure Storage) #38
  
  
  
  • [x] Backup to s3, gcp. See vmbackup.
  
  
  • [ ] Stateless Storage
  
  
• [x] Replication #118
• [ ] Data downsampling #36
• [x] Alert Manager Integration #119 See vmalert.
• [x] CLI tool for data migration, re-balancing and adding/removing nodes #103
  
  
  
  • [x] From influx to VM
  
  
  • [x] From prometheus snapshots to VM
  
  
  • [x] From vm single to VM cluster
  
  

Feel free to comment any item or add own one

[khansen1960]

Is there any estimation of when the data downsampling feature might be implemented?

[valyala]

There are no estimations for downsampling feature yet. This feature has lower priority comparing to object storage and replication. See https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/README.md#downsampling for the reasons behind low priority for downsampling feature.

@khansen1960 , do you have practical use case where downsampling would improve VictoriaMetrics performance or significantly reduce disk space usage? This could raise the priority for this feature.

[khansen1960]

Aliaksandr

Thank you for the quick response.

We are using Victoriametrics v1.18.1 for our Prometheus platforms data storage. We have multiple Prometheus hosts doing remote writes to Victoriametrics and we query Victoriametrics from Grafana for our dashboards. We are currently monitoring 760 hosts, which is 1/5th of the total hosts in network. We currently have 4 months of data and the disk usage for the victoriametrics/data/big directory is 143GB. We want to retain 1 year of historical data. If I estimate this out to 1 year we have 429GB disk space with current load and if we add our remaining hosts we come out to 2.145TB. Recently we were asked to also collect mysql and Cassandra metrics, and the addition of these will increase the usage even more.
Ideally, we would like full metrics for 1 months and downsampled metrics for the remaining 11 months.

Keith Hansen | Network Operations Center | [24]7.ai | [email protected][e-mail] | NOC: [phone] | www.247.aihttp://www.247.ai

From: Aliaksandr Valialkin [e-mail]
Sent: Tuesday, November 26, 2019 12:45 PM
To: VictoriaMetrics/VictoriaMetrics [e-mail]
Cc: Keith Hansen [e-mail]; Mention [e-mail]
Subject: Re: [VictoriaMetrics/VictoriaMetrics] Roadmap (#129)

[24]7.ai Security alert : External email; unless trust the sender and/or expect this email, do not open or click links.

There are no estimations for downsampling feature yet. This feature has lower priority comparing to object storage and replication. See https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/README.md#downsamplinghttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_VictoriaMetrics_VictoriaMetrics_blob_master_README.md-23downsampling&d=DwMCaQ&c=-MJkLfNHw3fLcSxc54P8Yg&r=SNrfT4HUuz-QfwV4K1_U0L3N_jVUlCy2gsVLzQcEUrY&m=CmKvukhBHc7aOM9w77Ugc7q5lL9Hv1o0kXH170aT_rE&s=MHgXahlNqoovI1COF3A2M-Am5tPwI8x6gsWBV7Eyv5s&e= for the reasons behind low priority for downsampling feature.

@khansen1960https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_khansen1960&d=DwMCaQ&c=-MJkLfNHw3fLcSxc54P8Yg&r=SNrfT4HUuz-QfwV4K1_U0L3N_jVUlCy2gsVLzQcEUrY&m=CmKvukhBHc7aOM9w77Ugc7q5lL9Hv1o0kXH170aT_rE&s=Db3yfFHlEMqAupfd5iU-qte7E6Zfomnj7CfMxJfEFeI&e= , do you have practical use case where downsampling would improve VictoriaMetrics performance or significantly reduce disk space usage?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_VictoriaMetrics_VictoriaMetrics_issues_129-3Femail-5Fsource-3Dnotifications-26email-5Ftoken-3DAMES4JBDZPWT5NULOVBIJ43QVWDGNA5CNFSM4IHJ4WB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFHMFZA-23issuecomment-2D558809828&d=DwMCaQ&c=-MJkLfNHw3fLcSxc54P8Yg&r=SNrfT4HUuz-QfwV4K1_U0L3N_jVUlCy2gsVLzQcEUrY&m=CmKvukhBHc7aOM9w77Ugc7q5lL9Hv1o0kXH170aT_rE&s=7OtQJxgrCHgI6QUBqX53PkMPrse7xiaUZ0YYcPSWCXQ&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AMES4JDFZ6C7O4HHCC6MTFTQVWDGNANCNFSM4IHJ4WBQ&d=DwMCaQ&c=-MJkLfNHw3fLcSxc54P8Yg&r=SNrfT4HUuz-QfwV4K1_U0L3N_jVUlCy2gsVLzQcEUrY&m=CmKvukhBHc7aOM9w77Ugc7q5lL9Hv1o0kXH170aT_rE&s=yPViNLKZ3aBsqB0L9szOO3xsrtTof5QRN1x7Sap5-uY&e=.

[mxdec]

Since Prometheus 2.13.0 has streaming remote_read API, it would be very nice to have the same thing in VictoriaMetrics. Is it planned on the roadmap ?

[valyala]

@khansen1960 , thanks for providing the detailed use case. There are plans to add tiered downsampling for this use case, but there is no hard deadline yet :(

In the mean time you can use the following setup:

To set up two VictoriaMetrics instances:

• The first one would store raw data with -retentionPeriod=1, i.e. for one month.
• The second one would store downsampled data with -retentionPeriod=12.

You need to set up an additional Prometheus, which would scrape /federate endpoint from the first VictoriaMetrics instance with scrape_interval suitable for the downsampled data, and replicate the scraped data into the second VictoriaMetrics instance.

Then you can put Promxy in front of two VictoriaMetrics instances in order to get transparent query view over all the 12 months.

[valyala]

Since Prometheus 2.13.0 has streaming remote_read API, it would be very nice to have the same thing in VictoriaMetrics. Is it planned on the roadmap ?

VictoriaMetrics supports PromQL via Prometheus querying API, so it can be used as drop-in replacement for Prometheus in Grafana.

There are no plans for implementing Prometheus remote_read API in VictoriaMetrics, because this API is broken even after adding streaming support:

• Every Prometheus instance expects the remote_read API endpoint returns only the data written by the given Pormetheus instance via remote_write API. It may return garbage response if remote_read API endpoint returns data written to remote storage by other Prometheus instances. See this issue for details.
• Prometheus expects receiving all the samples (aka raw data) from remote storage via remote_read API, which are required by the incoming PromQL query. The amount of samples can be quite large if the query touches big number of time series over big time ranges. This means big amounts of data must be transferred from remote storage to Prometheus on each query. This increases query latency and may occupy all the network bandwidth between remote storage and Prometheus for extended periods of time. This also can cost a lot of money if network bandwidth isn't free.

There is a hint field in remote_read request, which is aimed towards performing data aggregation on remote storage side. Unfortunately, it is unclear how to use the hint without breaking the resulting responses.

Given these issues, any remote storage solution that claims Prometheus remote_read API support, shouldn't be used in production.

If you need reading raw data from VictoriaMetrics, then take a look at /api/v1/export API.

[valyala]

FYI, we released vmagent, which can perform the following tasks:

• Scrape Prometheus-compatible targets.
• Add, remove and rename metrics and labels.
• Filter incoming metrics.
• Replicate incoming metrics into multiple destination services.
• Buffer collected metrics on disk until connection to destination service becomes available.

[valyala]

FYI, cluster version of VictoriaMetrics supports application-level replication starting from v1.36.0. See more details about the replication here.

Additionally, we recently released vmalert - a tool for alerting, which is backwards-compatible with alerting configs for Prometheus.