Victoriametrics: Downsampling data

[Feudoor]

Is there ability to downsample data?

For example, I need to store raw metrics for 1 month, and metrics aggregated by 30 minutes for 1 year.

[valyala]

VictoriaMetrics doesn't provide automatic downsampling at the moment. But it may be implemented using the following approach:

• To run multiple VictoriaMetrics instances (or clusters) with distinct retentions, since each VictoriaMetrics instance works with a single retention.
• To periodically scrape the required downsampled data via /federate API from the instance with raw data and store it in the instance with higher retention.

We are planning to add recording rules to VictoriaMetrics with the ability to export the recorded data into external storage such as another VictoriaMetrics instance with higher retention.

side notes

Downsampling is usually used for two purposes:

• reducing query time over long time ranges
• reducing the required storage size

VictoriaMetrics is optimized for both cases

• it may quickly perform queries over long time ranges containing hundreds of millions of datapoints. See benchmark results for heavy queries in this article
• it compresses data much better comparing to competitors. See more info in this article

So VictoriaMetrics work quite good without the downsampling. An additional benefit is that you can drill down old data to small time ranges without precision loss.

[thulle]

I'm commenting part as a +1 on this being a requested feature, and part answering questions posted on reddit by @valyala about reasons to use VM.

To be able to downsample data like RRD in combination with the current storageefficiency of large amounts of timeseries, and being able to query this downsampled data transparently is imho. the featureset required for VM to become the obvious choice for devops/dashboards.
By transparently I mean that older data should be returned in lower resolution, without the querier having to make anything different. This would allow debugging issues using high resolution data as they happen and view trends over long time just by changing timerange in a dashboard, instead of having to modify queries & datasources in existing/pre-made dashboards or storing huge amounts of data.

[jujo1]

There appears to be limited updates on this topic. The need for downsampling (in particular OHLC bars) is paramount for use cases in financial price data. The objective is to store data long term at high frequency and most commonly use data at varying lower frequencies.

[valyala]

@jujo1 , did you try using rollup_candlestick(m[d]) on the raw data? This function returns four time series per each input time series - open, high, low and close. See https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/ExtendedPromQL for details.

Could you answer the following questions in order to understand better your use case:
1) What is the interval between data points in a single time series? Note that the minimal interval between data points supported by VictoriaMetrics is 1 millisecond.
2) How many unique time series are queried by a single query on average and on maximum?
3) What is the average and the maximum query interval? (hour, day, week, month, year, etc.)
4) Could you provide slow query for your case, so we could optimize it without resorting to down-sampling?

VictoriaMetrics is able to scan up to 50 millions of data points per second per CPU core, and the performance scales almost linearly with the number of CPU cores. For instance, 20 CPU cores can result in scan speed of up to 1 billion of data points per second.

Example calculations: if new points for each time series arrive every 100ms, then a single time series for one year would contain 10*3600*24*365=315M data points. This means that VictoriaMetrics could process up to 3 such time series per second on 20 CPU cores for year-long time range.

In the mean time you can create a custom script, which would periodically fetch down-sampled OHLC numbers from raw data and put them into a separate VictoriaMetrics instance for fast query processing on long time ranges.

[valyala]

FYI, it is possible to (ab)use deduplication for simple downsampling. For instance, if scrape interval for the ingested data is 15s, while -dedup.minScrapeInterval is set to 5m, then VictoriaMetrics will leave only a single sample per each 5m interval in the storage. See these docs for details.

[valyala]

FYI, vmalert supports recording rules starting from v1.37.0.

[bojleros]

Hi, Would you please give at least rough estimate on downsampling feature release? It is very nice that VM is capable of storing such huge amounts of series but we'd prefer to use our resources in more optimal way. In general old data is far less likely to be queried and even if this gets queried it is enough to have just 5m aggregates instead of full resolution. Its likely that current data is going to be querried quite often so downsampling means more resources for the most important data -> more efficiency.

[valyala]

Would you please give at least rough estimate on downsampling feature release?

We already designed draft architecture for multi-level downsampling and are going to add it to VictoriaMetrics during the next 6 months.

[bojleros]

@valyala What is the overall plan? Are we going to be able to specify additional retention thresholds so datapoints will be downsampled as soon as their age breaches the thresholds? Would it be possible in simple single node deployments?

[raags]

Does downsampling here mean an aggregation across the sampled time range? How will the new data point be calculated?

For e.g. in graphite, every metric can be configured with an aggregation function (min, max, avg, sum, etc) that is applied to the sample period. i.e. if 1 min goes down to 1 hour, for http_requests_total it should sum(1h), while for http_requets_duration it should avg(1h).

[valyala]

What is the overall plan? Are we going to be able to specify additional retention thresholds so datapoints will be downsampled as soon as their age breaches the thresholds? Would it be possible in simple single node deployments?

Draft config will look like the following:

-downsample 1d:1m,7d:5m,30d:1h

This means that downsampling is applied in the following way:

• Samples with timestamps closer than 1d (one day) to the current time aren't downsampled
• Samples with timestamps from 1d to 7d to the current time are downsampled with 1m (one minute) interval
• Samples with timestamps from 7d to 30d to the current time are downsampled with 5m interval
• Older samples are downsampled with 1h interval.

The functionality will be available in both single-node and cluster versions of VictoriaMetrics.

Does downsampling here mean an aggregation across the sampled time range? How will the new data point be calculated?

The downsampling means leaving the first point on the configured interval. There is no any aggregation.

in graphite, every metric can be configured with an aggregation function (min, max, avg, sum, etc) that is applied to the sample period. i.e. if 1 min goes down to 1 hour, for http_requests_total it should sum(1h), while for http_requets_duration it should avg(1h).

This type of downsampling is easy to misconfigure and is hard to reason about, so we decided postponing its implementation after simple downsampling mentioned above will be implemented.

[hekmon]

That is great news !

So it seams it is a kind of dynamic deduplication windows ? How dedup will work once downsampling is here ? Will it be merged with DS as DS can be configured to act as dedup ?

[valyala]

How dedup will work once downsampling is here ? Will it be merged with DS as DS can be configured to act as dedup ?

The -dedup.minScrapeInterval command-line flag will be left for backwards compatibility, while simple downsampling will re-use the deduplication code.