Victoriametrics: Does vm support query.staleness-delta

[flashmouse]

Hi we recently try move data from influx to vm. it's a big performance improve after we do some test. thx!

these day we write the same data both to influxdb and vm,and find some difference makes us confused, so we hope to get some hint.

1.we have 1 mertic write point every 30s, and when we query from influxdb(group by(30s)) and vm (step=30s),we get 2 same time-series line.but when we query by 1m, 2lines are not exactly the same, whether 2 db has different semantic when they do aggreate?

2.we have some metrics that write every 30s, but when value is 0 we won't write any point, we see the line of influx is broken. although vm is still a line, obviously it do something like influx's fill(line) but what we need is fill(0).
i think it's may prometheus --query.staleness-delta config? does vm have any same configuration?

thx..

[valyala]

1. VictoriaMetrics follows Prometheus semantics when queried via /api/v1/query and /api/v1/query_range (aka Prometheus query API). The semantics is documented in the first paragraph here:

When queries are run, timestamps at which to sample data are selected independently of the actual present time series data. This is mainly to support cases like aggregation (sum, avg, and so on), where multiple aggregated time series do not exactly align in time. Because of their independence, Prometheus needs to assign a value at those timestamps for each relevant time series. It does so by simply taking the newest sample before this timestamp.

1. VictoriaMetrics fills gaps with the size up to 2x interval between data points for the given time series, while Prometheus fills gaps up to --query.staleness-delta or --query.lookback-delta (default value is 5 minutes). VictoriaMetrics' behavior is better suited for too short and too long scrape intervals, while Prometheus behavior works the best for medium-sized scrape intervals in the range [10s ... 5m]. Prometheus behavior is documented here:

If no sample is found (by default) 5 minutes before a sampling timestamp, no value is returned for that time series at this point in time. This effectively means that time series "disappear" from graphs at times where their latest collected sample is older than 5 minutes or after they are marked stale.

The next VictoriaMetrics release will provide -search.maxLookback command-line flag, which can be used for enabling Prometheus-compatible behavior for filling the gaps. It has the same meaning as -query.lookback-delta command-line flag in Prometheus. It will also accept max_lookback query arg for /api/v1/query and /api/v1/query_range handlers.

[valyala]

@flashmouse , you can build VictoriaMetrics from the following commits and try -search.maxLookback command-line flag or max_lookback query arg there:

• single-node version: c66a691593928ed0d6b5276e54ccddc55bce622d (build instructions).
• cluster version 99786c2864e32010b6d5ae3155aad4c8949bdc1e (build instructions).

[flashmouse]

thx @valyala, i compile new code , add max_lookback=10s and run it, seems new behavior meet our require. i'll keep running and check whether has any other problem. thx again guy!

[valyala]

The -search.maxLookback flag is available starting from v1.28.1.