Thanos: rate[5m] not showing downsampled data

[arnitolog]

Thanos, Prometheus and Golang version used
Thanos: v0.4.0
Prometheus: 2.9.2
no custom build

What happened
rate/irate functions are not working over the data returned from Thanos store

What you expected to happen
all functions work with any data
How to reproduce it (as minimally and precisely as possible):

Full logs to relevant components

Anything else we need to know

[arnitolog]

more info:
both functions work as expected in case if only raw data is used in the Thanos query. If I try to switch to Auto downsampling or Max 5m or Max1h the function is calculated for the data from the sidecar only.

[bwplotka]

Thanks for the report :+1:
Ok I think it's fair to scope this issue to "some function are not working with downsampling" right?

[bwplotka]

It might be tricky on how downsampled is done. To keep the math precision, we downsampled by reducing resolution and aggregating sumples in 5 different chunks per single raw chunk: https://github.com/improbable-eng/thanos/blob/da70cb0445b29dc614253e62ceb712243ecf718b/pkg/compact/downsample/downsample.go#L193

So those function are expected to work for sure: https://prometheus.io/docs/prometheus/latest/querying/functions/#aggregation_over_time

Rate/Irate however should work just fine as we keep count properly. However rate will work with at least 2 samples in it (: Since you ask for 5m and you use autodownsampling which is either raw, 5m or 1h resolution I am quite sure you don't have 2 samples within 5m, right?

[bwplotka]

This being said doing rate(.....[below 1h]) with auto downsampling does not make much sense. Wonder what would be the action item to help users here, other than better docs on it? (: @arnitolog

[arnitolog]

ok, make sense. But the same worked with the Thanos 0.3.2 and I'm curious has the calculation algorithm been changed

[bwplotka]

Nothing has changed AFAIK, so should be no difference, otherwise then blocks being now downsampled for you and previously you were hitting just raw data (:

[bwplotka]

We are working on adding min resolution https://github.com/improbable-eng/thanos/pull/1104 so you can specify to not query raw data (: Useful for debugging

[arnitolog]

Thanks a lot.

[bwplotka]

Just to sum up the rate[5m] will not work with lower resolutions (5m, 1h) as rate requires at least 2 samples (it is recommended to have at least 3) (:

So either something like rate[1h] or rate[$internval] has to be used in Grafana to have it working.

Also please read https://thanos.io/components/compact.md/#downsampling-resolution-and-retention about what downsampling is designed for. We are looking at ways to improve usage (: