Prometheus: avg_over_time() anomaly when input disappears

[belm0]

What did you do?

avg_over_time() for [1h] on a metric with values in the set {0, 1} and which eventually disappears

What did you expect to see?

output series should end at value 0, one hour after input disappears

What did you see instead? Under which circumstances?

output series has an anomaly, ramping up after input disappears and ending at the value 1

I noticed that the sum_over_time() output is fine, so I worked around this problem with sum_over_time() ÷ duration.

This graph shows the input, avg_over_time() output, and sum_over_time() ÷ duration output (with a slight Y offset for visibility).

Environment

• Prometheus version: 2.9.2

[roidelapluie]

Thanks for your report. It looks as if this is actually a question about usage and not development.

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

In your case, avg_over_time will only use the last datapoints at the end when the output disappears, which is expected.

[roidelapluie]

sum_over_time() / duration is not equivalent because you might not have datapoints in the whole duration.

Also, a value which disappears is not the same as 0 for Prometheus.

[belm0]

I see-- avg_over_time() is only counting time with datapoints. So as the majority of the time window covers undefined data, the output converges to the last datapoint.