Victoriametrics: histogram_quantile results sometimes doesn't work.

[InletOrder]

First is Prometheus result, and second one is VictoriaMetrics.

136 seems to be a similar issue.

Do you have any ideas?
The query is as follows.

histogram_quantile(0.5, sum(irate(nginx_http_request_duration_seconds_bucket{kubernetes_namespace=~"prd",host!="", kubernetes_pod_name=~".*prd-app-59748c9bdc-25zq5"}[2m])) by (app,le))

Thank you.

[valyala]

1. Which version of VictoriaMetrics do you use? It can be located at /metrics page via vm_app_version metric or it can be obtained via passing -version argument to VictoriaMetrics binary?

2. Could you attach Prometheus and VictoriaMetrics graphs for sum(irate(nginx_http_request_duration_seconds_bucket{kubernetes_namespace=~"prd",host!="", kubernetes_pod_name=~".*prd-app-59748c9bdc-25zq5"}[2m])) by (app,le) on the same time range?

3. Try substituting irate with rate in the query, since irate doesn't play well in this case - https://medium.com/@valyala/why-irate-from-prometheus-doesnt-capture-spikes-45f9896d7832

[InletOrder]

Thank you for the reply!

1. The VictoriaMetrics version is v1.25.2-cluster and it is running on Kubernetes.
   Prometheus version is v2.11.0.
   

2. First Prometheus, second VictoriaMetrics.

3. I substituted irate with rate, but still some data are dropped from the graph.
   First Prometheus, second VictoriaMetrics.
   The query is as follows.

histogram_quantile(0.5, histogram_quantile(0.5, sum(rate(nginx_http_request_duration_seconds_bucket{kubernetes_namespace=~"prd",host!="", kubernetes_pod_name=~".*prd-app-59748c9bdc-25zq5"}[2m])) by (app,le))

[valyala]

@ijij0516 , could you try building VictoriaMetrics from 1272e407b2f62dc4e7e15e3644b76c552eb6b16a (single-node version) or e9db22a551268cfbb2836946f7b756e66c35495e (cluster version) and verifying whether the issue is fixed?

[valyala]

@ijij0516 , the bugfix is available in v1.26.0. Could you verify it there?

[InletOrder]

@valyala I confirmed that the problem has been solved in v1.26.0. now.
Thank you!
Prometheus

VictoriaMetrics