Prometheus: rate()[1m] does not return any data

[krogon-dp]

What did you do?

1. Query single metric for last 10 minutes. Metric is coming from cadvisor (kubelet build-in)
   
   
   
   • used prometheus-query for CLI
   
   
2. Query the same metric using rate() function with range vector of 1m

What did you expect to see?
Rate of increase of the time series in the range vector.

What did you see instead? Under which circumstances?
No datapoints found.

Environment

Prometheus running on kubernetes 1.7.0 from helm chart (version stable/4.5.0).

• System information:

  Linux 4.4.65-k8s x86_64

• Prometheus version:

# prometheus -version
prometheus, version 1.7.1 (branch: master, revision: 3afb3fffa3a29c3de865e1172fb740442e9d0133)
  build user:       root@0aa1b7fc430d
  build date:       20170612-11:44:05
  go version:       go1.8.3

• Alertmanager version:

# alertmanager -version
alertmanager, version 0.8.0 (branch: HEAD, revision: 74e7e48d24bddd2e2a80c7840af9b2de271cc74c)
  build user:       root@439065dc2905
  build date:       20170720-14:14:06
  go version:       go1.8.3

• Prometheus configuration file:

- job_name: kubernetes-cadvisor
  scrape_interval: 1m
  scrape_timeout: 30s
  metrics_path: /metrics
  scheme: https
  kubernetes_sd_configs:
  - api_server: null
    role: node
    namespaces:
      names: []
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: false
  relabel_configs:
  - source_labels: []
    separator: ;
    regex: __meta_kubernetes_node_label_(.+)
    replacement: $1
    action: labelmap
  - source_labels: []
    separator: ;
    regex: (.*)
    target_label: __address__
    replacement: kubernetes.default.svc:443
    action: replace
  - source_labels: [__meta_kubernetes_node_name]
    separator: ;
    regex: (.+)
    target_label: __metrics_path__
    replacement: /api/v1/nodes/${1}:4194/proxy/metrics
    action: replace

• Additional information

# QUERY="container_cpu_usage_seconds_total{container_name='prometheus-server',cpu='cpu00'}"
# ./prometheus-query -query "$QUERY" -start "10 minute ago" -format csv | awk -F"," 'BEGIN {OFS=","} {if(NR==1){print "DATE_DELTA,DATE,VALUE"} if(NR>1){ if(ts==0){ts=$1} diff=$1-ts;ts=$1; $1=strftime("%Y-%m-%d %H:%M:%S", $1); last=$2; print diff, $1, $2}}' | head
DATE_DELTA,DATE,VALUE
0,2017-09-20 06:57:43,85348.758770
15,2017-09-20 06:57:58,85348.758770
15,2017-09-20 06:58:13,85348.758770
15,2017-09-20 06:58:28,85358.587517
15,2017-09-20 06:58:43,85358.587517
15,2017-09-20 06:58:58,85358.587517
15,2017-09-20 06:59:13,85358.587517
15,2017-09-20 06:59:28,85358.587517
15,2017-09-20 06:59:43,85358.587517

# QUERY="rate(container_cpu_usage_seconds_total{container_name='prometheus-server',cpu='cpu00'}[1m])"
# ./prometheus-query -query "$QUERY" -start "10 minute ago" -format csv
time,

[benclapp]

This looks more like a question rather than an issue, as this is how I'd expect Prometheus to behave.

Looks like you're trying to query with a rate of 1m, which is also your scrape interval. The rate and irate functions need at least two scrapes to return a result.

What it sounds like you want to use could be irate over a larger range. This would only use the two most recent scrapes. E.g.

irate(container_cpu_usage_seconds_total{container_name='prometheus-server',cpu='cpu00'}[5m])

[krogon-dp]

Scrape interval is 1m, that's true. However, first query shows that the metrics are every 15s, so rate[1m] should work.

[benclapp]

Had a look at prometheus-query. It looks like the default step value is 15 seconds, which is probably why you see a 15s step in the results you posted.

You'll still need to use rate() or irate() over a larger range to get the metric you're after :)

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.