Kube-state-metrics: How to effectively query job count and success rate?

[zhengl7]

In my case I guess it's because a job's metadata is in k8s for about 2 days, so a job's metric (like kube_job_created, kube_job_status_succeeded) in prometheus will last that long (because kube-state-metrics just simply keep polling k8s data right?). Thus I'm struggling to find an effective way to query things like,

1. In one day, how many job run and how many failed?
2. Job duration histogram or as a filterable time-series

I know this is probably just the way kube-state-metrics work, but curious if anyone know how I can better utilize the current gauge metrics. Either a Prometheus or Grafana query would be great, thanks!

[brancz]

You basically want an integral over the gauge graph. You can somewhat simulate this with sum_over_time or average_over_time and a recording rule.

[zhengl7]

Thanks for the answer, I tried to play with those functions before but didn't find an effective way. The main problem is a gauge metric lasts too long (3 days) while only the first 5 minutes of it is meaningful (a job's duration).

I did find a better query using the timestamp metric of jobs to filter out the most recent jobs using a window, something like this,

count(time() - kube_job_status_completion_time < 3600 and kube_job_status_succeeded == 1)