Kube-state-metrics: kube_pod_completion_time metrics not exported

[panhaitao]

What happened:
kube_pod_completion_time metirc can not get,Is this a BUG REPORT or FEATURE REQUEST?

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

1. install kube-state-metrics from https://github.com/kubernetes/kube-state-metrics/tree/master/examples/standard.
2. when Run command: curl http://10.10.15.26:8080/metrics | grep kube_pod_completion_time only return this result:

# HELP kube_pod_completion_time Completion time in unix timestamp for a pod.
# TYPE kube_pod_completion_time gauge

Environment:

• kube-state-metrics:v2.0.0-alpha.1
• Kubernetes version v1.17.4
• node os CentOS 7.6

[lilic]

@panhaitao thanks for the bug report! Do you have any terminated pods and the timestamp is present on them and they are being failed to be produced into metrics? Do you have a minimal example to reproduce? Thanks!

[yashvardhan-kukreja]

@panhaitao I tried to reproduce this issue by installing kube-state-metrics using helm.
But it worked perfectly fine for me.
I'd suggest you to install kube-state-metrics via helm (ideally it shouldn't differ) and see if the kube_pod_completion_time metrics for you are still unavailable.

This is the ouput I received

# HELP kube_pod_completion_time Completion time in unix timestamp for a pod.
# TYPE kube_pod_completion_time gauge
kube_pod_completion_time{namespace="kuberhealthy",pod="dns-status-internal-1611318555"} 1.611318576e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="dns-status-internal-1611320953"} 1.611320964e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="dns-status-internal-1611274355"} 1.611289682e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="dns-status-internal-1611355647"} 1.611355658e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="daemonset-1611322093"} 1.61132213e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="dns-status-internal-1611266435"} 1.611266449e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="dns-status-internal-1611259115"} 1.61125913e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="deployment-1611325544"} 1.611325717e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="dns-status-internal-1611262355"} 1.611262364e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="dns-status-internal-1611268235"} 1.611268244e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="daemonset-1611293293"} 1.61129335e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="dns-status-internal-1611245915"} 1.611245927e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="deployment-1611357568"} 1.611357686e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="daemonset-1611356968"} 1.611357017e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="deployment-1611318193"} 1.611318332e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="dns-status-internal-1611203439"} 1.611203498e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="dns-status-internal-1611353608"} 1.611353616e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="dns-status-internal-1611231355"} 1.611231366e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="dns-status-internal-1611229315"} 1.611229327e+09

[lilic]

@yashvardhan-kukreja hello, installing via helm vs regular manifests should not make a difference here. I have no issues and I do not use helm. :) But thank you for confirming!

[panhaitao]

@panhaitao I tried to reproduce this issue by installing kube-state-metrics using helm.
But it worked perfectly fine for me.
I'd suggest you to install kube-state-metrics via helm (ideally it shouldn't differ) and see if the kube_pod_completion_time metrics for you are still unavailable.

This is the ouput I received

# HELP kube_pod_completion_time Completion time in unix timestamp for a pod.
# TYPE kube_pod_completion_time gauge
kube_pod_completion_time{namespace="kuberhealthy",pod="dns-status-internal-1611318555"} 1.611318576e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="dns-status-internal-1611320953"} 1.611320964e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="dns-status-internal-1611274355"} 1.611289682e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="dns-status-internal-1611355647"} 1.611355658e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="daemonset-1611322093"} 1.61132213e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="dns-status-internal-1611266435"} 1.611266449e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="dns-status-internal-1611259115"} 1.61125913e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="deployment-1611325544"} 1.611325717e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="dns-status-internal-1611262355"} 1.611262364e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="dns-status-internal-1611268235"} 1.611268244e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="daemonset-1611293293"} 1.61129335e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="dns-status-internal-1611245915"} 1.611245927e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="deployment-1611357568"} 1.611357686e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="daemonset-1611356968"} 1.611357017e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="deployment-1611318193"} 1.611318332e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="dns-status-internal-1611203439"} 1.611203498e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="dns-status-internal-1611353608"} 1.611353616e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="dns-status-internal-1611231355"} 1.611231366e+09
kube_pod_completion_time{namespace="kuberhealthy",pod="dns-status-internal-1611229315"} 1.611229327e+09

Thank you for you replay! could show me the k8s cluster version ?
I try many times in last year Oct，still get the empty value, I will try again in my cluster at this weekend，may be a The wrong usage for myself～

[yashvardhan-kukreja]

@panhaitao I tested this on a toy cluster I set up on KinD. It was using v1.19.1 of kubernetes

[lilic]

I also see the metrics on my OpenShift cluster using an older Kubernetes version and kube-state-metrics 1.9.7. But the difference here @yashvardhan-kukreja might be that the reporter is using a newer version of kube-state-metrics, have you tried to reproduce with that one?

@panhaitao can you see other metrics? And can you confirm if there are any terminated pods on your cluster? Do you mind also using the latest beta release, thank you again for the report!

[panhaitao]

Thanks very much for erveyone !!! @lilic @yashvardhan-kukreja @gnufied @ncdc

I retry it in k8s cluster with kube-state-metrics:v2.0.0-beta, I can get the kube_pod_completion_time Now ,

ENV:
kube-state-metrics:v2.0.0-beta
Kubernetes version v1.17.4
node os CentOS 7.6

Result:

[root@10-10-69-26 Kube-state-metrics-standard]# curl http://172.17.12.2:8080/metrics | grep kube_pod_completion

TYPE kube_pod_completion_time gauge

kube_pod_completion_time{namespace="cattle-logging-system",pod="rancher-logging-fluentd-configcheck-55d9abff"} 1.612452211e+09

[lilic]

Yay happy to hear there is no bug in latest version, thanks for your report! Closing issue, if you have any other ones in the future please open a new one. 🎉