Kube-state-metrics: kube_pod_status_ready includes completed pods

[smarterclayton]

The metrics for kube_pod_status_ready are probably less useful than they can be because they include completed pods, which are dead. So all completed pods are always "unready", but to an end user looking at them to determine health of the cluster, they're meaningless.

Completed pods are never going to be ready, but when I'm trying to figure out whether a cluster upgrade caused outage on running pods, I want to measure the number of unready pods at the current moment. I think on the whole, conditions don't apply to completed pods (for the most part).

Conditions on a completed pod:

  - lastProbeTime: null
    lastTransitionTime: 2017-08-23T22:10:16Z
    reason: PodCompleted
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: 2017-08-23T22:11:29Z
    reason: PodCompleted
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: 2017-08-23T22:10:16Z
    status: "True"
    type: PodScheduled

Scheduled might be a counterexample (there are other ones too).

Is the point of a ksm metric to report what the API reports, or to provide operational value? If operational value, I'd argue that we should filter ready for completed pods.

[simonswine]

I think there are ways to do that by looking at two metrics. I am currently filtering jobs like that in my alerting rule:

kube_pod_info{created_by_kind!="Job"} AND ON (pod, namespace) kube_pod_status_ready{condition="true"} == 0

Not too sure, but I think it would be better to leave that filtering to prometheus/user, but I think a couple of exmaple rules would be great

[brancz]

kube-state-metrics is meant to mirror the kubernetes api objects as Prometheus metrics, I don't think we should do any type of filtering that we wouldn't naturally get out of the api objects. Combining two metrics as @simonswine mentioned could be a solution.

[smarterclayton]

It's making me consider removing the condition from completed pods in kube itself. It's fairly painful for conditions on pods to have to filter out the long tail of done pods (most large clusters are only about 1/3rd active pods). But that's a sufficient workaround for now.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[yuliyantsvetkov]

I think there are ways to do that by looking at two metrics. I am currently filtering jobs like that in my alerting rule:

kube_pod_info{created_by_kind!="Job"} AND ON (pod, namespace) kube_pod_status_ready{condition="true"} == 0

Not too sure, but I think it would be better to leave that filtering to prometheus/user, but I think a couple of exmaple rules would be great

This actually did the job for me to look for terminated pods, but reversed order was used here:
kube_pod_info{created_by_kind!="Job"} AND ON (pod, namespace) kube_pod_status_ready{condition="false"} == 1

[yuliyantsvetkov]

Here is bit more refined query built from the one that @simonswine shared:

sum by(namespace)( kube_pod_info{created_by_kind!="Job"} AND ON (pod, namespace) kube_pod_status_ready{condition="false"} == 1)

[dimm0]

Still no real fix for this?
Doesn't make much sense to include the metrics of completed pods into any stats...

[nid06]

I am unable to configure readiness alert . Has anybody got solution yet ?
@yuliyantsvetkov couldn't find field created_by_kind in kube_pod_info

[yuliyantsvetkov]

@nid06 well that works for me with the prometheus-operator.