Kube-state-metrics: kube_deployment_status_.* wrong

Created on 24 Apr 2019  路  13Comments  路  Source: kubernetes/kube-state-metrics

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:
Kube-state-metrics reports that the kube-state-metrics deployment has no available replicas.
More info:

How many pods should the deployment have:

query: kube_deployment_spec_replicas{namespace="kube-prometheus", deployment="kube-state-metrics"}
result: kube_deployment_spec_replicas{deployment="kube-state-metrics",instance="10.244.9.136:8443",job="kube-state-metrics",namespace="kube-prometheus",scrape_endpoint="main"} 1

How many pods have deployment:

query: kube_deployment_status_replicas{namespace="kube-prometheus", deployment="kube-state-metrics"}
result: kube_deployment_status_replicas{deployment="kube-state-metrics",instance="10.244.9.136:8443",job="kube-state-metrics",namespace="kube-prometheus",scrape_endpoint="main"}   1

Now let's see the deployment status available / unavailable:

query: kube_deployment_status_replicas_available{namespace="kube-prometheus", deployment="kube-state-metrics"}
result: kube_deployment_status_replicas_available{deployment="kube-state-metrics",instance="10.244.9.136:8443",job="kube-state-metrics",namespace="kube-prometheus",scrape_endpoint="main"} 0

query: kube_deployment_status_replicas_unavailable{namespace="kube-prometheus", deployment="kube-state-metrics"}
result: kube_deployment_status_replicas_unavailable{deployment="kube-state-metrics",instance="10.244.9.136:8443",job="kube-state-metrics",namespace="kube-prometheus",scrape_endpoint="main"}   1

Let's look at the state in Kubernetes:

# kubectl -n kube-prometheus get deploy kube-state-metrics
NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
kube-state-metrics   1         1         1            1           42d


# kubectl -n kube-prometheus get deploy kube-state-metrics -o json | jq '.status'
{
  "availableReplicas": 1,
  "conditions": [
    {
      "lastTransitionTime": "2019-03-12T14:57:47Z",
      "lastUpdateTime": "2019-03-12T14:57:47Z",
      "message": "Deployment has minimum availability.",
      "reason": "MinimumReplicasAvailable",
      "status": "True",
      "type": "Available"
    },
    {
      "lastTransitionTime": "2019-03-12T14:57:47Z",
      "lastUpdateTime": "2019-04-23T11:09:40Z",
      "message": "ReplicaSet \"kube-state-metrics-f75d74d69\" has successfully progressed.",
      "reason": "NewReplicaSetAvailable",
      "status": "True",
      "type": "Progressing"
    }
  ],
  "observedGeneration": 9,
  "readyReplicas": 1,
  "replicas": 1,
  "updatedReplicas": 1
}


# kubectl -n kube-prometheus get rs kube-state-metrics-f75d74d69 -o json | jq .status
{
  "availableReplicas": 1,
  "fullyLabeledReplicas": 1,
  "observedGeneration": 7,
  "readyReplicas": 1,
  "replicas": 1
}


# kubectl -n kube-prometheus get pod kube-state-metrics-f75d74d69-nr4vw -o json | jq .status
{
  "conditions": [
    {
      "lastProbeTime": null,
      "lastTransitionTime": "2019-04-23T20:16:38Z",
      "status": "True",
      "type": "Initialized"
    },
    {
      "lastProbeTime": null,
      "lastTransitionTime": "2019-04-23T20:16:43Z",
      "status": "True",
      "type": "Ready"
    },
    {
      "lastProbeTime": null,
      "lastTransitionTime": null,
      "status": "True",
      "type": "ContainersReady"
    },
    {
      "lastProbeTime": null,
      "lastTransitionTime": "2019-04-23T20:16:38Z",
      "status": "True",
      "type": "PodScheduled"
    }
  ],
  "containerStatuses": [
    {
      "containerID": "docker://cf2a58d3b802fd4753a95f42a2c8ba3737d62368f68d3364152ac680bb784f14",
      "image": "gcr.io/google_containers/addon-resizer:1.8.3",
      "imageID": "docker-pullable://gcr.io/google_containers/addon-resizer@sha256:07353f7b26327f0d933515a22b1de587b040d3d85c464ea299c1b9f242529326",
      "lastState": {},
      "name": "addon-resizer",
      "ready": true,
      "restartCount": 0,
      "state": {
        "running": {
          "startedAt": "2019-04-23T20:16:42Z"
        }
      }
    },
    {
      "containerID": "docker://ab8320ecaa7021075c2f9ee145ae164e1aeef04d5644a34edf98200994eca816",
      "image": "registry.flant.com/sys/antiopa/prometheus/kube-state-metrics:18a3e2bd7464a2fcfd3f33fe44d612a00a9ee9ee",
      "imageID": "docker-pullable://registry.example.com/sys/addon-operator/prometheus/kube-state-metrics@sha256:f21cecfa64be946a0212a9d3249b030d73875c387d266580499d9844db011f80",
      "lastState": {},
      "name": "kube-state-metrics",
      "ready": true,
      "restartCount": 0,
      "state": {
        "running": {
          "startedAt": "2019-04-23T20:16:42Z"
        }
      }
    }
  ],
  "hostIP": "192.168.203.23",
  "phase": "Running",
  "podIP": "10.244.9.136",
  "qosClass": "Burstable",
  "startTime": "2019-04-23T20:16:38Z"
}

More metrics:

query: kube_replicaset_status_replicas{namespace="kube-prometheus",replicaset=~"kube-state-metrics.*"}
result:
kube_replicaset_status_replicas{instance="10.244.9.137:8443",job="kube-state-metrics",namespace="kube-prometheus",replicaset="kube-state-metrics-86b66c656c",scrape_endpoint="main"}    0
kube_replicaset_status_replicas{instance="10.244.9.137:8443",job="kube-state-metrics",namespace="kube-prometheus",replicaset="kube-state-metrics-b44968b47",scrape_endpoint="main"} 0
kube_replicaset_status_replicas{instance="10.244.9.137:8443",job="kube-state-metrics",namespace="kube-prometheus",replicaset="kube-state-metrics-f75d74d69",scrape_endpoint="main"} 1

query: kube_pod_status_ready{namespace="kube-prometheus", pod=~"kube-state-metrics.*"}
result:
kube_pod_status_ready{condition="false",instance="10.244.9.137:8443",job="kube-state-metrics",namespace="kube-prometheus",pod="kube-state-metrics-f75d74d69-28rr4",scrape_endpoint="main"}  0
kube_pod_status_ready{condition="true",instance="10.244.9.137:8443",job="kube-state-metrics",namespace="kube-prometheus",pod="kube-state-metrics-f75d74d69-28rr4",scrape_endpoint="main"}   1
kube_pod_status_ready{condition="unknown",instance="10.244.9.137:8443",job="kube-state-metrics",namespace="kube-prometheus",pod="kube-state-metrics-f75d74d69-28rr4",scrape_endpoint="main"}    0

What you expected to happen:
I expected the kube_deployment_status_replicas_available and kube_deployment_status_replicas_unavailable metrics to be correct information

How to reproduce it (as minimally and precisely as possible):

This does not always happen. Restarting the kube-state-metrics hearth does not solve the problem.

Anything else we need to know?:
-

Environment:

  • Kubernetes version (use kubectl version):
# kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.8", GitCommit:"4e209c9383fa00631d124c8adcc011d617339b3c", GitTreeState:"clean", BuildDate:"2019-02-28T18:49:34Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.6", GitCommit:"b1d75deca493a24a2f87eb1efde1a569e52fc8d9", GitTreeState:"clean", BuildDate:"2018-12-16T04:30:10Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
  • Kube-state-metrics image version
version.Version{GitCommit:"", BuildDate:"2019-04-19T07:57:26Z", Release:"v1.6.0-rc.0", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
kinbug
Source
picture qw1mb0

All 13 comments

We have seen this problem on 10 clusters out of 80. We have a lot of alerts based on these metrics.

qw1mb0 picture qw1mb0 on 24 Apr 2019

Does this eventually solve itself or once it's in this state it always stays like that? Could you try giving kube-state-metrics more memory and cpu? This could tell us if the internal queues might be worked off faster as in not being worked off fast enough now.

brancz picture brancz on 24 Apr 2019

We will conduct some tests and report the results next week. Thanks for the idea.

qw1mb0 picture qw1mb0 on 24 Apr 2019
👍1

@qw1mb0 thanks! I am curious what is happening here.

mxinden picture mxinden on 26 Apr 2019

We continue to experience problems with incorrect data in kube-state-metrics

/ # curl -s localhost:8081/metrics | grep  site-feed-generator | grep deployment
kube_deployment_created{namespace="site-production",deployment="site-feed-generator"} 1.554587198e+09
kube_deployment_status_replicas{namespace="site-production",deployment="site-feed-generator"} 1
kube_deployment_status_replicas_available{namespace="site-production",deployment="site-feed-generator"} 0
kube_deployment_status_replicas_unavailable{namespace="site-production",deployment="site-feed-generator"} 1
kube_deployment_status_replicas_updated{namespace="site-production",deployment="site-feed-generator"} 1
kube_deployment_status_observed_generation{namespace="site-production",deployment="site-feed-generator"} 140
kube_deployment_spec_replicas{namespace="site-production",deployment="site-feed-generator"} 1
kube_deployment_spec_paused{namespace="site-production",deployment="site-feed-generator"} 0
kube_deployment_spec_strategy_rollingupdate_max_unavailable{namespace="site-production",deployment="site-feed-generator"} 1
kube_deployment_spec_strategy_rollingupdate_max_surge{namespace="site-production",deployment="site-feed-generator"} 1
kube_deployment_metadata_generation{namespace="site-production",deployment="site-feed-generator"} 140
kube_deployment_labels{namespace="site-production",deployment="site-feed-generator",label_service="site-feed-generator"} 1

The real state of the deployment:

# kubectl -n site-production get deploy site-feed-generator 
NAME                  DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
site-feed-generator   1         1         1            1           23d
# kubectl -n site-production get pod site-feed-generator-69684c69bb-sl49b 
NAME                                   READY     STATUS    RESTARTS   AGE
site-feed-generator-69684c69bb-sl49b   1/1       Running   0          8h

Container allocated so many resources resources:

      limits:
        cpu: 620m
        memory: 990Mi
      requests:
        cpu: 620m
        memory: 990Mi

via addon-resizer:

    - /pod_nanny
    - --config-dir=/etc/config
    - --container=kube-state-metrics
    - --cpu=200m
    - --extra-cpu=15m
    - --memory=150Mi
    - --extra-memory=30Mi
    - --threshold=5
    - --deployment=kube-state-metrics

Real CPU consumption:
http://screenshot.flant.ru/a.sidorov/91/65/9165b7e8adc7d69a676ffaf6ee9446386e4384d1.png

Real RAM consumption: http://screenshot.flant.ru/a.sidorov/dc/d0/dcd0bd89bf0021a89e0f33dba1ed069cc84948e1.png

After restarting kube-state-metrics everything is correct:

/ # curl -s localhost:8081/metrics | grep  site-feed-generator | grep deployment
kube_deployment_created{namespace="site-production",deployment="site-feed-generator"} 1.554587198e+09
kube_deployment_status_replicas{namespace="site-production",deployment="site-feed-generator"} 1
kube_deployment_status_replicas_available{namespace="site-production",deployment="site-feed-generator"} 1
kube_deployment_status_replicas_unavailable{namespace="site-production",deployment="site-feed-generator"} 0
kube_deployment_status_replicas_updated{namespace="site-production",deployment="site-feed-generator"} 1
kube_deployment_status_observed_generation{namespace="site-production",deployment="site-feed-generator"} 142
kube_deployment_spec_replicas{namespace="site-production",deployment="site-feed-generator"} 1
kube_deployment_spec_paused{namespace="site-production",deployment="site-feed-generator"} 0
kube_deployment_spec_strategy_rollingupdate_max_unavailable{namespace="site-production",deployment="site-feed-generator"} 1
kube_deployment_spec_strategy_rollingupdate_max_surge{namespace="site-production",deployment="site-feed-generator"} 1
kube_deployment_metadata_generation{namespace="site-production",deployment="site-feed-generator"} 142
kube_deployment_labels{namespace="site-production",deployment="site-feed-generator",label_service="site-feed-generator"} 1
qw1mb0 picture qw1mb0 on 30 Apr 2019

Could you completely remove the limits and see if this is still happening? Also you may be able to tell if the CPU is being throttled with this query if you are collecting cAdvisor metrics:

sum(increase(container_cpu_cfs_throttled_periods_total{container_name="kube-state-metrics", namespace="kube-system", pod=~"kube-state-metrics.*"}[5m])) by (container_name, pod_name, namespace)
brancz picture brancz on 30 Apr 2019

Throttling graph:
http://screenshot.flant.ru/a.sidorov/2c/62/2c62df84fe3da6d5e1cc0a6c4add76860c2fe318.png

Now I will remove the limits and follow the work.

qw1mb0 picture qw1mb0 on 30 Apr 2019

Yeah that looks like the process is pretty much starved of the CPU it needs, removing CPU limits (or at least doubling judging by this graph) should help.

brancz picture brancz on 30 Apr 2019

We removed the limits, but we still see a problem with kube-state-metrics.

10.244.7.190 - old kube-state-metrics
10.244.7.191 - new pod, after restart kube-state-metrics

kube_deployment_status_replicas_available{deployment="chrome",instance="10.244.7.190:8443",job="kube-state-metrics",namespace="prod",scrape_endpoint="main"} | 6
kube_deployment_status_replicas_available{deployment="chrome",instance="10.244.7.191:8443",job="kube-state-metrics",namespace="prod",scrape_endpoint="main"} | 20

kube_deployment_status_replicas_unavailable{deployment="chrome",instance="10.244.7.190:8443",job="kube-state-metrics",namespace="prod",scrape_endpoint="main"} | 14
kube_deployment_status_replicas_unavailable{deployment="chrome",instance="10.244.7.191:8443",job="kube-state-metrics",namespace="prod",scrape_endpoint="main"} | 0
qw1mb0 picture qw1mb0 on 8 May 2019

Another example from another cluster:

kube_deployment_status_replicas_available{deployment="kube-state-metrics",instance="10.244.8.185:8443",job="kube-state-metrics",namespace="kube-prometheus",scrape_endpoint="main"} | 0

kube_deployment_status_replicas_unavailable{deployment="kube-state-metrics",instance="10.244.8.185:8443",job="kube-state-metrics",namespace="kube-prometheus",scrape_endpoint="main"} | 1

But target exists: http://screenshot.flant.ru/a.sidorov/43/46/434617f629c43c3ef87f5ffeb8f2befdcd8bb276.png

up{instance="10.244.8.185:8443",job="kube-state-metrics",scrape_endpoint="main"} | 1
up{instance="10.244.8.185:9443",job="kube-state-metrics",scrape_endpoint="self"} | 1

Info about replicaset and pod correct:

kube_replicaset_status_ready_replicas{instance="10.244.8.185:8443",job="kube-state-metrics",namespace="kube-prometheus",replicaset="kube-state-metrics-54d585db77",scrape_endpoint="main"} | 1

kube_pod_status_ready{condition="true",instance="10.244.8.185:8443",job="kube-state-metrics",namespace="kube-prometheus",pod="kube-state-metrics-54d585db77-rs8dg",scrape_endpoint="main"} | 1

kube_pod_status_phase{instance="10.244.8.185:8443",job="kube-state-metrics",namespace="kube-prometheus",phase="Running",pod="kube-state-metrics-54d585db77-rs8dg",scrape_endpoint="main"} | 1

kube_pod_container_status_running{container="kube-state-metrics",instance="10.244.8.185:8443",job="kube-state-metrics",namespace="kube-prometheus",pod="kube-state-metrics-54d585db77-rs8dg",scrape_endpoint="main"} | 1

In logs of kube-state-metrics only:

E0508 15:26:51.178535       1 reflector.go:329] pkg/mod/k8s.io/[email protected]+incompatible/tools/cache/reflector.go:94: expected type *v1beta1.Deployment, but watch event object had type *v1.Deployment
E0508 15:26:51.179632       1 reflector.go:329] pkg/mod/k8s.io/[email protected]+incompatible/tools/cache/reflector.go:94: expected type *v1beta1.Deployment, but watch event object had type *v1.Deployment
E0508 15:27:06.024578       1 reflector.go:329] pkg/mod/k8s.io/[email protected]+incompatible/tools/cache/reflector.go:94: expected type *v1beta1.Deployment, but watch event object had type *v1.Deployment
E0508 15:27:33.635505       1 reflector.go:329] pkg/mod/k8s.io/[email protected]+incompatible/tools/cache/reflector.go:94: expected type *v1beta1.Deployment, but watch event object had type *v1.Deployment
E0508 15:27:34.675222       1 reflector.go:329] pkg/mod/k8s.io/[email protected]+incompatible/tools/cache/reflector.go:94: expected type *v1beta1.Deployment, but watch event object had type *v1.Deployment
qw1mb0 picture qw1mb0 on 8 May 2019

Could you upgrade to the stable release of v1.6.0, that error should be fixed there.

brancz picture brancz on 8 May 2019
👍1

Thank you very much.
We tested the stable version 1.6.0 on ~ 100 clusters and we are not seeing this problem anymore. If the problem persists, I will open the issue.

qw1mb0 picture qw1mb0 on 18 May 2019

We are seeing this issue even on 1.8.0
what was the root cause for this?

sagarbirla picture sagarbirla on 24 Mar 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

lilic picture lilic  路  3Comments
stvnwrgs picture stvnwrgs  路  7Comments
derekwaynecarr picture derekwaynecarr  路  4Comments
dopey picture dopey  路  5Comments
roobre picture roobre  路  7Comments