kube-state-metrics tries to start 2 pods on Master node
Is this a BUG REPORT or FEATURE REQUEST?:
Uncomment only one, leave it on its own line:
/kind bug
/kind feature
What happened:
kube-state-metrics deployment tries to start 2 pods when I schedule it to run on a master node despite replicas being set to 1.
What you expected to happen:
That only one pod is scheduled on a master node.
How to reproduce it (as minimally and precisely as possible):
Deploy kube-state-metrics using nodeSelector and tolerations as below:
nodeSelector:
beta.kubernetes.io/os: linux
kubernetes.io/hostname: master-node1
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
Anything else we need to know?:
One pods runs normally and the other pod which wasn't even supposed to be scheduled stays in a Pending state because it ignores the tolerations I set and tries to schedule it on a master node anyway because of nodeSelector:
[root@master-node1 ~]# kubectl get pods -n monitoring-prometheus -o wide | grep kube-state-metrics
kube-state-metrics-784996ddd9-vd2bk 4/4 Running 0 17h 10.244.1.69 master-node1 <none> <none>
kube-state-metrics-798cbf4476-lxvsl 0/4 Pending 0 17h <none> <none> <none> <none>
Here's the output of kubectl describe pod on the Pending pod:
Name: kube-state-metrics-798cbf4476-lxvsl
Namespace: monitoring-prometheus
Priority: 0
PriorityClassName: <none>
Node: <none>
Labels: app=kube-state-metrics
pod-template-hash=798cbf4476
Annotations: <none>
Status: Pending
IP:
Controlled By: ReplicaSet/kube-state-metrics-798cbf4476
Containers:
kube-rbac-proxy-main:
Image: quay.io/coreos/kube-rbac-proxy:v0.4.1
Port: 8443/TCP
Host Port: 0/TCP
Args:
--logtostderr
--secure-listen-address=:8443
--tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
--upstream=http://127.0.0.1:8081/
Limits:
cpu: 20m
memory: 40Mi
Requests:
cpu: 10m
memory: 20Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-state-metrics-token-rdj26 (ro)
kube-rbac-proxy-self:
Image: quay.io/coreos/kube-rbac-proxy:v0.4.1
Port: 9443/TCP
Host Port: 0/TCP
Args:
--logtostderr
--secure-listen-address=:9443
--tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
--upstream=http://127.0.0.1:8082/
Limits:
cpu: 20m
memory: 40Mi
Requests:
cpu: 10m
memory: 20Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-state-metrics-token-rdj26 (ro)
kube-state-metrics:
Image: quay.io/coreos/kube-state-metrics:v1.5.0
Port: <none>
Host Port: <none>
Args:
--host=127.0.0.1
--port=8081
--telemetry-host=127.0.0.1
--telemetry-port=8082
Limits:
cpu: 112m
memory: 330Mi
Requests:
cpu: 112m
memory: 330Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-state-metrics-token-rdj26 (ro)
addon-resizer:
Image: quay.io/coreos/addon-resizer:1.0
Port: <none>
Host Port: <none>
Command:
/pod_nanny
--container=kube-state-metrics
--cpu=100m
--extra-cpu=2m
--memory=150Mi
--extra-memory=30Mi
--threshold=5
--deployment=kube-state-metrics
Limits:
cpu: 50m
memory: 30Mi
Requests:
cpu: 10m
memory: 30Mi
Environment:
MY_POD_NAME: kube-state-metrics-798cbf4476-lxvsl (v1:metadata.name)
MY_POD_NAMESPACE: monitoring-prometheus (v1:metadata.namespace)
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-state-metrics-token-rdj26 (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
kube-state-metrics-token-rdj26:
Type: Secret (a volume populated by a Secret)
SecretName: kube-state-metrics-token-rdj26
Optional: false
QoS Class: Burstable
Node-Selectors: beta.kubernetes.io/os=linux
kubernetes.io/hostname=master-node1
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
Here's the yaml file I'm using to deploy it:
apiVersion: apps/v1beta2
kind: Deployment
metadata:
labels:
app: kube-state-metrics
name: kube-state-metrics
namespace: monitoring-prometheus
spec:
replicas: 1
selector:
matchLabels:
app: kube-state-metrics
template:
metadata:
labels:
app: kube-state-metrics
spec:
containers:
- args:
- --logtostderr
- --secure-listen-address=:8443
- --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
- --upstream=http://127.0.0.1:8081/
image: quay.io/coreos/kube-rbac-proxy:v0.4.1
name: kube-rbac-proxy-main
ports:
- containerPort: 8443
name: https-main
resources:
limits:
cpu: 20m
memory: 40Mi
requests:
cpu: 10m
memory: 20Mi
- args:
- --logtostderr
- --secure-listen-address=:9443
- --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
- --upstream=http://127.0.0.1:8082/
image: quay.io/coreos/kube-rbac-proxy:v0.4.1
name: kube-rbac-proxy-self
ports:
- containerPort: 9443
name: https-self
resources:
limits:
cpu: 20m
memory: 40Mi
requests:
cpu: 10m
memory: 20Mi
- args:
- --host=127.0.0.1
- --port=8081
- --telemetry-host=127.0.0.1
- --telemetry-port=8082
image: quay.io/coreos/kube-state-metrics:v1.5.0
name: kube-state-metrics
resources:
limits:
cpu: 100m
memory: 150Mi
requests:
cpu: 100m
memory: 150Mi
- command:
- /pod_nanny
- --container=kube-state-metrics
- --cpu=100m
- --extra-cpu=2m
- --memory=150Mi
- --extra-memory=30Mi
- --threshold=5
- --deployment=kube-state-metrics
env:
- name: MY_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: MY_POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: quay.io/coreos/addon-resizer:1.0
name: addon-resizer
resources:
limits:
cpu: 50m
memory: 30Mi
requests:
cpu: 10m
memory: 30Mi
nodeSelector:
beta.kubernetes.io/os: linux
kubernetes.io/hostname: master-node1
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
securityContext:
runAsNonRoot: true
runAsUser: 65534
serviceAccountName: kube-state-metrics
Environment:
- Kubernetes version (use
kubectl version):
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.2", GitCommit:"cff46ab41ff0bb44d8584413b598ad8360ec1def", GitTreeState:"clean", BuildDate:"2019-01-10T23:35:51Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.2", GitCommit:"cff46ab41ff0bb44d8584413b598ad8360ec1def", GitTreeState:"clean", BuildDate:"2019-01-10T23:28:14Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"linux/amd64"}
- Kube-state-metrics image version
v1.5.0
andre-franco
All 19 comments
Why do you think this is a problem with kube-state-metrics itself? This seems like for some reason Kubernetes is not doing the right thing. Did you check if eventually there were events in the monitoring namespace that might say why it says pending?
brancz
on 22 Feb 2019
@brancz sorry for the delay, I've been away from the office for a while.
I know what causes the Pod to stay in a pending state, it's because it correctly sets the NodeSelector configuration but ignores the Tolerations for NoSchedule that was set:
[root@kb-dev-master1 ~]# kubectl describe pod kube-state-metrics-56c656c546-jx6kg -n monitoring-prometheus
Name: kube-state-metrics-56c656c546-jx6kg
Namespace: monitoring-prometheus
Priority: 0
PriorityClassName: <none>
Node: <none>
Labels: app=kube-state-metrics
pod-template-hash=56c656c546
Annotations: <none>
Status: Pending
IP:
Controlled By: ReplicaSet/kube-state-metrics-56c656c546
Containers:
kube-rbac-proxy-main:
Image: quay.io/coreos/kube-rbac-proxy:v0.4.1
Port: 8443/TCP
Host Port: 0/TCP
Args:
--logtostderr
--secure-listen-address=:8443
--tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
--upstream=http://127.0.0.1:8081/
Limits:
cpu: 20m
memory: 40Mi
Requests:
cpu: 10m
memory: 20Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-state-metrics-token-rdj26 (ro)
kube-rbac-proxy-self:
Image: quay.io/coreos/kube-rbac-proxy:v0.4.1
Port: 9443/TCP
Host Port: 0/TCP
Args:
--logtostderr
--secure-listen-address=:9443
--tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
--upstream=http://127.0.0.1:8082/
Limits:
cpu: 20m
memory: 40Mi
Requests:
cpu: 10m
memory: 20Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-state-metrics-token-rdj26 (ro)
kube-state-metrics:
Image: quay.io/coreos/kube-state-metrics:v1.5.0
Port: <none>
Host Port: <none>
Args:
--host=127.0.0.1
--port=8081
--telemetry-host=127.0.0.1
--telemetry-port=8082
Limits:
cpu: 112m
memory: 330Mi
Requests:
cpu: 112m
memory: 330Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-state-metrics-token-rdj26 (ro)
addon-resizer:
Image: quay.io/coreos/addon-resizer:1.0
Port: <none>
Host Port: <none>
Command:
/pod_nanny
--container=kube-state-metrics
--cpu=100m
--extra-cpu=2m
--memory=150Mi
--extra-memory=30Mi
--threshold=5
--deployment=kube-state-metrics
Limits:
cpu: 50m
memory: 30Mi
Requests:
cpu: 10m
memory: 30Mi
Environment:
MY_POD_NAME: kube-state-metrics-56c656c546-jx6kg (v1:metadata.name)
MY_POD_NAMESPACE: monitoring-prometheus (v1:metadata.namespace)
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-state-metrics-token-rdj26 (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
kube-state-metrics-token-rdj26:
Type: Secret (a volume populated by a Secret)
SecretName: kube-state-metrics-token-rdj26
Optional: false
QoS Class: Burstable
Node-Selectors: beta.kubernetes.io/os=linux
node-role.kubernetes.io/master=master
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 22m (x2 over 22m) default-scheduler 0/6 nodes are available: 3 node(s) didn't match node selector, 3 node(s) had taints that the pod didn't tolerate.
However, the issue isn't about this Pod being on a pending state...
The problem is that two kube-state-metrics ReplicaSets are created despite me asking for only one in my Deployment, this is the yaml I'm using to deploy kube-state-metrics:
apiVersion: apps/v1beta2
kind: Deployment
metadata:
labels:
app: kube-state-metrics
name: kube-state-metrics
namespace: monitoring-prometheus
spec:
replicas: 1
selector:
matchLabels:
app: kube-state-metrics
template:
metadata:
labels:
app: kube-state-metrics
spec:
containers:
- args:
- --logtostderr
- --secure-listen-address=:8443
- --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
- --upstream=http://127.0.0.1:8081/
image: quay.io/coreos/kube-rbac-proxy:v0.4.1
name: kube-rbac-proxy-main
ports:
- containerPort: 8443
name: https-main
resources:
limits:
cpu: 20m
memory: 40Mi
requests:
cpu: 10m
memory: 20Mi
- args:
- --logtostderr
- --secure-listen-address=:9443
- --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
- --upstream=http://127.0.0.1:8082/
image: quay.io/coreos/kube-rbac-proxy:v0.4.1
name: kube-rbac-proxy-self
ports:
- containerPort: 9443
name: https-self
resources:
limits:
cpu: 20m
memory: 40Mi
requests:
cpu: 10m
memory: 20Mi
- args:
- --host=127.0.0.1
- --port=8081
- --telemetry-host=127.0.0.1
- --telemetry-port=8082
image: quay.io/coreos/kube-state-metrics:v1.5.0
name: kube-state-metrics
resources:
limits:
cpu: 100m
memory: 150Mi
requests:
cpu: 100m
memory: 150Mi
- command:
- /pod_nanny
- --container=kube-state-metrics
- --cpu=100m
- --extra-cpu=2m
- --memory=150Mi
- --extra-memory=30Mi
- --threshold=5
- --deployment=kube-state-metrics
env:
- name: MY_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: MY_POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: quay.io/coreos/addon-resizer:1.0
name: addon-resizer
resources:
limits:
cpu: 50m
memory: 30Mi
requests:
cpu: 10m
memory: 30Mi
nodeSelector:
beta.kubernetes.io/os: linux
node-role.kubernetes.io/master: master
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
securityContext:
runAsNonRoot: true
runAsUser: 65534
serviceAccountName: kube-state-metrics
I believe the issue is with kube-state-metrics itself because I've deployed all components (except for node-exporter that runs on both master and worker nodes, of course) of my prometheus stack on master nodes, using the same nodeSelector and tolerations settings for the other Pods such as prometheus server, prometheus-operator, grafana, alertmanager... this odd behavior only happens to kube-state-metrics.
Here's how the Pods look on my monitoring namespace:
[root@kb-dev-master1 ~]# kubectl get pods -n monitoring-prometheus -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
alertmanager-main-0 2/2 Running 0 18d 10.244.0.84 kb-dev-master1 <none> <none>
grafana-74b855797-5n7hv 1/1 Running 0 11d 10.244.2.88 kb-dev-master3 <none> <none>
kube-state-metrics-56c656c546-jx6kg 0/4 Pending 0 17d <none> <none> <none> <none>
kube-state-metrics-9fffd76d7-jlwtq 4/4 Running 0 17d 10.244.0.86 kb-dev-master1 <none> <none>
node-exporter-b7zw5 2/2 Running 2 18d 192.168.42.71 kb-dev-worker1 <none> <none>
node-exporter-gl8sv 2/2 Running 0 18d 192.168.42.63 kb-dev-master3 <none> <none>
node-exporter-svp8g 2/2 Running 0 18d 192.168.42.62 kb-dev-master2 <none> <none>
node-exporter-vbwgg 2/2 Running 0 18d 192.168.42.61 kb-dev-master1 <none> <none>
node-exporter-zhvqk 2/2 Running 2 18d 192.168.42.73 kb-dev-worker3 <none> <none>
node-exporter-zjd2f 2/2 Running 2 18d 192.168.42.72 kb-dev-worker2 <none> <none>
prometheus-k8s-0 3/3 Running 0 11d 10.244.1.74 kb-dev-master2 <none> <none>
prometheus-operator-5d75765645-4v2mw 1/1 Running 0 18d 10.244.1.66 kb-dev-master2 <none> <none>
Thanks!
andre-franco
on 4 Mar 2019
Thanks for the detailed report @andre-franco.
I am having difficulties understanding how this should relate to kube-state-metrics.
I am guessing the two kube-state-metrics pods and their respective replica sets originate from the same deployment, right? Would you mind removing the deployment all together, making sure both Pods are gone. Once that happened, can you create the Deployment again and see what happens?
Having two Replicasets for a single Deployments sounds like a rolling update after e.g. changing the tolerations in the spec. See https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/#deployment-recommended for more details on Depoyments and Replicasets.
mxinden
on 5 Mar 2019
Hello @mxinden, I too am unsure whether the problem is related to kube-state-metrics itself...
I am guessing the two kube-state-metrics pods and their respective replica sets originate from the same deployment, right?
Correct. The behaviour it shows is as follows:
1 - you create the Deployment with the yaml I provided.
2 - kubernetes creates a ReplicaSet for this Deployment and a Pod.
3 - upon successful container creation and the Pod is on a Running state, the other ReplicaSet gets created and its Pod stays in that Pending state.
Would you mind removing the deployment all together, making sure both Pods are gone. Once that happened, can you create the Deployment again and see what happens?
I've done that already and the result is the same. Not only have I removed the Deployment completely and created it again in this cluster, but I've also created it from scratch in different clusters, running different versions of k8s, namely v1.13.2 and v1.10.1.
In all of the environments the behaviour is the same.
If I just remove the nodeSelector configuration from this yaml and kube-state-metrics gets scheduled to run on a worker node, this behaviour doesn't happen. It simply creates a single ReplicaSet and a single Pod like it's supposed to.
Right now I'm completely clueless why this is happening...
Thanks.
andre-franco
on 6 Mar 2019
@andre-franco are you able to reproduce the same with a different deployment, e.g. some nginx webserver?
Thanks for bearing with us.
mxinden
on 6 Mar 2019
@mxinden, I was unable to reprocude this behaviour with a simple nginx deployment.
Here's the yaml for the nginx deployment I used:
apiVersion: apps/v1beta2
kind: Deployment
metadata:
labels:
app: nginx
name: nginx
namespace: monitoring-prometheus
spec:
selector:
matchLabels:
app: nginx
replicas: 1
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
nodeSelector:
beta.kubernetes.io/os: linux
node-role.kubernetes.io/master: master
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
Only 1 replica set is created, just as expected. Here's how my prometheus stack looks like after deploying nginx:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
alertmanager-main-0 2/2 Running 0 20d 10.244.0.84 kb-dev-master1 <none> <none>
grafana-74b855797-5n7hv 1/1 Running 0 13d 10.244.2.88 kb-dev-master3 <none> <none>
kube-state-metrics-56c656c546-jx6kg 0/4 Pending 0 19d <none> <none> <none> <none>
kube-state-metrics-9fffd76d7-jlwtq 4/4 Running 0 19d 10.244.0.86 kb-dev-master1 <none> <none>
nginx-6b47f6c4f8-mznq7 1/1 Running 0 58s 10.244.0.87 kb-dev-master1 <none> <none>
node-exporter-b7zw5 2/2 Running 2 20d 192.168.42.71 kb-dev-worker1 <none> <none>
node-exporter-gl8sv 2/2 Running 0 20d 192.168.42.63 kb-dev-master3 <none> <none>
node-exporter-svp8g 2/2 Running 0 20d 192.168.42.62 kb-dev-master2 <none> <none>
node-exporter-vbwgg 2/2 Running 0 20d 192.168.42.61 kb-dev-master1 <none> <none>
node-exporter-zhvqk 2/2 Running 2 20d 192.168.42.73 kb-dev-worker3 <none> <none>
node-exporter-zjd2f 2/2 Running 2 20d 192.168.42.72 kb-dev-worker2 <none> <none>
prometheus-k8s-0 3/3 Running 1 13d 10.244.1.74 kb-dev-master2 <none> <none>
prometheus-operator-5d75765645-4v2mw 1/1 Running 0 20d 10.244.1.66 kb-dev-master2 <none> <none>
+1 facing the same issue
alok87
on 6 Mar 2019
We also got bitten by a similar issue yesterday. In our cluster we have two instance groups, each having a taint - ondemand and spot. Every pod we run in the cluster needs to tolerate one of these taints.
We are running kube state metrics deployment https://gist.github.com/alok87/1d83d520437aeccc9302cd711c64322c having taint as spot.
On doing creating the above deployment a single replica set gets created and a single pod start coming up. As soon as this pod comes up, a new replica set on its own gets created without taint and tolerations in it. This results in one more pod coming up and it gets stuck in Pending state.
$ kubectl get pods -n monitoring | grep state
kube-state-metrics-5b9d46bc89-lp84t 0/4 Pending 0 3m
kube-state-metrics-cd55b8646-xqqh2 4/4 Running 0 3m
$ kubectl get rs -n monitoring | grep state
kube-state-metrics-56759cf9dc 0 0 0 5m
kube-state-metrics-7fcc4bc945 1 1 1 3m```
Issue
Both the replica set is managed by a single deployment object. And after the first replica set comes up we have seen deployment object to not have the taints in it. Something is changing the deployment object after the first replica set becomes ready.
For us, these Pending pods resulted in Spotinst controller continuously scaling our nodes to the maximum no of nodes allowed.
To solve this we have scheduled this deployment without toleration. Without toleration also we see many replica set getting created every once and again but then are in 0 Desired state, so this is not resulting in Pending pods.
Below is the output of the replica set after we moved the deployment to not have tolerations.
$ kubectl get rs -n monitoring | grep state
kube-state-metrics-56b55cdb97 1 1 1 3h
kube-state-metrics-56d88fd8fc 0 0 0 12h
kube-state-metrics-59dbc84c47 0 0 0 12h
kube-state-metrics-5d7566bcd9 0 0 0 11h
kube-state-metrics-648f64b477 0 0 0 12h
kube-state-metrics-6bc55f5989 0 0 0 12h
kube-state-metrics-6dc46869d9 0 0 0 12h
kube-state-metrics-6fc4898dbf 0 0 0 12h
kube-state-metrics-7654c5cdb7 0 0 0 12h
kube-state-metrics-877bbfb6f 0 0 0 12h
kube-state-metrics-f965b44c9 0 0 0 11h
Let me know if any new issue should be opened for this?
Also it looks like a deployment object issue and not kube state metrics issue particularly. So where should we open the issue?
alok87
on 7 Mar 2019
The replicasets being leftover is a feature of the Deployment object in Kubernetes. By default a Deployment keeps the last 10 versions of the replicasets it creates, for rollback purposes. This sounds to me like this is an issue with the addon-resizer not transferring tolerations properly when it "autoscales" kube-state-metrics.
brancz
on 7 Mar 2019
@brancz Thanks.
Should i open a separate issue somewhere else?
alok87
on 7 Mar 2019
@andre-franco you seem to be running an old version of the addon-resizer:
Image: quay.io/coreos/addon-resizer:1.0
This version probably still contains the issue described in https://github.com/kubernetes/autoscaler/issues/188.
Would you mind trying out a recent version. E.g. the one we use in current kube-prometheus: https://github.com/kubernetes/kube-state-metrics/blob/master/kubernetes/kube-state-metrics-deployment.yaml
mxinden
on 7 Mar 2019
I had similar issue, k8s.gcr.io/addon-resizer:1.8.4 works like a charm. But beware, look like this bug is back in 2.0 (kubernetes/autoscaler#1799).
szymonpk
on 21 Mar 2019
+1 facing the same issue锛寀se addon-resizer-amd64:2.1 ,and use in current kube-prometheus: https://github.com/coreos/prometheus-operator/blob/v0.29.0/contrib/kube-prometheus/manifests/kube-state-metrics-deployment.yaml
marvinpan8
on 11 Apr 2019
+1, encounter the same problem, use k8s.gcr.io/addon-resizer:1.7.
WakeupTsai
on 16 Apr 2019
1.8.4 is the version recommended here:
https://github.com/kubernetes/autoscaler/tree/master/addon-resizer
@mxinden If you're going to remove it, then might I suggest removing it from the documentation there and here as well (or file the relevant issues to have it removed):
https://coreos.com/operators/prometheus/docs/latest/user-guides/cluster-monitoring.html#exporters
braderhart
on 11 May 2019
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
fejta-bot
on 9 Aug 2019
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
fejta-bot
on 8 Sep 2019
Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
fejta-bot
on 8 Oct 2019
@fejta-bot: Closing this issue.
In response to this:
Rotten issues close after 30d of inactivity.
Reopen the issue with/reopen.
Mark the issue as fresh with/remove-lifecycle rotten.Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
k8s-ci-robot
on 8 Oct 2019
Related issues
jackzampolin
路
8Comments
chasemaier
路
5Comments
dopey
路
5Comments
therc
路
4Comments
derekwaynecarr
路
4Comments
Most helpful comment
+1 facing the same issue