Kube-prometheus: kube-state-metrics doesn't work after upgrade and removing addon-resizer

Created on 23 Dec 2019  路  6Comments  路  Source: prometheus-operator/kube-prometheus

After I did the upgrade of kube-state-metrics from v1.7.2 to v1.9.0 and removing addon-resizer container I am getting the following error in kube-state-metrics container:

E1223 12:06:46.860982       1 reflector.go:156] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:108: Failed to list *v1beta1.VolumeAttachment: volumeattachments.storage.k8s.io is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list resource "volumeattachments" in API group "storage.k8s.io" at the cluster scope
E1223 12:06:47.861880       1 reflector.go:156] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:108: Failed to list *v1.Namespace: networkpolicies.networking.k8s.io is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list resource "networkpolicies" in API group "networking.k8s.io" at the cluster scope
E1223 12:06:48.166988       1 reflector.go:156] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:108: Failed to list *v1beta1.MutatingWebhookConfiguration: mutatingwebhookconfigurations.admissionregistration.k8s.io is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list resource "mutatingwebhookconfigurations" in API group "admissionregistration.k8s.io" at the cluster scope
E1223 12:06:49.059870       1 reflector.go:156] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:108: Failed to list *v1beta1.ValidatingWebhookConfiguration: validatingwebhookconfigurations.admissionregistration.k8s.io is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list resource "validatingwebhookconfigurations" in API group "admissionregistration.k8s.io" at the cluster scope
E1223 12:06:49.266197       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 49 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x13c4de0, 0x2178810)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:74 +0xa3
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:48 +0x82
panic(0x13c4de0, 0x2178810)
    /usr/local/go/src/runtime/panic.go:679 +0x1b2
k8s.io/kube-state-metrics/internal/store.wrapHPAFunc.func1(0x153f360, 0xc0029631d8, 0xc0002a4b40)
    /go/src/k8s.io/kube-state-metrics/internal/store/hpa.go:293 +0xbe
k8s.io/kube-state-metrics/pkg/metric.(*FamilyGenerator).Generate(...)
    /go/src/k8s.io/kube-state-metrics/pkg/metric/generator.go:39
k8s.io/kube-state-metrics/pkg/metric.ComposeMetricGenFuncs.func1(0x153f360, 0xc0029631d8, 0x7f7921b388f0, 0xc0029631d8, 0x0)
    /go/src/k8s.io/kube-state-metrics/pkg/metric/generator.go:78 +0x122
k8s.io/kube-state-metrics/pkg/metrics_store.(*MetricsStore).Add(0xc0003a2e80, 0x153f360, 0xc0029631d8, 0x0, 0x0)
    /go/src/k8s.io/kube-state-metrics/pkg/metrics_store/metrics_store.go:76 +0xf6
k8s.io/kube-state-metrics/pkg/metrics_store.(*MetricsStore).Replace(0xc0003a2e80, 0xc00015b900, 0x49, 0x49, 0xc0027d60a0, 0x9, 0x6b7a922b, 0x219a560)
    /go/src/k8s.io/kube-state-metrics/pkg/metrics_store/metrics_store.go:138 +0xa4
k8s.io/client-go/tools/cache.(*Reflector).syncWith(0xc000399c80, 0xc00015aa00, 0x49, 0x49, 0xc0027d60a0, 0x9, 0x0, 0xc0005de000)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:354 +0xf8
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1(0xc000399c80, 0xc000508080, 0xc00016aa20, 0xc000727bb8, 0x0, 0x0)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:250 +0x8fa
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc000399c80, 0xc00016aa20, 0x0, 0x0)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:257 +0x1a9
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:155 +0x33
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc00019bf78)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152 +0x5e
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000727f78, 0x3b9aca00, 0x0, 0x1, 0xc00016aa20)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(...)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc000399c80, 0xc00016aa20)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:154 +0x16b
created by k8s.io/kube-state-metrics/internal/store.(*Builder).reflectorPerNamespace
    /go/src/k8s.io/kube-state-metrics/internal/store/builder.go:337 +0x265
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x128190e]
picture Aracki
👍2

Most helpful comment

I've changed ClusterRole kube-state-metrics for kube-state-metrics 1.9.0 and added:

- apiGroups:
  - admissionregistration.k8s.io
  resources:
  - mutatingwebhookconfigurations
  - validatingwebhookconfigurations
  verbs:
  - list
  - watch
- apiGroups:
  - networking.k8s.io
  resources:
  - networkpolicies
  verbs:
  - list
  - watch
- apiGroups:
  - storage.k8s.io
  resources:
  - volumeattachments
  verbs:
  - list
  - watch

So the complete ClusterRole looks like:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-state-metrics
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - secrets
  - nodes
  - pods
  - services
  - resourcequotas
  - replicationcontrollers
  - limitranges
  - persistentvolumeclaims
  - persistentvolumes
  - namespaces
  - endpoints
  verbs:
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - daemonsets
  - deployments
  - replicasets
  - ingresses
  verbs:
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - statefulsets
  - daemonsets
  - deployments
  - replicasets
  verbs:
  - list
  - watch
- apiGroups:
  - batch
  resources:
  - cronjobs
  - jobs
  verbs:
  - list
  - watch
- apiGroups:
  - autoscaling
  resources:
  - horizontalpodautoscalers
  verbs:
  - list
  - watch
- apiGroups:
  - authentication.k8s.io
  resources:
  - tokenreviews
  verbs:
  - create
- apiGroups:
  - authorization.k8s.io
  resources:
  - subjectaccessreviews
  verbs:
  - create
- apiGroups:
  - policy
  resources:
  - poddisruptionbudgets
  verbs:
  - list
  - watch
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests
  verbs:
  - list
  - watch
- apiGroups:
  - storage.k8s.io
  resources:
  - storageclasses
  verbs:
  - list
  - watch
- apiGroups:
  - admissionregistration.k8s.io
  resources:
  - mutatingwebhookconfigurations
  - validatingwebhookconfigurations
  verbs:
  - list
  - watch
- apiGroups:
  - networking.k8s.io
  resources:
  - networkpolicies
  verbs:
  - list
  - watch
- apiGroups:
  - storage.k8s.io
  resources:
  - volumeattachments
  verbs:
  - list
  - watch

With this change kube-stste-metrics 1.9.0. works fine.

picture r0bj on 27 Dec 2019
👍10 🎉3

All 6 comments

I've changed ClusterRole kube-state-metrics for kube-state-metrics 1.9.0 and added:

- apiGroups:
  - admissionregistration.k8s.io
  resources:
  - mutatingwebhookconfigurations
  - validatingwebhookconfigurations
  verbs:
  - list
  - watch
- apiGroups:
  - networking.k8s.io
  resources:
  - networkpolicies
  verbs:
  - list
  - watch
- apiGroups:
  - storage.k8s.io
  resources:
  - volumeattachments
  verbs:
  - list
  - watch

So the complete ClusterRole looks like:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-state-metrics
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - secrets
  - nodes
  - pods
  - services
  - resourcequotas
  - replicationcontrollers
  - limitranges
  - persistentvolumeclaims
  - persistentvolumes
  - namespaces
  - endpoints
  verbs:
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - daemonsets
  - deployments
  - replicasets
  - ingresses
  verbs:
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - statefulsets
  - daemonsets
  - deployments
  - replicasets
  verbs:
  - list
  - watch
- apiGroups:
  - batch
  resources:
  - cronjobs
  - jobs
  verbs:
  - list
  - watch
- apiGroups:
  - autoscaling
  resources:
  - horizontalpodautoscalers
  verbs:
  - list
  - watch
- apiGroups:
  - authentication.k8s.io
  resources:
  - tokenreviews
  verbs:
  - create
- apiGroups:
  - authorization.k8s.io
  resources:
  - subjectaccessreviews
  verbs:
  - create
- apiGroups:
  - policy
  resources:
  - poddisruptionbudgets
  verbs:
  - list
  - watch
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests
  verbs:
  - list
  - watch
- apiGroups:
  - storage.k8s.io
  resources:
  - storageclasses
  verbs:
  - list
  - watch
- apiGroups:
  - admissionregistration.k8s.io
  resources:
  - mutatingwebhookconfigurations
  - validatingwebhookconfigurations
  verbs:
  - list
  - watch
- apiGroups:
  - networking.k8s.io
  resources:
  - networkpolicies
  verbs:
  - list
  - watch
- apiGroups:
  - storage.k8s.io
  resources:
  - volumeattachments
  verbs:
  - list
  - watch

With this change kube-stste-metrics 1.9.0. works fine.

r0bj picture r0bj on 27 Dec 2019
👍10 🎉3

For me, it doesn't work. If I add missing apiGroups to ClusterRole I am getting:

I1228 23:18:03.762769       1 main.go:86] Using default collectors
I1228 23:18:03.762867       1 main.go:98] Using all namespace
I1228 23:18:03.762881       1 main.go:139] metric white-blacklisting: blacklisting the following items:
W1228 23:18:03.762914       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1228 23:18:03.861141       1 main.go:184] Testing communication with server
I1228 23:18:03.874687       1 main.go:189] Running with Kubernetes cluster version: v1.14. git version: v1.14.8. git tree state: clean. commit: 211047e9a1922595eaa3a1127ed365e9299a6c23. platform: linux/amd64
I1228 23:18:03.874754       1 main.go:191] Communication with server successful
I1228 23:18:03.959238       1 main.go:225] Starting metrics server: 127.0.0.1:8081
I1228 23:18:03.959432       1 main.go:200] Starting kube-state-metrics self metrics server: 127.0.0.1:8082
I1228 23:18:03.959436       1 metrics_handler.go:96] Autosharding disabled
I1228 23:18:04.060758       1 builder.go:146] Active collectors: certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,validatingwebhookconfigurations,volumeattachments
E1228 23:18:05.960659       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 70 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x13c4de0, 0x2178810)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:74 +0xa3
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:48 +0x82
panic(0x13c4de0, 0x2178810)
    /usr/local/go/src/runtime/panic.go:679 +0x1b2
k8s.io/kube-state-metrics/internal/store.wrapHPAFunc.func1(0x153f360, 0xc002510e68, 0xc002e5e600)
    /go/src/k8s.io/kube-state-metrics/internal/store/hpa.go:293 +0xbe
k8s.io/kube-state-metrics/pkg/metric.(*FamilyGenerator).Generate(...)
    /go/src/k8s.io/kube-state-metrics/pkg/metric/generator.go:39
k8s.io/kube-state-metrics/pkg/metric.ComposeMetricGenFuncs.func1(0x153f360, 0xc002510e68, 0x7f562b236968, 0xc002510e68, 0x0)
    /go/src/k8s.io/kube-state-metrics/pkg/metric/generator.go:78 +0x122
k8s.io/kube-state-metrics/pkg/metrics_store.(*MetricsStore).Add(0xc0003a3000, 0x153f360, 0xc002510e68, 0x0, 0x0)
    /go/src/k8s.io/kube-state-metrics/pkg/metrics_store/metrics_store.go:76 +0xf6
k8s.io/kube-state-metrics/pkg/metrics_store.(*MetricsStore).Replace(0xc0003a3000, 0xc00015b900, 0x49, 0x49, 0xc000d6e490, 0x9, 0x6e1eccfc, 0x219a560)
    /go/src/k8s.io/kube-state-metrics/pkg/metrics_store/metrics_store.go:138 +0xa4
k8s.io/client-go/tools/cache.(*Reflector).syncWith(0xc000397c80, 0xc000778000, 0x49, 0x49, 0xc000d6e490, 0x9, 0x0, 0xc0005120c0)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:354 +0xf8
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1(0xc000397c80, 0xc000506200, 0xc00016ca20, 0xc0007e9bb8, 0x0, 0x0)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:250 +0x8fa
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc000397c80, 0xc00016ca20, 0x0, 0x0)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:257 +0x1a9
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:155 +0x33
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc00036ef78)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152 +0x5e
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0007e9f78, 0x3b9aca00, 0x0, 0x1, 0xc00016ca20)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(...)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc000397c80, 0xc00016ca20)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:154 +0x16b
created by k8s.io/kube-state-metrics/internal/store.(*Builder).reflectorPerNamespace
    /go/src/k8s.io/kube-state-metrics/internal/store/builder.go:337 +0x265
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x128190e]

The same error I guess, but without RBAC errors. 馃 @r0bj

Aracki picture Aracki on 29 Dec 2019
👍3

@Aracki having the same issue w/ k-s-m 1.9.0

chiefy picture chiefy on 3 Jan 2020

Looks like this is https://github.com/kubernetes/kube-state-metrics/issues/1012

brancz picture brancz on 7 Jan 2020
👍2

I think this was already fixed. Please reopen if this is still an issue.

paulfantom picture paulfantom on 2 Mar 2021
👍1

Not happening anymore with v1.9.7.

Aracki picture Aracki on 7 Mar 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

brancz picture brancz  路  5Comments
argamanza picture argamanza  路  3Comments
grobie picture grobie  路  6Comments
lilic picture lilic  路  7Comments
RiRa12621 picture RiRa12621  路  6Comments