Kube-state-metrics: Segfault with 1.9.0
/kind bug
What happened:
Kube state metrics pod crashlops with a segfault.
I 2019-12-23T01:32:57.901573Z Using default collectors
I 2019-12-23T01:32:57.901633Z Using all namespace
I 2019-12-23T01:32:57.901643Z metric white-blacklisting: blacklisting the following items:
W 2019-12-23T01:32:57.901666Z Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I 2019-12-23T01:32:57.903415Z Testing communication with server
I 2019-12-23T01:32:57.912850Z Running with Kubernetes cluster version: v1.15+. git version: v1.15.4-gke.22. git tree state: clean. commit: a6ba43f5a24ac29e631bb627c9b2a719c4e93638. platform: linux/amd64
I 2019-12-23T01:32:57.912877Z Communication with server successful
I 2019-12-23T01:32:57.913028Z Starting metrics server: 0.0.0.0:8080
I 2019-12-23T01:32:57.913135Z Starting kube-state-metrics self metrics server: 0.0.0.0:8081
I 2019-12-23T01:32:57.913254Z Autosharding disabled
I 2019-12-23T01:32:57.914977Z Active collectors: certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,validatingwebhookconfigurations,volumeattachments
E 2019-12-23T01:32:57.917328Z pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:108: Failed to list *v1beta1.MutatingWebhookConfiguration: mutatingwebhookconfigurations.admissionregistration.k8s.io is forbidden: User "system:serviceaccount:core:prometheus-kube-state-metrics" cannot list resource "mutatingwebhookconfigurations" in API group "admissionregistration.k8s.io" at the cluster scope
E 2019-12-23T01:32:57.917723Z Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
E 2019-12-23T01:32:57.917867028Z goroutine 43 [running]:
E 2019-12-23T01:32:57.917872518Z k8s.io/apimachinery/pkg/util/runtime.logPanic(0x13c4de0, 0x2178810)
E 2019-12-23T01:32:57.917877652Z /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:74 +0xa3
E 2019-12-23T01:32:57.917883098Z k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
E 2019-12-23T01:32:57.917887996Z /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:48 +0x82
E 2019-12-23T01:32:57.917893026Z panic(0x13c4de0, 0x2178810)
E 2019-12-23T01:32:57.917897653Z /usr/local/go/src/runtime/panic.go:679 +0x1b2
E 2019-12-23T01:32:57.917902458Z k8s.io/kube-state-metrics/internal/store.wrapHPAFunc.func1(0x153f360, 0xc000062c38, 0xc00011b170)
E 2019-12-23T01:32:57.917918096Z /go/src/k8s.io/kube-state-metrics/internal/store/hpa.go:293 +0xbe
E 2019-12-23T01:32:57.917922515Z k8s.io/kube-state-metrics/pkg/metric.(*FamilyGenerator).Generate(...)
E 2019-12-23T01:32:57.917925700Z /go/src/k8s.io/kube-state-metrics/pkg/metric/generator.go:39
E 2019-12-23T01:32:57.917929049Z k8s.io/kube-state-metrics/pkg/metric.ComposeMetricGenFuncs.func1(0x153f360, 0xc000062c38, 0x7fcc182a4328, 0xc000062c38, 0x0)
E 2019-12-23T01:32:57.917932387Z /go/src/k8s.io/kube-state-metrics/pkg/metric/generator.go:78 +0x122
E 2019-12-23T01:32:57.917935668Z k8s.io/kube-state-metrics/pkg/metrics_store.(*MetricsStore).Add(0xc0003ea600, 0x153f360, 0xc000062c38, 0x0, 0x0)
E 2019-12-23T01:32:57.917939442Z /go/src/k8s.io/kube-state-metrics/pkg/metrics_store/metrics_store.go:76 +0xf6
E 2019-12-23T01:32:57.917942736Z k8s.io/kube-state-metrics/pkg/metrics_store.(*MetricsStore).Replace(0xc0003ea600, 0xc00000c340, 0x2, 0x2, 0xc000366100, 0x9, 0x17ad70c, 0x219a560)
E 2019-12-23T01:32:57.917946210Z /go/src/k8s.io/kube-state-metrics/pkg/metrics_store/metrics_store.go:138 +0xa4
E 2019-12-23T01:32:57.917949488Z k8s.io/client-go/tools/cache.(*Reflector).syncWith(0xc0003f8540, 0xc00000c320, 0x2, 0x2, 0xc000366100, 0x9, 0x0, 0xc000180c00)
E 2019-12-23T01:32:57.917952841Z /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:354 +0xf8
E 2019-12-23T01:32:57.917956143Z k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1(0xc0003f8540, 0xc000389d80, 0xc0003d43c0, 0xc0004dfbb8, 0x0, 0x0)
E 2019-12-23T01:32:57.917959399Z /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:250 +0x8fa
E 2019-12-23T01:32:57.917963333Z k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0003f8540, 0xc0003d43c0, 0x0, 0x0)
E 2019-12-23T01:32:57.917966597Z /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:257 +0x1a9
E 2019-12-23T01:32:57.917969894Z k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
E 2019-12-23T01:32:57.917972968Z /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:155 +0x33
E 2019-12-23T01:32:57.917976290Z k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0003a8f78)
E 2019-12-23T01:32:57.917979410Z /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152 +0x5e
E 2019-12-23T01:32:57.917982668Z k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0004dff78, 0x3b9aca00, 0x0, 0x1, 0xc0003d43c0)
E 2019-12-23T01:32:57.917985871Z /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153 +0xf8
E 2019-12-23T01:32:57.917989119Z k8s.io/apimachinery/pkg/util/wait.Until(...)
E 2019-12-23T01:32:57.917992192Z /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88
E 2019-12-23T01:32:57.917995402Z k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0003f8540, 0xc0003d43c0)
E 2019-12-23T01:32:57.917998525Z /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:154 +0x16b
E 2019-12-23T01:32:57.918004951Z created by k8s.io/kube-state-metrics/internal/store.(*Builder).reflectorPerNamespace
E 2019-12-23T01:32:57.918008276Z /go/src/k8s.io/kube-state-metrics/internal/store/builder.go:337 +0x265
E 2019-12-23T01:32:57.921970200Z panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x128190e]
goroutine 43 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:55 +0x105
panic(0x13c4de0, 0x2178810)
/usr/local/go/src/runtime/panic.go:679 +0x1b2
k8s.io/kube-state-metrics/internal/store.wrapHPAFunc.func1(0x153f360, 0xc000062c38, 0xc00011b170)
/go/src/k8s.io/kube-state-metrics/internal/store/hpa.go:293 +0xbe
k8s.io/kube-state-metrics/pkg/metric.(*FamilyGenerator).Generate(...)
/go/src/k8s.io/kube-state-metrics/pkg/metric/generator.go:39
k8s.io/kube-state-metrics/pkg/metric.ComposeMetricGenFuncs.func1(0x153f360, 0xc000062c38, 0x7fcc182a4328, 0xc000062c38, 0x0)
/go/src/k8s.io/kube-state-metrics/pkg/metric/generator.go:78 +0x122
k8s.io/kube-state-metrics/pkg/metrics_store.(*MetricsStore).Add(0xc0003ea600, 0x153f360, 0xc000062c38, 0x0, 0x0)
/go/src/k8s.io/kube-state-metrics/pkg/metrics_store/metrics_store.go:76 +0xf6
k8s.io/kube-state-metrics/pkg/metrics_store.(*MetricsStore).Replace(0xc0003ea600, 0xc00000c340, 0x2, 0x2, 0xc000366100, 0x9, 0x17ad70c, 0x219a560)
/go/src/k8s.io/kube-state-metrics/pkg/metrics_store/metrics_store.go:138 +0xa4
k8s.io/client-go/tools/cache.(*Reflector).syncWith(0xc0003f8540, 0xc00000c320, 0x2, 0x2, 0xc000366100, 0x9, 0x0, 0xc000180c00)
/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:354 +0xf8
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1(0xc0003f8540, 0xc000389d80, 0xc0003d43c0, 0xc0004dfbb8, 0x0, 0x0)
/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:250 +0x8fa
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0003f8540, 0xc0003d43c0, 0x0, 0x0)
/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:257 +0x1a9
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:155 +0x33
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0003a8f78)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152 +0x5e
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0004dff78, 0x3b9aca00, 0x0, 0x1, 0xc0003d43c0)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(...)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0003f8540, 0xc0003d43c0)
/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:154 +0x16b
created by k8s.io/kube-state-metrics/internal/store.(*Builder).reflectorPerNamespace
/go/src/k8s.io/kube-state-metrics/internal/store/builder.go:337 +0x265
What you expected to happen:
Run normally.
How to reproduce it (as minimally and precisely as possible):
I just ran it as usual after changing the image tag from v1.8.0 to v1.9.0.
Anything else we need to know?:
Environment:
- Kubernetes version (use
kubectl version): v1.15.4-gke.22 - Kube-state-metrics image version: v1.9.0
Deployment YAML for Kube state metrics
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "14"
creationTimestamp: "2019-05-27T01:46:37Z"
generation: 14
labels:
app: prometheus
chart: prometheus-9.3.2
component: kube-state-metrics
heritage: Tiller
release: prometheus
name: prometheus-kube-state-metrics
namespace: core
resourceVersion: "123555910"
selfLink: /apis/apps/v1/namespaces/core/deployments/prometheus-kube-state-metrics
uid: 4446bfd1-8021-11e9-89af-42010a01fe08
spec:
progressDeadlineSeconds: 2147483647
replicas: 1
revisionHistoryLimit: 2147483647
selector:
matchLabels:
app: prometheus
component: kube-state-metrics
release: prometheus
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: prometheus
chart: prometheus-9.3.2
component: kube-state-metrics
heritage: Tiller
release: prometheus
spec:
containers:
- image: quay.io/coreos/kube-state-metrics:v1.8.0
imagePullPolicy: IfNotPresent
name: prometheus-kube-state-metrics
ports:
- containerPort: 8080
name: metrics
protocol: TCP
resources:
limits:
memory: 300Mi
requests:
cpu: 100m
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
priorityClassName: essential
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
runAsNonRoot: true
runAsUser: 65534
serviceAccount: prometheus-kube-state-metrics
serviceAccountName: prometheus-kube-state-metrics
terminationGracePeriodSeconds: 30
status:
availableReplicas: 1
conditions:
- lastTransitionTime: "2019-05-27T01:46:37Z"
lastUpdateTime: "2019-05-27T01:46:37Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
observedGeneration: 14
readyReplicas: 1
replicas: 1
updatedReplicas: 1
kinbug
lawliet89
👍13
All 13 comments
I am getting the exact same error on my end.
natalysheinin
on 23 Dec 2019
Any confirmation from maintainer on this? We are considering upgrading too.
dongwangdw
on 24 Dec 2019
It's the Christmas/new year period. Give the maintainers some time until after the new year for a response.
lawliet89
on 24 Dec 2019
🎉4
❤3
👍3
The PR that should fix it has been merged in the 1.9 branch, @lawliet89 can you verify the fix? Thanks!
Also thanks @tariq1890 for the quick fix on this! 馃帀
@dongwangdw You can already build the image from the release branch or wait until the release candidate to be cut, but that won't happen until next year, as most of us are on PTO. :)
lilic
on 30 Dec 2019
Not sure if I did anything wrong:
- Checked out
release-1.9branch makemake container
Then I tagged the image built to lawliet89/kube-state-metrics:v1.9.0-gh1012
And I got more segfaults
I 2019-12-31T01:36:27.253562Z Using default collectors
I 2019-12-31T01:36:27.253665Z Using all namespace
I 2019-12-31T01:36:27.253687Z metric white-blacklisting: blacklisting the following items:
W 2019-12-31T01:36:27.253756Z Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I 2019-12-31T01:36:27.256858Z Testing communication with server
I 2019-12-31T01:36:27.264867Z Running with Kubernetes cluster version: v1.15+. git version: v1.15.4-gke.22. git tree state: clean. commit: a6ba43f5a24ac29e631bb627c9b2a719c4e93638. platform: linux/amd64
I 2019-12-31T01:36:27.264915Z Communication with server successful
I 2019-12-31T01:36:27.265255Z Starting metrics server: 0.0.0.0:8080
I 2019-12-31T01:36:27.265418Z Starting kube-state-metrics self metrics server: 0.0.0.0:8081
I 2019-12-31T01:36:27.265596Z Autosharding disabled
I 2019-12-31T01:36:27.268408Z Active collectors: certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,validatingwebhookconfigurations,volumeattachments
E 2019-12-31T01:36:27.288431Z pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:108: Failed to list *v1beta1.VolumeAttachment: volumeattachments.storage.k8s.io is forbidden: User "system:serviceaccount:core:prometheus-kube-state-metrics" cannot list resource "volumeattachments" in API group "storage.k8s.io" at the cluster scope
E 2019-12-31T01:36:27.290131Z Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
E 2019-12-31T01:36:27.290427856Z goroutine 55 [running]:
E 2019-12-31T01:36:27.290432905Z k8s.io/apimachinery/pkg/util/runtime.logPanic(0x13c4de0, 0x2178810)
E 2019-12-31T01:36:27.290436625Z /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:74 +0xa3
E 2019-12-31T01:36:27.290440492Z k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
E 2019-12-31T01:36:27.290444032Z /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:48 +0x82
E 2019-12-31T01:36:27.290447698Z panic(0x13c4de0, 0x2178810)
E 2019-12-31T01:36:27.290451012Z /usr/local/go/src/runtime/panic.go:679 +0x1b2
E 2019-12-31T01:36:27.290454338Z k8s.io/kube-state-metrics/internal/store.wrapHPAFunc.func1(0x153f360, 0xc0000b4000, 0xc000728ae0)
E 2019-12-31T01:36:27.290457692Z /go/src/k8s.io/kube-state-metrics/internal/store/hpa.go:298 +0xbe
E 2019-12-31T01:36:27.290481433Z k8s.io/kube-state-metrics/pkg/metric.(*FamilyGenerator).Generate(...)
E 2019-12-31T01:36:27.290485602Z /go/src/k8s.io/kube-state-metrics/pkg/metric/generator.go:39
E 2019-12-31T01:36:27.290489Z k8s.io/kube-state-metrics/pkg/metric.ComposeMetricGenFuncs.func1(0x153f360, 0xc0000b4000, 0x7efc1c3e85f0, 0xc0000b4000, 0x0)
E 2019-12-31T01:36:27.290492600Z /go/src/k8s.io/kube-state-metrics/pkg/metric/generator.go:78 +0x122
E 2019-12-31T01:36:27.290496068Z k8s.io/kube-state-metrics/pkg/metrics_store.(*MetricsStore).Add(0xc000432440, 0x153f360, 0xc0000b4000, 0x0, 0x0)
E 2019-12-31T01:36:27.290499440Z /go/src/k8s.io/kube-state-metrics/pkg/metrics_store/metrics_store.go:76 +0xf6
E 2019-12-31T01:36:27.290502938Z k8s.io/kube-state-metrics/pkg/metrics_store.(*MetricsStore).Replace(0xc000432440, 0xc000726c00, 0x2, 0x2, 0xc000055db0, 0x9, 0x302d962, 0x219a560)
E 2019-12-31T01:36:27.290506420Z /go/src/k8s.io/kube-state-metrics/pkg/metrics_store/metrics_store.go:138 +0xa4
E 2019-12-31T01:36:27.290509803Z k8s.io/client-go/tools/cache.(*Reflector).syncWith(0xc000452480, 0xc000726be0, 0x2, 0x2, 0xc000055db0, 0x9, 0x0, 0xc0004820c0)
E 2019-12-31T01:36:27.290513168Z /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:354 +0xf8
E 2019-12-31T01:36:27.290516483Z k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1(0xc000452480, 0xc00012c680, 0xc0004341e0, 0xc0004cdbb8, 0x0, 0x0)
E 2019-12-31T01:36:27.290519818Z /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:250 +0x8fa
E 2019-12-31T01:36:27.290524310Z k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc000452480, 0xc0004341e0, 0x0, 0x0)
E 2019-12-31T01:36:27.290527815Z /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:257 +0x1a9
E 2019-12-31T01:36:27.290531206Z k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
E 2019-12-31T01:36:27.290534446Z /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:155 +0x33
E 2019-12-31T01:36:27.290537906Z k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc00019c778)
E 2019-12-31T01:36:27.290541168Z /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152 +0x5e
E 2019-12-31T01:36:27.290544507Z k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0004cdf78, 0x3b9aca00, 0x0, 0x1, 0xc0004341e0)
E 2019-12-31T01:36:27.290554115Z /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153 +0xf8
E 2019-12-31T01:36:27.290557792Z k8s.io/apimachinery/pkg/util/wait.Until(...)
E 2019-12-31T01:36:27.290561054Z /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88
E 2019-12-31T01:36:27.290564442Z k8s.io/client-go/tools/cache.(*Reflector).Run(0xc000452480, 0xc0004341e0)
E 2019-12-31T01:36:27.290567767Z /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:154 +0x16b
E 2019-12-31T01:36:27.290584587Z created by k8s.io/kube-state-metrics/internal/store.(*Builder).reflectorPerNamespace
E 2019-12-31T01:36:27.290644592Z /go/src/k8s.io/kube-state-metrics/internal/store/builder.go:337 +0x265
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x128197e]
goroutine 55 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:55 +0x105
panic(0x13c4de0, 0x2178810)
/usr/local/go/src/runtime/panic.go:679 +0x1b2
k8s.io/kube-state-metrics/internal/store.wrapHPAFunc.func1(0x153f360, 0xc0000b4000, 0xc000728ae0)
/go/src/k8s.io/kube-state-metrics/internal/store/hpa.go:298 +0xbe
k8s.io/kube-state-metrics/pkg/metric.(*FamilyGenerator).Generate(...)
/go/src/k8s.io/kube-state-metrics/pkg/metric/generator.go:39
k8s.io/kube-state-metrics/pkg/metric.ComposeMetricGenFuncs.func1(0x153f360, 0xc0000b4000, 0x7efc1c3e85f0, 0xc0000b4000, 0x0)
/go/src/k8s.io/kube-state-metrics/pkg/metric/generator.go:78 +0x122
k8s.io/kube-state-metrics/pkg/metrics_store.(*MetricsStore).Add(0xc000432440, 0x153f360, 0xc0000b4000, 0x0, 0x0)
/go/src/k8s.io/kube-state-metrics/pkg/metrics_store/metrics_store.go:76 +0xf6
k8s.io/kube-state-metrics/pkg/metrics_store.(*MetricsStore).Replace(0xc000432440, 0xc000726c00, 0x2, 0x2, 0xc000055db0, 0x9, 0x302d962, 0x219a560)
/go/src/k8s.io/kube-state-metrics/pkg/metrics_store/metrics_store.go:138 +0xa4
k8s.io/client-go/tools/cache.(*Reflector).syncWith(0xc000452480, 0xc000726be0, 0x2, 0x2, 0xc000055db0, 0x9, 0x0, 0xc0004820c0)
/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:354 +0xf8
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1(0xc000452480, 0xc00012c680, 0xc0004341e0, 0xc0004cdbb8, 0x0, 0x0)
/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:250 +0x8fa
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc000452480, 0xc0004341e0, 0x0, 0x0)
/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:257 +0x1a9
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:155 +0x33
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc00019c778)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152 +0x5e
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0004cdf78, 0x3b9aca00, 0x0, 0x1, 0xc0004341e0)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(...)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc000452480, 0xc0004341e0)
/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:154 +0x16b
created by k8s.io/kube-state-metrics/internal/store.(*Builder).reflectorPerNamespace
/go/src/k8s.io/kube-state-metrics/internal/store/builder.go:337 +0x265
Thanks for testing the release-1.9 branch @lawliet89. It looks like we are unable to extract an int/int64 value from a resource.Quantity object when generating the kube_hpa_status_current_metrics_average_value metric. When we fail to extract numeric value, we end up adding a nil metric element to the pre-allocated metric family slice which is leading to a nil-dereference panic.
Would it be possible for you to set a debug point starting here and find the case where we don't seem to get the int value?
tariq1890
on 31 Dec 2019
I'm not very familiar with debugging in Go so I'll try something out on Thursday after New Year.
In the mean time, I will post the HPA I have in the cluster that might be causing issues. It makes use of external metrics.
HPA
{
"kind": "HorizontalPodAutoscaler",
"apiVersion": "autoscaling/v2beta1",
"metadata": {
"name": "hpa-app-api-staging",
"namespace": "app-staging",
"selfLink": "/apis/autoscaling/v2beta1/namespaces/app-staging/horizontalpodautoscalers/hpa-app-api-staging",
"uid": "265faff2-17f6-11ea-97d1-42010a01fe06",
"resourceVersion": "126681030",
"creationTimestamp": "2019-12-06T07:00:56Z",
"labels": {
"app.kubernetes.io/instance": "app-api-staging",
"app.kubernetes.io/managed-by": "Tiller",
"app.kubernetes.io/name": "app-api",
"helm.sh/chart": "app-api-0.11.21"
}
},
"spec": {
"scaleTargetRef": {
"kind": "Deployment",
"name": "app-api-staging",
"apiVersion": "apps/v1"
},
"minReplicas": 2,
"maxReplicas": 5,
"metrics": [
{
"type": "Resource",
"resource": {
"name": "memory",
"targetAverageUtilization": 75
}
},
{
"type": "External",
"external": {
"metricName": "traefik_backend_requests_per_second",
"targetAverageValue": "100"
}
},
{
"type": "Resource",
"resource": {
"name": "cpu",
"targetAverageUtilization": 80
}
}
]
},
"status": {
"lastScaleTime": "2019-12-31T02:34:20Z",
"currentReplicas": 2,
"desiredReplicas": 2,
"currentMetrics": [
{
"type": "Resource",
"resource": {
"name": "memory",
"currentAverageUtilization": 28,
"currentAverageValue": "847775744"
}
},
{
"type": "External",
"external": {
"metricName": "traefik_backend_requests_per_second",
"currentValue": "0",
"currentAverageValue": "2900m"
}
},
{
"type": "Resource",
"resource": {
"name": "cpu",
"currentAverageUtilization": 6,
"currentAverageValue": "62m"
}
}
],
"conditions": [
{
"type": "AbleToScale",
"status": "True",
"lastTransitionTime": "2019-12-19T02:37:49Z",
"reason": "ReadyForNewScale",
"message": "recommended size matches current size"
},
{
"type": "ScalingActive",
"status": "True",
"lastTransitionTime": "2019-12-30T05:21:42Z",
"reason": "ValidMetricFound",
"message": "the HPA was able to successfully calculate a replica count from memory resource utilization (percentage of request)"
},
{
"type": "ScalingLimited",
"status": "True",
"lastTransitionTime": "2019-12-31T02:39:28Z",
"reason": "TooFewReplicas",
"message": "the desired replica count is increasing faster than the maximum scale rate"
}
]
}
}
External Metric:
{
"kind": "ExternalMetricValueList",
"apiVersion": "external.metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/traefik_backend_requests_per_second"
},
"items": [
{
"metricName": "",
"metricLabels": {
"backend": "admin-api-docs.domain.com/"
},
"timestamp": "2019-12-31T04:13:36Z",
"value": "0"
},
{
"metricName": "",
"metricLabels": {
"backend": "admin-api.domain.com"
},
"timestamp": "2019-12-31T04:13:36Z",
"value": "0"
},
{
"metricName": "",
"metricLabels": {
"backend": "api-docs.domain.com/"
},
"timestamp": "2019-12-31T04:13:36Z",
"value": "0"
},
{
"metricName": "",
"metricLabels": {
"backend": "api.domain.com"
},
"timestamp": "2019-12-31T04:13:36Z",
"value": "0"
},
{
"metricName": "",
"metricLabels": {
"backend": "jaeger.domain.com"
},
"timestamp": "2019-12-31T04:13:36Z",
"value": "0"
},
{
"metricName": "",
"metricLabels": {
"backend": "prom-alertmanager.domain.com/"
},
"timestamp": "2019-12-31T04:13:36Z",
"value": "0"
},
{
"metricName": "",
"metricLabels": {
"backend": "traefik-external.domain.com"
},
"timestamp": "2019-12-31T04:13:36Z",
"value": "0"
},
{
"metricName": "",
"metricLabels": {
"backend": "traefik-internal.domain.com"
},
"timestamp": "2019-12-31T04:13:36Z",
"value": "0"
},
{
"metricName": "",
"metricLabels": {
"backend": "domain.com"
},
"timestamp": "2019-12-31T04:13:36Z",
"value": "0"
},
{
"metricName": "",
"metricLabels": {
"backend": "grafana.domain.com/"
},
"timestamp": "2019-12-31T04:13:36Z",
"value": "0"
},
{
"metricName": "",
"metricLabels": {
"backend": "museum.domain.com/"
},
"timestamp": "2019-12-31T04:13:36Z",
"value": "0"
},
{
"metricName": "",
"metricLabels": {
"backend": "prom.domain.com/"
},
"timestamp": "2019-12-31T04:13:36Z",
"value": "5933m"
}
]
}
FYI I am seeing the exact same behavior, however It's only effecting 1/5 shards (index 2 in case it matters, though it is probably just based on a metric that is selected by the hash).
Kube 1.16.2 KSM 1.9.0 (image: quay.io/coreos/kube-state-metrics:v1.9.0)
For reference here is my trace:
I1231 18:28:34.403162 1 main.go:86] Using default collectors
I1231 18:28:34.403243 1 main.go:98] Using all namespace
I1231 18:28:34.403259 1 main.go:139] metric white-blacklisting: blacklisting the following items:
W1231 18:28:34.403291 1 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I1231 18:28:34.405851 1 main.go:184] Testing communication with server
I1231 18:28:34.423265 1 main.go:189] Running with Kubernetes cluster version: v1.16. git version: v1.16.2. git tree state: clean. commit: c97fe5036ef3df2967d086711e6c0c405941e14b. platform: linux/amd64
I1231 18:28:34.423300 1 main.go:191] Communication with server successful
I1231 18:28:34.423476 1 main.go:225] Starting metrics server: 0.0.0.0:8080
I1231 18:28:34.423652 1 main.go:200] Starting kube-state-metrics self metrics server: 0.0.0.0:8081
I1231 18:28:34.423677 1 metrics_handler.go:96] Autosharding disabled
I1231 18:28:34.423689 1 metrics_handler.go:79] configuring sharding of this instance to be shard index 2 (zero-indexed) out of 5 total shards
I1231 18:28:34.426446 1 builder.go:146] Active collectors: certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,validatingwebhookconfigurations,volumeattachments
E1231 18:28:34.619135 1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 63 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x13c4de0, 0x2178810)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:74 +0xa3
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:48 +0x82
panic(0x13c4de0, 0x2178810)
/usr/local/go/src/runtime/panic.go:679 +0x1b2
k8s.io/kube-state-metrics/internal/store.wrapHPAFunc.func1(0x153f360, 0xc0014a4040, 0xc000cb89f0)
/go/src/k8s.io/kube-state-metrics/internal/store/hpa.go:293 +0xbe
k8s.io/kube-state-metrics/pkg/metric.(*FamilyGenerator).Generate(...)
/go/src/k8s.io/kube-state-metrics/pkg/metric/generator.go:39
k8s.io/kube-state-metrics/pkg/metric.ComposeMetricGenFuncs.func1(0x153f360, 0xc0014a4040, 0x7f2b85269010, 0xc0014a4040, 0x0)
/go/src/k8s.io/kube-state-metrics/pkg/metric/generator.go:78 +0x122
k8s.io/kube-state-metrics/pkg/metrics_store.(*MetricsStore).Add(0xc0004fc280, 0x153f360, 0xc0014a4040, 0x0, 0x0)
/go/src/k8s.io/kube-state-metrics/pkg/metrics_store/metrics_store.go:76 +0xf6
k8s.io/kube-state-metrics/pkg/metrics_store.(*MetricsStore).Replace(0xc0004fc280, 0xc0015fe000, 0x16b, 0x16b, 0x0, 0x0, 0xd5b01ba, 0x219a560)
/go/src/k8s.io/kube-state-metrics/pkg/metrics_store/metrics_store.go:138 +0xa4
k8s.io/client-go/tools/cache.(*Reflector).syncWith(0xc000510300, 0xc00013d800, 0x16b, 0x16b, 0x0, 0x0, 0x0, 0xc0001941e0)
/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:354 +0xf8
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1(0xc000510300, 0xc0005ca000, 0xc0001dca20, 0xc001445bb8, 0x0, 0x0)
/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:250 +0x8fa
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc000510300, 0xc0001dca20, 0x0, 0x0)
/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:257 +0x1a9
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:155 +0x33
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0004e1778)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152 +0x5e
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc001445f78, 0x3b9aca00, 0x0, 0x1, 0xc0001dca20)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(...)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc000510300, 0xc0001dca20)
/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:154 +0x16b
created by k8s.io/kube-state-metrics/internal/store.(*Builder).reflectorPerNamespace
/go/src/k8s.io/kube-state-metrics/internal/store/builder.go:337 +0x265
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x128190e]
goroutine 63 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:55 +0x105
panic(0x13c4de0, 0x2178810)
/usr/local/go/src/runtime/panic.go:679 +0x1b2
k8s.io/kube-state-metrics/internal/store.wrapHPAFunc.func1(0x153f360, 0xc0014a4040, 0xc000cb89f0)
/go/src/k8s.io/kube-state-metrics/internal/store/hpa.go:293 +0xbe
k8s.io/kube-state-metrics/pkg/metric.(*FamilyGenerator).Generate(...)
/go/src/k8s.io/kube-state-metrics/pkg/metric/generator.go:39
k8s.io/kube-state-metrics/pkg/metric.ComposeMetricGenFuncs.func1(0x153f360, 0xc0014a4040, 0x7f2b85269010, 0xc0014a4040, 0x0)
/go/src/k8s.io/kube-state-metrics/pkg/metric/generator.go:78 +0x122
k8s.io/kube-state-metrics/pkg/metrics_store.(*MetricsStore).Add(0xc0004fc280, 0x153f360, 0xc0014a4040, 0x0, 0x0)
/go/src/k8s.io/kube-state-metrics/pkg/metrics_store/metrics_store.go:76 +0xf6
k8s.io/kube-state-metrics/pkg/metrics_store.(*MetricsStore).Replace(0xc0004fc280, 0xc0015fe000, 0x16b, 0x16b, 0x0, 0x0, 0xd5b01ba, 0x219a560)
/go/src/k8s.io/kube-state-metrics/pkg/metrics_store/metrics_store.go:138 +0xa4
k8s.io/client-go/tools/cache.(*Reflector).syncWith(0xc000510300, 0xc00013d800, 0x16b, 0x16b, 0x0, 0x0, 0x0, 0xc0001941e0)
/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:354 +0xf8
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1(0xc000510300, 0xc0005ca000, 0xc0001dca20, 0xc001445bb8, 0x0, 0x0)
/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:250 +0x8fa
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc000510300, 0xc0001dca20, 0x0, 0x0)
/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:257 +0x1a9
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:155 +0x33
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0004e1778)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152 +0x5e
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc001445f78, 0x3b9aca00, 0x0, 0x1, 0xc0001dca20)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(...)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc000510300, 0xc0001dca20)
/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:154 +0x16b
created by k8s.io/kube-state-metrics/internal/store.(*Builder).reflectorPerNamespace
/go/src/k8s.io/kube-state-metrics/internal/store/builder.go:337 +0x265
SleepyBrett
on 31 Dec 2019
@lawliet89 Thanks for sharing your HPA details. That does help with replicating this issue. We are currently working on getting a bugfix out to solve this issue completely.
tariq1890
on 6 Jan 2020
👍2
@LiliC I've built #1019 and pushed it to lawliet89/kube-state-metrics:v1.9.0-gh1019 and it seems to work fine. I'll close this issue for now. 馃憤
lawliet89
on 10 Jan 2020
👍2
❤1
@lawliet89 Thanks so much for testing! 馃帀
lilic
on 10 Jan 2020
I'm seeing this with quay.io/coreos/kube-state-metrics:v1.9.1
invertigo
on 27 Feb 2020
@invertigo there were additional fixes added in the later versions, please try with latest release and open a new issue if you are still experiencing this problem, thanks.
lilic
on 27 Feb 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
terinjokes
路
4Comments
lilic
路
3Comments
jingzhao-self
路
4Comments
mikekap
路
8Comments
sayarg
路
4Comments
Most helpful comment
It's the Christmas/new year period. Give the maintainers some time until after the new year for a response.