Kube-state-metrics: Segfault with 1.9.2

[natalysheinin]

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:

Kube state metrics pod crashloops with a segfault.

I0121 18:12:25.922394       1 main.go:89] Using collectors cronjobs,daemonsets,deployments,horizontalpodautoscalers,jobs,nodes,persistentvolumeclaims,persistentvolumes,resourcequotas,statefulsets,verticalpodautoscalers
I0121 18:12:25.922465       1 main.go:98] Using all namespace
I0121 18:12:25.922478       1 main.go:139] metric white-blacklisting: blacklisting the following items:
W0121 18:12:25.922514       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0121 18:12:25.928459       1 main.go:184] Testing communication with server
I0121 18:12:25.938362       1 main.go:189] Running with Kubernetes cluster version: v1.11+. git version: v1.11.10-gke.4. git tree state: clean. commit: 767331edaa4a96ad2fb42eade03ab45199d883f7. platform: linux/amd64
I0121 18:12:25.938405       1 main.go:191] Communication with server successful
I0121 18:12:25.938584       1 main.go:225] Starting metrics server: 0.0.0.0:8080
I0121 18:12:25.938686       1 main.go:200] Starting kube-state-metrics self metrics server: 0.0.0.0:8081
I0121 18:12:25.938773       1 metrics_handler.go:96] Autosharding disabled
I0121 18:12:25.939181       1 builder.go:146] Active collectors: cronjobs,daemonsets,deployments,horizontalpodautoscalers,jobs,nodes,persistentvolumeclaims,persistentvolumes,resourcequotas,statefulsets,verticalpodautoscalers
E0121 18:12:25.947319       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 56 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x13c4de0, 0x2178810)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:74 +0xa3
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:48 +0x82
panic(0x13c4de0, 0x2178810)
    /usr/local/go/src/runtime/panic.go:679 +0x1b2
k8s.io/kube-state-metrics/internal/store.glob..func81(0xc000160000, 0xc000266c60)
    /go/src/k8s.io/kube-state-metrics/internal/store/hpa.go:284 +0x1d4
k8s.io/kube-state-metrics/internal/store.wrapHPAFunc.func1(0x153f360, 0xc000160000, 0xc0001168a0)
    /go/src/k8s.io/kube-state-metrics/internal/store/hpa.go:300 +0x5c
k8s.io/kube-state-metrics/pkg/metric.(*FamilyGenerator).Generate(...)
    /go/src/k8s.io/kube-state-metrics/pkg/metric/generator.go:39
k8s.io/kube-state-metrics/pkg/metric.ComposeMetricGenFuncs.func1(0x153f360, 0xc000160000, 0x7f0b6ff5a1e8, 0xc000160000, 0x0)
    /go/src/k8s.io/kube-state-metrics/pkg/metric/generator.go:78 +0x122
k8s.io/kube-state-metrics/pkg/metrics_store.(*MetricsStore).Add(0xc0003b4600, 0x153f360, 0xc000160000, 0x0, 0x0)
    /go/src/k8s.io/kube-state-metrics/pkg/metrics_store/metrics_store.go:76 +0xf6
k8s.io/kube-state-metrics/pkg/metrics_store.(*MetricsStore).Replace(0xc0003b4600, 0xc0000aff20, 0x3, 0x3, 0xc0003760a0, 0x9, 0x2a25b14, 0x219a560)
    /go/src/k8s.io/kube-state-metrics/pkg/metrics_store/metrics_store.go:138 +0xa4
k8s.io/client-go/tools/cache.(*Reflector).syncWith(0xc0003c0780, 0xc0000afef0, 0x3, 0x3, 0xc0003760a0, 0x9, 0x0, 0xc00018a720)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:354 +0xf8
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1(0xc0003c0780, 0xc000398d80, 0xc0003da480, 0xc000363bb8, 0x0, 0x0)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:250 +0x8fa
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0003c0780, 0xc0003da480, 0x0, 0x0)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:257 +0x1a9
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:155 +0x33
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000198778)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152 +0x5e
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000363f78, 0x3b9aca00, 0x0, 0x1, 0xc0003da480)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(...)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0003c0780, 0xc0003da480)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:154 +0x16b
created by k8s.io/kube-state-metrics/internal/store.(*Builder).reflectorPerNamespace
    /go/src/k8s.io/kube-state-metrics/internal/store/builder.go:337 +0x265
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x12601d4]

What you expected to happen:

Run normally.

How to reproduce it (as minimally and precisely as possible):

Changed image tag to v.1.9.2
quay.io/coreos/kube-state-metrics:v1.9.2

Environment:

• Kubernetes version (use kubectl version): v1.11.10-gke.4
• Kube-state-metrics image version: quay.io/coreos/kube-state-metrics:v1.9.2

[natalysheinin]

/kind bug

[lilic]

Thanks for the bug report, we reverted the hpa metrics that were causing this problem, we will cut a new release asap.

[natalysheinin]

Thank-you @LiliC

[lilic]

@natalysheinin release is out do you mind letting us know if this fixed the issue for you, thanks!

[natalysheinin]

@LiliC Resolved with https://github.com/kubernetes/kube-state-metrics/pull/1033

[lilic]

Thanks! Closing issue.