Kube-state-metrics: panic after fixing jobs rbac

[jurgenweber]

Hi

Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"clean", BuildDate:"2017-08-31T09:14:02Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.4", GitCommit:"793658f2d7ca7f064d2bdf606519f9fe1229c381", GitTreeState:"clean", BuildDate:"2017-08-17T08:30:51Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

after upgrading to v1.0.1 from 0.4.1 I found I needed some further rbac perimissions, no problems.

Once I added jobs as below:

- apiGroups:
  - batch
  resources:
  - cronjobs
  - jobs
  verbs:
  - list
  - watch

I started receiving:

[prometheus-prometheus-kube-state-metrics-1085824839-jvcz4] panic: runtime error: invalid memory address or nil pointer dereference
[prometheus-prometheus-kube-state-metrics-1085824839-jvcz4] [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x11e0172]
[prometheus-prometheus-kube-state-metrics-1085824839-jvcz4]
[prometheus-prometheus-kube-state-metrics-1085824839-jvcz4] goroutine 136 [running]:
[prometheus-prometheus-kube-state-metrics-1085824839-jvcz4] time.time.Time.Unix(...)
[prometheus-prometheus-kube-state-metrics-1085824839-jvcz4] /usr/local/google/home/kawych/go-tools/src/k8s.io/kube-state-metrics/collectors/job.go:173
[prometheus-prometheus-kube-state-metrics-1085824839-jvcz4] k8s.io/kube-state-metrics/collectors.(*jobCollector).collectJob(0xc420394bd0, 0xc4205cdaa0, 0x0, 0x0, 0x0, 0x0, 0xc4215c0c00, 0x1e, 0x0, 0x0, ...)
[prometheus-prometheus-kube-state-metrics-1085824839-jvcz4] /usr/local/google/home/kawych/go-tools/src/k8s.io/kube-state-metrics/collectors/job.go:173 +0x1f2
[prometheus-prometheus-kube-state-metrics-1085824839-jvcz4] k8s.io/kube-state-metrics/collectors.(*jobCollector).Collect(0xc420394bd0, 0xc4205cdaa0)
[prometheus-prometheus-kube-state-metrics-1085824839-jvcz4] /usr/local/google/home/kawych/go-tools/src/k8s.io/kube-state-metrics/collectors/job.go:141 +0x12f
[prometheus-prometheus-kube-state-metrics-1085824839-jvcz4] /usr/local/google/home/kawych/go-tools/src/k8s.io/kube-state-metrics/vendor/github.com/prometheus/client_golang/prometheus/registry.go:433 +0x61
[prometheus-prometheus-kube-state-metrics-1085824839-jvcz4] created by k8s.io/kube-state-metrics/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Gather
[prometheus-prometheus-kube-state-metrics-1085824839-jvcz4] /usr/local/google/home/kawych/go-tools/src/k8s.io/kube-state-metrics/vendor/github.com/prometheus/client_golang/prometheus/registry.go:431 +0x271

If I disable the rbac permissions and just leave this

[prometheus-prometheus-kube-state-metrics-1085824839-npk9s] E0918 01:29:10.670107       1 reflector.go:201] k8s.io/kube-state-metrics/collectors/job.go:106: Failed to list *v1.Job: User "system:serviceaccount:devops:prometheus-prometheus-kube-state-metrics" cannot list jobs.batch at the cluster scope. (get jobs.batch)

in the logs, at least kube-state-metrics stops crashing.

Thanks

[brancz]

I think this is a legit panic, not sure why this is only occurring now, but the problem is that we're not nil checking here.

[andyxning]

@jurgenweber @brancz This has been fix in #248 and it is in master now.

BTW, @brancz Maybe it is time for us to do a minor release since we have added some features and fixed problems.

[brancz]

248 is already released in v1.0.1. The code snippet I linked to is from current master, I believe there is a nil panic possible there, if I'm not totally wrong. I might get to writing a test for it tomorrow.

Agreed regarding a minor release, although I'd like to get the automatic detection of available APIs in first.

[jurgenweber]

just to clarify, #248 is not in v1.0.1...... I just double checked (http://take.ms/a7FTj).

A new minor release would be appreciated.

[andyxning]

@brancz v1.0.1 does not include the #248 . Automatic detection of available APIs seems not to be prepared well in a few days. I suggest we at least make a bugfix release for #248 .

[brancz]

You're right, #248 has not been released yet, however, https://github.com/kubernetes/kube-state-metrics/blob/6a95e48df4d462da08a5e863d2bfae65aa6c9cd3/collectors/job.go#L171

Is taken directly from master and that is definitely the panic being reported here. The panic output literally says the line number.

[alamaison]

+1 for a minor release. I'm not sure if this is the source of our panic but I'd love to test it.

[andyxning]

@alamaison @jurgenweber After an offline discussion with @brancz, we want to give #257 a try before we make a minor release if it can be resolved in recent days(should not exceed this week).

[jurgenweber]

Yeah, no problems on my end. No rush.

[jurgenweber]

ping :)

[andyxning]

@jurgenweber We have released v1.1.0-rc.0. Please give it a try. :)

[andyxning]

@alamaison We have released v1.1.0-rc.0. Please give it a try. :)

[jurgenweber]

Will do either today or early next week.

[jurgenweber]

yeah, this new release looks to have reoslved it.

[brancz]

Thanks! I'll close this here, but feel free to re-open or open a new issue, if you find it occurring again.