Kube-state-metrics: Cannot Create Role / Cluster Role

[chasemaier]

I am attempting to apply the contents of the kubernetes directory to a cluster (formed via kube-aws on AWS).

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.6", GitCommit:"7fa1c1756d8bc963f1a389f4a6937dc71f08ada2", GitTreeState:"clean", BuildDate:"2017-06-16T18:34:20Z", GoVersion:"go1.7.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.6+coreos.1", GitCommit:"42a5c8b99c994a51d9ceaed5d0254f177e97d419", GitTreeState:"clean", BuildDate:"2017-06-21T01:10:07Z", GoVersion:"go1.7.6", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl apply -f roles/kube-state-metrics/

clusterrolebinding "kube-state-metrics" configured
deployment "kube-state-metrics" configured
rolebinding "kube-state-metrics" configured
serviceaccount "kube-state-metrics" configured
service "kube-state-metrics" configured
Error from server (Forbidden): error when creating "roles/kube-state-metrics/kube-state-metrics-cluster-role.yaml": clusterroles.rbac.authorization.k8s.io "kube-state-metrics" is forbidden: attempt to grant extra privileges: [{[list] [] [nodes] [] []} {[watch] [] [nodes] [] []} {[list] [] [pods] [] []} {[watch] [] [pods] [] []} {[list] [] [services] [] []} {[watch] [] [services] [] []} {[list] [] [resourcequotas] [] []} {[watch] [] [resourcequotas] [] []} {[list] [] [replicationcontrollers] [] []} {[watch] [] [replicationcontrollers] [] []} {[list] [] [limitranges] [] []} {[watch] [] [limitranges] [] []} {[list] [] [persistentvolumeclaims] [] []} {[watch] [] [persistentvolumeclaims] [] []} {[list] [extensions] [daemonsets] [] []} {[watch] [extensions] [daemonsets] [] []} {[list] [extensions] [deployments] [] []} {[watch] [extensions] [deployments] [] []} {[list] [extensions] [replicasets] [] []} {[watch] [extensions] [replicasets] [] []} {[list] [apps] [statefulsets] [] []} {[watch] [apps] [statefulsets] [] []} {[list] [batch] [cronjobs] [] []} {[watch] [batch] [cronjobs] [] []} {[list] [batch] [jobs] [] []} {[watch] [batch] [jobs] [] []}] user=&{kube-admin  [kube-aws system:authenticated] map[]} ownerrules=[] ruleResolutionErrors=[]
Error from server (Forbidden): error when creating "roles/kube-state-metrics/kube-state-metrics-role.yaml": roles.rbac.authorization.k8s.io "kube-state-metrics-resizer" is forbidden: attempt to grant extra privileges: [{[get] [] [pods] [] []} {[get] [extensions] [deployments] [kube-state-metrics] []} {[update] [extensions] [deployments] [kube-state-metrics] []}] user=&{kube-admin  [kube-aws system:authenticated] map[]} ownerrules=[] ruleResolutionErrors=[]

The pods from the defined deployment seem to run despite the roles failing to create, but the roles are not created successfully.

$ kubectl logs -f kube-state-metrics-2331359496-ltkvj --namespace=kube-system -c kube-state-metrics
I1010 20:27:09.850349       1 main.go:154] Using default collectors
I1010 20:27:09.850677       1 main.go:202] service account token present: true
I1010 20:27:09.850692       1 main.go:203] service host: https://10.3.0.1:443
I1010 20:27:09.851385       1 main.go:229] Testing communication with server
I1010 20:27:10.052076       1 main.go:234] Communication with server successful
I1010 20:27:10.052328       1 main.go:279] Active collectors: deployments,limitranges,replicationcontrollers,statefulsets,services,cronjobs,persistentvolumeclaims,pods,replicasets,jobs,daemonsets,nodes,resourcequotas
I1010 20:27:10.052343       1 main.go:243] Starting metrics server: :8080

Is there another way the roles should be applied to ensure the pods have access to all the available metrics?

[brancz]

My guess is that you are on GKE, and this is likely the cause: https://coreos.com/operators/prometheus/docs/latest/troubleshooting.html

I believe GKE still falls back to legacy ABAC instead of deny-by-default RBAC, which is why it still runs successfully.

[KieranP]

@brancz That was the case. Thanks for the link. Might pay to pop this in the README, I imagine a lot of people, like myself, are using GKE and would experience this same issue.

[brancz]

@KieranP are you up for making this your first contribution to the Kubernetes org? :wink:

[KieranP]

@brancz Done :-) https://github.com/kubernetes/kube-state-metrics/pull/326

[brancz]

@KieranP the PR is merged, congrats on your first contribution to Kubernetes.

I’ll close this here. Happy holidays.