Kubeadm: Heapster default deployment does not support RBAC in v1.6.0 setup
Is this a BUG REPORT or FEATURE REQUEST?
BUG I suppose
What happened?
The default Heapster manifests available at:
https://github.com/kubernetes/heapster/tree/master/deploy/kube-config/influxdb
Are not supported for an RBAC setup, the heapster pod does not have access to read the metrics:
E0419 11:33:26.390515 1 reflector.go:203] k8s.io/heapster/metrics/sources/kubelet/kubelet.go:342: Failed to list *api.Node: the server does not allow access to the requested resource (get nodes)
E0419 11:33:26.390664 1 reflector.go:203] k8s.io/heapster/metrics/heapster.go:319: Failed to list *api.Pod: the server does not allow access to the requested resource (get pods)
E0419 11:33:26.390818 1 reflector.go:203] k8s.io/heapster/metrics/processors/node_autoscaling_enricher.go:100: Failed to list *api.Node: the server does not allow access to the requested resource (get nodes)
E0419 11:33:26.390940 1 reflector.go:203] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: Failed to list *api.Namespace: the server does not allow access to the requested resource (get namespaces)
E0419 11:33:27.389735 1 reflector.go:203] k8s.io/heapster/metrics/heapster.go:327: Failed to list *api.Node: the server does not allow access to the requested resource (get nodes)
E0419 11:33:27.403494 1 reflector.go:203] k8s.io/heapster/metrics/sources/kubelet/kubelet.go:342: Failed to list *api.Node: the server does not allow access to the requested resource (get nodes)
E0419 11:33:27.403603 1 reflector.go:203] k8s.io/heapster/metrics/heapster.go:319: Failed to list *api.Pod: the server does not allow access to the requested resource (get pods)
E0419 11:33:27.403605 1 reflector.go:203] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: Failed to list *api.Namespace: the server does not allow access to the requested resource (get namespaces)
E0419 11:33:27.403683 1 reflector.go:203] k8s.io/heapster/metrics/processors/node_autoscaling_enricher.go:100: Failed to list *api.Node: the server does not allow access to the requested resource (get nodes)
What you expected to happen?
that it worked out of the box by following the heapster deployment guide
How to reproduce it (as minimally and precisely as possible)?
Deploy the manifests in
https://github.com/kubernetes/heapster/tree/master/deploy/kube-config/influxdb
sebgoa
All 11 comments
it is an RBAC issue.
Working manifest at:
https://github.com/luxas/kubeadm-workshop/blob/master/demos/monitoring/heapster.yaml
but we need to have the default manifest updated and documented...
sebgoa
on 19 Apr 2017
cc @luxas
xilabao
on 20 Apr 2017
Another option is adding a clusterrolebinding for the heapster account to have the cluster-admin role. This should only be used in a test environment since it grants permission to the entire kube-system namespace.
kubectl create clusterrolebinding add-on-cluster-admin \
--clusterrole=cluster-admin \
--serviceaccount=kube-system:default
carlosedp
on 25 Apr 2017
Sorry @carlosedp, but that will give all kube-system Pods root access to the cluster which is nearly as turning of RBAC completely which is _extremely_ discouraged.
cc @liggitt @jbeda
The real fix for this problem is to create a ServiceAccount for heapster and grant the system:heapster ClusterRole to that ServiceAccount with a ClusterRoleBinding.
Please see: https://github.com/kubernetes/heapster/issues/1612
luxas
on 25 Apr 2017
Added a comment to my original one regarding this. This solution was added to the RBAC documentation as a temporary fix while migrating to RBAC.
carlosedp
on 25 Apr 2017
@sebgoa I fixed the heapster manifests: https://github.com/kubernetes/heapster/issues/1612
@carlosedp Thanks for editing the comment to say "test env only". The problem is that dev/test methods are sometimes likely to slip into production, therefore my relatively strong statement.
I hope you understand, I know the RBAC change is hard and kind of disruptive for apps since we're basically going from allow-all to deny-all. Please bear with us in this change towards a more Kubernetes.
luxas
on 28 Apr 2017
Actually official one has below ClusterRoleBinding created
https://raw.githubusercontent.com/luxas/heapster/1308dd71f0ba343895456b46d1bbf3238800b6f3/deploy/kube-config/rbac/heapster-rbac.yaml
why not put in one yaml? Thanks
~
your version was working as it had ClusterRoleBinding, the official fix still complains
~
k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: Failed to list *v1.Namespace: User "system:serviceaccount:kube-system:heapster" cannot list namespaces at the cluster scope. (get namespaces)
zhuroy
on 28 Apr 2017
@zhuroy It's expected that you apply both heapster.yaml and heapster-rbac.yaml if you use RBAC
luxas
on 28 Apr 2017
Hello,
using the suggested heapster.yaml file fixes the data collection in the dashboard, but it's not working with the horizontal pod autoscaler.
When I'm checking the HPA resource I'm getting this:
Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 34m 7s 70 horizontal-pod-autoscaler Warning FailedGetResourceMetric missing request for cpu on container ghost in pod default/ghost-2663835528-169fs 34m 7s 70 horizontal-pod-autoscaler Warning FailedComputeMetricsReplicas failed to get cpu utilization: missing request for cpu on container ghost in pod default/ghost-2663835528-169fs
Is this also an RBAC issue?
Thank you!
Laszlo
lbudai74
on 12 Jun 2017
Hi all,
Any updates to this issue, I still have this error in both version of 1.8.8 and 1.9.3.
E0514 07:51:32.974038 1 reflector.go:190] k8s.io/heapster/metrics/util/util.go:30: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:heapster" cannot list nodes at the cluster scope
E0514 07:51:32.980050 1 reflector.go:190] k8s.io/heapster/metrics/util/util.go:30: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:heapster" cannot list nodes at the cluster scope
E0514 07:51:32.981074 1 reflector.go:190] k8s.io/heapster/metrics/heapster.go:328: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:heapster" cannot list pods at the cluster scope
E0514 07:51:32.982340 1 reflector.go:190] k8s.io/heapster/metrics/util/util.go:30: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:heapster" cannot list nodes at the cluster scope
E0514 07:51:32.983079 1 reflector.go:190] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:89: Failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:kube-system:heapster" cannot list namespaces at the cluster scope
I0514 07:52:05.076147 1 influxdb.go:274] Created database "k8s" on influxDB server at "monitoring-influxdb.kube-system.svc:8086"
And HPA does not work neither.
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetResourceMetric 3m (x301 over 2h) horizontal-pod-autoscaler unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)
antran89
on 14 May 2018
the system:heapster role includes the permissions required by heapster:
whatever deploys heapster is responsible for binding that role to the heapster service account. for example:
liggitt
on 14 May 2018
Related issues
tmjd
路
4Comments
bruceauyeung
路
4Comments
chuckha
路
3Comments
cnmade
路
4Comments
helphi
路
3Comments
Most helpful comment
it is an RBAC issue.
Working manifest at:
https://github.com/luxas/kubeadm-workshop/blob/master/demos/monitoring/heapster.yaml
but we need to have the default manifest updated and documented...