Aks-engine: Dashboard permission errors from `kubectl proxy` on K8s 1.10.3 cluster deployed from acs-engine master

Created on 31 May 2018 · 27Comments · Source: Azure/aks-engine

Is this an ISSUE or FEATURE REQUEST?

Issue

What version of acs-engine?

Version: canary
GitCommit: a1fe789
GitTreeState: dirty

(I had one change to the Makefile so that make dev works on my system with a strange net config)

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)
Kubernetes

What happened:

The dashboard at http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/overview?namespace=default doesn't seem to have any permissions for the service account it's running as.

configmaps is forbidden: User "system:serviceaccount:kube-system:kubernetes-dashboard" cannot list configmaps in the namespace "default" 
close 
warning
persistentvolumeclaims is forbidden: User "system:serviceaccount:kube-system:kubernetes-dashboard" cannot list persistentvolumeclaims in the namespace "default" 
close 
warning
secrets is forbidden: User "system:serviceaccount:kube-system:kubernetes-dashboard" cannot list secrets in the namespace "default" 
close 
warning
services is forbidden: User "system:serviceaccount:kube-system:kubernetes-dashboard" cannot list services in the namespace "default" 
close 
warning
ingresses.extensions is forbidden: User "system:serviceaccount:kube-system:kubernetes-dashboard" cannot list ingresses.extensions in the namespace "default" 
close 
warning
daemonsets.apps is forbidden: User "system:serviceaccount:kube-system:kubernetes-dashboard" cannot list daemonsets.apps in the namespace "default" 
close 
warning
pods is forbidden: User "system:serviceaccount:kube-system:kubernetes-dashboard" cannot list pods in the namespace "default" 
close 
warning
events is forbidden: User "system:serviceaccount:kube-system:kubernetes-dashboard" cannot list events in the namespace "default" 
close 
warning
deployments.apps is forbidden: User "system:serviceaccount:kube-system:kubernetes-dashboard" cannot list deployments.apps in the namespace "default" 
close 
warning
replicasets.apps is forbidden: User "system:serviceaccount:kube-system:kubernetes-dashboard" cannot list replicasets.apps in the namespace "default" 
close 
warning
jobs.batch is forbidden: User "system:serviceaccount:kube-system:kubernetes-dashboard" cannot list jobs.batch in the namespace "default" 
close 
warning
cronjobs.batch is forbidden: User "system:serviceaccount:kube-system:kubernetes-dashboard" cannot list cronjobs.batch in the namespace "default" 
close 
warning
replicationcontrollers is forbidden: User "system:serviceaccount:kube-system:kubernetes-dashboard" cannot list replicationcontrollers in the namespace "default" 
close 
warning
statefulsets.apps is forbidden: User "system:serviceaccount:kube-system:kubernetes-dashboard" cannot list statefulsets.apps in the namespace "default"

What you expected to happen:

Dashboard should work, be able to show nodes, pods, and so on

How to reproduce it (as minimally and precisely as possible):

Create a JSON file with

"orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorVersion": "1.10.3"
    },

acs-engine generate, deploy it
Copy .kube/config from master node
kubectl proxy
browse to http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/overview?namespace=default

Source

PatrickLang

👍17

Most helpful comment

Run kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard
Everything worked for me.

kairok on 25 Aug 2018

👍134 ❤35 🎉28 😄22 🚀21 👀7 👎1

All 27 comments

Hmm, the role only seems to have access to heapster

$ kubectl -n kube-system describe rolebinding kubernetes-dashboard-minimal

Name:         kubernetes-dashboard-minimal
Labels:       addonmanager.kubernetes.io/mode=Reconcile
              k8s-app=kubernetes-dashboard
              kubernetes.io/cluster-service=true
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"rbac.authorization.k8s.io/v1","kind":"Role
Binding","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile",...
Role:
  Kind:  Role
  Name:  kubernetes-dashboard-minimal
Subjects:
  Kind            Name                  Namespace
  ----            ----                  ---------
  ServiceAccount  kubernetes-dashboard  kube-system

$ kubectl -n kube-system describe role kubernetes-dashboard-minimal
Name:         kubernetes-dashboard-minimal
Labels:       addonmanager.kubernetes.io/mode=Reconcile
              k8s-app=kubernetes-dashboard
              kubernetes.io/cluster-service=true
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"rbac.authorization.k8s.io/v1","kind":"Role
","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile","k8s-ap...
PolicyRule:
  Resources       Non-Resource URLs  Resource Names                     Verbs
  ---------       -----------------  --------------                     -----
  configmaps      []                 []                                 [create]
  configmaps      []                 [kubernetes-dashboard-settings]    [get update]
  secrets         []                 []                                 [create]
  secrets         []                 [kubernetes-dashboard-key-holder]  [get update delete]
  services        []                 [heapster]                         [proxy]
  services/proxy  []                 [heapster]                         [get]
  services/proxy  []                 [http:heapster:]                   [get]
  services/proxy  []                 [https:heapster:]                  [get]

PatrickLang on 31 May 2018

Applying the binding described at https://github.com/kubernetes/dashboard/wiki/Access-control#admin-privileges fixed it

PatrickLang on 31 May 2018

👍29

Looks like #2816 added RBAC for k8s 1.8+

The config for 1.7, 1.8 adds a ClusterRoleBinding of cluster-admin to Kubernetes-dashboard. The non-versioned one used for 1.9+ does not bind cluster-admin, instead Kubernetes-dashboard-minimal.

PatrickLang on 1 Jun 2018

👍1

This is so broken...

Cloudmersive on 2 Jul 2018

😄5

https://blogs.msdn.microsoft.com/kennethteo/2018/07/26/aks-with-aad/ hmm this shows a binding based on an AAD user. might work with acs-engine

PatrickLang on 8 Aug 2018

Run kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard
Everything worked for me.

kairok on 25 Aug 2018

👍134 ❤35 🎉28 😄22 🚀21 👀7 👎1

Alright - I did a lot more reading here. It seems that creating a service account and using a token is the right thing to do:
https://github.com/kubernetes/dashboard/wiki/Creating-sample-user

Binding kube-system:kubernetes-dashboard to the cluster-admin role means that anyone or anything that can access the dashboard has full control over the cluster.

PatrickLang on 31 Aug 2018

👍14 🎉1 👎1

It's not working on AKS as well.
@kairok thank you for a fix!!!

SychevIgor on 12 Sep 2018

This only happened to me when RBAC was enabled. Below fixed it

Run kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard
Everything worked for me.

ccit-spence on 12 Oct 2018

👍10 🚀1

kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard

its work for me

mgwulay on 9 Nov 2018

👍35 🎉5 😄1 👎1

Thank you mgwulay. This works for me.

JMagerram on 16 Feb 2019

@kairok thank you for a fix!!!

sklrsn on 1 Mar 2019

This looks fixed

jackfrancis on 21 Mar 2019

It's only fixed after running the above command though.

joshuataylor on 6 May 2019

👍3

Run kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard
Everything worked for me.

Fantastic. It worked for me too... Thanks :)

sujyot on 27 May 2019

@joshuataylor thank you for adding that important bit of information :) so that folks coming to this thread from older clusters have a clear path

jackfrancis on 28 May 2019

For reference: this is (now) documented in https://github.com/kubernetes/dashboard/wiki/Access-control#admin-privileges

Rick-Jongbloed on 22 Jun 2019

Run kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard
Everything worked for me.

that worked for me

glglak on 4 Aug 2019

lol pretty disgraceful this is broken over a year later.

KyleSanderson on 7 Sep 2019

👎4

nothing is working for me after trying all above

alkadevops on 24 Oct 2019

@alkadevops
Installed the Dashboard with Ingress just a few seconds ago, worked perfectly.
May delete you serviceaccount and try again: https://medium.com/@kanrangsan/creating-admin-user-to-access-kubernetes-dashboard-723d6c9764e4

BR Bernd

Berndinox on 25 Oct 2019

As it is an issue hit by so many ppl, although it's closed with some fix, this doc (or others) does not tell the customers about what to do or what happens.
https://docs.microsoft.com/en-us/dotnet/architecture/containerized-lifecycle/design-develop-containerized-apps/build-aspnet-core-applications-linux-containers-aks-kubernetes

gynet on 14 Feb 2020

Run kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard
Everything worked for me.

This! Solved my problem. Thank you!

paulohenriquesi on 25 Mar 2020

👍2

Run kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard
Everything worked for me.

Assigning "cluster-admin" role is not solution for every case, especially if you are building kubernetes multitenancy model

theazureguy007

CloudA2Z-Code on 6 Apr 2020

Run kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard

Nope. I ran this and I still can't get the permissions of an admin no matter what. Definitely not solved.
Thanks.

stampeder on 17 Apr 2020

👀1 👍1

Sorry for posting on closed issue, but IMHO you should never link a service account to a Cluster-admin role.
The Privilege is just way to big, it's like God on your cluster and if anyone uses it, that person can potentially destroy all ressources within your cluster.

You should use something like this instead:

kind: ClusterRole 
apiVersion: rbac.authorization.k8s.io/v1beta1 
metadata: 
  name: k8s-dashboard-clusterrole 
  namespace: kube-system
rules: 
  - apiGroups: 
      - "*" 
    resources: 
      - "*" 
    verbs: 
      - "list"
      - "get" 
      - "watch" 

--- 
kind: ClusterRoleBinding 
apiVersion: rbac.authorization.k8s.io/v1beta1 
metadata: 
  name: k8s-dashboard-clusterrole 
roleRef: 
  apiGroup: rbac.authorization.k8s.io 
  kind: ClusterRole 
  name: k8s-dashboard-clusterrole 
subjects: 
- kind: ServiceAccount 
  name: kubernetes-dashboard
  namespace: kube-system

This provides the service account kubernetes-dashboard the ability to List, get and watch all ressources on all API but can't modify anything. Hope it helps