Minikube: RBAC is broken

[gouthamve]

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Minikube version (use minikube version): v0.20.0

Environment:

• OS (e.g. from /etc/os-release): Mac
• VM Driver: virtualbox
• ISO version: minikube-v0.20.0.iso

What happened:
Enabled RBAC via --extra-config=apiserver.Authorization.Mode=RBAC.

kubedns fails:

E0724 10:00:32.313124       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: User "system:serviceaccount:kube-system:default" cannot list services at the cluster scope. (get services)
E0724 10:00:32.442709       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: User "system:serviceaccount:kube-system:default" cannot list endpoints at the cluster scope. (get endpoints)
E0724 10:00:32.444056       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.ConfigMap: User "system:serviceaccount:kube-system:default" cannot list configmaps in the namespace "kube-system". (get configmaps)
I0724 10:00:32.496263       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...

What you expected to happen:
It works and everything spins up successfully.

How to reproduce it (as minimally and precisely as possible):
Run minikube with RBAC and k8s version 1.7.0

[r2d4]

The kube-dns addon we package isn't configured to use RBAC. If you want to use RBAC, you need to supply your own addons, (that means turning off the addons with minikube addons disable ... and providing your own configuration.

[r2d4]

We are tracking enabling RBAC or bundling RBAC-enabled addons in #1722

[chancez]

You could also just give the kube-system service account admin permissions, then everything should work.

[mcwienczek]

Thank you @chancez for giving direction. For anyone that doesn't want to google it anymore, sample configuration below.

after applying these rules minikube works again with RBAC:
source: https://github.com/screwdriver-cd-test/config-examples
http://blog.screwdriver.cd/post/161863341372/set-up-screwdriver-in-kubernetes

# Wide open access to the cluster (mostly for kubelet)
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: cluster-writer
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["*"]
  - nonResourceURLs: ["*"]
    verbs: ["*"]

---

# Full read access to the api and resources
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: cluster-reader
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["get", "list", "watch"]
  - nonResourceURLs: ["*"]
    verbs: ["*"]
---
# Give admin, kubelet, kube-system, kube-proxy god access
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: cluster-write
subjects:
  - kind: User
    name: admin
  - kind: User
    name: kubelet
  - kind: ServiceAccount
    name: default
    namespace: kube-system
  - kind: User
    name: kube-proxy
roleRef:
  kind: ClusterRole
  name: cluster-writer
  apiGroup: rbac.authorization.k8s.io

---

# Setup sd-build as a reader. This has to be a
# ClusterRoleBinding to get access to non-resource URLs
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: cluster-read
subjects:
  - kind: ServiceAccount
    name: sd-build
    namespace: default
roleRef:
  kind: ClusterRole
  name: cluster-reader
  apiGroup: rbac.authorization.k8s.io

---

# Setup sd-build as a writer in its namespace
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: sd-build-write
subjects:
  - kind: ServiceAccount
    name: sd-build
    namespace: default
roleRef:
  kind: ClusterRole
  name: cluster-writer
  apiGroup: rbac.authorization.k8s.io

[wallrj]

Note that upstream kube-dns now configure the controller to use a separate service account:

• https://github.com/kubernetes/kubernetes/pull/38816

Perhaps Minikube should do the same.