Tf-operator: helm permission issue on 1.8.1

[jlewi]

When I try to install the operator on a 1.8.1 cluster (GKE) like so

helm install https://storage.googleapis.com/tf-on-k8s-dogfood-releases/latest/tf-job-operator-chart-latest.tgz -n tf-job --wait --replace --set cloud=gke

I get the error

Error: release tf-job failed: namespaces "default" is forbidden: User "system:serviceaccount:kube-system:default" cannot get namespaces in the namespace "default": Unknown user "system:serviceaccount:kube-system:default"

This looks like an RBAC issue. Previously I was using K8s 1.7 so I guess something changed with 1.8 which is why I'm hitting this now.

@sozercan Any idea what's going on? Is the problem that helm needs to be granted appropriate permissions as mentioned here

helm version
Client: &version.Version{SemVer:"v2.4.2", GitCommit:"82d8e9498d96535cc6787a6a9194a76161d29b4c", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.7.0", GitCommit:"08c1144f5eb3e3b636d9775617287cc26e53dba4", GitTreeState:"clean"}

[sozercan]

I am running it on k8s 1.8 with RBAC on Azure, let me check

[jlewi]

Thanks. I followed this SO post and that appears to have fixed the issue.

[jlewi]

@foxish what's the proper way to setup helm on a GKE cluster running 1.8? Should it just work or is it expected that I have to run commands like the following (from this post)

kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'      
helm init --service-account tiller --upgrade

[foxish]

@jlewi, it is expected with RBAC turned on that the permissions will need to be setup for the tiller deployment. There are steps in https://github.com/kubernetes/helm/blob/master/docs/service_accounts.md.
In addition, there may be separate permissions that the operator needs in a particular namespace, to create pods, edit pods, create configmaps/secrets, etc.

For example, with spark, we ship with some instructions and default clusterroles and clusterrolebinding configuration YAMLs.

[foxish]

Having tiller be cluster-admin is not uncommon, but it would make sense to lock down the operator itself a bit more with its own service account and namespace granting just enough permissions to operate correctly.

[sozercan]

Tiller that is bundled with Azure includes service account and role bindings (as cluster-admin). I am guessing this doesn't come with GKE?

Tfjob CRD sets up it's own serviceaccount and role bindings, so that shouldn't be an issue. Sounds like this is permissions for the tiller itself.
Maybe we can update the docs to include something like this in case it doesn't exist

kubectl create clusterrolebinding tiller-cluster-admin \
    --clusterrole=cluster-admin \
    --serviceaccount=kube-system:default

[jlewi]

@foxish Thanks. I'll send a PR adding to instructions. Just to confirm is RBAC automatically turned on in GKE with 1.8? I didn't see this mentioned in the release notes.

[bacongobbler]

Yes, RBAC is automatically turned on in GKE with kubernetes 1.8.

[bacongobbler]

See https://github.com/kubernetes/helm/issues/2962#issuecomment-331523916 for more in-depth instructions on how to configure tiller to work in a RBAC-enabled environment.

[foxish]

Sorry, yeah, didn't see your comment. Yes, RBAC turns on automatically with 1.8.

Container Engine clusters running Kubernetes version 1.8 and later disable the legacy authorization system by default, and thus role-based access control permissions take effect with no special action required.

[bacongobbler]

@sozercan's example works too, but the users should be made well aware that this means if a user has access to tiller, they may install/uninstall anything within their cluster.

[jlewi]

Thanks
I added links in #111. PTAL and comment if you are so inclined.

[naeemrashid]

I encountered the same issue. Insert these line to disable RBAC in config.yaml

rbac:
   enabled: false

[eGov-NewState]

Just these commands, it'll work

kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'
helm init --service-account tiller --upgrade

[falcocoris]

Hello

@Gajendran27

I've done just that but it doesn't seem to be enough.
Is there something I missed ?

I have a test cluster with 3 nodes and the only thing i did was helm init so far..

# kubectl create serviceaccount --namespace kube-system tiller
serviceaccount "tiller" created
# kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
clusterrolebinding "tiller-cluster-rule" created
# kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'
deployment "tiller-deploy" patched
# helm init --service-account tiller --upgrade
$HELM_HOME has been configured at /home/falcoris/.helm.

Tiller (the Helm server-side component) has been upgraded to the current version.
Happy Helming!
# helm ls
Error: configmaps is forbidden: User "system:serviceaccount:kube-system:default" cannot list configmaps in the namespace "kube-system"

[aganesan94]

Same here did that as per instructions still run into the same issue.

[falcocoris]

using these lines fixed my pb :

# kubectl create clusterrolebinding add-on-cluster-admin \
  --clusterrole=cluster-admin \
  --serviceaccount=kube-system:default
clusterrolebinding "add-on-cluster-admin" created
# helm reset     

Error: error unstalling Tiller: timed out waiting for the condition

Note the error at the end, i'm not sure this command did anything in the end but it worked with all of these.

[rnkhouse]

kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'
Error from server (BadRequest): invalid character 's' looking for beginning of object key string