Contour: Cannot deploy ClusterRoles on GKE 1.8.1 Cluster

[erasmus74]

Seems I run into this issue, after cluster is up, this is the first thing I tried. Let me know what you need to help me debug this.

$ kubectl get nodes
NAME                                         STATUS    ROLES     AGE       VERSION
gke-gke-cluster-default-pool-97bc52ee-076l   Ready     <none>    19h       v1.8.1-gke.1
gke-gke-cluster-default-pool-97bc52ee-b0wh   Ready     <none>    19h       v1.8.1-gke.1
gke-gke-cluster-default-pool-97bc52ee-g42r   Ready     <none>    19h       v1.8.1-gke.1
$ kubectl apply -f http://j.hept.io/contour-deployment-rbac
namespace "heptio-contour" created
serviceaccount "contour" created
deployment "contour" created
clusterrolebinding "contour" created
service "contour" created
Error from server (Forbidden): error when creating "http://j.hept.io/contour-deployment-rbac": clusterroles.rbac.authorization.k8s.io "contour" is forbidden: attempt to grant extra privileges: [PolicyRule{Resources:["configmaps"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["configmaps"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["endpoints"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["endpoints"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["nodes"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["nodes"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["secrets"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["secrets"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["nodes"], APIGroups:[""], Verbs:["get"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["get"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["ingresses"], APIGroups:["extensions"], Verbs:["get"]} PolicyRule{Resources:["ingresses"], APIGroups:["extensions"], Verbs:["list"]} PolicyRule{Resources:["ingresses"], APIGroups:["extensions"], Verbs:["watch"]}] user=&{[email protected]  [system:authenticated] map[]} ownerrules=[PolicyRule{Resources:["selfsubjectaccessreviews"], APIGroups:["authorization.k8s.io"], Verbs:["create"]} PolicyRule{NonResourceURLs:["/api" "/api/*" "/apis" "/apis/*" "/healthz" "/swaggerapi" "/swaggerapi/*" "/version"], Verbs:["get"]}] ruleResolutionErrors=[]
$ kubectl delete -f http://j.hept.io/contour-deployment-rbac
namespace "heptio-contour" deleted
serviceaccount "contour" deleted
deployment "contour" deleted
clusterrolebinding "contour" deleted
service "contour" deleted
Error from server (NotFound): error when deleting "http://j.hept.io/contour-deployment-rbac": clusterroles.rbac.authorization.k8s.io "contour" not found

[erasmus74]

Further reading seems I run into this issue.
https://honeycomb.io/docs/connect/kubernetes/gke/

It would also seem this is going to be the norm soon, and users will need to bind to roles that allow the required priv escalation. https://kubernetes.io/docs/admin/authorization/rbac/#privilege-escalation-prevention-and-bootstrapping

Still unsure how to make this work "out-of-box" yet.

[davecheney]

Hi @erasmus74

The root of the problem is GKE does not enabled RBAC if you choose the Legacy Authentication option -- which is the default when you create a GKE cluster.

The solution is to use the non GKE example

kubectl apply -f http://j.hept.io/contour-deployment-norbac

I'm going to close this, but please reopen it if you need further assistance.

Thanks

Dave

[erasmus74]

I did enable RBAC.

On Wed, Nov 8, 2017 at 6:43 PM, Dave Cheney [e-mail]
wrote:

Hi @erasmus74 https://github.com/erasmus74

The root of the problem is GKE does not enabled RBAC if you choose the Legacy
Authentication option -- which is the default when you create a GKE
cluster.

The solution is to use the non GKE example
https://github.com/heptio/contour#add-contour-to-your-cluster

kubectl apply -f http://j.hept.io/contour-deployment-norbac

I'm going to close this, but please reopen it if you need further
assistance.

Thanks

Dave

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/heptio/contour/issues/36#issuecomment-342999255, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AHd0fjUnL_jnBgVGVymstXaiRurRqb0aks5s0jyhgaJpZM4QU7oU
.

--
Regards,
Edward S

[davecheney]

Thanks for your reply. Looking at the response you posted that matches the error I see when I try to deploy the rbac example to a non rbac enbabled cluster.

If you use the non rbac version

kubectl apply -f http://j.hept.io/contour-deployment-norbac

You'll be fine.

[colemickens]

I'm seeing the same behavior with a GKE cluster running 1.9.2-gke.0. RBAC is enabled.

+ kubectl apply -f https://j.hept.io/contour-deployment-rbac
namespace "heptio-contour" created
serviceaccount "contour" created
deployment "contour" created
clusterrolebinding "contour" created
service "contour" created

Error from server (Forbidden): error when creating "https://j.hept.io/contour-deployment-rbac": clusterroles.rbac.authorization.k8s.io "contour" is forbidden: attempt to grant extra privileges: [PolicyRule{Resources:["configmaps"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["configmaps"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["endpoints"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["endpoints"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["nodes"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["nodes"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["secrets"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["secrets"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["nodes"], APIGroups:[""], Verbs:["get"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["get"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["ingresses"], APIGroups:["extensions"], Verbs:["get"]} PolicyRule{Resources:["ingresses"], APIGroups:["extensions"], Verbs:["list"]} PolicyRule{Resources:["ingresses"], APIGroups:["extensions"], Verbs:["watch"]}] user=&{[email protected] [system:authenticated] map[]} ownerrules=[PolicyRule{Resources:["selfsubjectaccessreviews" "selfsubjectrulesreviews"], APIGroups:["authorization.k8s.io"], Verbs:["create"]} PolicyRule{NonResourceURLs:["/api" "/api/*" "/apis" "/apis/*" "/healthz" "/swagger-2.0.0.pb-v1" "/swagger.json" "/swaggerapi" "/swaggerapi/*" "/version"], Verbs:["get"]}] ruleResolutionErrors=[]

Also, I was able to install cert-manager with rbac enabled.

I have installed helm as follows:

kubectl create serviceaccount -n kube-system tiller
kubectl create clusterrolebinding tiller-binding --clusterrole=cluster-admin --serviceaccount kube-system:tiller
helm init --service-account tiller

[colemickens]

I've found my mistake. Contour isn't using helm. The default RBAC for users in GKE is insufficiently privileged to deploy this. I've resolved this by creating a role binding to 'cluster-admin' for my user.

[davecheney]

Yeah, there is a magic incantation you need to do to allow the gke admin user to create cluster roles.

I kinda see this as a gke problem, but I’d be happy to be convinced otherwise, especially if this is something we can fix in our deployment yaml.

On 7 Feb 2018, at 15:33, Cole Mickens [e-mail] wrote:

I've found my mistake. Contour isn't using helm. The default RBAC for users in GKE is insufficiently privileged to deploy this. I've resolved this by creating a role binding to 'cluster-admin' for my user.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub, or mute the thread.

[marcwuk]

What is the magic incantation for GKE. Can we add this to the install guide.

[colemickens]

@marcwuk https://cloud.google.com/kubernetes-engine/docs/how-to/role-based-access-control#prerequisites_for_using_role-based_access_control

[nicovogelaar]

I had the same problem on GKE 1.8.8 with RBAC enabled. This helped me: https://github.com/kubernetes-incubator/external-dns/issues/514

[davecheney]

I’d be glad of a PR they links to this obscure part of the GCE documentation. I hold GCE entirely responsible for this UX paper cut, but until thy fix it (I’ve no idea if the can or will) then we should at least make it possible for people to find the answer they need.

On 21 Apr 2018, at 20:44, Nico Vogelaar [e-mail] wrote:

I had the same problem on GKE 1.8.8 with RBAC enabled. This helped me: kubernetes-incubator/external-dns#514

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub, or mute the thread.

[djensen47]

What's the downside of not using RBAC?

[davecheney]

The downside of not using RBAC is anyone who has credentials to use your
cluster has admin on your cluster. That may be ok if its just a cluster for
experimentation, but as a company, Heptio have decided that we won't
support or recommend non RBAC deployments.

On 3 May 2018 at 00:51, Dave Jensen [e-mail] wrote:

What's the downside of not using RBAC?

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/heptio/contour/issues/36#issuecomment-386145620, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcA8j3rgLliWViKSmpftuNcmA7xBS3ks5tujh2gaJpZM4QU7oU
.