Kops: Issues adding RBAC support

[amitkgupta]

Use Case

I'd like to provision and maintain an up-to-date Kubernetes cluster on AWS, and provide it to my colleagues so they can experiment with using it to deploy applications. I don't want them provisioning ELBs within my account so I'd like to give them non-admin access to create and manage resources except ELBs, and I'd also like to apply resource quota limits to what they can do. I'd also like to configure access dynamically through the API (no SSH'ing in, editing files, restarting the entire cluster, etc.), hence the desire to use RBAC (see also #203, #312).

Purpose of this issue

• Have a record of current work in progress, and confirm my understanding on the remaining work required.
• Ask some questions to address confusion I ran into doing my current work in progress
• Highlight some blockers to completing this work
• Offer to complete the work assuming my understanding of the remaining work is correct and the blockers can be fixed.

Current work

I simply added:

AuthorizationMode          string `json:"authorizationMode,omitempty" flag:"authorization-mode"`
AuthorizationRbacSuperUser string `json:"authorizationRbacSuperUser,omitempty" flag:"authorization-rbac-super-user"`

to both componentconfig.go and v1alpha1/componentconfig.go. I was then able to make and kops edit cluster, setting the auth mode to RBAC,AlwaysAllow, the super user to kubecfg, and adding runtime config with rbac.authorization.k8s.io/v1alpha1: "true". I was able to kops update cluster, saving the configuration in the state store, however this didn't actually update the real cluster, I suspect due to #925.

Question: When I didn't make these changes to both files, I got some strange errors when running kops edit cluster, e.g. complaints about the vi editor. My cluster config mentions Version: v1alpha1 at the top. Why did I need to udpate both of those source files? Why do the two even exist?

Question: Does this seem correct so far? If this were a new cluster, would you expect this to just work?

Remaining work

I included AlwaysAllow because I figure all the control plane components using service accounts would break otherwise. My plan was to then use the RBAC API to grant sufficient permissions to those service accounts, then edit my cluster again to only have RBAC as the auth mode, and update the cluster. Does this sound correct?

Because it's probably not safe to directly switch a running Kubernetes cluster from AlwaysAllow to RBAC, or even bootstrap a new cluster with just RBAC, I wouldn't add any actual flags to the kops commands.

Question: So the remaining work would simply document the procedure to enable RBAC, probably in docs/cluster_spec.md, right?

My personal next steps would be to:

• create a namespace
• apply quota limits to that namespace
• figure out if it's possible to restrict the ability to create ELBs at either the namespace, user, or group level
• use the RBAC API to create a user or group that only has access to that namespace, and hopefully can't create ELBs,
• issue client certs and keys to my colleagues signed by the CA key stored in my $KOPS_STATE_STORE with the appropriate users/groups in the cert metadata.

Question: Does it even seem likely that there's a fine-grained way to restrict ability to create ELBs? Where is the right place to look for documentation on this?

Blockers

So far, I think it's just the one mentioned above:

• #925: Making changes to the cluster configuration in the KubeAPIServer section seems to have no effect, and so it doesn't work with kops {rolling-,}update cluster if the cluster already exists. Maybe it only works for fresh clusters?

[amitkgupta]

Yup, not long term, just in the interim while I use RBAC API to give the
service accounts sufficient permissions. Because if I go straight from
AlwaysAllow to RBAC without an interim having both, everything except the
super user won't be able to do anything. In particular I assume I won't
even be able to access any APIs served by components that rely on
functioning service account tokens.

On Mon, Dec 12, 2016 at 3:42 PM Keith Mitchell [e-mail]
wrote:

I don't think setting both RBAC and AlwaysAllow will be useful. Per the k8s
docs http://kubernetes.io/docs/admin/authorization/:

If multiple modes are provided the set is unioned, and only a single
authorizer is required to admit the action. This means the flag:
--authorization-mode=AlwaysDeny,AlwaysAllow will always allow.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/kops/issues/1130#issuecomment-266589708,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABgeadtcF7ugCD3SPncMHdOBB_A18X2dks5rHdvQgaJpZM4LKEiq
.

[chrislovecnm]

How does this relate to https://github.com/kubernetes/kops/pull/896

I appreciate you patience, since it took a bit to respond

[Amit-PivotalLabs]

Added some inline questions to #896

[chrislovecnm]

@kris-nova I am going to assign this to you.

[itskingori]

@amitkgupta were you able to to get this working at all?

spec:
+  kubeAPIServer:
+    authorizationMode: RBAC,AlwaysAllow
+    authorizationRbacSuperUser: admin
+    runtimeConfig:
+      rbac.authorization.k8s.io/v1alpha1: "true"

[Amit-PivotalLabs]

@itskingori sort of. I had to make 2 changes to the version of kops I was using at the time: https://github.com/kubernetes/kops/commit/6e267a3c6bdab17e9eb10e2340e33217135c09cb and https://github.com/kubernetes/kops/commit/9b2e87970804d6d4a84d387fcceddc96d4991ebf. Then it still didn't work, you can see details of that here: https://github.com/kubernetes/kubernetes/issues/38997

[sstarcher]

@itskingori It's not a good idea to use AlwaysAllow. That will allow it to skip RBAC. From the kubernetes documentation

If multiple modes are provided the set is unioned, and only a single authorizer is required to admit the action. This means the flag:

https://kubernetes.io/docs/admin/authorization/

[itskingori]

@sstarcher yes ... this allows you to bootstrap your cluster and create the policies. Ideally, once done you should be able to remove the AlwaysAllow and restart.

[bobbychef64]

how to enable rbac on an existing k8 cluster. Is it possible ?

[jurgenweber]

a quote from the slack channel, assuming you are kube 1.6.

justinsb [2:08 PM] @jurgen we don't enable RBAC by default, but you can change `authorization` from `alwaysAllow: {}` to `rbac: {}`

[henning]

@jurgenweber
I just tried this "switching from alwaysAllow to rbac" via "kops edit config" and then did update cluster --yes and roling-update --yes - but the apiserver is still started with --authorization-mode=AlwaysAllow

I didn't had to ask in Slack, the docs suggest that this should be possible - but seems not so.

@bobbychef64 did you try it and have you been successful somehow?

UPDATE:

That worked after I ensured that I had the latest kops version installed!

The other one was maybe a month or two old, did let me change the settings, not complain about an unknown or unsupported setting,
but seemed to do nothing to apply the changes authorization to the cluster...

[jurgenweber]

worked for me, no problems. I have RBAC, etc.

[KashifSaadat]

@amitkgupta @chrislovecnm looks like this has been resolved, can we close the issue?

[chrislovecnm]

please let me know if we should re-open