Gardener: Project Tolerations to support limiting access to Seed usage

[rfranzke]

What would you like to be added:
Today, the Seed can have several taints specified in the .spec.taints[] list. One of them is seed.gardener.cloud/protected. If set, only shoots in the garden namespace are allowed to specify the seed in the .spec.seedName. However, the garden namespace is hard-coded (see #401) which is not good. Also, there is a new use case where specific seeds shall be restricted to specific (multiple) projects only.

Let's adapt our Project resource with a new .spec.tolerations[] field:

---
apiVersion: core.gardener.cloud/v1beta1
kind: Project
metadata:
  name: my-project
spec:
  ...
  tolerations:
  - key: <some-key>

If all taints of a Seed are tolerated by the Project then the Shoot is allowed to use it. Today, we also have other taints (invisible, disable-dns, excess-capacity). These are more settings of the seed and don't fit to the taint/toleration concept. Hence, we should move them out of the .spec.taints[] to a new .spec.settings[] field.

Now a Seed can have the <seed-name>.seed.gardener.cloud/protected taint. If the project has the <seed-name>.seed.gardener.cloud/protected toleration then a Shoot is allowed to be scheduled on it.

In order to prevent that end-users (which are having update,patch privileges) add arbitrary tolerations we should use a custom RBAC verb. Only with it it would be allowed to change the .spec.tolerations field:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: gardener.cloud:project-tolerations
rules:
- apiGroups:
  - core.gardener.cloud
  resources:
  - projects
  verbs:
  - tolerations
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: gardener.cloud:project-tolerations
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: gardener.cloud:project-tolerations
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: gardener-admin

The scheduler and our admission should be enhanced to respect these new taints/tolerations concepts.

PS: In a future iteration we could think about introducing a new namespaced Seed resource, but this is out of scope of this issue.

Why is this needed:
Exclude projects from using various seeds to make them exclusive to some of them.

[timuthy]

Something that I'm a bit concerned about is that the Project is the only resource which carries Tolerations. For me it sounds like the namespace of a pod defined the tolerations instead of the pod itself. What about if we follow the PodTolerationRestriction admission plugin which would mean that the project defines a list of possible tolerations but the shoot is the actual resource which defines the toleration?

[rfranzke]

Sure, sounds great, thanks @timuthy. I see that with the PodTolerationRestriction it's possible to specify default tolerations. This would be helpful such that not every Shoot needs to specify certain tolerations for protected seeds they want to use.

[rfranzke]

After @timuthy's proposal I checked again the PodTolerationRestriction admission plugin and would adapt above proposal as follows:

---
apiVersion: core.gardener.cloud/v1beta1
kind: Project
metadata:
  name: my-project
spec:
  ...
  tolerations:
    defaults:
    - key: <some-key>
    whiteList:
    - key: <some-key>  

While the PodTolerationRestriction is using annotations on the Namespace our ShootTolerationRestriction could use .spec.tolerations in the Project resource.

• The .spec.tolerations.defaults can be edited by everybody, but only those tolerations that are part of .spec.tolerations.whiteList may be added (like in the PodTolerationRestriction).
• The .spec.tolerations.whiteList can only be edited by users having the spec-tolerations-whitelist custom verb for project resources (similarly like outlined above).