Linkerd2: Admission Controllers don't run over the proxy and init injected containers. Check consequences.

[alpeb]

At the moment the proxy injector injects the proxy and init containers, most of the Admission Controllers have already run (on the owner ref (Deployment, StatefulSet, etc) ) and they won't run again against the injected containers.

This has forced us to, during injection:
1) Do the job of the ServiceAccount MWC, by manually mounting the token mount
2) Do the job of the LimitRanger by manually setting the resource requests/limits on the init container (and forcing people to set them for the proxy through injection flags/annotations) #2763

This issue is to track if there are other possible issues with other MWCs.

[grampelberg]

@alpeb is there a way to change the ordering at all?

[alpeb]

@grampelberg I couldn't find any docs specifying this, so after lots of k8s code scavenging I could confirm the order is hard-coded:

https://github.com/kubernetes/kubernetes/blob/5268f69405251a4a74130fa903e055a59071179a/pkg/kubeapiserver/options/plugins.go#L61-L94

The mutating webhook runs almost at last.

[ihcsim]

Other relevant issues:

https://github.com/kubernetes/kubernetes/issues/55435
https://github.com/kubernetes/kubernetes/issues/65716
https://github.com/kubernetes/kubernetes/issues/64333

[annismckenzie]

Because @alpeb just commented on my PR in our fork I'm gonna link to it here because it's relevant: alfatraining/linkerd2#1 – disabling AutomountServiceAccountToken in the pod spec will break the sidecar.

[alpeb]

I went through the list of admission controller plugins in the k8s docs, and these are the plugins that can cause problems because of the proxy injector ignoring them:

AlwaysPullImages

The proxy injector can't work with this plugin. It forces the container's ImagePullPolicy to be Always (during the controllers validation phase), which conflicts with the value IfNotPresent that we use in the injected sidecar.

PSP

The proxy injector can't work with PSP restrictions. During the mutating phase, after the containers are injected the PSP plugin won't be triggered so it won't add its restrictions into the containers. Then during the validation phase the same PSP plugin won't find those restrictions and it will deny the creation of the pod.

LimitRanger and ResourceQuota

Already handled in #2763 by making sure we have resource requests and limits set on the injected containers.

ServiceAccount

This plugin creates the volume mounts for the SA tokens. It doesn't run over the injected containers, but it doesn't matter because we manually add those mounts during injection, by copying the mount path from the main container. But if AutomountServiceAccountToken is false, the main container won't have that mount, so the injected containers can't either. In this case the only solution is to disable identity in these pods.

SecurityContextDeny

This a less configurable alternative to PSP. It forbids having securityContext.runAsUser, so not even the control plane can be installed in such an environment.

[alpeb]

I've just realized the AlwaysPullImages doesn't work with the proxy injector, so I've included it in my comment above.

[alpeb]

Looks like this will get fixed in a future Kubernetes version by allowing to re-run the mutating webhooks more than once. Mentioned under the "goals" section" of the KEP for graduating admission webhooks to GA:
https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/00xx-admission-webhooks-to-ga.md

And this is the issue tracking that specific enhancement:
https://github.com/kubernetes/kubernetes/issues/73291

[liggitt]

Looks like this will get fixed in a future Kubernetes version by allowing to re-run the mutating webhooks more than once.

this is implemented in kubernetes v1.15.

• design: https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/00xx-admission-webhooks-to-ga.md#mutating-plugin-ordering
• implementation: https://github.com/kubernetes/kubernetes/pull/78080/

the in-tree admission plugins will rerun on top of any changes mutating webhooks make, and mutating webhooks that opt-in will get called a second time if webhooks after them make further modifications

[alpeb]

Thanks @liggitt !

I successfully verified this is working for us using the branch from kubernetes/kubernetes#78505

I tested by assigning a PSP with readOnlyRootFilesystem: true to the service account attached to some sample pod spec.
In k8s v1.14, persisting that pod fails with a PSP validation error.
When using the branch referred above, persisting the pod works, and I can see readOnlyRootFilesystem: true being added into the proxy and proxy-init containers that got injected by the webhook :+1:

[liggitt]

Fantastic, thanks for the feedback

[alpeb]

I think we can close this one now. For k8s older than 1.15, #2911 provides some minimal SecurityContext values during injection that address the PSP problem.