Linkerd2: Design `inject` replacement using Admission Controllers

[olix0r]

Now that Kubernetes 1.9 is generally available, we should figure out how we can use the new Admission Controllers features so that users don't have to use conduit inject. This probably needs a small design document covering how this would be productionized, with special attention to user experience.

[komljen]

We should probably follow the same path like Istio https://istio.io/docs/setup/kubernetes/sidecar-injection.html#automatic-sidecar-injection and here https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.9/#mutatingwebhookconfiguration-v1beta1-admissionregistration.

When all is set the users would just need to label some resources if they want sidecar injected. In Istio it can be added as default to the namespace, but again it can be disabled or enabled for each particular resource using annotations.

[alenkacz]

I'll try to tackle this one... :)

[rmars]

@alenkacz sweet! a while back, I cobbled together a barebones admisison webhook example that doesn't do any real injecting, but can be a reference if you're trying to get started. It's at https://github.com/rmars/admission-webhook-example.

[coufalja]

Hi, any progress here? We would like to adopt Linkerd2 but this makes it a blocker for us (or more in general not being Helm friendly). I am not sure what is the expected outcome of this issue if that is a PR or design document or something else. If needed I am able to help with AdmissionWebhook work given there is some spec how it should be plugged in already.

[grampelberg]

@CoufalJa interested in helping out? The next step would be getting you paired up with someone to build out a design doc.

[coufalja]

Yep, If someone helps me through the design process then yes.

[ihcsim]

I will be interested in helping out with this issue too, if there are no takers yet. I'm in the PDT timezone, if anyone will like to collaborate.

[c4milo]

I really want to use linkerd2 but this issue seems to be a major roadblock for us. Is there a workaround to add running helm-installed services to linkerd2's service mesh?

[grampelberg]

Yup, use helm template | linkerd inject | kubectl apply.

[c4milo]

@grampelberg thanks! I tried this:

$ git clone https://github.com/helm/charts.git
$ cd stable/kubernetes-dashboard
$ helm template .  --namespace kube-system --name kube-dash | linkerd inject - | kubectl apply -f -

Unfortunately, it doesn't seem to work. I see the deployment but I don't see any pods being created.

[grampelberg]

@c4milo you missed the kubectl apply -f - on the end.

[c4milo]

@grampelberg it was copy/paste mistake in my post, I did run kubectl apply -f -

[grampelberg]

Should have worked ... might be some config issue with the chart. helm template simply renders what it tiller would have applied, that text gets piped through inject (where the sidecar gets added) and then k8s is supposed to start it all up.

[wmorgan]

@c4milo maybe you can post a sample of the final yaml after running helm template and linkerd inject?

[ihcsim]

I have been doing some poc to try to get the webhook to work with the CA controller. So far, I have gotten the proxy and proxy-init auto-injection part to work. Then I realized I probably need to configure the WebhookMutatingConfiguration rules to mutate the deployment spec (to account for the TLS identity of the pods), not just the pod spec. Will reach out again when I have something more concrete to discuss.

[akhy]

Yup, use helm template | linkerd inject | kubectl apply.

@grampelberg this should works to create/update the resources, but I doubt it'll update Helm/Tiller's release metadata as well (e.g. release number, etc) 🤔

helm template only outputs release's resources manifest, but no Helm's internal resources... that means we'll lost ability to revert Helm release among other things.

The more correct (but hacky) workaround implementation I can think of is by coupling it with helm install/upgrade command like this:

release=my-service
helm upgrade --install -f values.yaml $release stable/myapp
helm get manifest $release | linkerd inject - | kubectl apply -f -

the deployments will be updated twice though

[maorfr]

What about something like:

for files; do helm template . -x $file -f ... | linkerd inject > temp && mv temp $file; done
helm upgrade my-release --namespace ...

[kfox1111]

helm template doen't store state enough that you can upgrade as there is no release.
even if it did, an upgrade would notice the templates are different then the injected ones, and revert the sidecars.

a mutating web admission hook is really the right way to fix it I think.

[maorfr]

The command i suggested earlier works, but messes up the chart directory.

Use this plugin:
https://github.com/maorfr/helm-inject

it works :)

[dendle]

Guys, isn't this done now? @ihcsim has committed beta functionality into linkerd2, and its available to use in the edge build.
Are we waiting for someone to do this for itsio, also?
With this marked as done, it will prompt talk in other issues regarding how helm interacts with injected sidecars.

Forgive me if I've jumped the gun!

Thanks,
Matt

[klingerf]

@dendle Thanks for following up about this. You're correct that the first implementation of auto-inject shipped last week. There are a few small auto-inject fixes going out with tomorrow's edge-18.10.3 release, after which we'll mark this issue as resolved.

[klingerf]

Experimental support for the proxy auto injection is now available in the edge-18.10.3 release. More documentation about this feature can be found here:

https://linkerd.io/2/proxy-injection/

Will close out this issue.