Linkerd2: There appears to be no clear path for integration with Helm

[leopoldodonnell]

Currently, conduit is injected by analyzing a deployment _yaml_ and generating output that is piped into kubectl apply. This breaks somewhat when you start deploying via helm/tiller.

I've been able to do something like this, but its very clunky

kubectl get deployment --namespace app-namespace my-deployment -o yaml |conduit inject - |kubectl apply -f -

Is there another solution I'm missing, or one that is planned?

[olix0r]

Thanks for starting this discussion, @leopoldodonnell.

It may be possible to simply store the output of the inject command in tools like helm/tiller.

I think we'd eventually like to remove the inject command as it is and replace it with something like Mutating Webhook Admission plugins, or whatever Kubernetes primitive eventually meets this need. (We had previously looked at Initializers, but it seems they are not the proper path forward.)

I'm curious if others have thoughts on how they'd want to manage this sort of thing.

[robdefeo]

Helm does have hooks although I dont know if this can replace what inject does. If inject is effectivly creating a yaml template for kubernetes then helm can fulfil this. What does Inject do exactly? Looking at the code it appears to supplement the yaml. It does though have complexity and trying to support this by something custom built by myself suggests there will be errors / feature gap either now or in the future. Plus having lots of different implementations by users.

[leopoldodonnell]

@robdefeo I share the same fears. I really like the simplicity of conduit, but at the end of the day it needs to fit into a deployment pipeline. The pipelines that I typically roll out are Helm based, so this is important.

I was initially too lazy to try using a hook. I'll look into that to see if it could be made fairly seemless

[wmorgan]

Mutating webhooks are beta in 1.9, and 1.9 is now available on GKE and minikube. If you folks are able to use 1.9 in the near future, we may be able to avoid any Helm magic, or indeed any need to use inject at all.

https://kubernetes.io/docs/admin/admission-controllers/#mutatingadmissionwebhook-beta-in-19

[robdefeo]

I agree with @leopoldodonnell migrating to 1.9 is great if it removes the need for inject @wmorgan what type of solution are you thinking of? Would this mean helm does not need to know about this conduit?

[robdefeo]

@wmorgan Any idea what milestone / timeframe we could see the mutating webhooks used to not need to work around helm / inject issues.

[wmorgan]

@robdefeo we're still tossing this idea around around a bit, but given the widespread availability of 1.9 this feature is potentially a good candidate for 0.4 release which would be mid March. PRs welcome in the meantime, of course!

[briansmith]

There are lots of parts to this:

• Should we provide a Helm chart for installing the Conduit control plane and/or setting the default settings for Conduit proxies? Is it OK to ask the user to run conduit install to generate the Helm chart or do people want the Helm chart provided directly?

• We need to consider both installation orders: Conduit installed before Tiller vs. Tiller installed before Conduit. In particular we need to consider the implications of injecting the Conduit proxy into Tiller.

• Would Helm users actually accept that Conduit modifies their Helm-managed services in a way that has limited visibility through Helm? If we only implement the mutating web hook, and don't implement a Tiller plugin, then (AFAICT) what Helm users see through the Helm tool will no longer match what is actually produced. I doubt Helm users would like that. Is Helm doing something to support mutating webhooks, or is it expected that the mutating webhooks need to do something to explicitly help Helm?

PRs welcome in the meantime, of course!

Right away, it would be useful to see what Helm users would want a Helm chart for the Conduit control plane itself to look like, in particular the chart/sub-chart structure given the various components (see https://github.com/runconduit/conduit/blob/master/BUILD.md#components).

[robdefeo]

I reached out on a helm user group and this was the feedback

Following istio's approach, the mutating webhook should inject the sidecars it needs into Pod resources, which should make it work irrespective of how you deploy your applications

Creating a helm chart for conduit itself is a completely separate issue from injecting a sidecar into standard workload pods. WRT helm not 'seeing' the same thing as is the actuality, that should not be an issue as Helm does not create pods directly, only Deployment/StatefulSet (i.e. workload api) resource. if it does create Pod resources, Helm will be aware of it as the mutating webhook is on the 'hot path' to storing the pod resource in the apiserver, and so the pod returned to tiller when it makes the Create call will contain the pod resource with the sidecar injected https://github.com/istio/istio/blob/master/pilot/cmd/sidecar-injector/main.go <- the entrypoint for the istio sidecar-injector

[wmorgan]

@robdefeo thanks for pinging them. That matches my understanding as well. So for full Helm support, it sounds like we need mutating webhook support for proxy injection, and a chart for the control plane. Please feel free to create issues for both of these. I'm also interested in whether you have opinions on @briansmith's questions above about how to best structure the Helm chart.

[robdefeo]

@briansmith let me know if you want further clarification but these are my thoughts on your questions.

1. It would be preferred to install conduit control plane via helm rather than having a CLI to do it. One less tool to work with (that is used as a one off)
2. If you are using helm its probably the first thing you install with tiller then you manage your future installations with helm. I suggest focusing on that use case first.
3. The most important aspect is that I can do a helm upgrade and helm delete etc. I want to be able to manage the deployments, if part of the helm deployment is to configure conduit then that great. I'd rather have one tool to manage it.

@wmorgan I will create the issues for this. Thanks

[briansmith]

OK. I'm going to close this in favor of the more specific issues #452 (Create helm chart for installing conduit) and #453 (cooperating with Helm when injecting the proxy into pods). Thanks for the feedback here and please comment in those two issues with any more details!

[wmorgan]

Current status of Helm work (since we have some issue sprawl): #452 is tracking the control plane chart, and #561 is tracking the admission controller work.