Pulumi-kubernetes: Support helm v3 chart releases and helm v3 CLI

[jaxxstorm]

Problem description

When we install a helm chart, we simply template the chart locally and then install the YAML into the cluster. Because of this, it's not possible to interact with helm charts using the helm CLI because the release information is never created
Since helm v3, helm no longer has a server-side component (tiller) and all helm CLI interaction happens with the k8s API.

When a helm release is created, it simply creates a secret object in the specified namespace for each release:

NAME                        TYPE                                  DATA   AGE
default-token-cc5pg         kubernetes.io/service-account-token   3      16d
sh.helm.release.v1.opa.v1   helm.sh/release.v1                    1      2m6s
sh.helm.release.v1.opa.v2   helm.sh/release.v1                    1      37s

This secret is base64 encoded by the k8s API, and then gzipped and base64 encoded by helm as well. The contents can be found like so:

k get secret "sh.helm.release.v1.opa.v1" -o json | jq .data.release -r | base64 -D | base64 -D | gzip -d | jq . 

And you can see an example output here:

https://gist.github.com/jaxxstorm/e808b7dc00e108c66495bd5762842cfe

With all this in mind, it seems feasible that we can support creating a release when pulumi installs a chart with the following workflow:

• On adding a helm resource, template the YAML as usual
• Build a JSON string with the contents of the helm chart, most of which is available to us already
• gzip it and base 64 encoded it
• Create the secret

We could optionally also support releases in line with pulumi state upgrades - so each state change would create a helm release by creating a new aq version of the secret - however this creates the question of what we would do someone updates a release outside of Pulumi (ie via the helm CLI)? How would we handle this drift?

[lblackstone]

@jaxxstorm Do you have a link to Helm docs about this behavior?

I think this approach sounds promising. I expect we'd create this secret as a subcomponent of the Chart ComponentResource.

Curious if the same secret is reused, or if they create a new secret for every release?

[jaxxstorm]

Surprisingly, I couldn't find any docs about this behaviour. You can find information about their built-in objects here

https://helm.sh/docs/chart_template_guide/builtin_objects/

I also took a look at this operator: https://github.com/fluxcd/helm-operator/blob/master/pkg/helm/v3/release.go which manages helm releases as a crd.

Each secret is new for a release as you can see here:

sh.helm.release.v1.opa.v1   helm.sh/release.v1                    1      2m6s
sh.helm.release.v1.opa.v2   helm.sh/release.v1                    1      37s

[mfar]

This is also deal breaker for us.if we use pulumi to deploy our charts,
we can't use helm client to continuously update charts anymore

[confiq]

This would be a perfect way to control HELM charts for us too.
it could avoid silly mistakes like this

[iridian-ks]

Something that seemed very odd to me when I first tried to use the Pulumi Helm provider is that the Helm CLI is required. Terraform's Helm Provider uses the Helm Go packages and effectively embeds Helm into the provider: https://github.com/hashicorp/terraform-provider-helm/blob/master/helm/resource_release.go

I believe the Pulumi provider is also written in Go and could also do this to remove the requirement of the Helm CLI being installed. Not that it's necessarily bad to be a wrapper around a command line tool, but I think it should improve robustness by removing additional external dependencies. The pre-built Pulumi docker images don't have the Helm CLI installed, for example.

There's a little bit of duplicated functionality between Helm and Pulumi, but I think having Pulumi manage Helm values.yaml files still has power even if it's deferring rendered YAML state to Helm. This is the approach Terraform takes and was quite wonderful to work with. The downside of Terraform is that I couldn't do post-provisioning of deployed Charts whereas I can do that with Pulumi. There were also times where I accidentally had secrets in my values.yaml and those were left in plaintext in the state files. Pulumi could fix that too.

[lblackstone]

Something that seemed very odd to me when I first tried to use the Pulumi Helm provider is that the Helm CLI is required. Terraform's Helm Provider uses the Helm Go packages and effectively embeds Helm into the provider

Yep, we'd like to do this eventually. This is tracked in https://github.com/pulumi/pulumi-kubernetes/issues/920

[lkt82]

I would also like to use pulumi but it's a deal breaker that I can't use helm chart the same way as in terraform 👎 where its only the values that are tracked in state.

Almost all the charts I have tried can not be used duo to different errors when parsing the template like [System.InvalidOperationException: Expected System.String but got System.Double deserializing Pulumi.Kubernetes.Types.Outputs.Core.V1.ResourceRequirements(limits)]

The current implementation does not work for me if I need to find a unique workaround for every chart

It's way simpler to let helm do what is has been created for.

[alexbechmann]

Would it make sense for the Pulumi engine to use the helm CLI to do the actual deployment? This would make it consistent with how pulumi creates other cloud resources e.g. azure using az cli.

Then Pulumi would just have to compose the correct helm CLI command for the pulumi resource lifecycles:

• Create/Update

helm upgrade my-release repository-name/name-of-chart --create --namespace my-namespace

Delete

helm uninstall my-release --namespace my-namespace

You would also be able to interact with the release from the helm CLI and rollback commands will not effect the integrity of the Pulumi state. Also charts tend to assume you are using helm CLI & CRD management is designed around this assumption.

I often find that pulumi importing each k8s resource from a chart can lead to tens of thousands of lines of JSON in state file, and I often get "X already exists" errors. Errors like this seem avoidable when k8s itself lets you kubectl apply -f ... or helm upgrade ... multiple times without any issues.

[lblackstone]

Would it make sense for the Pulumi engine to use the helm CLI to do the actual deployment?

We're considering making that an option, but there are some downsides to that approach:

1. Transformations wouldn't work. This is a very popular feature because it allows you to "monkey patch" resources without requiring changes to upstream charts/manifests.
2. Await logic wouldn't work. When you run helm install, it creates the resources in the cluster, but doesn't do additional checking to make sure the resources are working as expected (same behavior as kubectl). By default, our k8s provider does readiness checking for most resources, which often catches configuration problems as soon as you run an update.
3. There's more room for "works on my machine" type errors since some of the state is being offloaded to an external tool. Most of the heavy lifting in our providers happens natively using relevant Go SDKs rather than shelling out to tools like kubectl or az cli.

That said, we hear the feedback, and will be looking at further improving the user experience for k8s users this year.

[alexbechmann]

Hi, thanks for your feedback :)

1. I agree the transform functionality is really useful. Maybe you could do something like the following?:
   a) Re-create the chart in temp directory
   yaml tempdir templates/ manifest.yaml (empty) Chart.yaml values.yaml
   b) Populate manifest.yaml like: helm template ... | run each k8s item through the user's transform functions | save to manifest.yaml
   c) run all k8s items through the policy engine!
   d) helm upgrade --install ./tempdir --values ....
2. helm cli has --wait and --timeout args which will wait for pods to be ready. So you should be able to communicate a failure here?
3. Helm is written in GO? maybe there is some possibilities to use helm functionality without relying on the CLI shell?

The reliability benefits of helm CLI deployments is something that would be amazing to be able to leverage from within my pulumi programs. Right now i use Pulumi for absolutely everything minus kubernetes.

Could also be that the strategy of using Helm to leverage deploying is something you can opt into, (like a different import?) so if any of the concerns you raised are not resolvable, people can choose between them?

[lblackstone]

Helm is written in GO? maybe there is some possibilities to use helm functionality without relying on the CLI shell?

Our Helm v3 support actually uses the Helm SDK under the hood, so it's a possibility. Currently, we're just using the fetch and template operations.

Could also be that the strategy of using Helm to leverage deploying is something you can opt into, (like a different import?) so if any of the concerns you raised are not resolvable, people can choose between them?

Yes, I think having a separate import would be the most likely option if we go down that route.

[maxromanovsky]

This would also allow easy identification of outdated charts using helm whatup: https://github.com/fabmation-gmbh/helm-whatup