Flux: helm-op: refer to secrets in FluxHelmRelease resources

[squaremo]

Some charts have parameters which are secrets. In normal Helm use, you might keep (or generate) such a secret as an environment variable and supply it in a CI pipeline. But with a FluxHelmRelease you'd have to include the secret value in the manifest itself -- and in git, if you are going Full GitOps.

If a FluxHelmRelease could refer to a kubernetes secret, and use the entries therein in its values, they would not need to leave the cluster. And it would mean you could use sealed secrets and so on.

There are drawbacks to this -- it means another source of changes we have to account for in the operator. Needs some thinking through.

[mpareja]

I share the same concern because it prevents me from using some Helm charts with Flux. If you _do_ take this on, please consider supporting a plugin-based secrets provider model so folks using other secret managers, like Vault / AWS Secrets Manager, can leverage them.

[hiddeco]

Want to add a note that most 'mature' charts get around this problem by giving you the option to manually supply the name of a secret that contains the secret values the chart normally creates for you. This gives you the option to combine a FluxHelmRelease with a SealedSecret.

[stephenmoloney]

I know this thread is relevant to me as I am using secret references and sealed-secrets but I am a bit confused by it. In other words, I don't understand the issue.

Can someone explain it in very simple terms (the issue) ?

[hiddeco]

@stephenmoloney

I know this thread is relevant to me as I am using secret references and sealed-secrets but I am a bit confused by it. In other words, I don't understand the issue.

At times you need to supply sensitive data (e.g. passwords) to a chart. Some charts do not offer a secret override like you are probably using (providing the name of the secret that has the correct key/value pairs for the sensitive data) but expect you to provide them in plain text in the values.yaml.

The solution proposed above would allow you to provide those values encrypted in a SealedSecret, by merging values from a given secret in to a FluxHelmRelease. Working around the shortcomings these charts have.

[stephenmoloney]

@hiddeco

Some charts do not offer a secret override like you are probably using (providing the name of the secret that has the correct key/value pairs for the sensitive data) but expect you to provide them in plain text in the values.yaml

I haven't really had this problem because the charts that I am using offer the solution you described here below:

Want to add a note that most 'mature' charts get around this problem by giving you the option to manually supply the name of a secret that contains the secret values the chart normally creates for you

Yes, there are good examples of that such as the postgresql stable chart in the helm repo.

The solution proposed above would allow you to provide those values encrypted in a SealedSecret, by merging values from a given secret in to a FluxHelmRelease. Working around the shortcomings these charts have.

I understand now thanks. I suppose this makes sense to some extent but would making a pull request against the pertinent helm chart be a better solution?

[justinbarrick]

This feels like a bit of scope creep to me, since you can pretty easily modify a non-compliant chart to do what you need (especially since you need to check them in with helm-operator). That said, secret key references in CRDs might be better implemented as a proper Kubernetes feature, rather than a one-off for this resource? I could see it being more generally useful.

[pcfens]

In our puppet manifests we've liked the approach taken by the hiera-eyaml project. It let's us keep _everything_ in git except for one private decryption key.

The short description is that eyaml drops public/private key encrypted values in a yaml file so that the puppet server can decrypt them. We keep the public key in our git repositories so that values can be added and replaced by anyone with push access while keeping sensitive data out of git. It's not as fully featured as something like vault, but is easier to maintain and accomplishes our goal of everything in git.

Today in kubernetes land, the only place we need to keep secrets alongside non-secrets is with our Helm configs, but we'd happily store more in git if there were a safe way to do it.

I'm not sure how complicated this approach would be to implement, but it would let us keep everything in version control and not require modification of helm charts (keeping it easy for new adopters like me).

[hiddeco]

@pcfens what you describe is already possible by using 'Sealed Secrets'.

[pcfens]

The sealed secrets approach looks close, and will be a big help. I was thinking a little more generically so that any value in a yaml file could be encrypted. Of course the approach would require some intermediary tool, but flux sits at a good spot in the pipeline to handle general decryption.

[squaremo]

I think this use case is covered by the recently merged #1468 -- if someone can try it out and post here that'd be much appreciated.

[kozejonaz]

@squaremo I have done my first test using the functionality from https://github.com/weaveworks/flux/pull/1468 to get the keycloak.password from a Kubernetes Secret that was deployed as a Sealed Secret by Flux, and it works beautifully.

This functionality solves a lot of annoying problems when you want to do proper GitOps using Helm charts. Thank you.

[squaremo]

I think I can close this, given the feedback on #1468 above. Yay!