Flux: Vault Integration with Helm

[ghost]

Describe the feature
One of the issues I have with Helm is the ability to pass secrets. You usually have to do this part through the shell using --set and then have separate values file for the non sensitive values. We can't do this when we use the helm operator and we can't commit sensitive values.

It would be great if the vault operator could support a way to go and get those secrets from a secret management system like hashicorps vault, then pass them into the values.

This is more of a question if you think this feature would be worth while and if so I could look to write the code if I could be pointed in the right direction.

A rough example of how it could be used.

* Example *

apiVersion: flux.weave.works/v1beta1
kind: HelmRelease
metadata:
  name: myapp
  namespace:  mynamespace
spec:
  releaseName: myapp
  chart:
    git: https://github.com/helm/charts.git
    path: stable/jenkins
    ref: master
  vault:
    address: https://myvault.example.com
    auth_type: kubernetes
    secret_map:
      - admin
        path: secret/jenkins
        helm_value: administratorPassword 

1. What are the prerequisites for this?
   Maybe a configuration section to turn this feature on.

[squaremo]

There is some room for this kind of feature, in the way that the valuesFrom field is used. But before diving into that: can Vault put values into an ordinary Kubernetes secret? Then you could just use .spec.valuesFrom.secretKeyRef to refer to such a secret from the HelmRelease.

[squaremo]

/cc @ncabatoff re Vault

[ghost]

Thanks @squaremo

I am not aware of any vault functionality that could put a value from vault into a k8 secret. Although that would be awesome.

For our own applications we build our apps to speak directly to vault or do something in a init container.

This ask would be great for community charts, where sensitive values are required.

[ncabatoff]

But before diving into that: can Vault put values into an ordinary Kubernetes secret?

The whole idea of Vault is central management and control of secrets. If you start pushing those secrets into k8s secrets, you lose the benefits that come with that centralization. The secrets are now at greater risk, since they're stored less securely in etcd - by default without even encryption. There's no longer an audit trail recording how they're accessed and by what. Revocation can no longer be done immediately or reliably. And so on.

That said, I think the proposal makes sense for the community charts use case. Probably the reason no one's created a vault->k8ssecret gateway (besides the drawbacks I just listed) is that there's no mechanism to signal applications that they need to re-read an updated secret. But here the mechanism to update secrets is to redeploy the application, which is heavy-handed but works.

Is the plan is to have helm-operator poll Vault for the secrets, and changes would trigger a new deployment? I guess that's not really required, as the user could always setup consul-template or something like that to monitor secrets and trigger a release when they change. For now just being able to inject the secrets into a release would be awesome.

There are two ways Flux could bridge Vault and Kubernetes secrets. The first is what you're proposing, i.e. providing values to charts so they can create secrets based on their templates. I think that means that the secrets will be recorded in plaintext in helm's release history, which of course isn't great. Alternatively, helm-operator could itself create the secrets to be consumed by charts, without persisting them in the release history. This would be preferable from a security standpoint but I'm guessing that would decrease the number of community charts that could benefit from this feature.

Last thought: can we make the solution more generic rather than Vault specific? Let's say we target REST APIs that yield JSON results, and we tweak your example a bit, with the assumption that a Vault Agent sidecar is deployed with Flux that save us from having to deal with authentication:

spec:
  releaseName: myapp
  chart:
    git: https://github.com/helm/charts.git
    path: stable/jenkins
    ref: master
  secret_source: https://example.com
  secrets:
    admin:
      request_path: v1/secret/data/jenkins
      response_jsonpath: data.data
    dbpass:
      request_path: v1/database/creds/mypg
      response_jsonpath: data

This would GET https://example.com/v1/secret/data/jenkins, parse the result as JSON, index the jsonpath data.data which would yield the JSON map corresponding to the KV-V2 secret named jenkins, and build a secret named admin containing the corresponding key-value pairs that would be included in the release.

Then it would do similarly to create a secret named dbpass containing the username and password keyvals coming from a DB secret engine named mypg.

[ghost]

@ncabatoff

Very valid point about pulling secrets from vault to a k8 secret. I don't think it would fly in most companies security policies.

I really like the idea of a generic solution using a sidecar.

[squaremo]

Thanks for that thorough reply @ncabatoff!

It's possible to get values from a URL (but at present, you can't project the resulting value, as in your example Nick): https://github.com/weaveworks/flux/blob/master/site/helm-integration.md#external-sources.
These will get recorded in the Helm release history, as they would for helm install --values. But some charts don't have any other way of supplying secret values.

If the chart is able to just refer to a secret (rather than requiring the secret _values_), it doesn't really matter whether it's flux or the helm-operator or something else that's creating the secrets -- though granted, it would be quite natural to run that process as a sidecar to the helm-operator since they have a shared interest.

[ncabatoff]

It's possible to get values from a URL (but at present, you can't project the resulting value, as in your example Nick): https://github.com/weaveworks/flux/blob/master/site/helm-integration.md#external-sources.
These will get recorded in the Helm release history, as they would for helm install --values. But some charts don't have any other way of supplying secret values.

Let's call that Use Case 1: worst security, most convenience. It sounds like all the pieces are already there to accommodate that one: deploy Vault Agent, use externalSourceRef to pull secrets from it into values, and you're done. Except maybe you also want to set up something to trigger a release when the secrets change, like I spoke of before, but that need not be in Flux.

If the chart is able to just refer to a secret (rather than requiring the secret values), it doesn't really matter whether it's flux or the helm-operator or something else that's creating the secrets -- though granted, it would be quite natural to run that process as a sidecar to the helm-operator since they have a shared interest.

At my previous job we would've loved to have this functionality, which I'll call Use Case 2. We wound up using sealed secrets as an alternative but I think this would be better, especially if you're already a Vault user.

[ghost]

Thanks @ncabatoff and @squaremo

I will have a play around

Simon

[amstee]

We're facing the same issue at Babylon Health using flux.

I was wondering if this approach might be reproducible with an Admission Controller that would be able to impersonate the service account that will eventually use those secrets, fetch them from Vault and populate the resource as needed.

This would prevent to replicate those secrets in two different location vault <--> kubernetes but then comes the issue of updating the resource when the secret is changed.

What I'm mixed about is the Use Case 2 that @ncabatoff pointed out. Because we would end up with a duplication of the secrets in Vault to Kubernetes.

An elegant solution would be a .spec.valuesFrom.vault I suppose, what do you guys think ?

[ncabatoff]

I recently discovered a couple of Vault CSI implementations, which is another interesting way of exposing Vault secrets in pods for use case #2 (though not as k8s secrets, but I don't think that's essential). If I were still trying to address my former company's needs I'd probably be exploring that option.

An elegant solution would be a .spec.valuesFrom.vault I suppose, what do you guys think ?

If the result is that secrets get persisted in cleartext in the release, which I think they'd have to be, it's not a solution I can get behind.