Flux: How to use Flux to release Helm charts with secret sops-encrypted values?

[wmiller112]

Describe the bug

I cannot seem to figure out a way to consume the decrypted values of a sops file via --sops=true or with a .flux.yaml patch update command: sops -d as values in a helm chart. For other files containing values that the helm chart consumes, we need to provide a valuesFrom. I am able to decrypt to another file (via .flux.yaml patch updated method), and then with a valuesFrom pointing to that file get them into a helm, but this results in the unencrypted file in the git repo.

One method I have considered is using kustomize with the encrypted file structured as a secret, which would get decrypted and applied outside the helm chart, and be referenced by the helmrelease, but this loses the flexibility of the helm chart for things like naming, checksum, etc.

To Reproduce

Steps to reproduce the behavior:

1. A directory with a helmrelease and sops encrypted (test.enc.yaml) file with the contents:

mySecrets:
    foo: bar

1. A helm chart with a configmap that consumes the values of mySecrets:

apiVersion: v1
kind: ConfigMap
metadata:
  name: test-config
data:
  {{- range $key, $value := .Values.mySecrets }}
  {{ $key }}: {{ tpl $value $ | quote }}
  {{- end }}

1. Flux configured with either:
2. sops enabled and a --git-path pointing to a directory that contains the above files
3. manifest generation enabled, --git-path pointing to a directory that contains the above files, .flux.yaml with patchUpdated command: sops -d test.enc.yaml

As I understand, both of the above decrypt the file, but since the output is not associated with the helmrelease(as a file, configmap, secret, etc) it cannot be consumed by the chart.

Expected behavior

Im not exactly sure what the expected behavior here should be. I may be approaching this wrong, or it might be that Im not using the right tools to accomplish this, but have tried countless different approaches and hoping someone has a suggestion.

[2opremio]

I am missing quite a few details on what you are trying to do. Please confirm if this is correct:

1. You have a chart definition in a Git repository
2. That is the repository provided to Flux for syncing
3. You deploy the chart through the Helm Operator
4. You want to release that chart using values which are secret and you want to somehow encrypt them with sops (and get them decrypted when the HelmRelease is deployed)

You could do this with a .flux.yaml file which generates the HelmRelease. The generator command would decrypt sops-encrypted secrets living in the repo and fill in the values section of the HelmRelease. This can be done in different ways, one of them being Kustomize, but you can use whatever commands you think are more appropiate.

[2opremio]

An alternative (and probably simpler) to achieve the same thing without sops is to use https://git-secret.io/ (--git-secret flag) and encrypt the full HelmRelease.

Note, however, that regardless of using sops or git-secret, once Flux reads the HelmRelease and applies it to the cluster, the values would be in clear-text and anybody with permissions to list/get HelmReleases from the cluster will be able to read them ... which isn't exactly great.

Another solution to solve this problem would be to extend the HelmRelease definition to support encrypted value entries. The Helm Operator could decrypt them when needed.

I am not thrilled about tying Flux to specific technologies (which we are already doing by specifically supporting sops and git secrets), but maybe we can do thisin a vendor-neutral way by generically specifying how to decrypt them in the HelmRelease.

There may be other options, but I don't know the Helm ecosystem that well ( @hiddeco @stefanprodan , I invoke you).

[wmiller112]

1. You have a chart definition in a Git repository
2. That is the repository provided to Flux for syncing
3. You deploy the chart through the Helm Operator
4. You want to release that chart using values which are secret and you want to somehow encrypt them with sops (and get them decrypted when the HelmRelease is deployed)

Yes, spot on. Apologies for the vagueness on the intent, will work on that. I appreciate the response and suggestions. Adding them directly to the HelmRelease makes sense, and I understand the risk associated there. I will implement this for the time being.

I definitely understand not wanting to tie this project to other technologies, and the suggestion to extend the HelmRelease definition is really interesting. I'll take a look at that and see what I can do there.

[wmiller112]

I'm fining that the generated decrypted file from the second method described here ends up in git after the first generation: https://github.com/fluxcd/flux/pull/2580#issuecomment-553372361

.flux.yaml:

version: 1
patchUpdated:
  generators:
    - command: sops -d --output secrets.yaml secrets.enc.yaml && kustomize build .
  patchFile: helm-release-patch.yaml

If I don't output to a different file, and instead just sops -d secrets.enc.yaml && kustomize build ., kustomize gets the encrypted values and fails, which is what I'd expect since it just outputs the file to stdout.

For now adding a - command: rm secrets.yaml will do the trick though.

[2opremio]

I'm fining that the generated decrypted file from the second method described here ends up in git after the first generation

Yes, after release/update operations whatever is put in the git repository will be committed. Maybe we should only do that for commandUpdated config files.

[2opremio]

Is the solution good enough for you?

[wmiller112]

Yes, it is. Thanks again for your guidance on this.

[hasc]

I cannot figure out a way how to provide values from a SOPS decrypted secrets.yaml to a helmrelease manifest.

There are some kustomize plugins but I wonder whether there's a simpler or better way to achieve this.

Any help or advice on this would be great.

[Ant59]

I'd also like some guidance on how to use values from a sops-encrypted secrets.yaml in a HelmRelease.

[wmiller112]

I ended up going with the method suggested above, which is to add them to the helmrelease as values after decrypting. Create a patch that contains secrets under spec.values, and then you can encrypt just the data with sops --encrypted-regex (not necessary, but convenient to preserve the file readability. Then when you run something like below to decrypt and output the generated helmrelease.

- command: sops -d --output dec.secret.yaml secret.yaml
- command: rm dec.secret.yaml

[wmiller112]

Also note that I'm having to create a decrypted file so that it can be used for kustomize, which I then have to remove so it doesn't get committed back to the repo on image update. There may be a better way to do this, but this is what ended up working for me.

[Ant59]

I also ended up using --encrypted-regex to encrypt just the secrets in the HelmRelease manifests. Seems like a good solution.

[Djiit]

Hey, I'm trying to achieve the same thing here but Flux complains that there is two releases with the same name (which is normal because I want to merge them). Did I miss anything ? I'm happy to open a new issue but I think this is quite related to this and I must be missing something dumb... Thanks

[Djiit]

@wmiller112, what directory structure did you use here ?

[cristianburca]

I'm fining that the generated decrypted file from the second method described here ends up in git after the first generation: #2580 (comment)

.flux.yaml:

version: 1
patchUpdated:
  generators:
    - command: sops -d --output secrets.yaml secrets.enc.yaml && kustomize build .
  patchFile: helm-release-patch.yaml

If I don't output to a different file, and instead just sops -d secrets.enc.yaml && kustomize build ., kustomize gets the encrypted values and fails, which is what I'd expect since it just outputs the file to stdout.

For now adding a - command: rm secrets.yaml will do the trick though.

Hi. What's inside helm-release-patch.yaml?