Flux: Manifest Generation issues with Helm, SOPS and Kustomize

[Djiit]

Describe the bug

Hey I'm trying to use SOPS encrypted secrets on top of HelmReleases. Currently I'm trying to (as seen in https://github.com/fluxcd/flux/issues/2804) :

• Use SOPS to decrypt a secrets.enc.yaml file, and ouput to secrets.yaml << working like a charm, I can see my decrypted values in the logs
• Use Kustomize to merge this with a base HelmRelease << Works On My Machine ™️. But sort of fail silently on Flux.
• Clean-up the generated secrets.yaml so it's not commited in my repo.

While the kustomize build command works fine locally, it doesnt on flux. I'm guessing it's not "merging" all the part with Kustomize, but I'm not sure how to confirm/debug this.

To Reproduce

Follow the Helm Operator install. I used sops.enabled=true, manifest-generation=true, git-path=releases.

Create this structure, here to deploy cert-manager, extracted from my actual setup :

bases/
  cert-manager/
    kustomization.yaml
    cert-manager.yaml # Full HelmRelease
releases/
  cert-manager/
    kustomization.yaml
    secrets.enc.yaml # HelmRelease with same metadata.name (for Kustomize) and only a subset of spec.values (only secrets values)
    .flux-yaml

Here is my .flux.yaml :

version: 1
patchUpdated:
  generators:
    - command: sops -d --output secrets.yaml secrets.enc.yaml
    - command: kustomize build
    - command: rm secrets.yaml
  patchFile: flux-patch.yaml

Expected behavior

I expected flux to apply my HelmRelease with my secrets merged with it.

Logs

ts=2020-05-25T13:25:42.075743796Z caller=sync.go:167 component=daemon err="default:helmrelease/toto: running kubectl: The HelmRelease \"toto\" is invalid: []: Invalid value: map[string]interface {}{\"apiVersion\":\"helm.fluxcd.io/v1\", \"kind\":\"HelmRelease\", \"metadata\":map[string]interface {}{\"annotations\":map[string]interface {}{\"fluxcd.io/automated\":\"false\", \"fluxcd.io/sync-checksum\":\"c2d4b448385daa573ed4770520d13de62ac042b2\", \"kubectl.kubernetes.io/last-applied-configuration\":\"{\\\"apiVersion\\\":\\\"helm.fluxcd.io/v1\\\",\\\"kind\\\":\\\"HelmRelease\\\",\\\"metadata\\\":{\\\"annotations\\\":{\\\"fluxcd.io/automated\\\":\\\"false\\\",\\\"fluxcd.io/sync-checksum\\\":\\\"c2d4b448385daa573ed4770520d13de62ac042b2\\\"},\\\"labels\\\":{\\\"fluxcd.io/sync-gc-mark\\\":\\\"sha256.He7nhmBH-XffnTykW26Kk8KQtsjdbXIu5ujxVEpJM10\\\"},\\\"name\\\":\\\"toto\\\",\\\"namespace\\\":\\\"default\\\"},\\\"spec\\\":{\\\"values\\\":{\\\"fooz\\\":\\\"barz\\\",\\\"ingressShim\\\":{\\\"defaultIssuerKind\\\":\\\"ClusterIssuer\\\",\\\"defaultIssuerName\\\":\\\"letsencrypt-prod\\\"},\\\"podAnnotations\\\":{\\\"iam.amazonaws.com/role\\\":\\\"codefi-common_cert-manager\\\",\\\"test.julien/oof\\\":\\\"SUPER-SECRET\\\"}}}}\\n\"}, \"creationTimestamp\":\"2020-05-25T13:25:41Z\", \"generation\":1, \"labels\":map[string]interface {}{\"fluxcd.io/sync-gc-mark\":\"sha256.He7nhmBH-XffnTykW26Kk8KQtsjdbXIu5ujxVEpJM10\"}, \"name\":\"toto\", \"namespace\":\"default\", \"uid\":\"0703164a-21f1-44d2-b091-eec5750d3c5a\"}, \"spec\":map[string]interface {}{\"values\":map[string]interface {}{\"fooz\":\"barz\", \"ingressShim\":map[string]interface {}{\"defaultIssuerKind\":\"ClusterIssuer\", \"defaultIssuerName\":\"letsencrypt-prod\"}, \"podAnnotations\":map[string]interface {}{\"iam.amazonaws.com/role\":\"codefi-common_cert-manager\", \"test.julien/oof\":\"SUPER-SECRET\"}}}}: validation failure list:\nspec.chart in body is required"

Looks
Additional context

• Flux version: 1.19.0
• Kubernetes version: 1.16
• Git provider: Gitlab
• Container registry provider: Gitlab

Thanks for your help

[briangordonmills]

I've been trying to use get Kustomize to apply changes to a HelmRelease kind, and have had no luck either.

[sharkymcdongles]

I have same problem. When I try it says they are duplicate definitions and won't deploy. :(

When I run kustomize build . though it is 100% correct. It seems their implementation doesn't patch the spec values portions of helmreleases properly for some reason.