Helmfile: Potentially nondeterministic rendering of state secrets

[yurrriq]

Providing a MFE here would be difficult, but we seem to be encountering nondeterministic rendering of state secrets (:rofl:).

With a setup like the following, where defaults.yaml and overrides.yaml have some of the same keys, we seem to sometimes end up with the value from defaults.yaml and other times from overrides.yaml.

environments:
  example:
    secrets:
    - secrets/defaults.yaml
    - secrets/example/overrides.yaml

The same pattern with values works fine, so perhaps this has something to do with secrets. It also seems to work ok with AWS KMS as the key store, but not always with our self-hosted Hashicorp Vault as the key store. The sops/vault --debug logs (with HELM_SECRETS_DRIVER_ARGS=--verbose too) seem to imply everything worked fine, and I do see the decrypted values from both defaults.yaml and overrides.yaml.

Does that make sense? Any idea what's going on here and how we might prevent the nondeterminism?

[yurrriq]

Grasping at more straws:

• Maybe the time it takes to decrypt defaults.yaml and overrides.yaml varies and thus determines the order/precedence.
• Maybe mergo doesn't work how I think it does.
• Maybe secrets are parsed concurrently.

[yurrriq]

Comparing the logs of a successful run vs unsuccessful, we've noticed that the environment secrets files are processed in a nondeterministic order, which results in differing precedence in the merge.

[mumoshu]

@yurrriq Hey! Thanks for reporting.

That's interesting... Helmfile does decrypt environment secrets concurrently, but it also sorts results by ids so that the result is deterministic and merging happens in the order of definitions.

https://github.com/roboll/helmfile/blob/92ba347d6f899bd4b29a4279efef3199d0fa36b0/pkg/state/create.go#L341

That said, I cant' think of any potential bug that may result in secrets being processed in nondeterministic order.

Or perhaps helm-secrets has some limitation preventing it from being run concurrently? 🤔 I have never heard about such limitation though.

[yurrriq]

I first noticed it in GitLab CI jobs, but was able to reproduce it locally today. Two runs of helmfile with the same values etc showed it decrypting the two secret files in reverse order, and then giving precedence to one or the other. We noticed it in a helmfile diff run, in case that makes a difference...

Here's some more info:

λ helmfile --version
helmfile version v0.138.7

λ helm version --short
v3.5.3+g041ce5a

λ helm plugin list
NAME    VERSION     DESCRIPTION                                                                  
diff    3.1.3       Preview helm upgrade changes as a diff                                       
secrets 3.7.0-dev   This plugin provides secrets values encryption for Helm charts secure storing

λ sops --version
sops 3.7.1 (latest)

λ vault -version
Vault v1.7.1 (cgo)

If you think it'd be helpful, I could try to purge any secrets from the log files and email them to you or something.

[yurrriq]

Looks like this was fixed in #1726, and we just need to update to 0.139.x. I'll do that and report back. :crossed_fingers:

[yurrriq]

Confirmed that #1726 fixed it. Many thanks, @astorath and @mumoshu!