Helmfile: Ability to diff values from helmfile

[mattymo]

I'm not sure if this should go in helm-diff or not, but it would be really valuable to get a diff between the combined user-provided values from a helmfile defined release vs current state in Helm for a given release.

Here is an example scenario:
Step 1: Deploy myapp with helmfile

releases:
- name: myapp
  chart: mychart
  values:
   - foo: bar

helmfile apply



md5-7b9fec14a7c50d4972bedfbf47c621c9



helm upgrade myapp mychart --reuse-values --set bar=baz



md5-7605e758c29f06676462ec46b5eabc18



$ helmfile diff --values-only
default, myapp, User-provided values has been added:
foo: bar
+ bar: baz



md5-6786bfb337e48bac05ebc26aa4479e26



for param in `helmfile --log-level=debug template 2>&1 | grep -- --value | head -1`; do if echo $param | grep /tmp; then cat $param; fi; done



md5-5d452af0f0c45c3f6e30bcd5c9ed0c4c



helmfile --log-level=debug template 2>&1 | grep -- --values | head -1

[mumoshu]

@mattymo Hey! Thanks for reporting. I'm still thinking about how diff between values work. Maybe we add some facility to Helmfile that can generate values for specific release and diff it against helm get values output?

Also, in the meantime, do you think helmfile build --embed-values added in #1436 could help reviewing values to be diff'ed?

[mattymo]

@mumoshu You write new features so often that it's hard to track what changed recently. You are a great developer and I'm glad you've written something so useful.

I have some pre-deploy helm charts as an included helmfile in theactual helmfile I use, so using helmfile build will render both as 2 yaml documents. I use https://github.com/mikefarah/yq as my preferred yq binary for handling YAMLs.. I can get my values out with this:

helmfile -l name=mychart build --embed-values | yq r -d'*' - 'releases[0].values[*]'

For now, that does what I need in a parseable way. I still have to do something like this to get a full list of helm releases:

helmfile build | yq -r -d'*' - 'releases[*].name'

I have one last issue to solve.. which is comparing helm chart version installed by helmfile vs current state. I want to know based on helm-diff output if the changes are due to different values or due to different helm chart version.

[mattymo]

Any news on this bug? Actually using helmfile build --embed-values when you have several values files leads to this issue:

$ helmfile -l name=myapp build --embed-values 2>/dev/null | yq -r -d'*' - 'releases[0].values'
- foo:
    bar: value
- bar:
    something: value
- foo:
    baz: value

I can't easily merge the two values files that have "foo" key. If I take away the array from the output, I lose foo.bar=value:

$ helmfile -l name=myapp build --embed-values 2>/dev/null | yq -r -d'*' - 'releases[0].values[*]'
- bar:
    something: value
- foo:
    baz: value

I think my remaining workaround is to transform the array into separate YAML docs, but https://github.com/mikefarah/yq doesn't support merging all documents into a single document. Is this a feature you could add to helmfile build --embed-values?

(edit: I filed a bug against yq to see if it could be solved from that side: https://github.com/mikefarah/yq/issues/535 )

[mumoshu]

@mattymo Is this a bug in Helmfile? 😃 Anyway, I don't have a complete idea yet. I'm not even sure what's the technically correct way to do this.

helm get values gives us user-provided values used for installing the live helm release. So our diff target should be that.

For the diff source, Helmfile indeed has a set of values embedded in helmfile.yaml and also values.yaml.
Maybe Helmfile needs to merge those in the exactly same way as Helm? But I'd need some time to read Helm code and see how it merges values exactly.

Lastly, I believe we need to sort the keys in YAML dicts, so that the diff result is stable regardless of the key order, which can be randomized(it isn't strictly random of course, but I believe the order isn't guaranteed anyway).

[mattymo]

@mumoshu Sorting keys isn't too bad. I found a project, dyff, that can diff YAMLs and it auto-sorts keys.

My problem lies in merging the --embed-values YAML documents that get rendered.

[mumoshu]

@mattymo dyff seems useful. So- would it be enough if we could just enhance helmfile build --embed-values to somehow merge values for you, in the meantime?

[mattymo]

@mumoshu Yes, that is exactly what I wanted in the first place. I just wanted to demonstrate how I could do it as a workaround using external tools.

[mumoshu]

@mattymo Thanks. It's close. Are you using releases[].secrets, or ref+ URLs in values to inject secrets into values? Probably we need to decide either to merge secrets into values, or not. helm get values does produce values containing helmfile values and secrets, because these are the same thing from helm's perspective.

[mumoshu]

Perhaps we'd better add a dedicated command for producing merged values, like helmfile -l name=whatever write-values --output-file-template "{{ .Release.Name}}.values.yaml"? Helmfile already has -l key=value for filtering the releases to be processed so we reuse that. write-values is intended to be a new command that writes values.yaml by merging targeted release's values and secrets(after decryption), and writing resulting files to the path of your choice.

[mumoshu]

Btw the --output-file-template GOTEMPLATE flag is designed after helmfile template --output-dir-template I've added recently.

[mattymo]

@mumoshu I'm not currently using the secrets plugin or referring to any external URLs. All of my values come from helmfile.yaml explicitly or from local files. I trust your judgement here for how best to implement this.

[mumoshu]

@mattymo Thanks for confirming. I'll give it a try with the write-values idea. I think it shouldn't take so much to be implented. But throughout testing is what I miss all time. I'd appreciate it if you could help testing it once released 😃

[mattymo]

@mumoshu You can count on me testing it after it's available.

[mumoshu]

Waiting for CI #1469

[mumoshu]

@mattymo Would you mind giving it a try with v0.129.0? Note that I couldn't add support for --set and set: in helmfile.yaml for now. You need to translate them to values: to work around, in case you use set.

[mattymo]

@mumoshu Here's some feedback:
1) --write-values shouldn't trigger dependency checks since we aren't actually installing anything. I get this error if I try to run helmfile write-values with a chart that has a dependency:

in helmfile.yaml: "mychart" depends on nonexistent release "otherchart"

helmfile build --embed-values does this without any issue.

2) --output-file-template /dev/stdout works great. I have no concerns there. However, I feel like the default value would usually be current working directory, not the location where helmfile.yaml is located. So it would be ./{{ .State.AbsPathSHA1 }}/{{ .Release.Name}}.yaml

3) Small stylistic bug here: {{ .State.BaseName }}-{{ .State.AbsPathSHA1 }}/{{ .Release.Name}}.yaml (missing space after .Release.Name

[mumoshu]

@mattymo Thanks a lot for your feedbacks! The 1st and the 3rd items are addressed in #1533.

For the 2nd item, it seems take some time to implement as Helmfile internally run "chdir" for each sub-helmfile and we don't carry the original work dir for the entire Helmfile run.

As a workaround, you can add the absolute path to the current directory in the template like helmfile write-values --output-file-template "$(pwd)/{{ .State.AbsPathSHA1 }}/{{ .Release.Name}}.yaml".