Helmfile: Feature Request: Add `--dry-run` option

[osterman]

what

• Add option to process helmfile.yaml, but not execute helm

why

• This would be helmfile for development

use-case

We're working on a master helmfile, kind of like a "distribution" of charts. Part of this involves a lot of parameterization. It would make it easier to safely iterate using a --dry-run flag as is common with many command line tools.

[mumoshu]

@osterman Hi! Thanks for trying helmfile.

Would you mind elaborating a bit more about your expected output from helmfile?

I guess it suffices your expectation if helmfile diff is extended to show diff even in case there's no installed releases yet?(Btw it requires an enhancement in helm-diff side)

[mumoshu]

Or maybe you'd want something like ver=0.4.0 bash -c 'helm fetch stable/mysql --version $ver && helm template mysql-$ver.tgz' for all the releases defined in helmfile.yml?

[osterman]

In this particular use-case, I don't even have access to kubernetes or helm. I just want to validate the helmfile.yaml template results is valid yaml and that the --set arguments are correct.

[mumoshu]

@osterman Thanks for the clarification!

Just to be certain, would you mind running ver=0.4.0 bash -c 'helm fetch stable/mysql --version $ver && helm template mysql-$ver.tgz'?

The output contains the rendered kubernetes manifests which should reflect everything you provided - chart and values. So I guess you can use that for the validation? If it makes sense, I'm eager to implement --dry-run which outputs something like that for every release.

[osterman]

Just to be extra clear, with the upcoming 0.13.0 release, the entire helmfile.yaml is a template. We're using the current master. No mechanism exists to validate the output from the sprig template and thus the execution of helm upgade..... from the generated yaml.

I want to know what helmfile will execute before helmfile executes anything.

This is kind of like rsync --dry-run. I am familiar with helm template as I use that when I iterate on my charts. Right now, I'm developing a very advanced helmfile.yaml but I lack the ability to validate the templatized yaml.

Currently, helmfile will output: (and a subsequent call to helm)

exec: helm upgrade --install charts stable/chartmuseum .............

Instead, I want to see: (with no execution of helm)

[dry-run] exec: helm upgrade --install charts stable/chartmuseum .............

For example, I've tried using sprigcli to render the template, but it doesn't implement coalesce or uses an old version.

[osterman]

Here's an example of the complex helmfile.yaml: https://github.com/cloudposse/geodesic/pull/124/files

[mumoshu]

Thanks! Got it. Yeah, such feature would be useful 👍

[sstarcher]

Would it make sense to add a helmfile lint similar to helm lint. That could do validating and maybe other checking?

[mumoshu]

@sstarcher I think so. Actually, we have the exact discussion in #58 😉

[osterman]

Yes, that sounds like a great idea. The lint would satisfy the helmfile validation. That said, I still want to know what the command looks like before executing it as the helmfile syntax is a big divergence from a standard values.yaml.

[sstarcher]

@osterman Agreed I was thinking we needed a command to output a generated helmfile. We could either treat this as a part of lint. helmfile lint --generate or as a separate command helmfile generate

I guess the equivalent would be helms helm install --dry-run --debug. Possibly a helmfile lint --debug to lint and output the generated helmfile?

[mumoshu]

@sstarcher @osterman Just to be extra clear, I was thinking that --dry-run and lint to be two different things.

So, I'd expect:

• 👍 helmfile sync --dry-run to do everything but running helm upgrade --install - it renders helmfile and print the commands without actually running them. I believe this is what @osterman like via this feature request.
• 👍 helmfile lint to render helmfile and run helm lint. It doesn't print the rendered helmfile normally, but it does print helmfile if the lint failed. Also, we probably won't add helmfile lint --dry-run as there seems like no motivation to not actually running helm lint.
• 🤔 helmfile lint --debug would print the rendered helmfile even if there was no lint error. But given the vanilla helmfile lint prints the rendered helmfile on a lint error, I think we have no specific need for this feature?
• 🤔 helmfile template(after helm template) to output a generated helmfile. But I'm not sure about an exact use-case for this feature too, because the vanilla helmfile lint would print the rendered helmfile when necessary.

[sstarcher]

so helmfile sync --dry-run would not do helm upgrade --install --dry-run

[mumoshu]

@sstarcher Thanks! That's exactly my first thought. But given helm upgrade --install -dry-run doesn't bloat up the output much:

$ helm upgrade --install --dry-run foo stable/mysql
Release "foo" does not exist. Installing it now.
NAME:   foo

I started to think I can alter my proposal to:

• 👍 helmfile sync --dry-run to do everything but actually running helm upgrade --install - it renders helmfile and print the commands, run helm upgrade --install -dry-run without actually ~running~installing them. I believe this is what @osterman like via this feature request.
• 👍 helmfile sync --dry-run --debug to do everything but actually running helm upgrade --install - it renders helmfile and print the commands, run helm upgrade --install -dry-run --debug to print the whole k8s manifests rendered by helm, without actually ~running~installing them.
• 👍 helmfile lint to render helmfile and run helm lint. It doesn't print the rendered helmfile normally, but it does print helmfile if the lint failed. Also, we probably won't add helmfile lint --dry-run as there seems like no motivation to not actually running helm lint.

How about this?

[osterman]

I think that sounds good.

For clarification, are you saying that helmfile sync --dry-run would also call helm upgrade --install --dry-run?

Basically, in my use-case, I don't even have helm installed and just want to see the commands, so I would not want it to call helm upgrade --install --dry-run. I think executing the helm upgrade --install --dry-run could be accomplished instead by running:

helmfile sync --args '--dry-run'

Thus my conclusion is helmfile sync --dry-run should never call exec.

[mumoshu]

@osterman

For clarification, are you saying that helmfile sync --dry-run would also call helm upgrade --install --dry-run?

Exactly!

Basically, in my use-case, I don't even have helm installed and just want to see the commands

Yes, I think I understand the value of that. Good point!

However, to me it seems like that we can't achieve that perfectly by just suppressing helm upgrade install --dry-run calls.

You'd occasionally need to encrypt/decrypt secrets referenced via secrets: section, which results in helm secret dec calls.

So, even we stopped calling helm upgrade install --dry-run in helmfile sync --dry-run, helmfile stills needs helm binary to call helm-secrets.

I can't list every edge-case like that right now but I do worry about them.
Is the ability to dry-run helmfile without a helm binary a hard requirement for you?

Thanks!

[osterman]

you'd occasionally need to encrypt/decrypt secrets referenced via secrets: section, which results in helm secret dec calls.

Aha, yes, we use chamber instead, which is why this didn't occur to me.

So, even we stopped calling helm upgrade install --dry-run in helmfile sync --dry-run, helmfile stills needs helm binary to call helm-secrets.

Makes sense. I don't have a workaround for that.

Is the ability to dry-run helmfile without a helm binary a hard requirement for you?

No, it's just a preference. =)

[sstarcher]

It seems reasonable for people to want to dry-run the helmfile without having a helm server.

[mumoshu]

@sstarcher In my understanding, helm upgrade --install --dry-run ddepends on tiller(helm server) but helm lint is not.

So, how about serving people who want to dry-run helmfile without tiller with helmfile lint?

[Nuru]

@mumoshu I want to see the rendered helmfile so I can validate that template conditionals are working as expected. With a complicated helmfile that generates multiple charts, it can be tough to trace back errors to the helmfile. And by errors I do not mean syntax or lint errors, I mean bugs where the helmfile is not doing what I want it to do.

[bitsofinfo]

any progress on this, would love to have --dry-run to just see the generated yaml

[mumoshu]

I need to settle on a spec to make progress 😃

@bitsofinfo What do you want for the "generated yaml"? Is it a dump of rendered helmfile.yaml, or basically a collection of helm template outputs, or anything else?

[mumoshu]

@Nuru @bitsofinfo Can I assume that what you need is something like helmfile --log-level=debug sync --dry-run, that dumps the rendered helmfile.yaml(not k8s manifests generated by helm templates), without actually applying them?

[bitsofinfo]

In my particular use case, I just wanted the same out put as helm install --dry-run --debug ... which yes, the generated templates w/out running them. Maybe new args for helmfile like --helm-dry-run and --helm-debug which just pass through? (as to not collide w/ your own args)

[mumoshu]

@bitsofinfo Thx! Perhaps helmfile template helps in that case?

[Nuru]

@mumoshu As a general rule, any tool that does multiple processing passes should have a way to stop at the end of any given pass and output what will be the input to the next pass. That includes outputting the output of the last input processing phase before acting on it. It might help you both architecturally and from UI and logical modeling perspective to think of it that way.

[mumoshu]

@Nuru Thanks for sharing your insight!

Would you mind also sharing an example of a helmfile command output that conforms to the rule?

[Nuru]

@mumoshu I'm not sure what you are asking for. I think for an example you can look at Gnu C++, which will output the result of the preprocessor, (back in the days when it was translated into C, the output of the translation to C), the translation into machine language, and the output as object files before being fed into the linker. So for helmfile I'd at least expect to be able to see the output it plans to send to helm without it actually sending it to helm.

[mumoshu]

@Nuru Thanks! So helmfile sync --dry-run may only print:

skipped: helmfile upgrade --install ....
skipped: helmfile upgrade --install ....
skipped: helmfile upgrade --install ....
skipped: helmfile upgrade --install ....

helmfile destroy --dry-run may print:

helmfile list ^yourrelease1$
skipped: helm delete --purge yourrelease1
helmfile list ^yourrelease2$
helmfile list ^yourrelease3$

Is that what you actually want? I'm just not sure what's you're exactly asking for!

[mumoshu]

In other words, I'm not asking for theory, but wanting to discuss about UI design and implementation idea for --dry-run.

[Nuru]

Well the repo gives the example helmfile:

repositories:
  - name: yourorg
    url: https://yourorg.example.com/charts

releases:
  - name: dbmigrator
    labels:
      job: dbmigrator
    chart: ./dbmigrator
    # DB host, port, and connection opts for the environment
    values:
    - "deploy/environments/{{ env "RAILS_ENV" }}/values.yaml"
    # DB username and password encrypted with helm-secrets(mozilla/sops)
    secrets:
    - "deploy/environments/{{ env "RAILS_ENV" }}/secrets.yaml"

So I'd like to be able to have a command line option that will output:

repositories:
  - name: yourorg
    url: https://yourorg.example.com/charts

releases:
  - name: dbmigrator
    labels:
      job: dbmigrator
    chart: ./dbmigrator
    # DB host, port, and connection opts for the environment
    values:
    - "deploy/environments/production/values.yaml"
    # DB username and password encrypted with helm-secrets(mozilla/sops)
    secrets:
    - "deploy/environments/production/secrets.yaml"

I would like another command line option that runs whatever helmfile is going to do but then calls helm get or something like that, so I can see the manifests and values files before they get applied. That is what I would call the --dry-run. I am not sure if that is what helmfile template is supposed to be doing or not, but I don't see how, for example, to get the values.yaml that is going to be fed into helm. It's possible you already have implemented the features I'm asking for and I just did not recognize it from the way they were named.

[mumoshu]

@Nuru Thanks for the explanation! Let me recap and summarize how we can possibly proceed.

dry-run (TODO)

So for --dry-run, I expect it to behave exactly the same as the original command without --dry-run, except it skips the "final" side-effective steps that affects the actual cluster.

That being said, I'd expect helmfile sync --dry-run to still does the following side-effective operations:

• Decrypt secrets and writes temporary YAML files under /tmp to be fed to helm upgrade --install
• Run custom helmfile hooks, as we have no way to determine if a hook is side-effective or not. But we can later enhance helmfile to pass an envvar like HELMFILE_DRY_RUN=1 so that each hook can implement their own dry-run mode.

But I expect the following side-effective operations to not happen:

• helm upgrade --install on each desired release
• presync hooks

I also expect the following things to not happen as they aren't executed when --dry-run isn't enabled:

• Run helm get [hooks|manifests|values|notes]
• Run helm template

More logs in --log-level=debug

helmfile --log-level=debug sync currently dumps a bunch of debug messages that includes the rendered helmfile.yaml.

In addition to that, I'd guess adding the following contents to the debug logs would help in regard to this issue and @Nuru's issue:

• Source paths and the resulting contents of all the values files that is rendered and being fed to helm upgrade
  
  
  
  • In @Nuru's case, the contents of "deploy/environments/production/values.yaml" and "deploy/environments/production/secrets.yaml" would be included in the debug log
  
  

helmfile --debug sync (TODO)

so I can see the manifests and values files before they get applied. That is what I would call the --dry-run

This isn't present in helmfile. But I'd suggest adding a helmfile --debug, rather than --dry-run, so that helmfile is able to add --debug flag to helm calls, that works as its name advertises.

You may already know but for example helm install --dry-run --debug stable/mysql adds the followings to the output:

• Hostname of the tiller helm is trying to connect
• Original chart version
• The local file path to the helm chart archive being used
• Release name
• Revision
• Date
• Chart name and the version number
• User supplied values (in YAML)
• Computed values (in YAML)
• Hook manifests
• Release manifests

[debug] SERVER: "MYTILLERHOST:44134"

[debug] Original chart version: ""
[debug] Fetched stable/mysql to /Users/myname/.helm/cache/archive/mysql-0.15.0.tgz

[debug] CHART PATH: /Users/myname/.helm/cache/archive/mysql-0.15.0.tgz

NAME:   lucky-terrier
REVISION: 1
RELEASED: Thu Apr  4 13:39:14 2019
CHART: mysql-0.15.0
USER-SUPPLIED VALUES:
{}

COMPUTED VALUES:
busybox:
  image: busybox
  tag: 1.29.3
#snip

HOOKS:
---
# lucky-terrier-mysql-test
apiVersion: v1
kind: Pod
metadata:
  name: lucky-terrier-mysql-test
  labels:
    app: lucky-terrier-mysql
    chart: "mysql-0.15.0"
    heritage: "Tiller"
    release: "lucky-terrier"
  annotations:
    "helm.sh/hook": test-success
spec:
  initContainers:
# snip

MANIFEST:

---
# Source: mysql/templates/secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: lucky-terrier-mysql
  labels:
    app: lucky-terrier-mysql
    chart: "mysql-0.15.0"
    release: "lucky-terrier"
    heritage: "Tiller"
type: Opaque
data:
# snip

@Nuru I believe adding --dry-run AND --debug to all the helmfile commands would help your usecase.

Do you also need what I summarized in the More logs in --log-level=debug section?

[Nuru]

@mumoshu Thank you for giving this so much attention!

I think you have a handle on all the information I would want, but I have some further suggestions about how to provide it.

There is no point to doing --dry-run except to produce output, so the output should be the default and not require --log-level=debug. I think most tool writers simply stop early, skipping the part where they execute the final step and then delete the temp files. You might add a step to move the temp files into the current directory or a subdirectory of the current directory.

I think by default you should not decrypt secrets during --dry-run but there should be a separate flag enabling that.

Also, I do not want to get lost in a sea of debug messages. It's hard enough getting lost in a sea of manifests. :-). I just want to see the files and commands hemlfile generates. It would be nice if it would extend that capability to actually running helm it it's dry-run mode(s), but I don't really know what those are.

[mumoshu]

@Nuru Hey! Thanks for the response.

I think by default you should not decrypt secrets during --dry-run but there should be a separate flag enabling that.

I'm a bit confused on this, because helmfile needs to decrypt secrets to generate temporary values.yaml files that are fed to helm upgrade commands. Without actually decrypting secrets, we can't print the pathes to actual values.yaml files consumed by helm upgrade. But I thought that's what you wanted.

[mumoshu]

So I got to think the best thing we could do would be that:

1. helmfile --debug <SUBCOMMAND sets the log-level to debug, logs the rendered helmfile.yaml and temporary values.yaml files to STDERR with the "debug" log-level(actually helmfile does this already), AND add --debug flags to helm commands.
2. helmfile --dry-run [sync|apply] skips running helm upgrade and presync hooks(helmfile has various hooks but only presync hooks are considered side-effective), but it still decrypts secrets, run helm diff, etc.
3. helmfile --debug --dry-run [sync|apply] prints rendered helmfile.yaml and temporary values.yaml files and NOT run any side-effective operations.

WDYT?

[sgandon]

this is a good idea, I was always missing this feature but apply is aleady a nice feature to see the changes.
One question though, why output the log to STDERR ?

[mumoshu]

@sgandon Hey! Thx for chiming in.

why output the log to STDERR ?

It turned out that doing so makes it easy to fix helmfile template without bloating the code-base #551

[mumoshu]

apply is aleady a nice feature to see the changes.

Probably after this feature apply will be even more useful, because helmfile --dry-run --debug apply would give both what will be affected according to helmfile diff, and rendered helmfile.yaml/values.yaml files.

[Nuru]

@mumoshu Your proposal for helmfile --debug <SUBCOMMAND> sounds good.

What would be the output of helmfile --dry-run sync? It seems from your description that it would not output much of anything, and would not do anything, which would make it not that useful.

Dry run should print out what hook commands would be run, but not run any of them, not even "cleanup" hooks. You have to treat any opaque command (like a hook) as having side effects.

What I'm really looking for is the ability to see and save:

1. The helmfile after the first pass of template substitution
2. The helmfile after the second pass of template substitution (similar to what is output by --log-level=debug but without line numbers or other debugging information)
3. The generated values.yaml that will be fed into helm.
4. All the commands that will be run, in order.

I think that should be what is enabled by the --dry-run flag, along with inhibiting (but still printing out) any helm commands with side effects.

It would be nice if I could specify a directory and helmfile would put the templated files in that directory, maybe named something like helmfile.out.1.yaml, helmfile.out.2.yaml and values.out.yaml.

[mumoshu]

@Nuru Thanks! I think we're close to something that works.

The helmfile after the first pass of template substitution

This is what --log-level debug mainly give you, so as helmfile --log-level debug sync --dry-run

The helmfile after the second pass of template substitution (similar to what is output by --log-level=debug but without line numbers or other debugging information)

Yeah that's what --log-level=debug mainly give you. Why line numbers matter here? I'm interested!

The generated values.yaml that will be fed into helm.

I think we can enhance --debug to to this. I that ok?

It would be nice if I could specify a directory and helmfile would put the templated files in that directory, maybe named something like helmfile.out.1.yaml, helmfile.out.2.yaml and values.out.yaml.

Sounds interesting. What do you use it for when --debug- or --log-level=debug gives all?

[mumoshu]

Dry run should print out what hook commands would be run, but not run any of them, not even "cleanup" hooks. You have to treat any opaque command (like a hook) as having side effects.

I think prepare and cleanup should be run even in the --dry-run mode, because they are meant for generating temporary local helm charts and cleaning it up.

In case you're using prepare for any work that affects your actual cluster not only local file system, you should use presync hook instead.

[Morriz]

@mumoshu Is there any way of getting the output of helmfile currently? I am searching but can't find any method that just shows me the templated output...what am I missing?

I am using the tiller plugin btw

[Morriz]

@mumoshu please? can we just get the template output?

[Morriz]

I really don't understand why it's not possible to just get the rendered manifests from helm with --dry-run. The myriad of use cases is huge. I need it to just inspect what resources are generated. I am pulling my hair out (I am now bald) because I can't do this now, and have to manually create values files to feed to helm, to be able to test local charts with helm template. What a drag.

[bitsofinfo]

def would like this to be able to store a copy of what helmfile generates and check it into git as part of our deployment audit capability

[mumoshu]

@Morriz Hey! Sorry for the silence.

What's your goal here?

can we just get the template output?
...
I need it to just inspect what resources are generated

helmfile --log-level=debug template would give (1)the final k8s results generated by helm template and (2)the rendered helmfile.yaml.

Does it help?

[mumoshu]

def would like this to be able to store a copy of what helmfile generates and check it into git as part of our deployment audit capability

Interesting! Do you mean that you want to store copies of your deployments including even decrypted secrets?

Perhaps you could share the exact points you want to audit so that we can discuss what's we really need for your usecase

[osterman]

1127 should help in debugging too

[drimailorr]

Hello @mumoshu
Some time ago in this conversation you've been mentioning implementation of own dry-run mode for hooks by using envvar like HELMFILE_DRY_RUN=1
In which state this work at the moment ?

[abdennour]

Wow! 3 years so far. any update ?

i wanna see the final computed values without showing the computed manifests... Just computed values

[Morriz]

we use an alias/function for that:

function hf_values() {
  [ -z "$VERBOSE" ] && quiet='--quiet'
  helmfile ${quiet-} -e "$CLOUD-$CLUSTER" -f helmfile.tpl/helmfile-dump.yaml build |
    grep -Ev $helmfile_output_hide |
    sed -e $replace_paths_pattern |
    yq read -P - 'releases[0].values[0]'
}

[mumoshu]

@abdennour Hey! It's 3 years and the point is that no one has ever successfully defined the definition of "dry-run". Is seeing computed values without showing manifests a dry-run, for example? I think that's arguable. What you should here would be to raise concrete topics like that and help everyone figure out the design for this dry-run thing.

BTW, for what you want, which values are you referring, helm release values or helmfile values?

If helm release values, @Morriz's script looks awesome. Adding --embed-values to invoke helmfile build --embed-values would enhance that further.

If helmfile values, there's no straight forward way to do that. So you'd better submit another separate feature request.

[mumoshu]

So, what if helmfile apply --dry-run added --dry-run only to helm upgrade -install, so that the dry run does call actual helm repo up, helm dep build, helm list, chartify-related things, and finally helm diff upgrade, but actual helm diff upgrade --install?

That way, you're very likely to cover everything that can be tested without affecting the cluster. It does wirtes to your local file system and modifies your local helm repositories cache.

[mumoshu]

Do we need helmfile sync --dry-run, helmfile delete --dry-run, and helmfile destroy --dry-run? If so, why would you think so? Maybe you some output from the command that indicates which releases were being synced or deleted but skipped due to dry-run?

[notjames]

[COPIED FROM https://github.com/roboll/helmfile/issues/1806 as requested by @mumoshu]

My only concern is losing helmfile --args specifically for performing a dry-run. I understand the reason behind
deprecating the --args option, but can we find a compromise in adding a --dry-run ability? That is an incalculable
tool for our purposes.

I haven't had a chance to read through this whole issue yet, but I felt it important to get the concept raised. It might already be addressed here, though.