Helmfile: Lock file

[ausov]

Summary

In order to maintain predictable deployments, as developer I want to generate and use lock file for all chart versions retrieved from a helmfile.

Rationale

We have an environment git repo for deployment with Helmfile. Helmfiles typically specify version range, e.g. "^1.0.0". Charts are developed separately. We're deploying from environment repo to "dev" automatically whenever some chart repo is updated, then to "staging" when chart repo is bumped to a stable version and then to "production" manually after "staging" is verified.

Since there's a chance that a new chart version arrives between deploying to staging and production, it is possible to unexpectedly deploy to production a version that is different from staging (i.e. untested). It is possible because of using version ranges.

Before we were using a huge helm umbrella chart for that purpose.

Therefore, it would be great if we could use helmfile to generate a "lock file" that can be pushed to git repo and then used for deploying _exact_ versions idempotently on "helmfile apply".

Workaround

As a possible workaround, we could maintain a fake umbrella chart in a local dir with all requirements. Instead of a repo in helmfiles, reference local dir of a subchart from umbrella chart. Requires "helm dep build" and "tar -x *.tar.gz" on subcharts dir every time before deployment.

Is this use-case valid for Helmfile usage ...or can you recommend an alternative way to organize an automated CD?

Thank you in advance!

[mumoshu]

@ausov Hey! Thanks for writing this. Yes, I agree that a lock file or a state file(like terraform's) for your helmfile.yaml would be useful for reproducible deployments.

Assuming you'll lock each chart's dependency with requirements.lock, I'd expect a lock file for helmfile.yaml to contain only version numbers of charts referenced in all the releases in all the (sub-)helmfiles. Does that sounds correct?

Then, my question is - how would the user-story from locking to deployment look like? helmfile plan that produces a lock(or state) file, and helmfile apply --lock-file path/to/lock/file.yaml to apply it?

[ausov]

Thank you for quick reply!

Yes, that sounds good to me. Looks similar to terraform style.

There're some edge cases to think about. For example, if the same chart is used in multiple versions. Taking this into account, I would expect the version stored per release name. By the way, how helmfile would process duplicate release names in different helmfiles referenced from another helmfile?

[mumoshu]

Thanks for your response.

how helmfile would process duplicate release names in different helmfiles referenced from another helmfile?

There's really no concept of "duplicate releases" in helmfile. Each (sub-)helmfile.yaml is processed one by one, independently.

Based on that,

the version stored per release name

Sounds almost ok! I'd add the id of helmfiel.yaml, and the relative path to each helmfilele.yaml would work.

[ausov]

That's great! Would be much safer to use in automated deployments. Looking forward to trying this...

[mumoshu]

Just curious, but how would your resulting CI pipeline look like after the suggested feature is available? I'm still not very sure how it will be used in practice.

We're deploying from environment repo to "dev" automatically whenever some chart repo is updated, then to "staging" when chart repo is bumped to a stable version and then to "production" manually after "staging" is verified.

Does the same repo contain the env and the charts(monorepo), or not? All of these works by automatically triggering the new env repo CI build by each new commit made to one of charts repositories?

[ausov]

We have charts and envs stored in different repos for reusability. Chart repo builds trigger dependent env repo builds.

But you've got a point about the CI pipelines. When we use terraform, we have specific branches per environment. The environment plan is therefore not shared between the branches.

So, the lock file is not exactly like the plan. I'm thinking that we almost should commit the file into Git rather than use it only as a deployment plan within the environment branch (which is also useful anyway), otherwise there's no way to re-use it from staging to production.

What do you think?

[ausov]

The question is what the version lock file would be? A configuration (Git) or a deployment plan (artifact)?

I think that most likely, if enabled, it should be the part of config. On the other hand, it really depends on project CI type. So I would call it "requirements lock" like in Helm and not the "plan". Makes sense?

[ausov]

I was thinking that introducing something like deployment plan would be too heavy for the tool. Right now it's really simple and nice. It helps to reduce complexity of custom bash scripting for custom deployment process. If a "plan" or "state" is needed then Terraform or similar tools can be used.

Maybe just a simple addition to consider: optionally generate requirements.lock file for each helmfile and put it into the same folder of the helmfile. This would reduce complexity and would be somewhat similar to Helm behavior.

I suggest to wait for votes from other users and think in the perspective of further development strategy of the tool.

[sstarcher]

If the lockfile does not contain all configurations I'm not seeing what a lockfile really provides vs setting version: for each chart. That's essentially what the lockfile is going to do for you.

[mumoshu]

Yep, I basically agree.

Probably the problem here is that we don't have an automated way to update only chart version numbers in your helmfile? @ausov

For that, I recently conjured up a command named syaml. Running syaml releases[0].version $(somehow-get-the-latest-version-number-of-the-chart yourchartname) would allow you to automate it.

Or more simply, creating an env file containing only the version numbers of the charts, updating version numbers with a way similar to the above wold work?

[ausov]

Yes, that was exactly the use case to update chart versions in CI automatically.

But I agree that the feature is not really needed/expected since Helmfile is not a packaging tool. The workarounds with env/syaml for version bumping is good enough. Thank you!

[sstarcher]

@mumoshu I fully agree having helmfile be able to update itself would be great. Something of a helmfile upgrade. I have no clue what a syaml would even be.

One issue I see with helmfile upgrading itself currently is if you use anchors those will all be blown away and the resulting file will be terrible. So we would either need to stop using anchors or use a different file for controlling the versions...such as a lockfile.

Which actually could make sense in a world where you do version: > 1.2.0 in your helmfile

[mumoshu]

@ausov Thanks a lot for confirmation! Forgive me keeping this open for a while until the whole discussion settles somewhere :)

[mumoshu]

For inspiration, to discover newer chart versions that conforms to a specific semver, you can use:

helm search --regexp '\vstable/mysql\v' --version ">= 0.0.0, < 2.4.0" --versions

For the --regpexp '\v...\v' workaround(?), see https://github.com/helm/helm/issues/3890#issuecomment-382148930.

[mumoshu]

One issue I see with helmfile upgrading itself currently is if you use anchors those will all be blown away and the resulting file will be terrible

@sstarcher Good point! Then we'd need an another file other than helmfile.yaml to store locked chart versions.

[mumoshu]

Would it be okay if helmfile dependency update that is currently used only for updating requirements.lock in helmfile-managed local charts, to additionally update the lock file for helmfile, named e.g. helmfile.lock.yaml?

[mumoshu]

Sketching details on how it works:

The lock file will be updated when and only when helm search --regexp '\vsYOUR_CHART_NAME\v' --version "YOUR_CHART_VERSION_CONSTRAINT" | tail -n 1 | awk '{ print $2 }' differs from the version number stored in the lock file.

helmfile apply would automatically read helmfile.lock.yaml so that it can merge all the version numbers in the lock file into the releases defined in helmfile.yaml.

[sstarcher]

So I have a team that has a usecase.

• Dev gets the latest pre-release - 1.0.0-123
• Stg gets a RC tagged pre-release - 1.0.0-rc.1
• Prod gets a release

They would like to do this in a automated fashion and I believe a lockfile would help with this. Even more so if we could specify to that lockfile update a regex of some sort as semver does not allow you to search for say rc. Just < and >.

Thoughts?

[mumoshu]

@sstarcher The only way to search helm chart versions helm search --regexp can't be used off the shelf for that use-case - --regexp isn't capable of searching version numbers.

So probably the only way would be to pattern match OR parse/filter the version numbers shown in helm search outputs :) And we should also need to enhance helmfile.yaml to accept a pattern for version number.

Can we extend the helm's chart version specifier like the below, for example?

releases:
- name: myapp
   chart: charts/myapp
   ## This is supported by helm
   #version: ""> 1.2.0"
   ## This is supported by helm
   #version: ""> 1.2.0, < 2.0.0"
   ## These are NOT supported by helm. So add a special logic for these to helmfile?
   #version: "1.2.0-\d+"
   #version: "1.2.0-rc.\d+"
   ## Do we want a more machine readable syntax for this? like
   #version: "regexp=1\.2\.0 -rc\.\d+"

[sstarcher]

You bring up a good point that this is getting fairly out of spec for helm/semver. Maybe my plan is a bad idea :)

[mumoshu]

@sstarcher Would it be possible for you to use different version constraints and a lock file per stage?

In helmfile.yaml you would have version: "> {{ .Environment.Values.myapp.version }}{{ .Environment.Values.myapp.minPrereleaseVersionFragment }}, < {{ .Environment.Values.myapp.version }}{{ .Environment.Values.myapp.maxPrereleaseTersionFragment }}" where the version is one behind the next version e.g. 0.9.0 and the fragment min/max can be fixed to.0/.999 for the dev env/stg.

Then you can use whatever tool to bump "myapp.version" after each final release and "helmfile dependency update" to up the proposed lock file?

[sstarcher]

The overall goal is to remove the human out of the version decisions and not have them touching a helmfile version. I think that proposal makes it very complicated and I would be more apt to implement the prior suggestion of an enhanced version constraint.

Another alternative instead of directly enhancing version I could add an additional version-regex that could be applied on top of a version search?

releases:
- name: myapp
    chart: charts/myapp
    version-regex: '.*-rc\..*

[sstarcher]

Another version scheme I could implement is

• Master is a fixed revision such as 0.0.1-latest
• RC tags are generated off of master when ready for staging
• Tags are generated when staging is verified

Talking alternative thoughts and considerations.

[mumoshu]

@sstarcher In your another alternative, I think you'd use version-regex: '.*-rc\..* just to filter the list of all the rc versions, and the filtering happens only when you run helmfile dependency update, not helmfile sync/apply/etc, right?

[mumoshu]

@sstarcher Would there be any downside of using a separate chart or a separate chart repo for dev?

like charts/myapp-dev or dev-charts/myapp for dev, with version: '>0.0.0-0' or devel: true, and charts/myapp and version: '>0.0.0-0' or devel: true for stg, and finally charts/myapp and devel: false for prod?

[sstarcher]

@mumoshu agreed the filtering would only be applied to the lockfile so maybe something specific to that makes more sense. lockfile-filter

Let me think of the pros and cons of doing a separate chart repo.

Pros:

• Accomplishes my goal
• Less confusing than overwriting 0.0.1-latest over and over again
• Full history of all dev-charts
• Can switch between dev/stable fairly easily

Cons:

• ?

Let me pitch this to the team I like it.

[mumoshu]

One more thing to consider: helmfile works on each helmfile.yaml sequentially and independently, and each helmfile.yaml is recommended to be self-contained.

That is, if a sub-helmfile helmfiles/myapp.yaml is referenced from the parent ./helmfile.yaml. You should be able to run them at once with helmfile -f ./helmfile.yaml, AND independently like helmfile -f helmfiles/myapp.yaml sync.

Would that mean that we need to have a "lock file" per helmfile.yaml, rather than having only one globally?

[sstarcher]

I would think you would want a lockfile per helmfile located adjacent to the helmfile. You can ignore my previous pitch about a lockfile-filter. I decided it was too bad of an idea.

[mumoshu]

Thanks for the update!

I've read through all the comments in this thread again. It turns out that helmfile should better align with what @ausov's suggested earlier:

Maybe just a simple addition to consider: optionally generate requirements.lock file for each helmfile and put it into the same folder of the helmfile. This would reduce complexity and would be somewhat similar to Helm behavior.

I think this is a nice idea. I'm now going to make helmfile dependency update to additionally generate requirements.lock adjacent to each helmfile.yaml, and other helmfile commands to read chart version numbers to be used from the lock file.

The implementation would look very similar to the "workaround" @ausov has summarized in the original issue description:

Workaround

As a possible workaround, we could maintain a fake umbrella chart in a local dir with all requirements. Instead of a repo in helmfiles, reference local dir of a subchart from umbrella chart. Requires "helm dep build" and "tar -x *.tar.gz" on subcharts dir every time before deployment.

Probably helmfile will generate a fake umbrella chart, probably a pair of Chart.yaml and requirements.yaml in a temporary directory, copy requirements.lock adjacent to helmfile.yaml to the directory, then run helm to update the requirements.lock.

It isn't elegant(i think) but likely to work, because it is built on the foundation of the "umbrella chart" strategy that we've used and battle-tested for this specific use-case :)

Any thoughts?

[mumoshu]

Note that helmfile dependency update currently work by iterating each local chart referenced by each helmfile, and then running helm dependency update to update requirements.lock for each local chart.

The enhancement extends it to helmfiles themselves.

[mumoshu]

I think there are use-cases that you have multiple helmfile.yaml files in a single directory. So using the single unique name like requirements.lock results in collisions.

I propose that requirements.lock adjacent to helmfile.yaml should be renamed helmfile.lock. It should be helmfile.2.lock when the helmfile config is named helmfilel.2.yaml, helmfile.2.yaml.gotmpl, or helmfile.2.gotmpl.

[mumoshu]

Oops, I forgot that helmfile dependency update is actually helmfile deps today!

[mumoshu]

I noticed that helmfile deps was implemented but was not visible in the cli 😂

So #593 enables helmfile deps, enhances it to produce lock files for helmfile state files, adds documentation and tests for it.