Helmfile: Feature Request/Discussion: Remote Helmfiles?

[osterman]

what

helmfiles:
  - https://raw.githubusercontent.com/cloudposse/helmfiles/0.6.1/helmfile.d/0020.kiam.yaml
  - https://raw.githubusercontent.com/cloudposse/helmfiles/0.7.0/helmfile.d/0100.external-dns.yaml

why

• Create definitions that stitch together different versions of helmfile definitions
• Do not force upgrades of all helmfiles just because one file was updated

use-case

We write a lot of helmfiles used by different organizations. We version pin to a release of our helmfiles distribution, however, sometimes users need to update only one specific service and are not ready to upgrade all services in a given release of helmfiles.

Using the approach above works more like terraform modules empowering the user to surgically import resources into their environment, while keeping it DRY and CODEOWNERS friendly by using a single source of truth.

references

• https://github.com/roboll/helmfile/pull/266

[mumoshu]

@osterman Yeah! This should be very useful.

One gotcha would be that you can't use any values.yaml(.gotmpl) files from within your remote helmfiles, until we have #195.

Do you like this feature implemented even before implementing #195?

[osterman]

@mumoshu it's not urgent at this time. We'd use it if it were there, but we are not blocked by it.

[mumoshu]

@osterman Noted 👍 Thanks.

[johnmarcou]

Hi,

I like the idea of remote helmfiles. These helmfiles can be stored on a remote git repo, and we can point to a specific tag version of the config.

Waiting for remote helmfile support, I am doing this workaround:

values.yaml:

helmfile: https://**************/raw/kube-system/helmfile.yaml?at=refs%2Ftags%2F0.0.2

helmfile:

environments:
  default:
    values:
      - values.yaml

bases:
  - {{exec "sh" (list "-c" "file=`mktemp` && curl -q $0 -o $file && echo $file" .Environment.Values.helmfile)}}

[mumoshu]

@johnmarcou Awesome workaround!

Yeah, this feature gets more interesting as we added bases recently.

One thing I'm missing before going forwad is that we still miss a kind of #195.

That is, remotebase file would allow us to reuse common parts of your helmfile state files. That's great.

But there's no way to distribute "common" values.yaml or even shell scripts that is referenced from the base file, along with it.

Any thoughts?

[johnmarcou]

To be explicit, I'd like to introduce my use case here.

I want to deploy N charts on M clusters to deploy cluster addons (logging, monitoring, ingress). To avoid to rewrite or copy/past the config for each cluster, I use bases.
The cluster addons are declared in different helmfiles, and each clusters has a helmfile using all addons as base. The problem is, when you update a addons helmfile (let's say the Kibana version of the logging helmfile), all M clusters will be impacted. We want to decouple that.

Inspired by Terraform and its module feature, we would do:

• a terraform addons module to deploy generic resources, versionned on a git repo with tags
• have a terraform file for each cluster pointing to a specific version of the terraform addons module

For example:

Cluster A -> Version 1.0
Cluster B -> Version 1.0
Cluster C -> Version 1.1
ClusteDev -> Branch dev

With helmfile, we can apply the same logic.

• a helmfile addons module to deploy generic resources, versionned on a git repo with tags
• have a helmfile file for each cluster pointing to a specific version of the helmfile addons module

GIT_CLUSTER_ADDONS (tags: v1.0, v1.1):
monitoring.yaml
logging.yaml
ingress.yaml
system.yaml

GIT_CLUSTER_MGT:
clusterA/
  helmfiles.yaml (based on the specific version of addons helmfiles:1.0)
clusterB/
  helmfiles.yaml (based on the specific version of addons helmfiles:1.0)
clusterC/
  helmfiles.yaml (based on the specific version of addons helmfiles:1.1)
clusterDev/
  helmfiles.yaml (based on the specific version of addons helmfiles:dev)

===
Now, regarding the config.
In a Terraform scenario, we can set config:

• module data provided by the module: hardcoded in the terraform module, setting are set as default values (usually called variables.tf)
• injected data provided by the consumer: in the terraform, we would override the default values OR give the path to actually data/file via values

For module data, I see two implementations:
a - hardcoded value in the remote helmfile
b - {{exec curl get ""PUBLIC-URL-OF-REMOTE-HELMFILE/values.yaml}} - which I really don't like because of the auto-reference of the public url. Note: Terraform can do that easily since we do a terraform init which git clone all the terraform modules assets locally.

For the injected data, I see two implementations:
c - the helmfile consumer puts data (cluster: clusterA) in the env-values.yaml file, and the remote helmfile expect this data {{.Environment.Values.cluster}} (we can still give a default with getOrNil/default...)
d - the helmfile consumer puts data path (elastAlertRules: myrules.yaml), and the remote helmfile expect this data path {{readFile ...}}
d' - Of course, the env values can be specified at the cluster level (/clusterA/env-values.yaml) or global (/_global/env-values.yaml).

I am using a, c, d and d' so far, depends of the nature of the config.

[johnmarcou]

I guess your main point is about b.

Terraform has the same problematic: load a file from the caller or from the module?
They addressed it like this:

file(path) - Reads the contents of a file into the string.
Variables in this file are not interpolated.
The contents of the file are read as-is.
The path is interpreted relative to the working directory.
Path variables can be used to reference paths relative to other base locations.
For example, when using file() from inside a module,
you generally want to make the path relative to the module base,
like this: file("${path.module}/file").

https://www.terraform.io/docs/configuration-0-11/interpolation.html#file-path-
For example: https://github.com/poseidon/typhoon/blob/2019177b6b823e8b557c5faee3ebf7eb34921937/bare-metal/container-linux/kubernetes/profiles.tf#L35

An (ugly?) way to implement that would be:

- name: mynginx
  chart: nginx
  values:
    - {{moduleFile values.yaml}} (would use the remote values.yaml in the module)
    - values.yaml (would use the local values.yaml file of the caller)

With moduleFile being a Go template function which says: if that helmfile has been used as remote based, then extract and use the BaseUrl to get the remote values file. Which might be really tricky since the raw path depends on the Git provider :/

Again, Terraform doesn't need remote assets path resolving since everything is git-cloned locally during the init.

[osterman]

Also related: https://github.com/roboll/helmfile/issues/469

[osterman]

195 is interesting and might simplify this ask. Perhaps helmfiles with their values ought to be thought of as packages, although that's not a requirement for us. The simple interface of just supporting a list of URLs would work wonders for us as we don't have the same requirements maybe as others.

[mumoshu]

The problem is, when you update a addons helmfile (let's say the Kibana version of the logging helmfile), all M clusters will be impacted. We want to decouple that.

I'm soooo glad to see you had the same concern with me! I'm reading through your whole comment now...

[mumoshu]

@johnmarcou

With moduleFile being a Go template function which says: if that helmfile has been used as remote based, then extract and use the BaseUrl to get the remote values file. Which might be really tricky since the raw path depends on the Git provider :/

I didn't feel your idea of {{moduleFile ...}} ugly.

I might be still missing something, but anyway - I was just thinking about making helmfile (1)git-clone the whole Git repository containing the module into a local module directory, and (2)translate every occurrences of URLs in values:, helmfiles:, bases: to point to respective files in the local module directory.

b - {{exec curl get ""PUBLIC-URL-OF-REMOTE-HELMFILE/values.yaml}} - which I really don't like because of the auto-reference of the public url. Note: Terraform can do that easily since we do a terraform init which git clone all the terraform modules assets locally.

Perhaps my above idea aligns with how terraform's handling modules, especially in the part terraform init which git clone all the terraform modules assets locally, right?

With it, @osterman's example:

helmfiles:
  - https://raw.githubusercontent.com/cloudposse/helmfiles/0.6.1/helmfile.d/0020.kiam.yaml
  - https://raw.githubusercontent.com/cloudposse/helmfiles/0.7.0/helmfile.d/0100.external-dns.yaml

will end up helmfile git-cloning the 0.7.0 and 0.6.1 tag/branches of cloudposse/helmfiles repository to local module directories like:

• $HELMFILE_MODULE_CACHE/github-cloudposse-helmfiles-0.7.0 and
• $HELMFILE_MODULE_CACHE/github-cloudposse-helmfiles-0.6.1

respectively.

For module data, I see two implementations:
a - hardcoded value in the remote helmfile

This would work as before.

For the injected data, I see two implementations:
c - the helmfile consumer puts data (cluster: clusterA) in the env-values.yaml file, and the remote helmfile expect this data {{.Environment.Values.cluster}} (we can still give a default with getOrNil/default...)

Yep.

This works forbases, as bases are designed to propagates the env values from the parent to the child=module.

For helmfiles, probably we want #523. Do you like #523 as well, @johnmarcou?

d - the helmfile consumer puts data path (elastAlertRules: myrules.yaml), and the remote helmfile expect this data path {{readFile ...}}

Great! I think this works as before even after the above enhancement.

d' - Of course, the env values can be specified at the cluster level (/clusterA/env-values.yaml) or global (/_global/env-values.yaml).

I couldn't understand this - How does the module helmfile discover clusterA? Or isn't discovery necessary?

I am using a, c, d and d' so far, depends of the nature of the config.

Awesome. I think you've almost given us the solution I couldn't come up 👍

[mumoshu]

I didn't feel your idea of {{moduleFile ...}} ugly.

@johnmarcou What made you think it may be ugly? Was it related to the syntax, or anything else?

[osterman]

@mumoshu I think you're on to something!!

I think what you are proposing sounds quite familiar to .terraform/modules and how it caches them locally. That could work. Perhaps different invocation syntax than my original example. Instead something like....

helmfiles:
- git::https://github.com/cloudposse/helmfiles.git//releases/kiam.yaml?ref=tags/0.40.0
- git::https://github.com/cloudposse/helmfiles.git//releases/external-dns.yaml?ref=tags/0.39.0

What this says is to clone https://github.com/cloudposse/helmfiles.git and checkout tags/0.40.0 then run the helmfile in releases/kiam.yaml (using terraform's source notation). It would then use all the other files relative to that checkout.

This can support private repos using (a) ssh or (b) using git-credentials helper and setting the appropriate envs.

[mumoshu]

@osterman Thanks for confirming!

It would then use all the other files relative to that checkout.

Yeah. I think this is an important part.

[mumoshu]

This feature is implemented for helmfiles in #648.

The helmfile module cache is currently hard-coded to something like $PWD/.helmfile/cache/$MODULE_ID whereas MODULE_ID would look like https_github_com_cloudposse_helmfiles_git.ref\=0.40.0/.

Ideas for further feature requests:

• Ability to customize the module cache dir by an envvar(HELMFILE_CACHE? HELMFILE_HOME?)
• Support for remote URLs in bases

[johnmarcou]

Hi @mumoshu

@johnmarcou What made you think it may be ugly? Was it related to the syntax, or anything else?

I was thinking about a function which take a URL from the sub-helmfile to rewrite/generate the url on the assets, from that URL. I feel it ugly since all git providers doesn't use the same way/structure to provide access to the raw version of the assets.

For example:

• bitbucket: https://bitbucket.company.org/projects/DPI/repos/applications-management/raw/README.md?at=refs%2Fheads%2Fmaster
• github: https://raw.githubusercontent.com/roboll/helmfile/master/README.md

So it would be hard to have a consistent function for that.

But if helmfile is using git clone to have the sub-helmfile (assets included) locally, it sounds really good to me.

For helmfiles, probably we want #523. Do you like #523 as well, @johnmarcou?

Regarding the config propagation, I would say there is 2 different needs:

1 - helmfile module (bases:): give a helmfile-module some values, to override the default set in the helmfile-module. We want to use a helmfile as a module, so each config needs to be overridden. We would use environment values so far as a workaround for that.

2 - sub-helmfile (helmfiles:): propagate values from helmfiles chain. Here, we want the "closest" config takes precedence.

Let's take that example:

# helmfileA:
helmfiles:
- helmfileB

# helmfileB:
helmfiles:
- helmfileC

# helmfileC:
bases:
- helmfileD

# helmfileD:
releases:
- name: myrelease

If we set myenvval in many place like this:

helmfileA: myenvval=1
helmfileB: myenvval=2
helmfileC: myenvval=3 <-- wins
helmfileD: myenvval=4

My opinion is it should result as myenval=3.

In practice, for example, myenvval could be a SlackChannel value:

helmfileA: myenvval=#it-group
helmfileB: myenvval=#software-team
helmfileC: myenvval=#project-team
helmfileD: myenvval=#your-slack-channel

[mumoshu]

@johnmarcou Thanks for the response!

But if helmfile is using git clone to have the sub-helmfile (assets included) locally, it sounds really good to me.

Glad you liked it :) To be extra clear, that's how #648 has been implemented.

1 - helmfile module (bases:): give a helmfile-module some values, to override the default set in the helmfile-module. We want to use a helmfile as a module, so each config needs to be overridden. We would use environment values so far as a workaround for that.

I believe it's a valid use-case.

One thing I'm not decided yet is that how we should pass values to the module(I call it a "layer" or a "base" btw).

The current bases implementation added via #587 works by implicitly inheriting environment values from the parent to the child(=base). That seems to deviate from what we have for helmfiles, cuz you must explicitly specify values to be overrode for sub-helmfile like:

helmfiles:
- path: path/to/sub/helmfile.yaml #or go-getter url
  values:
  - key1: val1
  - overrides.yaml

I'm wondering if it's ok to change bases to require explicitness as well, like:

bases:
- path: path/to/module.yaml
   values:
  - key1: val1
  - overrides.yaml # all these overrides are available via e.g. {{ .Values.key1 }} within the module.yaml

Does that sound good to you?

[johnmarcou]

That sound's absolutely perfect to me.

Environment values should be propagated from parent to child (the closest wins) and use as ... env values as it was the initial purpose.
Modules should gets the values via values:. It is really similar to Terraform-way. I like it!

I look forward to see that features happen! Thank you for that.

[mumoshu]

@johnmarcou To be extra clear, I'm assuming in https://github.com/roboll/helmfile/issues/347#issuecomment-498908612 myenvval=3 in helmfileC wins when and only when an expression to be evaluated like {{ .Values.myenvval }} is appeared WITHIN helmfileC, right?

[mumoshu]

It really depend on how/where you set values but:

Change bases to require explicitness

means that your example must be rewritten as follows to propagate values at all:

# helmfileA:
values:
- myenvval: 1

helmfiles:
- path: helmfileB
  valuesInherited: true
  # Alternatively:
  #values: {myenvval: 1}}

whatvever: {{ .Values.myenvval }}
// => evaluates to "whatever: 1"

# helmfileB:
values:
- myenvval: 2

whatvever: {{ .Values.myenvval }}
// => evaluates to "whatever: 1"

And for bases:

# helmfileA:
values:
- myenvval: 1

bases:
- path: helmfileB
  valuesInherited: true
  # Alternatively:
  #values: {myenvval: 1}}

whatvever: {{ .Values.myenvval }}
// => evaluates to "whatever: 1"

# helmfileD:
values:
- myenvval: 2

whatvever: {{ .Values.myenvval }}
// => evaluates to "whatever: 1"

And the "cloest" one wins when no values are passed explicitly:

# helmfileA:
values:
- myenvval: 1

helmfiles:
- path: helmfileB

whatvever: {{ .Values.myenvval }}
// => evaluates to "whatever: 1"

# helmfileB:
values:
- myenvval: 2

whatvever: {{ .Values.myenvval }}
// => evaluates to "whatever: 2"

The "cloest" one wins also when no values are passed explicitly for bases (the change i'm proposing):

# helmfileA:
values:
- myenvval: 1

bases:
- path: helmfileB

whatvever: {{ .Values.myenvval }}
// => evaluates to "whatever: 1"

# helmfileB:
values:
- myenvval: 2

whatvever: {{ .Values.myenvval }}
// => evaluates to "whatever: 2"

[mumoshu]

@bitsofinfo Sharing this thread with you, as I believe you're the biggest user of bases feature 😃

TL;DR; I'm making a breaking change on bases, that stops implicit inheritance of parents' values to bases, so that the u/x becomes consistent with helmfiles.

[bitsofinfo]

for what im doing now, will there be a workaround? can read this right at the moment but what changes?

so what you mean is if one "base" declares a {{ $var := something }} it won't be visible in subsequent bases?

[mumoshu]

@bitsofinfo

so what you mean is if one "base" declares a {{ $var := something }} it won't be visible in subsequent bases?

Yes. Helmfile is already working as such for that point, and remains so.

can read this right at the moment but what changes?

Values will not be inherited implicitly after the proposed change.

for what im doing now, will there be a workaround?

You'll be able to pass values explicitly from the parent by:

bases:
- path: base.yaml
  values:
    foo: foo_overrode

So that in base.yaml you get:

{{ .Values.foo }} => evaluates to foo_overrode

Or you can specifically instruct helmfile to inherit values like this:

values:
- foo: foo_inherited

bases:
- path: base.yaml
   valuesInherited: true

So that in base.yaml you get:

{{ .Values.foo  }} => evaluates to "foo_inverited"

[bitsofinfo]

ok, so from what I can tell, it should all keep working, especially w/ that valuesInherited flag. Nice.

[mumoshu]

@bitsofinfo Great! Thanks for confirming.