Helmfile: Using helmfiles, but unable to pass Environment down to helmfile.yaml

[nikolajbrinch]

When using

{{ readFile "environments.yaml }}}
--- 
helmfiles:
- ./*/helmfile.yaml

Environment.Values are not passed to the globbed helmfiles. I would expect that to happen/work, in order to keep helmfiles DRY.

[Tarick]

The same issue, using helmfile version v0.40.1

{{ readFile "common.yaml" }}
---
{{ readFile "environments.yaml" }}

With debug:

Processing helmfile.yaml
could not deduce environment: block, configuring only .Environment.Name. error: failed to read helmfile.yaml: environment "dev" is not defined
error in first-pass rendering: result of "helmfile.yaml":

Once I put environments block in helmfile.yaml, readFile of environments.yaml succedes as well and I have 2 environments blocks in the output.

[mumoshu]

@nikolajbrinch Hey! Your case isn't supposed to work as of today. That's because each helmfile is meant to be self-contained. Injecting environments like prod/staging/testing from the "parent" helmfile into "child" helmfiles is considered harmful because it works like an implicitly dependency from children to the parent.

I'll shortly reconsider though, about how we could support "inline environment values" and "injecting environment values from the command-line". Once we support either or both of them, I think it will justify passing environment values from the parent to the child.

[mumoshu]

@Tarick Hey! Just to clarify, are you talking about the fact that helmfile doesn't propagate the environment and the environment values from the parent to children? Or you're maybe talking about a possible bug(?) that multiple yaml docs declared within a single helmfile.yaml doesn't get merged?

[nikolajbrinch]

Yes. In my environments.yaml I declare environments and load defaults for environments. This is host-names etc. Data/variables are global for the complete cluster environment. It would be very tedious an error prone to have those replicated across many projects.
This also happens to be certificates etc, that needs updating once in a while. It would be nice to have these set globally, and change them, and run the admin-helmfile orchestra to upgrade all certificates, hostnames etc.

[Tarick]

@Tarick Hey! Just to clarify, are you talking about the fact that helmfile doesn't propagate the environment and the environment values from the parent to children? Or you're maybe talking about a possible bug(?) that multiple yaml docs declared within a single helmfile.yaml doesn't get merged?

The latter case. Basically, I tried to do what https://github.com/roboll/helmfile/blob/master/docs/writing-helmfile.md#layering recommends and this fails.

[nikolajbrinch]

@mumoshu I think one way could be conventions. i.e. environment.yaml declares environment. Just like values.yaml in Helm Charts declares values.
I really like this project and uses it for our infrastructure, but I have a couple of questions:

1. Why the xyz.yaml.gotmpl extension, when Helm uses _xyz.tpl? It seems that it would be easier to adopt if conventions where shared between helm and helmfile.
2. Propagation of values could be handled with global as in Helm.

Environment is handled specially, and I think this should be the case. But a convention would be nice, where I do not need to readFile a helmfile, but a file with a special name, prefix/postfix or something else is automatically merged into .Environment.
One example is Spring Boot that has application.yml as default configuration, and application-dev.yml for the dev environment, configurations are then merged.
This could be a convention in helmfile, where environment.yaml , and environment-dev.yaml are the analogs, and are merged, and available thoughout the chain of helmfiles executed.

[nikolajbrinch]

@Tarick I couldn't get Layering to work either.

[Tarick]

One example is Spring Boot that has application.yml as default configuration, and application-dev.yml
for the dev environment, configurations are then merged.

I use somethink like that in example below. The values files that are supplied to Helm are then merged by Helm.

- name: {{ .Environment.Name }}-api
    labels:
      app_name: "api"
      tier: "apps"
    chart: "somerepo/springboot-app"
    version: 0.1.0
    namespace: {{ .Environment.Values.namespace }}
    values:
      - "environments/default/api.yaml.gotmpl"   # Initial default settings
      - "environments/{{ .Environment.Name }}/api.yaml.gotmpl"   # Overrides

[sruon]

After debugging for a couple hours, I don't see how Layering can work the way it's described in the documentation.

https://github.com/roboll/helmfile/blob/master/tmpl/tmpl.go#L24 parses the file and does not know about the environment yet. If you use a value loaded from an environment, it will just choke on the unknown value and stop loading there i.e.

{{ readFile "environments.yaml"}}
---
releases:
  - name: my-app-{{ .Environment.Values.releaseName }}

I toyed around with splitting the input on --- and parsing each part as a separate fragment / reloading the environment on each loop and that appears to be almost entirely working.
The only issue I'm seeing is the templated values cannot be used as part of a release name, I think the first pass renderer needs to be modified similarily.

```diff --git a/main.go b/main.go
index 399109f..00fc2de 100644
--- a/main.go
+++ b/main.go
@@ -750,20 +750,26 @@ func (r *twoPassRenderer) renderEnvironment(content []byte) environment.Environm

func (r twoPassRenderer) renderTemplate(content []byte) (bytes.Buffer, error) {
// try a first pass render. This will always succeed, but can produce a limited env
- firstPassEnv := r.renderEnvironment(content)
+ splitContent := bytes.Split(content, []byte("---"))
+ var yamlBuf bytes.Buffer

• secondPassRenderer := tmpl.NewFileRenderer(r.reader, filepath.Dir(r.filename), firstPassEnv, r.namespace)
• yamlBuf, err := secondPassRenderer.RenderTemplateContentToBuffer(content)
• if err != nil {
• firstPassEnv := r.renderEnvironment(content)
• for _, subContent := range splitContent {
• secondPassRenderer := tmpl.NewFileRenderer(r.reader, filepath.Dir(r.filename), firstPassEnv, r.namespace)
• subBuf, err := secondPassRenderer.RenderTemplateContentToBuffer(subContent)
• if err != nil {
• if r.logger != nil {
• r.logger.Debugf("second-pass rendering failed, input of \"%s\":\n%s", r.filename, prependLineNumbers(string(subContent)))
• }
• return nil, err
• }
  if r.logger != nil {
• r.logger.Debugf("second-pass rendering failed, input of \"%s\":\n%s", r.filename, prependLineNumbers(string(content)))
• r.logger.Debugf("second-pass rendering result of \"%s\":\n%s", r.filename, prependLineNumbers(subBuf.String()))
  }
• return nil, err
• }
• if r.logger != nil {
• r.logger.Debugf("second-pass rendering result of \"%s\":\n%s", r.filename, prependLineNumbers(yamlBuf.String()))
• }
• return yamlBuf, nil
• yamlBuf.WriteString(subBuf.String())
• firstPassEnv = r.renderEnvironment(yamlBuf.Bytes())
• }

• return &yamlBuf, nil
  }

  func (a app) VisitDesiredStates(fileOrDir string, converge func(state.HelmState, helmexec.Interface) (bool, []error)) error {```

[sruon]

To amend my previous comment, the loadEnv function always passes an empty environment to the renderer https://github.com/roboll/helmfile/blob/98088499867ed4deab4430e2cdae4776336236e3/state/create.go#L123

Changing environment.EmptyEnvironment to environment.Environment{Name: name, Values: envVals} gives me the behavior I'm looking for.

Wonder if we could add that behavior as a flag?

[mumoshu]

Hey! Sorry for the delayed response. I'll comment one by one.

@sruon

{{ readFile "environments.yaml"}}
---
releases:
  - name: my-app-{{ .Environment.Values.releaseName }}

This doesn't work because at the time of the first-render(also see #308 for more context) the whole yaml file including before/after the separater --- are rendered altogether. At that time, {{.Environment.Values.releaseName }} isn't populated yet. The second-render works. So you need a special treatment to pass the first-render anyway. That is, use getOrNil like name: my-app-{{ .Environment.Values | getOrNil "releaseName" }}.

[mumoshu]

I use somethink like that in example below. The values files that are supplied to Helm are then merged by Helm.

@Tarick Your example looks awesome! I think you're correctly using the helmfile features as I have expected :)

[mumoshu]

I think one way could be conventions. i.e. environment.yaml declares environment. Just like values.yaml in Helm Charts declares values.

@nikolajbrinch Sounds nice! I do want more conventions so that every helmfile project looks familiar, which is great in regard to maintainability.

Only thing I'm missing is the good convention. Assuming there are a bunch of yaml files in the top-level of every project, giving it a too generic name would express less connection to helmfile. But helmfile.environments.yaml does look too verbose either..

Why the xyz.yaml.gotmpl extension, when Helm uses _xyz.tpl? It seems that it would be easier to adopt if conventions where shared between helm and helmfile.

Just because I thought it is clearer. When I named it, .tpl was thought to be used by not only helm but other template engines as well.

But I'm fine with adding .yaml.tpl as an alias if you like it. But note that helm's .tpl is used by only template helpers. Templated manifest files under chart's template/ directory themselves are suffixed with .yaml even though they are templates.

[mumoshu]

I have a vague idea of allowing helmfile to layer multiple helmfile.yaml files before parsing and calling, like kustomize and kasane do, if it helps:

helmfiles:
- layers:
   - global/environments.yaml
   - helmfiles/myapp-that-depends-on-the-global-env.yaml
- hemlfiles/myanotherapp-that-has-its-own-env.yaml

[joshwand]

I am also having difficulty with this-- I have a situation where I have many production environments (one per customer), and I want to put them in separate helmfiles to keep the file readable, but then I can't use the environment vars in the base file brought in via readfile:

helmfile-dev.yaml

environments: 
  ci-master:
    values: 
    - values-ci-master.yaml 
  dev1:
    values: 
    - values-dev1.yaml

{{ readFile "../Helmfile-app-base.yaml" }}

Helmfile-prod-customer1:

environments: 
  prod-customer1:
    values: 
      - prod-customer1/prod-customer1.yaml
  prod-customer1-sandbox:
    values: 
      - prod-customer1/prod-customer1-sandbox.yaml

{{ readFile ../Helmfile-app-base.yaml }}

but the values from the evnironment aren't rendered in Helmfile-app-base.yaml, which looks like this:

releases:
- name: app-{{ .Environment.Name }}
  namespace: {{ .Environment.Values.namespace }}
  chart: ../charts/myapp
  values: 
    - {{ .Environment.Values }} 

because the context is not passed to readFile ?

[rajiteh]

I was also thrown off by this bug, the readme specifically says this should be possible.

Let's assume that your helmfile.yaml looks like:

{ readFile "commons.yaml" }}
---
{{ readFile "environments.yaml" }}
---
releases:
- name: myapp
  chart: mychart

Whereas commons.yaml contained a monitoring agent:

releases:
- name: metricbaet
  chart: stable/metricbeat
````
And environments.yaml contained well-known environments:

environments:
development:
production:

but when I run `helmfile -e test lint` I see the following error:

could not deduce environment: block, configuring only .Environment.Name. error: failed to read helmfile.yaml: environment "test" is not defined
```

perhaps the readme should be updated to remove the invalid example.

[royjs]

I also stumbled an issue related to this.

I have a working helmfile that looks like this:

environments:
  dev:
  prod:

templates:
  default: &default
    missingFileHandler: Debug
    values:
      - values/common.yaml.gotmpl 
      {{- range $overrideFolder := .Environment.Values.overrideFolders }}
      - values/{{ $overrideFolder }}/{{`{{  .Release.Name }}`}}.yaml
      {{- end }}

releases:
  ....

I tried to move the environments out of this file and use '{{ readFile "environments.yaml" }}' to include it. When I do that, it stops working because the environment values are not available yet. I also tried delaying the parsing of the templates to the 2nd pass but the issue is that the yaml won't be valid (because of the range loop part) and the parsing fails, it needs to be done on the 1st pass.

I think it would make more sense to handle the environments separately so that they are always available in the state for the first pass. I'm not sure about the exact convention we should use but having a (optionally?) separate file for it makes sense.

[sboschman]

Would be great to be able to combine layering with the directory layout from glob patterns (multiple independent files), helmfile environments and release templates. :heart:

Our almost 300 lines of templated helmfile.yaml, with 24 templated value files and 26 environment specific value files (of which some are templated again) are not very DRY :rofl:
(counting 2817 lines of yaml in total :roll_eyes:)

Non-working idea:

• helmfile.yaml
• environments.yaml
• helmdefaults.yaml
• repositories.yaml
• templates.yaml
• apps
  
  
  
  • app1
  
  
  • helmfile.yaml
  
  
  • app2
  
  
  • helmfile.yaml
  
  

Example environments.yaml:

environments:
  default:
    values:
    - dev.yaml
  previder:
    values:
    - prod.yaml

The helm defaults are specified for a particular environment. Without layering this works. Example helmdefaults.yaml:

helmDefaults:
  kubeContext: "{{ .Environment.Values | getOrNil "kubeContext" }}"
  tillerNamespace: "{{ .Environment.Values | getOrNil "tillerNamespace" }}"

Without layering the release templates are rendered. With layering I get stuck on unknown anchor 'default' referenced. Example templates.yaml:

templates:
  default: &default
    chart: stable/{{`{{ .Release.Name }}`}}
    namespace: {{ .Environment.Values | getOrNil 'namespace' }}

    # this might be oke instead of defining the values on the environment and fix the namespace difference between environments instead
    values:
    - "./config/kubernetes-dashboard/{{ .Environment.Name }}.yaml"

Example apps/app1/helmfile.yaml:

{{ readFile "../../environments.yaml" }}
---
{{ readFile "../../repositories.yaml" }}
---
{{ readFile "../../helmdefaults.yaml" }}
---
{{ readFile "../../templates.yaml" }}
---

releases:
- name: consul
  version: ">= 3.5.0"
  <<: *default

[ufou]

I was about to log a new issue with this precise problem, is there any traction on this feature? It seems like an obvious thing to support and it would really help with DRY

[embik]

I'm not sure it's necessary, but here's a use case that misses this feature as far as I can tell. We are managing multiple clusters via helmfile and there are certain "global" values that I would like to pass only once to helmfile. For example, a cluster ID / name that is used for all ingress hostnames.

I tried serveral things. This looks like it does not work because of this issue:

values.yaml:

clusterID: 010101

helmfile.yaml:

environments:
  - default:
    values:
      - values.yaml

helmfiles:
  - helmfiles/prometheus.yaml

helmfiles/prometheus.yaml:

releases:
  - name: prometheus
    namespace: prometheus
    chart: stable/prometheus
    values:
      - values.yaml.gotmpl

helmfiles/values.yaml.gotmpl:

server:
  ingress:
    hosts:
      - prometheus.{{ .Environment.Values.clusterID }}.clusters.lcoal

I also tried including the helmfiles via readFile, but unfortunately their configuration will not get templated and therefore this takes away a considerable amount of features (the hook example from the README won't work with that, for example).

Passing down the configured environment via helmfiles would really be awesome!

[mumoshu]

I started to think we need #96 proposed by @gtaylor for this.

With that we could reuse whatever helmfile.yaml fragment(s) like:

includes:
- common-environments.yaml

And you'll explicitly repeat it in every sub-helmfile. Btw, I think the explicitness is a must-have, because implicitness means that we rely on "globals" #398, and I believe globals should be implemented by environment variables(not values) and the {{ env ... }} template function.

Once this comes reality, I'd deprecate the current layering system with multi-doc YAML as unnecessary.

WDYT?

[mumoshu]

includes seems a bit ambiguous whether it is able to override contents of a helmfile.yaml or becomes the "base" of it. The latter is what I think we need.

So should it better be bases?

bases:
- common-environments.yaml

[royjs]

Yeah for me includes would mean that the content of these files would be included in the helmfile.yaml, that implies that it might overwrite it. We need to use a word that it makes it clear that it's the other way around. I think that bases work

[mumoshu]

@embik Hey, thanks for chiming in!

For global variables, I'd suggest using environment variables with {{ env ... }} template expressions as suggested in #398.

Otherwise you can define environments: section in helmfiles/prometheus.yaml, rather than helmfile.yaml.

[mumoshu]

@royjs Thanks for your confirmation ☺️

[embik]

@mumoshu I'm aware, but defining environments: sections in around 20 files (and counting) gets quite repetitive over time, so I was looking for a way to pass down values. I'll take a look at increased use of environment variables for that, thank you.

Regarding the bases keyword, just another data point: Looks fine to me, doesn't solve my problem the way I'd prefer it but I definitely think it would be useful.

[mumoshu]

@embik Thx!

Would you mind clarifying a bit more about your use-case?

I was thinking bases would also help your use-cases by allowing you to reduce the amount of repetitions, from a few lines of environments: in around 20 files, to one or two lines of bases: ["your-common-environment"].

[mumoshu]

@embik FYI, I was considering to allow passing adhoc environment values to a helmfile in #523. But it would work only for a parent helmfiles. You'll be able to explicitly map part/all of adhoc environment values from the parent to sub-helmfiles as explained in #523, but it won't work as global variables.

Envvars(not helmfile environment values) will remain the only way in case what you want is basically globals.

I'm just summarizing my idea. If you have any suggestion how we could improve helmfile further, please feel free to share! Thx.

[embik]

Hey @mumoshu, we're using helm + helmfile as tools to manage the base configuration of many clusters. Several things (e.g. the ingress urls) make sense to be built on shared values.

I think #523 doesn't cover my approach (either symlinking helmfiles into a helmfile.d/ directory or load helmfiles via regex in a central helmfile using the helmfiles directive) but bases: looks better the more I look at it, something like this:

environment.yaml:

clusterID: id1234

base.yaml:

environments:
  default:
    values:
      - environment.yaml

helmfile.d/01-prometheus.yaml: (this is a symlink to a "helmfile repository" which contains all helmfiles):

bases:
  - ../base.yaml
releases:
  - name: prometheus
    namespace: prometheus
    chart: stable/prometheus
    values:
      - values.yaml.gotmpl

helmfile.d/values.yaml.gotmpl:

server:
  ingress:
    hosts:
      - prometheus.{{ .Environment.Values.clusterID }}.clusters.local

Is that how it is supposed to work?

[ufou]

I'm sure I'm at risk of being shot down by suggesting this (and I'm probably missing the bigger picture here), but If I don't say it, I'll regret it.

What about the possibility of having a pluggable backend for config? My primary suggestion would be to use hiera, it is super powerful - it allows you to control behaviour of how objects/properties are merged (or not), allows overrides at a granular level specified in a config file. It also has the option of supporting encrypted values.

These guys are using it https://github.com/cststack/k8comp

If there is no chance of doing something like that, then I would still opt for having someway to have some common.yaml that can be included, personally I think that an included file should have lower priority - meaning that if a property is set in both common.yaml and yaml file including the common.yaml, then the property value in common.yaml would be overridden - annoyingly this still probably means having duplication because afaik there would be no way to merge a nested property (which would be ideal)?

The short reply is that it would be great to make the helmfile.yaml more DRY and I'd be open to helping if this gains traction

[mumoshu]

@ufou Hey! Thanks for the suggestion.

In short I'm personally fine to introduce a pluggable backend(everyone wants different things, like vault for config store, lua, spruce for templating engine, gomplate for both and so on...)

What I'm not sure is what the plugin interface should be.

For the pluggable config store interface, I'd love it if it can be integrated to the secrets cache #444.

For the pluggable templating engine interface, I'd love it if we could specify one a engine per file.

If you have any concrete idea, please feel free to share!

I would still opt for having someway to have some common.yaml that can be included, personally I think that an included file should have lower priority - meaning that if a property is set in both common.yaml and yaml file including the common.yaml, then the property value in common.yaml would be overridden -

Is it basically bases: I've suggested above?

annoyingly this still probably means having duplication because afaik there would be no way to merge a nested property (which would be ideal)?

Yep, we have no way to deal with it as of today.

A tool like spruce has a feature-rich YAML-based DSL that supports operations like "merge", "prepend" and "append". But there are dozens of choices and I think we can never agree on a single solution!

it would be great to make the helmfile.yaml more DRY

Yeah I agree!

Actually, you can use a tool of your choice to generate helmfile.yaml and values.yaml files if your concern is mostly DRY.

For auditing purpose, you'd want to see the final, rendered helmfile.yaml and values.yaml files, anyway, before applying. So that you and your colleagues are confident in how it affects, e.g., your production cluster. That being said, the more you use helmfile's out-of-box templating, the more it gets hard to audit.

I even considered to add helmfile build, that renders helmfile templates to produce plain helmfile.yaml and values.yaml files, which is then commited to your Git repository for GitOps. In this picture, helmfile build is solely used to make configs DRY, which can be replaced with a tool of your choice, and you can still use helmfile to apply the resulting plain helmfiles.

[ufou]

For the pluggable config store interface, I'd love it if it can be integrated to the secrets cache #444.

Hiera would likely solve that one - in that it would be hiera decrypting the yaml and that would (in theory) only be called once when invoked by helmfile, I need to read up about the internals of helmfile to see if I could actually help with this or if it's beyond my capabilities!

[mumoshu]

Hiera would likely solve that one - in that it would be hiera decrypting the yaml and that would (in theory) only be called once when invoked by helmfile

@ufou Thanks for confirming! My point was that, helmfile currently calls one command per one secret decryption, which results in decrypting a secret over and over if it's used from many helmfile.yaml files.

So, in case hiera doesn't have a persistent cache of decrypted secrets(unlikely due to the security reason!), we'll probably still need to enhance something on helmfile-side. But I'm not sure how hiera can be integrated actually(the store interface thing i wrote above

[mumoshu]

@embik

Is that how it is supposed to work?

Exactly!

[mumoshu]

@embik I think I need to add a small correction.

I'm considering the specification of the feature as, any files defined under bases: would be merged as is to the callee helmfile.yaml.

That is, as you write base.yaml like:

environments:
  default:
    values:
      - environment.yaml

environment.yaml must be placed within the same directory the callee helmfile.yaml is in.

Is that ok for you?

[embik]

@mumoshu I guess that makes sense as the base.yaml is supposed to be a blueprint providing "defaults" for the callee helmflile. Totally fine be me, thank you for keeping me in sync.

[orbiran88]

I'm truly waiting for the 'bases' feature! Thanks @mumoshu

[mumoshu]

@embik @orbiran88 Thanks for your responses!

I think bases makes certain value by itself so I'm inclined to add it to helmfile.

But let me clarify what it solves and not, so that we don't settle on a sub-optimal solution to all the different use-cases mentioned in this issue. I'd appreciate any comment, as always.

---

My biggest concern for now is that bases itself doesn't solve @sboschman 's use-case described in https://github.com/roboll/helmfile/issues/388#issuecomment-468718386. That is, you can't use YAML anchors to refer to objects marked with *id by &id ACROSS YAML files.

So the example below won't work:

base.yaml:

templates:
  envoy: &envoy
    chart: stable/envoy
    values:
    - foo: bar

helmfile.yaml:

bases:
- base.yaml
releases:
- name: myenvoy1
  <<: *id #<-fails because the anchor &envoy is lost when `base.yaml` gets loaded

One way to solve this is to introduce a dedicated syntax for cross-file reference.

For example, spruce has inject for that.

For illustration purpose, integrating spruce into helmfile would allow you to rewrite the above example with:

templates:
  envoy:
    chart: stable/envoy
    values:
    - foo: bar
releases:
- .: (( inject templates.envoy ))
  name: myenvoy1

Now how templates: can be reused across multi helmfile.yaml files?

Perhaps we leverage the multi yaml doc support that helmfile already have, in combination with bases:

bases:
- base.yaml
---
releases:
- .: (( inject templates.envoy ))
  name: myenvoy1

This works by helmfile calling spruce to merge all the yaml files listed under bases, and then calling the second spruce to merge the first YAML doc and the second.

Otherwise I think we can lean toward to the approach of generating helmfile.yaml by spruce without introducing bases nor utilizing the multi-yaml-docs support. For example https://github.com/JulzDiverse/aviator can be used. But doing so would greatly change the U/X of helmfile.

[mumoshu]

bases won't support templates. Are you guys ok with that?

It is possible to support templates that requires Environment.Name only, but harder for ones requires .Environment.Values, because we need to load bases to build the values. That's a chicken-and-egg problem :)

[mumoshu]

Helmfile has double-rendering #308 that may complicate implementing bases.

That is, we would need this processing order to make bases work like we would expect:

1. Load helmfile.yaml as a go text/template. It isn't parsed as YAML yet.
2. Run the "first-render" with the template context contains only .Environment.Name, so that it (is expected to) becomes a valid YAML.
3. Parse the YAML. Now that we can read bases: from the YAML data, load files listed under bases: one by one, and merge it to the single golang map(we call this base map).
4. Merge the YAML data produced by step 2 to the base map.
5. Run the second-render. This time helmfile uses the outcome of step 4 above as the template context.

cc/ @davidovich

UPDATE: The above turned out unintuitive that I was unable to fully understand how it's supported to work. So I've simplified it as follows:

1. Load helmfile.yaml as a go text/template. It isn't parsed as YAML yet.
2. Double-render the template
3. Parse the YAML. Load "base" helmfiles referenced from it. Merge all the "base` helmfiles and the helmfile being loaded. Done

[mumoshu]

Also see #584

[mumoshu]

To summarize, the current Layering system doesn't work as advertised, as it relies on helmfile to template each "part" of your helmfile.yaml THEN merge them one by one.

The reality was that helmfile template all the parts of your helmfile.yaml at once, and then merge those YAML documents. So @sruon was making a GREAT point that we may need to change helmfile to render templates earlier - that is to evaluate a template per each helmfile.yaml part separated by ---. Sorry I missed my expertise to follow your great idea you raised it last year @sruon 😭

This, in combination with the wrong documentation, confused so many people. To finally overcome this situation, I'm authoring a fairly large PR that introduces the 2 enhancements:

• bases: for easier layering without gotemplates(but not perfect)
• Split before template, and then merge helmfile.yaml parts(inspired by/enhanced version of @sruon's work)

After that, everyone's use-cases can be addressed like follows:

(1) @nikolajbrinch https://github.com/roboll/helmfile/issues/388#issue-373930524

before

{{ readFile "environments.yaml }}}
--- 
helmfiles:
- ./*/helmfile.yaml

after: just add {{ readFile "environments.yaml" }} or bases: [environments.yaml] to each sub-helmfile. helmfile is still trying hard not to introduce global state and global variables other than envvars. So this might be the best thing we can do at the moment.

(2) @Tarick's https://github.com/roboll/helmfile/issues/388#issuecomment-433372631 and @rajiteh's https://github.com/roboll/helmfile/issues/388#issuecomment-467972236

The below would just work after the upcoming enhancement.

{{ readFile "common.yaml" }}
---
{{ readFile "environments.yaml" }}

(3) @joshwand https://github.com/roboll/helmfile/issues/388#issuecomment-450203933

because the context is not passed to readFile ?

I think that's almost correct. While evaluating the helmfile.yaml as a go template, readFile is evaluated to "inject" the file into the caller helmfile.yaml, and that's all. There's no further go template evaluations to render expressions in your helmfile-app-base.yaml, hence it isn't rendered.

I think it should be edited to use Release Template.

Instead of:

releases:
- name: app-{{ .Environment.Name }}
  namespace: {{ .Environment.Values.namespace }}
  chart: ../charts/myapp
  values: 
    - {{ .Environment.Values }} 

Use:

releases:
- name: app-{{` {{ .Environment.Name }} `}}
  namespace: {{` {{ .Environment.Values.namespace }} ``}
  chart: ../charts/myapp
  values: 
    - {{` {{ .Environment.Values }} ``}

This "defers" evaluations of template expressions immediately before processing each release.

(4) @royjs's https://github.com/roboll/helmfile/issues/388#issuecomment-468670403

I think you have one of the most advanced use-cases ☺️

environments:
  dev:
  prod:

templates:
  default: &default
    missingFileHandler: Debug
    values:
      - values/common.yaml.gotmpl 
      {{- range $overrideFolder := .Environment.Values.overrideFolders }}
      - values/{{ $overrideFolder }}/{{`{{  .Release.Name }}`}}.yaml
      {{- end }}

releases:
  ....

This requires you environments to be "already parsed" before evaluating .Environment.Values within your helmfile.yaml. Externalizing environments with {{ readFile ... }} does the opposite - because {{ readFile }} is executed only after environments are loaded. That's why your example didn't work.

After the upcoming change, you can make it a multi-part template with ---.

{{ readFile "environments.yaml" }}
---
templates:
  default: &default
    missingFileHandler: Debug
    values:
      - values/common.yaml.gotmpl 
      {{- range $overrideFolder := .Environment.Values.overrideFolders }}
      - values/{{ $overrideFolder }}/{{`{{  .Release.Name }}`}}.yaml
      {{- end }}

releases:
  ....

I think I've reviewed and answered all the use-cases introduced in this thread. But please feel free to ask if I missed any!

[mumoshu]

Forgot to mention yours, @sboschman https://github.com/roboll/helmfile/issues/388#issuecomment-468718386 😄

before:

{{ readFile "../../environments.yaml" }}
---
{{ readFile "../../repositories.yaml" }}
---
{{ readFile "../../helmdefaults.yaml" }}
---
{{ readFile "../../templates.yaml" }}
---

releases:
- name: consul
  version: ">= 3.5.0"
  <<: *default

This should be rewritten to:

bases:
- ../../environments.yaml
- ../../repositories.yaml
- ../../helmdefaults.yaml

{{ readFile "../../templates.yaml" }}

releases:
- name: consul
  version: ">= 3.5.0"
  <<: *default

The important point is that, to make *default successfully reference to the anchor defined in templates.yaml, you MUST import templates.yaml as a plain text BEFORE parsing the whole file content as a YAML. I used readFile for that.

[mumoshu]

Note that bases implementation is being simplified as described in https://github.com/roboll/helmfile/issues/388#issuecomment-491139877

[embik]

@mumoshu This sounds like a great set of changes and improvements. Thank you for investing such much time and energy and keeping everyone updated here!

[mumoshu]

587 is ready to be reviewed 😄

[mumoshu]

The enahcements will be included since helmfile v0.60.0

[galindro]

@mumoshu The approach that you mentioned here isn't working for me (my helmfile version is v0.98.1). I'm getting this error:

$ helmfile --log-level debug -e test diff
processing file "helmfile.yaml" in directory "."
first-pass rendering starting for "helmfile.yaml.part.0": inherited=&{test map[] map[]}, overrode=<nil>
first-pass uses: &{test map[] map[]}
first-pass rendering output of "helmfile.yaml.part.0":
 0: bases:
 1:   - ./common/environments.yaml
 2:   - ./common/repositories.yaml
 3:   - ./common/defaults.yaml
 4: 
 5: 
 6: 
 7: releases:
 8:   - name: gloo
 9:     <<: *default
10:     chart: gloo/{{ .Release.Name }}
11:     namespace: gloo-system
12:     version: <no value>
13: 

replaced <no value>s to workaround https://github.com/golang/go/issues/24963 to address https://github.com/roboll/helmfile/issues/553:
  strings.Join({
        ... // 10 identical lines
        "    chart: gloo/{{ .Release.Name }}",
        "    namespace: gloo-system",
-       "    version: <no value>",
+       "    version: ",
        "",
  }, "\n")

could not deduce `environment:` block, configuring only .Environment.Name. error: failed to read helmfile.yaml.part.0: reading document at index 1: yaml: unknown anchor 'default' referenced
error in first-pass rendering: result of "helmfile.yaml.part.0":
 0: bases:
 1:   - ./common/environments.yaml
 2:   - ./common/repositories.yaml
 3:   - ./common/defaults.yaml
 4: 
 5: 
 6: 
 7: releases:
 8:   - name: gloo
 9:     <<: *default
10:     chart: gloo/{{ .Release.Name }}
11:     namespace: gloo-system
12:     version: <no value>
13: 

first-pass produced: &{test map[] map[]}
first-pass rendering result of "helmfile.yaml.part.0": {test map[] map[]}
second-pass rendering failed, input of "helmfile.yaml.part.0":
 0: bases:
 1:   - ./common/environments.yaml
 2:   - ./common/repositories.yaml
 3:   - ./common/defaults.yaml
 4: 
 5: {{ readFile "./common/templates.yaml" }}
 6: 
 7: releases:
 8:   - name: gloo
 9:     <<: *default
10:     chart: gloo/{{ `{{ .Release.Name }}` }}
11:     namespace: gloo-system
12:     version: {{ .Values.GlooVersion }}
13: 

err: error during helmfile.yaml.part.0 parsing: template: stringTemplate:13:23: executing "stringTemplate" at <.Values.GlooVersion>: map has no entry for key "GlooVersion"
in ./helmfile.yaml: error during helmfile.yaml.part.0 parsing: template: stringTemplate:13:23: executing "stringTemplate" at <.Values.GlooVersion>: map has no entry for key "GlooVersion"

Directory structure

.
├── common
│   ├── defaults.yaml
│   ├── environments.yaml
│   ├── repositories.yaml
│   └── templates.yaml
├── environments
│   ├── prod.yaml
│   └── test.yaml
├── gloo
│   ├── namespace.yaml
│   └── values.test.yaml
└── helmfile.yaml

helmfile.yaml

bases:
  - ./common/environments.yaml
  - ./common/repositories.yaml
  - ./common/defaults.yaml

{{ readFile "./common/templates.yaml" }}

releases:
  - name: gloo
    <<: *default
    chart: gloo/{{ `{{ .Release.Name }}` }}
    namespace: gloo-system
    version: {{ .Values.GlooVersion }}

To temporally fix it, I just changed my helmfile.yaml to this:

helmfile.yaml

bases:
  - ./common/environments.yaml
  - ./common/repositories.yaml
  - ./common/defaults.yaml

---

templates:
  default: &default
    chart: stable/{{ `{{ .Release.Name }}` }}
    missingFileHandler: Warn
    values:
      - ./{{ `{{ .Release.Name }}` }}/values.{{ `{{ .Environment.Name }}` }}.yaml.gotmpl
    hooks:
      - events: ["prepare"]
        showlogs: true
        command: "kubectl"
        args: ["apply", "-f", "{{ `{{ .Release.Name }}` }}/namespace.yaml"]

releases:
  - name: gloo
    <<: *default
    chart: gloo/{{ `{{ .Release.Name }}` }}
    namespace: gloo-system
    version: {{ .Values.GlooVersion }}

Other files

./common/defaults.yaml

helmDefaults:
  tillerless: true
  kubeContext: eks-services
  wait: true
  timeout: 600
  recreatePods: false
  force: true

./common/environments.yaml

environments:
  prod:
    values:
      - ./environments/prod.yaml
  test:
    values:
      - ./environments/test.yaml

./common/repositories.yaml

repositories:
  - name: stable
    url: https://kubernetes-charts.storage.googleapis.com

./common/templates.yaml

templates:
  default: &default
    chart: stable/{{ `{{ .Release.Name }}` }}
    missingFileHandler: Warn
    values:
      - ./{{ `{{ .Release.Name }}` }}/values.{{ `{{ .Environment.Name }}` }}.yaml.gotmpl
    hooks:
      - events: ["prepare"]
        showlogs: true
        command: "kubectl"
        args: ["apply", "-f", "{{ `{{ .Release.Name }}` }}/namespace.yaml"]

./environments/prod.yaml

GlooVersion: 1.2.10

./environments/test.yaml

GlooVersion: 1.2.10

./gloo/namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: gloo-system

./gloo/values.test.yaml

settings:
  linkerd: false
gatewayProxies:
  gatewayProxy:
    service:
      type: LoadBalancer
      extraAnnotations:
        service.beta.kubernetes.io/aws-load-balancer-internal: "true"

[mumoshu]

@galindro Hey! Thanks for the info.

As you've tested and written, I believe we should always add --- between bases and releases sections of your helmfile.yaml practically. Otherwise we may or may not get strange errors that sometimes makes your env values disappear and sometimes not.

[galindro]

Yeah @mumoshu , but the problem is that I can't split out the templates section from helmfile.yaml into another file, even using the section separator characters ---. Which means that this doesn't works as well. Please, try yourself.

bases:
  - ./common/environments.yaml
  - ./common/repositories.yaml
  - ./common/defaults.yaml
---
{{ readFile "./common/templates.yaml" }}
---
releases:
  - name: gloo
    <<: *default
    chart: gloo/{{ `{{ .Release.Name }}` }}
    namespace: gloo-system
    version: {{ .Values.GlooVersion }}

[mumoshu]

Sorry to reply without testing it as you suggested, but are you saying that it is failing on *default?

If so, yeah it doesn't work. You shouldn't split readFile and releases. Otherwise it fails due to you can't refer to YAML anchors defined in another YAML document.

Your config should look like:

bases:
  - ./common/environments.yaml
  - ./common/repositories.yaml
  - ./common/defaults.yaml
---
{{ readFile "./common/templates.yaml" }}

releases:
  - name: gloo
    <<: *default
    chart: gloo/{{ `{{ .Release.Name }}` }}
    namespace: gloo-system
    version: {{ .Values.GlooVersion }}

[galindro]

@mumoshu I already tested using the mentioned method but it doesn't works also. Take a look:

$ helmfile --log-level debug -e test diff
processing file "helmfile.yaml" in directory "."
first-pass rendering starting for "helmfile.yaml.part.0": inherited=&{test map[] map[]}, overrode=<nil>
first-pass uses: &{test map[] map[]}
first-pass rendering output of "helmfile.yaml.part.0":
 0: bases:
 1:   - ./common/environments.yaml
 2:   - ./common/repositories.yaml
 3:   - ./common/defaults.yaml

error in first-pass rendering: result of "helmfile.yaml.part.0":
 0: bases:
 1:   - ./common/environments.yaml
 2:   - ./common/repositories.yaml
 3:   - ./common/defaults.yaml

first-pass produced: &{test map[] map[]}
first-pass rendering result of "helmfile.yaml.part.0": {test map[] map[]}
second-pass rendering result of "helmfile.yaml.part.0":
 0: bases:
 1:   - ./common/environments.yaml
 2:   - ./common/repositories.yaml
 3:   - ./common/defaults.yaml

first-pass rendering starting for "common/environments.yaml.part.0": inherited=&{test map[] map[]}, overrode=<nil>
first-pass uses: &{test map[] map[]}
first-pass rendering output of "common/environments.yaml.part.0":
 0: environments:
 1:   prod:
 2:     values:
 3:       - ./environments/prod.yaml
 4:   test:
 5:     values:
 6:       - ./environments/test.yaml
 7: 
 8: 

envvals_loader: loaded ./environments/test.yaml:map[GlooVersion:1.2.10]
first-pass produced: &{test map[GlooVersion:1.2.10] map[]}
first-pass rendering result of "common/environments.yaml.part.0": {test map[GlooVersion:1.2.10] map[]}
vals:
map[GlooVersion:1.2.10]
defaultVals:[]
second-pass rendering result of "common/environments.yaml.part.0":
 0: environments:
 1:   prod:
 2:     values:
 3:       - ./environments/prod.yaml
 4:   test:
 5:     values:
 6:       - ./environments/test.yaml
 7: 
 8: 

envvals_loader: loaded ./environments/test.yaml:map[GlooVersion:1.2.10]
merged environment: &{test map[GlooVersion:1.2.10] map[]}
first-pass rendering starting for "common/repositories.yaml.part.0": inherited=&{test map[] map[]}, overrode=<nil>
first-pass uses: &{test map[] map[]}
first-pass rendering output of "common/repositories.yaml.part.0":
 0: repositories:
 1:   - name: stable
 2:     url: https://kubernetes-charts.storage.googleapis.com
 3: 
 4: 

first-pass produced: &{test map[] map[]}
first-pass rendering result of "common/repositories.yaml.part.0": {test map[] map[]}
vals:
map[]
defaultVals:[]
second-pass rendering result of "common/repositories.yaml.part.0":
 0: repositories:
 1:   - name: stable
 2:     url: https://kubernetes-charts.storage.googleapis.com
 3: 
 4: 

merged environment: &{test map[] map[]}
first-pass rendering starting for "common/defaults.yaml.part.0": inherited=&{test map[] map[]}, overrode=<nil>
first-pass uses: &{test map[] map[]}
first-pass rendering output of "common/defaults.yaml.part.0":
 0: helmDefaults:
 1:   tillerless: true
 2:   kubeContext: eks-services
 3:   wait: true
 4:   timeout: 600
 5:   recreatePods: false
 6:   force: true
 7: 
 8: 

first-pass produced: &{test map[] map[]}
first-pass rendering result of "common/defaults.yaml.part.0": {test map[] map[]}
vals:
map[]
defaultVals:[]
second-pass rendering result of "common/defaults.yaml.part.0":
 0: helmDefaults:
 1:   tillerless: true
 2:   kubeContext: eks-services
 3:   wait: true
 4:   timeout: 600
 5:   recreatePods: false
 6:   force: true
 7: 
 8: 

merged environment: &{test map[] map[]}
envvals_loader: loaded ./environments/test.yaml:map[GlooVersion:1.2.10]
merged environment: &{test map[GlooVersion:1.2.10] map[]}
first-pass rendering starting for "helmfile.yaml.part.1": inherited=&{test map[GlooVersion:1.2.10] map[]}, overrode=<nil>
first-pass uses: &{test map[GlooVersion:1.2.10] map[]}
first-pass rendering output of "helmfile.yaml.part.1":
 0: 
 1: 
 2: releases:
 3:   - name: gloo
 4:     <<: *default
 5:     chart: gloo/{{ .Release.Name }}
 6:     namespace: gloo-system
 7:     version: <no value>
 8: 
 9: 

replaced <no value>s to workaround https://github.com/golang/go/issues/24963 to address https://github.com/roboll/helmfile/issues/553:
  strings.Join({
        ... // 5 identical lines
        "    chart: gloo/{{ .Release.Name }}",
        "    namespace: gloo-system",
-       "    version: <no value>",
+       "    version: ",
        "",
        "",
  }, "\n")

could not deduce `environment:` block, configuring only .Environment.Name. error: failed to read helmfile.yaml.part.1: reading document at index 1: yaml: unknown anchor 'default' referenced
error in first-pass rendering: result of "helmfile.yaml.part.1":
 0: 
 1: 
 2: releases:
 3:   - name: gloo
 4:     <<: *default
 5:     chart: gloo/{{ .Release.Name }}
 6:     namespace: gloo-system
 7:     version: <no value>
 8: 
 9: 

first-pass produced: &{test map[GlooVersion:1.2.10] map[]}
first-pass rendering result of "helmfile.yaml.part.1": {test map[GlooVersion:1.2.10] map[]}
second-pass rendering failed, input of "helmfile.yaml.part.1":
 0: {{ readFile "./common/templates.yaml" }}
 1: 
 2: releases:
 3:   - name: gloo
 4:     <<: *default
 5:     chart: gloo/{{ `{{ .Release.Name }}` }}
 6:     namespace: gloo-system
 7:     version: {{ .Values.GlooVersion }}
 8: 
 9: 

err: error during helmfile.yaml.part.1 parsing: template: stringTemplate:8:23: executing "stringTemplate" at <.Values.GlooVersion>: map has no entry for key "GlooVersion"
in ./helmfile.yaml: error during helmfile.yaml.part.1 parsing: template: stringTemplate:8:23: executing "stringTemplate" at <.Values.GlooVersion>: map has no entry for key "GlooVersion"

You can download this archive and test by yourself:
https://gist.github.com/galindro/c9b9bafe18b30f0e54748f38121bf47d#file-helmfile-test-tar-gz

[mumoshu]

@galindro Thanks! Ah that makes sense. Then you have no way to use anchors like *default over readFile. This is a fundamental issue coming from how double-rendering works. See also #932.

In the meantime, could you try using readFile to carry the default?

Like:

releases:
  - name: gloo
    {{ readFile "./common/default.yaml" | nindent 4}}
    chart: gloo/{{ `{{ .Release.Name }}` }}
    namespace: gloo-system
    version: {{ .Values.GlooVersion }}

[galindro]

@mumoshu , unfortunately, it not worked as well. Take a look:

First I changed the templates.yaml to this:

chart: stable/{{ `{{ .Release.Name }}` }}
missingFileHandler: Warn
values:
  - ./{{ `{{ .Release.Name }}` }}/values.{{ `{{ .Environment.Name }}` }}.yaml.gotmpl
hooks:
  - events: ["prepare"]
    showlogs: true
    command: "kubectl"
    args: ["apply", "-f", "{{ `{{ .Release.Name }}` }}/namespace.yaml"]

Then, I changed helmfile.yaml with your proposed change:

bases:
  - ./common/environments.yaml
  - ./common/repositories.yaml
  - ./common/defaults.yaml

releases:
  - name: gloo
    {{ readFile "./common/templates.yaml" | nindent 4 }}
    chart: gloo/{{ `{{ .Release.Name }}` }}
    namespace: gloo-system
    version: {{ .Values.GlooVersion }}

In the end, I executed helmfile, and this was the result:

$ helmfile --log-level debug -e test diff
processing file "helmfile.yaml" in directory "."
first-pass rendering starting for "helmfile.yaml.part.0": inherited=&{test map[] map[]}, overrode=<nil>
first-pass uses: &{test map[] map[]}
first-pass rendering output of "helmfile.yaml.part.0":
 0: bases:
 1:   - ./common/environments.yaml
 2:   - ./common/repositories.yaml
 3:   - ./common/defaults.yaml
 4: 
 5: releases:
 6:   - name: gloo
 7:     
 8:     
 9:     chart: gloo/{{ .Release.Name }}
10:     namespace: gloo-system
11:     version: <no value>
12: 
13: 

replaced <no value>s to workaround https://github.com/golang/go/issues/24963 to address https://github.com/roboll/helmfile/issues/553:
  strings.Join({
        ... // 9 identical lines
        "    chart: gloo/{{ .Release.Name }}",
        "    namespace: gloo-system",
-       "    version: <no value>",
+       "    version: ",
        "",
        "",
  }, "\n")

error in first-pass rendering: result of "helmfile.yaml.part.0":
 0: bases:
 1:   - ./common/environments.yaml
 2:   - ./common/repositories.yaml
 3:   - ./common/defaults.yaml
 4: 
 5: releases:
 6:   - name: gloo
 7:     
 8:     
 9:     chart: gloo/{{ .Release.Name }}
10:     namespace: gloo-system
11:     version: <no value>
12: 
13: 

first-pass produced: &{test map[] map[]}
first-pass rendering result of "helmfile.yaml.part.0": {test map[] map[]}
second-pass rendering failed, input of "helmfile.yaml.part.0":
 0: bases:
 1:   - ./common/environments.yaml
 2:   - ./common/repositories.yaml
 3:   - ./common/defaults.yaml
 4: 
 5: releases:
 6:   - name: gloo
 7:     {{ readFile "./common/templates.yaml" | nindent 4 }}
 8:     chart: gloo/{{ `{{ .Release.Name }}` }}
 9:     namespace: gloo-system
10:     version: {{ .Values.GlooVersion }}
11: 
12: 

err: error during helmfile.yaml.part.0 parsing: template: stringTemplate:11:23: executing "stringTemplate" at <.Values.GlooVersion>: map has no entry for key "GlooVersion"
in ./helmfile.yaml: error during helmfile.yaml.part.0 parsing: template: stringTemplate:11:23: executing "stringTemplate" at <.Values.GlooVersion>: map has no entry for key "GlooVersion"

P.S.: the defaults.yaml carries helmDefaults content, not the defaults that should be used by releases.

[galindro]

@mumoshu should we continue our talk here or here?

[mumoshu]

@galindro I think the last missing piece is that --- between bases and releases, as suggested in https://github.com/roboll/helmfile/issues/388#issuecomment-571367074

[vavdoshka]

Does passing the state down to helmfile through the inline yaml object looks like a viable approach?

```
environments:
stage:
values:
- env/common.yaml
- env/stage.yaml
prod:
values:
- env/common.yaml
- env/prod.yaml

helmfiles:

• path: releases/scdf/helmfile.yaml
  values:
  
  
  
  • {{ toYaml .Values | nindent 8 }}
    
    ```
  
  

It seems it works, at least on POC surface. It allows having only one point of the environment load and hence keeps code DRYer .
I'm new to helmfile so would be very happy to get feedback to know if and why the technique I shared is a no-go. If that matters our use-case is just to use helmfile template and the rest we delegate to ArgoCD.

[mumoshu]

@vavdoshka It should work! I've also came up with that myself in https://github.com/roboll/helmfile/issues/762#issuecomment-540444261

Please feel free to add your voice to #762 if you find that helps your use case.

[mumoshu]

@vavdoshka BTW a general recommendation is that you should put --- between environments and the dynamic part of your helmfie.yaml to avoid nasty things from happening due to the chicken-and-egg problem rendering helmfile. (you need to render helmfile template to load environments, but you need to render the helmfile as yaml before parsing it and loading environments)

environments:
  stage:
    values:
    - env/common.yaml
    - env/stage.yaml
  prod:
    values:
    - env/common.yaml
    - env/prod.yaml

---

helmfiles:
- path: releases/scdf/helmfile.yaml
  values:
    - {{ toYaml .Values | nindent 8 }}

[vavdoshka]

@mumoshu thanks for such a quick feedback!

Cool! I think this approach is clear and explicit enough, it just probably needs to be documented somewhere. Thanks for highlighting the --- issue.

[mumoshu]

@vavdoshka Thanks for confirming! I agree very much. Just not sure where to start. Perhaps adding that to the bottom of https://github.com/roboll/helmfile/blob/master/docs/writing-helmfile.md makes sense?

[vavdoshka]

yes, @mumoshu sounds good to me
I've opened a draft PR on this https://github.com/roboll/helmfile/pull/1808, please feel free to amend it.