Helmfile: Releases from `helmfiles` are ignored when the environment is specified

[andrewnazarov]

I'm facing the issue when releases defined in separate .yaml files and referenced via helmfiles: are ignored when I try to install the full stack for a certain environment.

Example:

#environments.yaml
environments:
  dev:
  staging:
  production:

#helmfiles/namespaces.yaml
repositories:
  - name: incubator
    url: https://kubernetes-charts-incubator.storage.googleapis.com/

releases:
  - name: namespaces
    namespace: default
    chart: incubator/raw
    values:
      - resources:
        - apiVersion: v1
          kind: Namespace
          metadata:
            name: redis
          spec:

#helmfile.yaml
bases:
- environments.yaml
---
helmfiles:
  - helmfiles/namespaces.yaml

repositories:
  - name: stable
    url: https://kubernetes-charts.storage.googleapis.com

helmDefaults:
  verify: false
  wait: false
  timeout: 600
  recreatePods: false
  force: true

releases:
  - name: redis
    namespace: redis
    chart: stable/redis
    version: ~9.2.1
    values:
      - usePassword: false
        cluster:
          enabled: false

Then, let's try to output the list of releases

ip113:helmfile andrewnazarov$ helmfile -e staging --log-level=debug list
processing file "helmfile.yaml" in directory "."
first-pass rendering starting for "helmfile.yaml.part.0": inherited=&{staging map[] map[]}, overrode=<nil>
first-pass uses: &{staging map[] map[]}
first-pass rendering output of "helmfile.yaml.part.0":
 0: bases:
 1: - environments.yaml

error in first-pass rendering: result of "helmfile.yaml.part.0":
 0: bases:
 1: - environments.yaml

first-pass produced: &{staging map[] map[]}
first-pass rendering result of "helmfile.yaml.part.0": {staging map[] map[]}
second-pass rendering result of "helmfile.yaml.part.0":
 0: bases:
 1: - environments.yaml

first-pass rendering starting for "environments.yaml.part.0": inherited=&{staging map[] map[]}, overrode=<nil>
first-pass uses: &{staging map[] map[]}
first-pass rendering output of "environments.yaml.part.0":
 0: environments:
 1:   dev:
 2:   staging:
 3:   production:

first-pass produced: &{staging map[] map[]}
first-pass rendering result of "environments.yaml.part.0": {staging map[] map[]}
vals:
map[]
defaultVals:[]
second-pass rendering result of "environments.yaml.part.0":
 0: environments:
 1:   dev:
 2:   staging:
 3:   production:

merged environment: &{staging map[] map[]}
merged environment: &{staging map[] map[]}
first-pass rendering starting for "helmfile.yaml.part.1": inherited=&{staging map[] map[]}, overrode=<nil>
first-pass uses: &{staging map[] map[]}
first-pass rendering output of "helmfile.yaml.part.1":
 0: helmfiles:
 1:   - helmfiles/namespaces.yaml
 2: 
 3: repositories:
 4:   - name: stable
 5:     url: https://kubernetes-charts.storage.googleapis.com
 6: 
 7: helmDefaults:
 8:   verify: false
 9:   wait: false
10:   timeout: 600
11:   recreatePods: false
12:   force: true
13: 
14: releases:
15:   - name: redis
16:     namespace: redis
17:     chart: stable/redis
18:     version: ~9.2.1
19:     values:
20:       - usePassword: false
21:         cluster:
22:           enabled: false

first-pass produced: &{staging map[] map[]}
first-pass rendering result of "helmfile.yaml.part.1": {staging map[] map[]}
vals:
map[]
defaultVals:[]
second-pass rendering result of "helmfile.yaml.part.1":
 0: helmfiles:
 1:   - helmfiles/namespaces.yaml
 2: 
 3: repositories:
 4:   - name: stable
 5:     url: https://kubernetes-charts.storage.googleapis.com
 6: 
 7: helmDefaults:
 8:   verify: false
 9:   wait: false
10:   timeout: 600
11:   recreatePods: false
12:   force: true
13: 
14: releases:
15:   - name: redis
16:     namespace: redis
17:     chart: stable/redis
18:     version: ~9.2.1
19:     values:
20:       - usePassword: false
21:         cluster:
22:           enabled: false

merged environment: &{staging map[] map[]}
processing file "namespaces.yaml" in directory "helmfiles"
changing working directory to "/Users/andrewnazarov/Repo/iris/test/helmfile/helmfiles"
first-pass rendering starting for "namespaces.yaml.part.0": inherited=&{staging map[] map[]}, overrode=<nil>
first-pass uses: &{staging map[] map[]}
first-pass rendering output of "namespaces.yaml.part.0":
 0: repositories:
 1:   - name: incubator
 2:     url: https://kubernetes-charts-incubator.storage.googleapis.com/
 3: 
 4: releases:
 5:   - name: namespaces
 6:     namespace: default
 7:     chart: incubator/raw
 8:     values:
 9:       - resources:
10:         - apiVersion: v1
11:           kind: Namespace
12:           metadata:
13:             name: redis
14:           spec:

first-pass produced: &{staging map[] map[]}
first-pass rendering result of "namespaces.yaml.part.0": {staging map[] map[]}
vals:
map[]
defaultVals:[]
second-pass rendering result of "namespaces.yaml.part.0":
 0: repositories:
 1:   - name: incubator
 2:     url: https://kubernetes-charts-incubator.storage.googleapis.com/
 3: 
 4: releases:
 5:   - name: namespaces
 6:     namespace: default
 7:     chart: incubator/raw
 8:     values:
 9:       - resources:
10:         - apiVersion: v1
11:           kind: Namespace
12:           metadata:
13:             name: redis
14:           spec:

changing working directory back to "/Users/andrewnazarov/Repo/iris/test/helmfile"
NAME    NAMESPACE   INSTALLED   LABELS
redis   redis       true              

Without environment specified everything looks ok:

ip113:helmfile andrewnazarov$ helmfile --log-level=debug list
processing file "helmfile.yaml" in directory "."
first-pass rendering starting for "helmfile.yaml.part.0": inherited=&{default map[] map[]}, overrode=<nil>
first-pass uses: &{default map[] map[]}
first-pass rendering output of "helmfile.yaml.part.0":
 0: bases:
 1: - environments.yaml

error in first-pass rendering: result of "helmfile.yaml.part.0":
 0: bases:
 1: - environments.yaml

first-pass produced: &{default map[] map[]}
first-pass rendering result of "helmfile.yaml.part.0": {default map[] map[]}
second-pass rendering result of "helmfile.yaml.part.0":
 0: bases:
 1: - environments.yaml

first-pass rendering starting for "environments.yaml.part.0": inherited=&{default map[] map[]}, overrode=<nil>
first-pass uses: &{default map[] map[]}
first-pass rendering output of "environments.yaml.part.0":
 0: environments:
 1:   dev:
 2:   staging:
 3:   production:

first-pass produced: &{default map[] map[]}
first-pass rendering result of "environments.yaml.part.0": {default map[] map[]}
vals:
map[]
defaultVals:[]
second-pass rendering result of "environments.yaml.part.0":
 0: environments:
 1:   dev:
 2:   staging:
 3:   production:

merged environment: &{default map[] map[]}
merged environment: &{default map[] map[]}
first-pass rendering starting for "helmfile.yaml.part.1": inherited=&{default map[] map[]}, overrode=<nil>
first-pass uses: &{default map[] map[]}
first-pass rendering output of "helmfile.yaml.part.1":
 0: helmfiles:
 1:   - helmfiles/namespaces.yaml
 2: 
 3: repositories:
 4:   - name: stable
 5:     url: https://kubernetes-charts.storage.googleapis.com
 6: 
 7: helmDefaults:
 8:   verify: false
 9:   wait: false
10:   timeout: 600
11:   recreatePods: false
12:   force: true
13: 
14: releases:
15:   - name: redis
16:     namespace: redis
17:     chart: stable/redis
18:     version: ~9.2.1
19:     values:
20:       - usePassword: false
21:         cluster:
22:           enabled: false

first-pass produced: &{default map[] map[]}
first-pass rendering result of "helmfile.yaml.part.1": {default map[] map[]}
vals:
map[]
defaultVals:[]
second-pass rendering result of "helmfile.yaml.part.1":
 0: helmfiles:
 1:   - helmfiles/namespaces.yaml
 2: 
 3: repositories:
 4:   - name: stable
 5:     url: https://kubernetes-charts.storage.googleapis.com
 6: 
 7: helmDefaults:
 8:   verify: false
 9:   wait: false
10:   timeout: 600
11:   recreatePods: false
12:   force: true
13: 
14: releases:
15:   - name: redis
16:     namespace: redis
17:     chart: stable/redis
18:     version: ~9.2.1
19:     values:
20:       - usePassword: false
21:         cluster:
22:           enabled: false

merged environment: &{default map[] map[]}
processing file "namespaces.yaml" in directory "helmfiles"
changing working directory to "/Users/andrewnazarov/Repo/iris/test/helmfile/helmfiles"
first-pass rendering starting for "namespaces.yaml.part.0": inherited=&{default map[] map[]}, overrode=<nil>
first-pass uses: &{default map[] map[]}
first-pass rendering output of "namespaces.yaml.part.0":
 0: repositories:
 1:   - name: incubator
 2:     url: https://kubernetes-charts-incubator.storage.googleapis.com/
 3: 
 4: releases:
 5:   - name: namespaces
 6:     namespace: default
 7:     chart: incubator/raw
 8:     values:
 9:       - resources:
10:         - apiVersion: v1
11:           kind: Namespace
12:           metadata:
13:             name: redis
14:           spec:

first-pass produced: &{default map[] map[]}
first-pass rendering result of "namespaces.yaml.part.0": {default map[] map[]}
vals:
map[]
defaultVals:[]
second-pass rendering result of "namespaces.yaml.part.0":
 0: repositories:
 1:   - name: incubator
 2:     url: https://kubernetes-charts-incubator.storage.googleapis.com/
 3: 
 4: releases:
 5:   - name: namespaces
 6:     namespace: default
 7:     chart: incubator/raw
 8:     values:
 9:       - resources:
10:         - apiVersion: v1
11:           kind: Namespace
12:           metadata:
13:             name: redis
14:           spec:

merged environment: &{default map[] map[]}
changing working directory back to "/Users/andrewnazarov/Repo/iris/test/helmfile"
NAME        NAMESPACE   INSTALLED   LABELS
namespaces  default     true              
redis       redis       true

The same story for helmfile diff and helmfile apply.

helmfile version v0.94.0

Also, the v0.94.1 has been checked for helmfile list. This issue looks like #1009, but not exactly the same.

[andrewnazarov]

The example could be simplified a bit by eliminating bases::

#helmfiles/namespaces.yaml
repositories:
  - name: incubator
    url: https://kubernetes-charts-incubator.storage.googleapis.com/

releases:
  - name: namespaces
    namespace: default
    chart: incubator/raw
    values:
      - resources:
        - apiVersion: v1
          kind: Namespace
          metadata:
            name: redis
          spec:

#helmfile.yaml
helmfiles:
  - helmfiles/namespaces.yaml

repositories:
  - name: stable
    url: https://kubernetes-charts.storage.googleapis.com

helmDefaults:
  verify: false
  wait: false
  timeout: 600
  recreatePods: false
  force: true

environments:
  dev:
  staging:
  production:

releases:
  - name: redis
    namespace: redis
    chart: stable/redis
    version: ~9.2.1
    values:
      - usePassword: false
        cluster:
          enabled: false

[andrewnazarov]

It seems I need to define environments for the subhelmfile as well like:

#helmfiles/namespaces.yaml
repositories:
  - name: incubator
    url: https://kubernetes-charts-incubator.storage.googleapis.com/

environments:
  dev:
  staging:
  production:

releases:
  - name: namespaces
    namespace: default
    chart: incubator/raw
    values:
      - resources:
        - apiVersion: v1
          kind: Namespace
          metadata:
            name: redis
          spec:

In this case, all releases are taken into consideration.

[mumoshu]

@andrewnazarov Hey! I've reread your original example and now realized the issue.

Yes, you need to define environments in every sub-helmfile involved, and that's how it works.

The expected use-case of this feature is "fail on a missing env detected". Like, say you had two environments prod and preview and two helmfile for app1 and app2 respectively. What if app2 missed prod env when you run helmfile -e prod? You'll probably want Helmfile to fail saying app2 has no environment named "prod".

[mumoshu]

Regardless of how it works today, I spotted two possible enhancements to Helmfile thanks to your report:

1. Based on https://github.com/roboll/helmfile/issues/1010#issuecomment-561358232, in your case helmfile -e prod should have failed due to an error like helmfiles/namespaces.yaml has no releases for env "prod", but it didn't. Maybe this is a bug?
2. Can we enhance Helmfile so that helmfile -e prod won't fail in your example? To make it possible, I would change Helmfile to NOT fail on helmfile -e prod when helmfile.yaml doesn't include environments at all.

[andrewnazarov]

@mumoshu In my case, the releases from sub-helmfiles were silently ignored. Thus, I was confused.

Basically there could be a few use cases for sub-helmfiles:

1. The umbrella helmfile with only references to external helmfiles
2. Regular helmfile with releases and stuff, but at the same time referencing to some external helmfiles.

I got the point about processing the sub-helmfile before the root one, so it's strange to take some part of the payload from the base.

On the other hand, it feels natural and DRY-compliant to define environments once and use them everywhere. If I want to ignore the release for a certain env I can do if-logic.

So, I really don't know what is better. I can understand both approaches.

I would prefer either using "global" environments (taken from the "root" file) or at least be warned by the output somehow.

[mumoshu]

1. The umbrella helmfile with only references to external helmfiles
2. Regular helmfile with releases and stuff, but at the same time referencing to some external helmfiles.

Agreed.

so it's strange to take some part of the payload from the base.

What do you mean by here exactly?

it feels natural and DRY-compliant to define environments ones and use them everywhere. If I want to ignore the release for a certain env I can do if-logic.

Yeah, but the downside of this is that it makes sub-helmfiles less modularized i.e. depends on environments provided externally.

[mumoshu]

at least be warned by the output somehow.

How would you like to be warned in your case?

[andrewnazarov]

Helmfile never processes, say, the namespaces releases in the sub-helmfile. It just double-render it, merge the parent helmfile.yaml onto it, then processes(sync,apply,etc) releases in the merged helmfile.yaml.

Oh, I've got lost a bit.

It's stated:

helmfiles:
- # Path to the helmfile state file being processed BEFORE releases in this state file
  path: path/to/subhelmfile.yaml
  # Label selector used for filtering releases in the nested state.
  # For example, `name=prometheus` in this context is equivalent to processing the nested state like
  #   helmfile -f path/to/subhelmfile.yaml -l name=prometheus sync

I thought bases: are for merging.

[andrewnazarov]

What do you mean by here exactly?

Probably that was due to my misunderstanding. I thought sub-helmfile is run first, then the main file. In this case it's strange that some things (blocks, whatever) from the main helmfile are taken into consideration when the sub-helmfile is applying. It seems I didn't understand the concept of helmfiles:.

[andrewnazarov]

Hm, actually from what I see in logs it seems it runs sub-helmfile first. Then, the main helmfile.

[mumoshu]

I thought sub-helmfile is run first, then the main file

Yes, you are correct.

some things (blocks, whatever) from the main helmfile are taken into consideration when the sub-helmfile is applying

What did you see for example? I thought nothing other than Environment.Name is inherited automatically.

[andrewnazarov]

Yes, you are correct.

Great! I was a bit embarrassed, now recovered:)

What did you see for example? I thought nothing other than Environment.Name is inherited automatically.

Nevermind this:).
Initially I meant that from one hand it's ok that one should explicitly define environments for sub-helmfiles process-wise, but on the other hand, it seems natural to grab things from the main helmfile to follow dry. But as you stated, you don't want such dependencies).

[andrewnazarov]

How would you like to be warned in your case?

Regarding the error message, probably a slight modification of the existed one would be sufficient. Like:

err: no releases found that matches specified selector() and environment(staging), in <subhelmfile_name>.

Yeah, but the downside of this is that it makes sub-helmfiles less modularized i.e. depends on environments provided externally.

But how to deal with third-party sub-helmfiles then?)

[mumoshu]

But how to deal with third-party sub-helmfiles then?)

@andrewnazarov I was considering the second point in my prev comment for that https://github.com/roboll/helmfile/issues/1010#issuecomment-561364261

[mumoshu]

That is, helmfile -e prod sync won't fail when namespaces.yaml missing environments: at all, but will fail when it has any non-empty environments like environments:{"test": { whatever... }} that misses a prod env in it.

WDYT @andrewnazarov ? Does it make sense?

[andrewnazarov]

Sorry for such a delay.

I really don't know what would be better. It was very confusing at first that releases defined in sub-helmfiles were ignored without any error or warning. Probably just a warning will be sufficient. One the other hand, if you know the actual behaviour, you'll be ready and implement things accordingly. So, it should be mentioned in docs for sure.

As I see, it's not only me expecting environments to be inherited (#1045). And probably it's better to concentrate on what you are proposing in #1048.

Right now I found a way how to deal with that, not the elegant though.

[albertrdixon]

To revive this issue:

• I think it would be really handy to be able to define environments in a remote / shared helmfile. This would reduce the boilerplate we run into having to copy/paste the environment block around in every helmfile.
• Maybe rethink how releases are included in environments then? Releases are included in an environment through label selector matching perhaps?