Helmfile: Bug with environment values merge

Created on 1 Oct 2018  ·  10Comments  ·  Source: roboll/helmfile

Hi! I think this is a bug or, maybe, I have missing something.

Imagine the following structure:

├── charts
│   └── environment
│       ├── Chart.yaml
│       ├── templates
│       │   └── configmap.yaml
│       └── values.yaml
├── environments
│   ├── dev
│   │   └── values.yaml
│   └── default.yaml.gotmpl
├── helmfile.yaml
└── values.yaml.gotmpl

helmfile.yaml:

environments:
  default:
    values:
    - environments/default.yaml.gotmpl
  dev:
    values:
    - environments/dev/values.yaml
    - environments/default.yaml.gotmpl

releases:
  - name: environment
    chart: ./charts/environment
    labels:
      tier: infrastructure
    values:
    - values.yaml.gotmpl

values.yaml.gotmpl:

baseUrl: {{ .Environment.Values | get "baseUrl" }}
backend:
  debug: {{ .Environment.Values | get "backend.debug" }}

environments/default.yaml.gotmpl

baseUrl: {{ .Environment.Values | getOrNil "baseUrl" | default "DEFAULT_ENV_BASEURL" }}
backend:
  debug: {{ .Environment.Values | getOrNil "backend.debug" | default "DEFAULT_ENV_DEBUG" }}

environments/dev/values.yaml

baseUrl: DEV_ENV_BASEURL
backend:
  debug: DEV_ENV_DEBUG

charts/environment/values.yaml

baseUrl: CHART_BASEURL
backend:
  debug: CHART_FALSE

charts/environment/templates/configmap.yaml

kind: ConfigMap
apiVersion: v1
metadata:
  name: configuration
data:
  BASE_URL: {{ .Values.baseUrl }}
  BACKEND_DEBUG: {{ .Values.backend.debug }}

helmfile -l tier=infrastructure -e default template

# Source: environment/templates/configmap.yaml
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: configuration
data:
  BASE_URL: DEFAULT_ENV_BASEURL
  BACKEND_DEBUG: DEFAULT_ENV_DEBUG

helmfile -l tier=infrastructure -e dev template

# Source: environment/templates/configmap.yaml
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: configuration
data:
  BASE_URL: DEFAULT_ENV_BASEURL <-- WHY? Needs to be DEV_ENV_BASEURL
  BACKEND_DEBUG: DEV_ENV_DEBUG

If I change order in helmfile.yaml from this

dev:
    values:
    - environments/dev/values.yaml
    - environments/default.yaml.gotmpl

to this:

dev:
    values:
    - environments/default.yaml.gotmpl
    - environments/dev/values.yaml

new result:

# Source: environment/templates/configmap.yaml
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: configuration
data:
  BASE_URL: DEV_ENV_BASEURL
  BACKEND_DEBUG: DEFAULT_ENV_DEBUG <--- WHY? Needs to be DEV_ENV_DEBUG

What am I doing wrong?

picture asychev

Most helpful comment

Awesome! Thanks again for your confirmation 👍

picture mumoshu on 3 Oct 2018
🎉2

All 10 comments

Hey!
The points are: (1) the latter environment values file wins over the former ones, and (2) environment values are not designed to be referenced from within other in environment values file.

So, change this:

  dev:
    values:
    - environments/dev/values.yaml
    - environments/default.yaml.gotmpl

to

  dev:
    values:
    - environments/default.yaml.gotmpl
    - environments/dev/values.yaml

and change this:

baseUrl: {{ .Environment.Values | getOrNil "baseUrl" | default "DEFAULT_ENV_BASEURL" }}
backend:
  debug: {{ .Environment.Values | getOrNil "backend.debug" | default "DEFAULT_ENV_DEBUG" }}

to

baseUrl: DEFAULT_ENV_BASEURL
backend:
  debug: DEFAULT_ENV_DEBUG
mumoshu picture mumoshu on 2 Oct 2018

@mumoshu Thank you for the clarification, but it is still unclear for me.

About (1) - as I show before - only part of values wins.

In provided example seems that environments/default.yaml.gotmpl not needed anymore, because now I need to put the full set of variables into default environment values and to each other environment values files. I tried to avoid code duplication, when all values located in environments/default.yaml but in dev - only overrides of part of them, specific to dev environment.

If so, is it better to put default values logic to values.yaml.gotmpl in values of releases and get rid of default environment files?

asychev picture asychev on 2 Oct 2018

Hey! Thanks for the response.

environments/default.yaml.gotmpl not needed anymore, because now I need to put the full set of variables into default environment values

Yes.

but in dev - only overrides of part of them, specific to dev environment.

Yes. This is possible. Sorry If I'm not following you correctly, but you can of course write your environments/dev/values.yaml like this:

baseUrl: DEV_ENV_BASEURL
#Omit `backend` from your yaml, and then only `baseUrl` is overrode
#backend:
#  debug: DEV_ENV_DEBUG

You now have no need to complicate things with getOrNil ... | default ....
And there seems like no code duplication in my example.

Is there anything else you wanted to do on top of this? Any input is welcomed. Thanks.

mumoshu picture mumoshu on 2 Oct 2018

@mumoshu it is still not working for me:

.
├── charts
│   └── environment
│       ├── Chart.yaml
│       ├── templates
│       │   └── configmap.yaml
│       └── values.yaml
├── environments
│   ├── default.yaml
│   └── dev
│       └── values.yaml
├── helmfile.yaml
└── values.yaml.gotmpl

helmfile.yaml

environments:
  default:
    values:
    - environments/default.yaml
  dev:
    values:
    - environments/default.yaml
    - environments/dev/values.yaml

releases:
  - name: environment
    chart: ./charts/environment
    labels:
      tier: infrastructure
    values:
    - values.yaml.gotmpl

environments/default.yaml

baseUrl: DEFAULT_ENV_BASEURL
backend:
  debug: DEFAULT_ENV_DEBUG

environments/dev/values.yaml

baseUrl: DEV_ENV_BASEURL
backend:
  debug: DEV_ENV_DEBUG

values.yaml.gotmpl

baseUrl: {{ .Environment.Values | get "baseUrl" }}
backend:
  debug: {{ .Environment.Values | get "backend.debug" }}

$ helmfile -l tier=infrastructure -e default template

# Source: environment/templates/configmap.yaml
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: configuration
data:
  BASE_URL: DEFAULT_ENV_BASEURL
  BACKEND_DEBUG: DEFAULT_ENV_DEBUG

$ helmfile -l tier=infrastructure -e dev template

```

# Source: environment/templates/configmap.yaml

kind: ConfigMap
apiVersion: v1
metadata:
name: configuration
data:
BASE_URL: DEV_ENV_BASEURL
BACKEND_DEBUG: DEFAULT_ENV_DEBUG <---

asychev picture asychev on 2 Oct 2018

Thanks a lot for detailed feedback! I'm trying to reproduce it now.

mumoshu picture mumoshu on 3 Oct 2018

@asychev It seems to be working as expected to me:

https://github.com/mumoshu/helmfile-issue-377

Would you mind reviewing my reproduction and compare it with your setup? Thanks!

mumoshu picture mumoshu on 3 Oct 2018

@mumoshu it is working in your example just because you commented out debug: DEFAULT_ENV_DEBUG in environments/default.yaml, and this leads to bugged -e default behaviour:

error calling get: no value exist for key "backend" in map[baseUrl:DEFAULT_ENV_BASEURL]
failed to render values files "values.yaml.gotmpl"

Please, uncomment and you clearly see the bug.

asychev picture asychev on 3 Oct 2018

@asychev Thanks for your support! I've successfully reproduced the issue. After some investigation, I found that we've been using a pretty old version of mergo that is used for merging values. I've bumped it to the latest release and the issue seems to be resolved.

Could you try again with the new helmfile release v0.40.1? Thanks!

mumoshu picture mumoshu on 3 Oct 2018
👍1

@mumoshu Yep, now everything works as expected, thank you!

asychev picture asychev on 3 Oct 2018

Awesome! Thanks again for your confirmation 👍

mumoshu picture mumoshu on 3 Oct 2018
🎉2
Was this page helpful?
0 / 5 - 0 ratings

Related issues

michaelpporter picture michaelpporter  ·  3Comments
ppawiggers picture ppawiggers  ·  3Comments
daaain picture daaain  ·  3Comments
willejs picture willejs  ·  4Comments
mumoshu picture mumoshu  ·  4Comments