Helmfile: prepare hooks executed on apply even if the release is not fit into selector

[dsuievalov]

Have upgraded from old version 0.82.0 to 0.130.0 and found that prepare hooks are not filtered by selectors any more and execute each time on apply/sync. According to README they should not execute:

prepare hooks are triggered on the release as long as it is not excluded by the helmfile selector(e.g. helmfile -l key=value).

Behavior of old version 0.82.0:
helmfile-0.82.0.yaml

helmDefaults:
  verify: false
  wait: true
  timeout: 600
  recreatePods: false
  atomic: true
  force: false

repositories:
  - name: stable
    url: https://kubernetes-charts.storage.googleapis.com
---

releases:
  - name: mongodb1
    labels:
      app_name: "mongodb1"
    chart: "stable/mongodb"
    version: 7.8.10
    namespace: {{ .Namespace }}
    hooks:
      - events: ["prepare"]
        showlogs: true
        command: "echo"
        args: ["mongodb1"]

  - name: mongodb2
    labels:
      app_name: "mongodb2"
    chart: "stable/mongodb"
    version: 7.8.10
    namespace: {{ .Namespace }}
    hooks:
      - events: ["prepare"]
        showlogs: true
        command: "echo"
        args: ["mongodb2"]

  - name: mongodb3
    labels:
      app_name: "mongodb3"
    chart: "stable/mongodb"
    version: 7.8.10
    namespace: {{ .Namespace }}
    hooks:
      - events: ["prepare"]
        showlogs: true
        command: "echo"
        args: ["mongodb3"]

$ helmfile-old -f helmfile-0.82.0.yaml -l app_name=mongodb1 sync --args='--dry-run' 1>/dev/null
Adding repo stable https://kubernetes-charts.storage.googleapis.com
"stable" already exists with the same configuration, skipping

Updating repo
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈Happy Helming!⎈

helmfile-0.82.0.yaml: basePath=.

hook[prepare] logs | mongodb1
hook[prepare] logs |
Upgrading release=mongodb1, chart=stable/mongodb

List of releases in error :
RELEASE
mongodb1
in ./helmfile-0.82.0.yaml: failed processing release mongodb1: helm exited with status 1:
  Error: invalid argument "600" for "--timeout" flag: time: missing unit in duration 600

Behavior of new version 0.130.0:
helmfile-0.130.0.yaml

helmDefaults:
  verify: false
  wait: true
  timeout: 600
  recreatePods: false
  atomic: true
  force: false
  historyMax: 10

apiVersions:
- example/v1

repositories:
  - name: stable
    url: https://kubernetes-charts.storage.googleapis.com
---

releases:
  - name: mongodb1
    labels:
      app_name: "mongodb1"
    chart: "stable/mongodb"
    version: 7.8.10
    namespace: {{ .Namespace }}
    hooks:
      - events: ["prepare"]
        showlogs: true
        command: "echo"
        args: ["mongodb1"]

  - name: mongodb2
    labels:
      app_name: "mongodb2"
    chart: "stable/mongodb"
    version: 7.8.10
    namespace: {{ .Namespace }}
    hooks:
      - events: ["prepare"]
        showlogs: true
        command: "echo"
        args: ["mongodb2"]

  - name: mongodb3
    labels:
      app_name: "mongodb3"
    chart: "stable/mongodb"
    version: 7.8.10
    namespace: {{ .Namespace }}
    hooks:
      - events: ["prepare"]
        showlogs: true
        command: "echo"
        args: ["mongodb3"]

$ helmfile -f helmfile-0.130.0.yaml -l app_name=mongodb1 sync --args='--dry-run' 1>/dev/null
Adding repo stable https://kubernetes-charts.storage.googleapis.com
"stable" has been added to your repositories

helmfile-0.130.0.yaml: basePath=.
helmfile-0.130.0.yaml: basePath=.

hook[prepare] logs | mongodb1
hook[prepare] logs |
helmfile-0.130.0.yaml: basePath=.

hook[prepare] logs | mongodb2
hook[prepare] logs |

hook[prepare] logs | mongodb3
hook[prepare] logs |
Affected releases are:
  mongodb1 (stable/mongodb) UPDATED

Upgrading release=mongodb1, chart=stable/mongodb
Listing releases matching ^mongodb1$

UPDATED RELEASES:
NAME       CHART            VERSION
mongodb1   stable/mongodb

Part of debug output of a new version:

$ helmfile --log-level debug -f helmfile-0.130.0.yaml -l app_name=mongodb1 sync --args='--dry-run' 1>/dev/null
processing file "helmfile-0.130.0.yaml" in directory "."
first-pass rendering starting for "helmfile-0.130.0.yaml.part.0": inherited=&{default map[] map[]}, overrode=<nil>
first-pass uses: &{default map[] map[]}
first-pass rendering output of "helmfile-0.130.0.yaml.part.0":
 0: ---
 1: helmDefaults:
 2:   verify: false
 3:   wait: true
 4:   timeout: 600
 5:   recreatePods: false
 6:   atomic: true
 7:   force: false
 8:   historyMax: 10
 9:
10: apiVersions:
11: - example/v1
12:
13: repositories:
14:   - name: stable
15:     url: https://kubernetes-charts.storage.googleapis.com

first-pass produced: &{default map[] map[]}
first-pass rendering result of "helmfile-0.130.0.yaml.part.0": {default map[] map[]}
vals:
map[]
defaultVals:[]
second-pass rendering result of "helmfile-0.130.0.yaml.part.0":
 0: ---
 1: helmDefaults:
 2:   verify: false
 3:   wait: true
 4:   timeout: 600
 5:   recreatePods: false
 6:   atomic: true
 7:   force: false
 8:   historyMax: 10
 9:
10: apiVersions:
11: - example/v1
12:
13: repositories:
14:   - name: stable
15:     url: https://kubernetes-charts.storage.googleapis.com

merged environment: &{default map[] map[]}
first-pass rendering starting for "helmfile-0.130.0.yaml.part.1": inherited=&{default map[] map[]}, overrode=<nil>
first-pass uses: &{default map[] map[]}
first-pass rendering output of "helmfile-0.130.0.yaml.part.1":
 0:
 1: releases:
 2:   - name: mongodb1
 3:     labels:
 4:       app_name: "mongodb1"
 5:     chart: "stable/mongodb"
 6:     version: 7.8.10
 7:     namespace:
 8:     hooks:
 9:       - events: ["prepare"]
10:         showlogs: true
11:         command: "echo"
12:         args: ["mongodb1"]
13:
14:   - name: mongodb2
15:     labels:
16:       app_name: "mongodb2"
17:     chart: "stable/mongodb"
18:     version: 7.8.10
19:     namespace:
20:     hooks:
21:       - events: ["prepare"]
22:         showlogs: true
23:         command: "echo"
24:         args: ["mongodb2"]
25:
26:   - name: mongodb3
27:     labels:
28:       app_name: "mongodb3"
29:     chart: "stable/mongodb"
30:     version: 7.8.10
31:     namespace:
32:     hooks:
33:       - events: ["prepare"]
34:         showlogs: true
35:         command: "echo"
36:         args: ["mongodb3"]
37:

first-pass produced: &{default map[] map[]}
first-pass rendering result of "helmfile-0.130.0.yaml.part.1": {default map[] map[]}
vals:
map[]
defaultVals:[]
second-pass rendering result of "helmfile-0.130.0.yaml.part.1":
 0:
 1: releases:
 2:   - name: mongodb1
 3:     labels:
 4:       app_name: "mongodb1"
 5:     chart: "stable/mongodb"
 6:     version: 7.8.10
 7:     namespace:
 8:     hooks:
 9:       - events: ["prepare"]
10:         showlogs: true
11:         command: "echo"
12:         args: ["mongodb1"]
13:
14:   - name: mongodb2
15:     labels:
16:       app_name: "mongodb2"
17:     chart: "stable/mongodb"
18:     version: 7.8.10
19:     namespace:
20:     hooks:
21:       - events: ["prepare"]
22:         showlogs: true
23:         command: "echo"
24:         args: ["mongodb2"]
25:
26:   - name: mongodb3
27:     labels:
28:       app_name: "mongodb3"
29:     chart: "stable/mongodb"
30:     version: 7.8.10
31:     namespace:
32:     hooks:
33:       - events: ["prepare"]
34:         showlogs: true
35:         command: "echo"
36:         args: ["mongodb3"]
37:

merged environment: &{default map[] map[]}
helm:XVlBz> v3.3.4+ga61ce56
Adding repo stable https://kubernetes-charts.storage.googleapis.com
exec: helm repo add stable https://kubernetes-charts.storage.googleapis.com --force-update
helm:gbaiC> "stable" has been added to your repositories
exec: helm repo add stable https://kubernetes-charts.storage.googleapis.com --force-update: "stable" has been added to your repositories

"stable" has been added to your repositories

helmfile-0.130.0.yaml: basePath=.
helmfile-0.130.0.yaml: basePath=.
hook[echo]: triggered by event "prepare"

hook[echo]: triggered by event "prepare"

echo:MRAjW> mongodb2
hook[echo]: mongodb2



hook[prepare] logs | mongodb2
hook[prepare] logs |
helmfile-0.130.0.yaml: basePath=.
hook[echo]: triggered by event "prepare"

echo:whTHc> mongodb1
hook[echo]: mongodb1



hook[prepare] logs | mongodb1
hook[prepare] logs |
echo:tcuAx> mongodb3
hook[echo]: mongodb3



hook[prepare] logs | mongodb3
hook[prepare] logs |
1 release(s) matching app_name=mongodb1 found in helmfile-0.130.0.yaml

Affected releases are:
  mongodb1 (stable/mongodb) UPDATED

processing 1 groups of releases in this order:
GROUP RELEASES
1     mongodb1

processing releases in group 1/1: mongodb1
Upgrading release=mongodb1, chart=stable/mongodb
exec: helm upgrade --install --reset-values mongodb1 stable/mongodb --version 7.8.10 --wait --timeout 600s --atomic --create-namespace --history-max 10 --dry-run
...

[dsuievalov]

Hello guys, any update? Maybe helpfile logic was changed and this is the expected behavior?

[mumoshu]

Thanks for reporting. Let me check this tomorrow!

[rdrim]

The same on my side. Seems like just prepare hooks are affected.
Presync hooks are getting filtered by selector just fine.

[mumoshu]

@dsuievalov @rdrim I believe this has been fixed via #1566. Would you mind trying v0.132.3?

[rdrim]

@mumoshu Thank you for the fix, works like a charm.

[dsuievalov]

@mumoshu yes, working fine. Thank you very much for fixing it and for your time on this :)