Helmfile: destroy not reversing the order of releases

Created on 13 Apr 2021  路  6Comments  路  Source: roboll/helmfile

helmfile.yaml

repositories:
  - name: cert-manager
    url: https://charts.jetstack.io
  - name: prometheus
    url: https://prometheus-community.github.io/helm-charts
  - name: kiali
    url: https://kiali.org/helm-charts
  - name: autoscaler
    url: https://kubernetes.github.io/autoscaler
  - name: bitnami
    url: https://charts.bitnami.com/bitnami
  - name: grafana
    url: https://grafana.github.io/helm-charts
  - name: elastic
    url: https://helm.elastic.co

helmDefaults:
  cleanupOnFail: true
  # wait for k8s resources via --wait. (default false), 300 seconds
  wait: true


releases:
  - name: cert-manager
    namespace: test
    chart: cert-manager/cert-manager
    version: v1.2.0
  - name: metrics-server
    namespace: test
    needs:
      - test/cert-manager
    chart: bitnami/metrics-server
    version: 5.8.0
  - name: prometheus
    needs:
      - test/cert-manager
    chart: prometheus/prometheus
    namespace: test
    version: 13.3.1
  - name: autoscaler
    namespace: test
    needs:
      - test/prometheus
    chart: autoscaler/cluster-autoscaler
    version: 9.4.0
  - name: prometheus-adapter
    namespace: test
    needs:
      - test/autoscaler
    chart: prometheus/prometheus-adapter
    version: 2.12.0
  - name: grafana
    namespace: test
    needs:
      - test/prometheus-adapter
    chart: grafana/grafana
    version: 6.6.3
  - name: external-dns
    namespace: test
    needs:
      - test/grafana
    chart: bitnami/external-dns
    version: 4.7.0
  - name: logstash
    namespace: test
    needs:
      - test/external-dns
    chart: elastic/logstash
    version: 7.11.1

Install order

processing 7 groups of releases in this order:
GROUP RELEASES
1     test/cert-manager
2     test/metrics-server, test/prometheus
3     test/autoscaler
4     test/prometheus-adapter
5     test/grafana
6     test/external-dns
7     test/logstash

uninstall order

processing 7 groups of releases in this order:
GROUP RELEASES
1     test/cert-manager
2     test/external-dns
3     test/grafana
4     test/prometheus-adapter
5     test/autoscaler
6     test/prometheus, test/metrics-server
7     test/logstash

bash-3.2$ helmfile --version
helmfile version v0.138.7
picture bseenu

Most helpful comment

Thanks everyone! I was finally able to reproduce it. I've also realized that the reversing logic had been incorrect since the addition of the reverse ordering and it sometimes resulted in a wrong order. #1777 fixes it.

picture mumoshu on 14 Apr 2021
🎉2

All 6 comments

@bseenu Hey! Thanks for reporting. Unfortunately I had no luck reproducing this. What was the exact helmfile destroy command you tried?

mumoshu picture mumoshu on 13 Apr 2021

install

helmfile --debug apply --skip-diff-on-install

uninstall

helmfile --debug destroy

bseenu picture bseenu on 13 Apr 2021
👍1

I can confirm on a fresh 1.18 cluster, from macos using brew's helm/helmfile:

$ helm version
version.BuildInfo{Version:"v3.5.3", GitCommit:"041ce5a2c17a58be0fcd5f5e16fb3e7e95fea622", GitTreeState:"dirty", GoVersion:"go1.16"}
$ helmfile version
helmfile version v0.138.7

same output

processing 7 groups of releases in this order:
GROUP RELEASES
1     test/cert-manager
2     test/external-dns
3     test/grafana
4     test/prometheus-adapter
5     test/autoscaler
6     test/prometheus, test/metrics-server
7     test/logstash

processing releases in group 1/7: test/cert-manager
Deleting cert-manager

which is unexpected

ldemailly picture ldemailly on 13 Apr 2021

I reproduced in a simpler (and faster to test) way:

helmfile

repositories:
  - name: envoy
    url: https://slamdev.github.io/helm-charts


releases:
  - name: envoy1
    namespace: test
    chart: envoy/envoy
  - name: envoy2
    namespace: test
    chart: envoy/envoy
    needs:
      - test/envoy1
  - name: envoy3
    namespace: test
    chart: envoy/envoy
    needs:
      - test/envoy2
  - name: envoy4
    namespace: test
    chart: envoy/envoy
    needs:
      - test/envoy3
  - name: envoy5
    namespace: test
    chart: envoy/envoy
    needs:
      - test/envoy4
  - name: envoy6
    namespace: test
    chart: envoy/envoy
    needs:
      - test/envoy5
  - name: envoy7
    namespace: test
    chart: envoy/envoy
    needs:
      - test/envoy6

$ helmfile --debug apply --skip-diff-on-install
processing 7 groups of releases in this order:
GROUP RELEASES
1     test/envoy1
2     test/envoy2
3     test/envoy3
4     test/envoy4
5     test/envoy5
6     test/envoy6
7     test/envoy7

^ as expected

$ helmfile --debug destroy
processing 7 groups of releases in this order:
GROUP RELEASES
1     test/envoy1
2     test/envoy6
3     test/envoy5
4     test/envoy4
5     test/envoy3
6     test/envoy2
7     test/envoy7
[...]
DELETED RELEASES:
NAME
envoy1
envoy6
envoy5
envoy4
envoy3
envoy2
envoy7

(Big!) BUG :-)

Note: it works with a dependency chain of 6 somehow and starts breaking at 7

ldemailly picture ldemailly on 14 Apr 2021

If it helps here is the full output of destroy until bug appears; there isn't much debugging of the ordering actually:

$ helmfile --debug destroy
processing file "helmfile.yaml" in directory "."
first-pass rendering starting for "helmfile.yaml.part.0": inherited=&{default map[] map[]}, overrode=<nil>
first-pass uses: &{default map[] map[]}
first-pass rendering output of "helmfile.yaml.part.0":
 0: repositories:
 1:   - name: envoy
 2:     url: https://slamdev.github.io/helm-charts
 3: 
 4: 
 5: releases:
 6:   - name: envoy1
 7:     namespace: test
 8:     chart: envoy/envoy
 9:   - name: envoy2
10:     namespace: test
11:     chart: envoy/envoy
12:     needs:
13:       - test/envoy1
14:   - name: envoy3
15:     namespace: test
16:     chart: envoy/envoy
17:     needs:
18:       - test/envoy2
19:   - name: envoy4
20:     namespace: test
21:     chart: envoy/envoy
22:     needs:
23:       - test/envoy3
24:   - name: envoy5
25:     namespace: test
26:     chart: envoy/envoy
27:     needs:
28:       - test/envoy4
29:   - name: envoy6
30:     namespace: test
31:     chart: envoy/envoy
32:     needs:
33:       - test/envoy5
34:   - name: envoy7
35:     namespace: test
36:     chart: envoy/envoy
37:     needs:
38:       - test/envoy6
39: 

first-pass produced: &{default map[] map[]}
first-pass rendering result of "helmfile.yaml.part.0": {default map[] map[]}
vals:
map[]
defaultVals:[]
second-pass rendering result of "helmfile.yaml.part.0":
 0: repositories:
 1:   - name: envoy
 2:     url: https://slamdev.github.io/helm-charts
 3: 
 4: 
 5: releases:
 6:   - name: envoy1
 7:     namespace: test
 8:     chart: envoy/envoy
 9:   - name: envoy2
10:     namespace: test
11:     chart: envoy/envoy
12:     needs:
13:       - test/envoy1
14:   - name: envoy3
15:     namespace: test
16:     chart: envoy/envoy
17:     needs:
18:       - test/envoy2
19:   - name: envoy4
20:     namespace: test
21:     chart: envoy/envoy
22:     needs:
23:       - test/envoy3
24:   - name: envoy5
25:     namespace: test
26:     chart: envoy/envoy
27:     needs:
28:       - test/envoy4
29:   - name: envoy6
30:     namespace: test
31:     chart: envoy/envoy
32:     needs:
33:       - test/envoy5
34:   - name: envoy7
35:     namespace: test
36:     chart: envoy/envoy
37:     needs:
38:       - test/envoy6
39: 

merged environment: &{default map[] map[]}
helm:XVlBz> v3.5.3+g041ce5a
7 release(s) found in helmfile.yaml

Listing releases matching ^envoy7$
exec: helm list --filter ^envoy7$ --namespace test --deployed --failed --pending
helm:gbaiC> NAME    NAMESPACE   REVISION    UPDATED                                 STATUS      CHART       APP VERSION
envoy7  test        1           2021-04-13 16:38:57.131682 -0700 PDT    deployed    envoy-0.0.6 v1.17.1
envoy7  test        1           2021-04-13 16:38:57.131682 -0700 PDT    deployed    envoy-0.0.6 v1.17.1    

Listing releases matching ^envoy6$
exec: helm list --filter ^envoy6$ --namespace test --deployed --failed --pending
helm:MRAjW> NAME    NAMESPACE   REVISION    UPDATED                             STATUS      CHART       APP VERSION
envoy6  test        1           2021-04-13 16:38:51.30436 -0700 PDT deployed    envoy-0.0.6 v1.17.1
envoy6  test        1           2021-04-13 16:38:51.30436 -0700 PDT deployed    envoy-0.0.6 v1.17.1    

Listing releases matching ^envoy5$
exec: helm list --filter ^envoy5$ --namespace test --deployed --failed --pending
helm:whTHc> NAME    NAMESPACE   REVISION    UPDATED                                 STATUS      CHART       APP VERSION
envoy5  test        1           2021-04-13 16:38:45.594236 -0700 PDT    deployed    envoy-0.0.6 v1.17.1
envoy5  test        1           2021-04-13 16:38:45.594236 -0700 PDT    deployed    envoy-0.0.6 v1.17.1    

Listing releases matching ^envoy4$
exec: helm list --filter ^envoy4$ --namespace test --deployed --failed --pending
helm:tcuAx> NAME    NAMESPACE   REVISION    UPDATED                                 STATUS      CHART       APP VERSION
envoy4  test        1           2021-04-13 16:38:40.040154 -0700 PDT    deployed    envoy-0.0.6 v1.17.1
envoy4  test        1           2021-04-13 16:38:40.040154 -0700 PDT    deployed    envoy-0.0.6 v1.17.1    

Listing releases matching ^envoy3$
exec: helm list --filter ^envoy3$ --namespace test --deployed --failed --pending
helm:hxKQF> NAME    NAMESPACE   REVISION    UPDATED                                 STATUS      CHART       APP VERSION
envoy3  test        1           2021-04-13 16:38:34.571427 -0700 PDT    deployed    envoy-0.0.6 v1.17.1
envoy3  test        1           2021-04-13 16:38:34.571427 -0700 PDT    deployed    envoy-0.0.6 v1.17.1    

Listing releases matching ^envoy2$
exec: helm list --filter ^envoy2$ --namespace test --deployed --failed --pending
helm:DaFpL> NAME    NAMESPACE   REVISION    UPDATED                                 STATUS      CHART       APP VERSION
envoy2  test        1           2021-04-13 16:38:28.891763 -0700 PDT    deployed    envoy-0.0.6 v1.17.1
envoy2  test        1           2021-04-13 16:38:28.891763 -0700 PDT    deployed    envoy-0.0.6 v1.17.1    

Listing releases matching ^envoy1$
exec: helm list --filter ^envoy1$ --namespace test --deployed --failed --pending
helm:SjFbc> NAME    NAMESPACE   REVISION    UPDATED                                 STATUS      CHART       APP VERSION
envoy1  test        1           2021-04-13 16:38:22.918428 -0700 PDT    deployed    envoy-0.0.6 v1.17.1
envoy1  test        1           2021-04-13 16:38:22.918428 -0700 PDT    deployed    envoy-0.0.6 v1.17.1    

processing 7 groups of releases in this order:
GROUP RELEASES
1     test/envoy1
2     test/envoy6
3     test/envoy5
4     test/envoy4
5     test/envoy3
6     test/envoy2
7     test/envoy7
ldemailly picture ldemailly on 14 Apr 2021

Thanks everyone! I was finally able to reproduce it. I've also realized that the reversing logic had been incorrect since the addition of the reverse ordering and it sometimes resulted in a wrong order. #1777 fixes it.

mumoshu picture mumoshu on 14 Apr 2021
🎉2
Was this page helpful?
0 / 5 - 0 ratings

Related issues

madAndroid picture madAndroid  路  3Comments
RafalMaleska picture RafalMaleska  路  3Comments
ppawiggers picture ppawiggers  路  3Comments
willejs picture willejs  路  4Comments
ivandardi picture ivandardi  路  3Comments