Helmfile: Waiting to install release until another release is finished

[maratoid]

Something like:

releases:
  # Published chart example
  - name: vault
    namespace: vault 
    labels:                                 
      foo: bar
    chart: roboll/vault-secret-manager
    version: ~1.24.1
  - name: grafana
    namespace: another
    chart: ../my-charts/grafana
    wait:
      vault       # wait for vault release to be "finished" before running this helm install

I've found myself in a situation where I was installing vault and then a bunch of things that depended on vault to be up, unsealed and populated with secrets. I ended up writing a large-ish bash script that drove helmfile with a bunch of while loops querying status of vault pods.

Could combine this with something like:

releases:
  # Published chart example
  - name: vault
    namespace: vault 
    labels:                                 
      foo: bar
    chart: roboll/vault-secret-manager
    version: ~1.24.1
    pre:
      shell: /bin/sh
      command: pre.sh          # pre orchestrate some stuff
    post: 
      shell: /bin/sh
      command: post.sh         # post orchestrate some stuff
  - name: grafana
    namespace: another
    chart: ../my-charts/grafana
    wait:
      vault       # wait for vault release to be "finished", pre and post included, before running this helm install

Maybe this is out of scope for a tool like helmfile though.

[sstarcher]

@maratoid I have solved this for myself using labels. I have a label called stage and I have pre, and post.

When I install a brand new cluster I do helmfile -l stage=pre sync && helmfile -l stage=post sync

Overall if you have control of the chart ordering should be a non-issue and the release should be smart enough to wait for the other service or to crash until the other service is up.

[mumoshu]

@maratoid @sstarcher Thanks for the feedback and the info!

I love talking about this kind of use-cases. I believe this operational issue - how to define/script a workflow for rolling out a set of helm releases - should be somehow resolved.

I'm just unsure how helmfile can/should help it yet.

helm once tended to add a simple workflow system in the form of hooks, but currently, the general direction is to recommend uses to implement real workflows using other external tools.

writing a large-ish bash script that drove helmfile with a bunch of while loops querying status of vault pods.

Yes, this is one way of implementing such workflow.

Talking personally, I'm investing on brigade which is intended for complementing helm. In theory, any workflow engine can be used instead. Airflow, Argo, CircleCI, Drone, and so on. But your mileage may vary according to your requirements.

Possible requirements include:

1. Reliability: Transient failures of K8S API connectivity, networking issue, etc. doesn't affect a workflow run. Automatic retries.
2. Durability: A process/container/node crashed in the middle of a workflow doesn't leave the half-done workflow forever.
3. Less verbosity: A correct distributed system should work regardless of startup ordering. Regardless, just relying on each microservice's ability to wait/retry forever until the dependent services come up results in a lot of "connection error" messages bloating your log stream and monitoring system. Just let your external workflow system to handle ordering and log filtering.
4. Scriptability: I'm not sure how I should name this but, many workflow systems end up reimplementing turing-complete language on top of markup languages like yaml. Then, why don't you just start with a real programming language and implement a workflow DSL on top if necessary?

writing a large-ish bash script that drove helmfile with a bunch of while loops querying status of vault pods.

Yes, I guess this is a feasible way to achieve the point 3 and 4 above.

[mumoshu]

@maratoid @sstarcher Btw, I guess that documenting idioms like helmfile -l stage=pre sync && helmfile -l stage=post sync would be helpful for helmfile users. WDYT about that?

[sstarcher]

@mumoshu I need to put in a PR to resolve #99 preferably first

[maratoid]

brigade does seem like a better fit. Thank you.

[baurmatt]

Helmfile seems to have this feature now => DAG-aware installation/deletion ordering