Helmfile: feat: Automated rollback of failed release

[mumoshu]

Once a helm release is failed, further effective helm upgrade --install runs can be prevented forever.

To work-around the issue, on every install/upgrade failure, helmfile should automatically run either helm delete --purge or helm rollback against the failed release according to the current revision number(1 or greater).

Relevant upstream issues are:

• https://github.com/helm/helm/pull/3097#issuecomment-417178786
• https://github.com/helm/helm/issues/3353#issuecomment-385222233

[osterman]

We’ve been using the —force helm arg to mitigate this. It will upgrade a failed release.

[mumoshu]

@osterman Thanks! Makes sense.

Then only thing remaining unclear to me is how upgrade --install --force works when it failed on first run.

According to @whereisaaron's comment at https://github.com/helm/helm/issues/3353#issuecomment-385222233, we seem to need helm delete --purge in that case, instead of rolling back. If upgrade --install --force works well in both cases, it can be a superior solution!

[stealthybox]

It's unclear to me whether --force is ever destructive.

An example of undesirable behavior would be if the --force flag ever causes a Deployment in a failed release to be deleted before it's "re-upgraded".

[whereisaaron]

You need the --purge because you can't (couldn't) upgrade a release that has never been successfully deployed. Even if you helm delete it. If you still can't upgrade a release that fails on its first attempt, then --purge is the only available way to handle that state I think?

[stealthybox]

@whereisaaron, there is a --force flag available on newer versions of helm upgrade that seems to auto-purge under certain conditions. I think it was pretty much explicitly added for that case, but the details escape me.

The lack of a clearly documented, safe, idempotent update in the newer versions of helm is somewhat frustrating.

[Evesy]

https://github.com/roboll/helmfile/pull/491 should fix this also

[Stono]

Is this in now? As I've just been burned by it:

UPGRADE FAILED
ROLLING BACK
Error: no CustomResourceDefinition with the name "meshpolicies.authentication.istio.io" found
Error: UPGRADE FAILED: no CustomResourceDefinition with the name "meshpolicies.authentication.istio.io" found

It seemed to auto rollback and I wasn't expecting it.

[mumoshu]

Could you try adding --atomic? The behaviour is currently opt-in. But if
you like I'd be eager to make it on by default!
2019年3月21日(木) 5:26 Karl Stoney [e-mail]:

Is this in now? As I've just been burned by it:

UPGRADE FAILED
ROLLING BACK
Error: no CustomResourceDefinition with the name "meshpolicies.authentication.istio.io" found
Error: UPGRADE FAILED: no CustomResourceDefinition with the name "meshpolicies.authentication.istio.io" found

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/roboll/helmfile/issues/256#issuecomment-475014141,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABV-QwalqK5F5ZyO_lZjZS4v529JxNlks5vYplqgaJpZM4WSuAd
.

[mumoshu]

@Stono Does setting atomic: true for your releases solves your issue? I'm going to make it default to true in the future #512

[Stono]

@mumoshu hey - i haven't had chance to check but i don't think you should change the default behaviour of helmfile to be different to that of helm.

For context as you know we have 200+ applications deployed and our ci/cd tooling manages rollbacks automatically for us, we wouldn't want helmfile taking that responsibility and we would also rather not go and edit 200+ helmfiles to deal with this API change!

[mikespokefire]

If you are going to change the default behaviour, would it be possible to provide a flag to opt out of this behaviour instead of having to specify it in a helmfile.yaml? i.e. helmfile sync --no-atomic

[mumoshu]

@Stono @mikespokefire Thanks for the feedbacks!

I'm happy with just adding the --atomic to helmfile and keeping the default behavior as-is.
Would it work for you both?

[Stono]

That would be much preferred.
I think a general rule of not deviating from default helm behaviour where
possible will make helmfile adoption simpler!

On Thu, 30 May 2019, 07:22 KUOKA Yusuke, [e-mail] wrote:

@Stono https://github.com/Stono @mikespokefire
https://github.com/mikespokefire Thanks for the feedbacks!

I'm happy with just adding the --atomic to helmfile and keeping the
default behavior as-is.
Would it work for you both?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/roboll/helmfile/issues/256?email_source=notifications&email_token=AALK7CNMBF5TDZOCVGX24H3PX5XB3A5CNFSM4FSK4AO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWRPKOY#issuecomment-497218875,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AALK7CKVGO2AP6VO24PF3W3PX5XB3ANCNFSM4FSK4AOQ
.

[mumoshu]

Noted. Thanks for your confirmation and the support ☺️