Flux: Rolling back by removing tag

[alexanderbuhler]

Hey guys,

just want to get some feedback on this thought. Since we're using a full declarative process and don't want to roll back by interacting with flux, I was looking at a process which would remove a docker image tag on "rolling back". I was hoping that flux will jump back to the latest tag before but just ended up with it telling me:

warning="image with zero created timestamp"
current="quay.io/XXX/XXX-XXX-REMOVED-TAG (0001-01-01 00:00:00 +0000 UTC)"
latest="quay.io/XXX/LATEST-TAG-BEFORE-WHICH-SHOULD-BE-RESTORED-TO (2019-06-17 09:39:06.613095408 +0000 UTC)"
action="skip container"

I'm guessing quay doesn't remove image tags from their API after they get deleted and/or "garbage collection" is delayed here. Is there a way to tell flux to ignore images with zero created timestamp? Because it already seems to know what the latest valid image is based on the output above.

Besides that of course, is there another approaches/best practices on how to trigger rollbacks?

Thanks!

[stefanprodan]

@alexanderbuhler does this work if you use sem ver filters? e.g.

  annotations:
    flux.weave.works/automated: "true"
    flux.weave.works/tag.podinfod: semver:~1.4

[alexanderbuhler]

@stefanprodan Like the idea, but no change. Seems he's stuck inspecting the timestamp of the currently applied tag and once he gets the zero he skips no matter what.

[hiddeco]

@alexanderbuhler Flux does not make blind leaps like that by design, and to go one step back, human intervention (or a process explicitly telling Flux to perform the rollback) is required.

One of the reasons for this is to protect your running applications from faulty Docker registry responses. One registry vendor for example had a bug a while ago that would only list a selection of the tags available, if Flux would not have been designed to ignore those 'removals' (or more specifically, zero timestamps), this could have resulted in applications rolling back due to the tag no longer being listed in the response Flux receives from the API.

Would a --previous flag for fluxctl release, which rolls back to the previous recorded and/or valid tag help you achieve the same result?

[alexanderbuhler]

@hiddeco Sounds very reasonable. Actually I have nothing to hold against that.

I guess --previous would be a feasible solution for those looking to rollback conveniently to the previous release. This is not really declarative, though, but I guess you have to give in somewhere.

Currently, we're looking at incoporating Flagger into our CI process so manual steps might not be relevant any more.

[hiddeco]

@alexanderbuhler great, I hope Flagger will serve you well! I created a new issue for the --previous flag (#2221).

As your question has been answered I am going to close this issue to get it of our radar, you are welcome to open a new issue if you encounter new problems and/or have questions, or to join #flux. :sunflower: