Flux: `automated` and `ignore` annotations clash

Created on 19 Feb 2019 · 5Comments · Source: fluxcd/flux

Having a resource annotated as both automated and ignored appears to be treated as both and seems to break automation of all cluster resources.

In our case, as long as we had some HelmRelease manifests annotated with both, none of the automated resources in our cluster were getting their images updated. Note that we were seeing the latest images available (via fluxctl list-images), they were just never used.

Upon removing one of the two, everything started flowing as normal.

One of our HelmReleases looked like this:

---
apiVersion: flux.weave.works/v1beta1
kind: HelmRelease
metadata:
  name: my-webapi
  namespace: default
  labels:
    chart: webapi
  annotations:
    flux.weave.works/automated: "true"
    flux.weave.works/tag.chart-image: glob:release-*
    flux.weave.works/ignore: true
...

With both of these annotations in place, we'd have log entries that seem to indicate it being both ignored and automated:

ts=2019-02-18T22:02:34.277718708Z caller=sync.go:66 component=sync-loop resource=default:helmrelease/my-webapi ignore=apply
ts=2019-02-18T22:07:27.170619954Z caller=images.go:33 component=sync-loop error="checking services for new images: helmreleases.flux.weave.works \"my-webapi\" not found"

After doing some digging, I think this occurs due to an issue in pollForNewImages.

It looks like GetUnlockedAutomatedResources doesn't take the policy.Ignore into account, so the image check fails when comparing with a non-existent flux controller on line 33. Since that error then causes a return, it short circuits image updates for everything else.

bug ☠ high user impact

Source

whitlaaa

All 5 comments

Thanks a lot for your high quality bug report and proposed solution. :1st_place_medal:

I validated the bug (valid) and your proposed solution (correct) and turned it into #1749. I am aiming to get a patch release out as soon as possible after this has been merged into master.