Flux: Invalid image ID on HelmRelease

[alastairs]

Describe the bug

Flux incorrectly updates HelmReleases when the tag is specified separately from the image, resulting in an invalid image ID error. I think also there's been a regression of the fix for #1290, or it wasn't fixed as I expected from reading that issue description.

To Reproduce

1. Configure FluxCD and the Helm Operator.
2. Define a HelmRelease for a chart with an imageTag value, e.g. stable/traefik or elastic/filebeat. A sample HelmRelease is included below
3. Set Flux automation annotations, including to point Flux at the imageTag value
4. Watch Flux update the image reference to the latest tag (1.7.19 at the time of writing)
5. Watch further Helm deployments fail

Sample HelmRelease

apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: traefik
  namespace: dev
  annotations:
    fluxcd.io/automated: "true"
    fluxcd.io/tag.chart-image: semver:~1.7
    # Tell Flux where to find the tag value
    tag.fluxcd.io/traefik: imageTag
spec:
  releaseName: belligerent-dog
  chart:
    repository: https://kubernetes-charts.storage.googleapis.com
    name: traefik
    version: 1.79.0
  rollback:
    enable: true
  values:
    ## Default values for Traefik
    image: traefik
    imageTag: v1.7

Expected behavior
Flux overwrites the imageTag value with the new tag version. Instead, however, it overwrites the image value, appending the tag. This results in Helm attempting to deploy an image like traefik:v1.7.19:v1.7. Flux logs a message stating expected image name as either <image>:<tag> or just <image>: invalid image ID.

Additional context
Add any other context about the problem here, e.g

• Flux version: 1.16
• Helm Operator version: 1.0.0-rc4
• Kubernetes version: 1.13 (GKE)
• Git provider: GitHub
• Container registry provider: Docker Hub

[alastairs]

I wondered if this might be because the traefik tags are v<semver>, rather than just <semver>. However, the following HelmRelease for filebeat displays the same behaviour

apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: filebeat
  namespace: waives-system
  annotations:
    fluxcd.io/automated: "true"
    fluxcd.io/tag.chart-image: semver:~7.4
    # Tell Flux where to find the tag value
    tag.fluxcd.io/traefik: imageTag
spec:
  releaseName: filebeat
  chart:
    repository: https://helm.elastic.co
    name: filebeat
    version: 7.4.1
  values:
    image: docker.elastic.co/beats/filebeat
    imageTag: 7.4.1

[stefanprodan]

You're missing the repository and filter annotations https://docs.fluxcd.io/en/stable/references/helm-operator-integration.html#annotations

[stefanprodan]

Try this:

apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: filebeat
  namespace: waives-system
  annotations:
    filter.fluxcd.io/filebeat: semver:~7.4
    tag.fluxcd.io/filebeat: imageTag
    repository.fluxcd.io/filebeat: image
spec:
  releaseName: filebeat
  chart:
    repository: https://helm.elastic.co
    name: filebeat
    version: 7.4.1
  values:
    image: docker.elastic.co/beats/filebeat
    imageTag: 7.4.1

[alastairs]

Thanks @stefanprodan, much appreciated. Good shout on the repository annotation; I'd missed tha tthis was required. I don't understand the relationship between the filter.fluxcd.io/* annotation and the fluxcd.io/tag.chart-image annotation, though, and the docs don't really make this clear AFAICT. Do I not need the fluxcd.io/automated annotation either?

[alastairs]

The changes @stefanprodan proposed have resolved the immediate issues, so many thanks for that. I still don't understand why this set worked and the previous one didn't, given the documentation here.

[alastairs]

I'm disappointed this was closed without addressing the documentation concerns I raised in my last comment. I've found Flux quite difficult to grasp so far, not least because I'm having to pull together information from 3-4 different (and somewhat sketchy) sources, with inconsistencies across them all. I still don't understand whether the annotations I'd initially used were valid but used incorrectly, or whether they were non-existent now (cf. with the references to the weave.works annotations now replaced with fluxcd.io annotations).

[2opremio]

@alastairs Sorry, I thought your query was resolved. Your input is very valuable as a new user, since you are looking at Flux with fresh eyes (and ours are biased after working with Flux for a while).

The tag/repository annotations indicate what image to update (and where to find it in the HelmRelease values) and the filter annotation indicates what registry images are considered as candidates for the update.

You do need the automated annotation.

I'm disappointed this was closed without addressing the documentation concerns I raised in my last comment

I didn't interpret those as concerns, my bad.

I am sorry about your frustrating experience with the documentation. It can always be improved, of course!

Regarding your disappointment, as you know this is a (rather understaffed) open source project, so, I would be really happy if you could channel that frustration/disappointment into documentation improvements so that the whole community wins as a result.

Re-opening the issue until you are satisfied with the outcome, let me know.

[2opremio]

The tag/repository annotations indicate what image to update (and where to find it in the HelmRelease values) and the filter annotation indicates what registry images are considered as candidates for the update.

Also, fluxcd.io/tag.chart-image is a legacy and alternative way to supply a filter, where you implicitly refer to an image placed at the top level of the values section.

[2opremio]

. I still don't understand whether the annotations I'd initially used were valid but used incorrectly, or whether they were non-existent now

Valid, but they didn't lead to the result you wanted

[alastairs]

Regarding your disappointment, as you know this is a (rather understaffed) open source project, so, I would be really happy if you could channel that frustration/disappointment into documentation improvements so that the whole community wins as a result.

Of course! I made a start earlier in the week with #2653 and am more than happy to contribute improvements when and where they’re clear. I’ll make some notes for a more substantial contribution—the main thing missing from a new user’s perspective is a “getting started” guide or similar, and I’m happy to add that when all the steps are straight in my head 😄

[2opremio]

the main thing missing from a new user’s perspective is a “getting started” guide or similar,

Have you taken a look at https://github.com/fluxcd/flux-get-started ?

[alastairs]

Have you taken a look at https://github.com/fluxcd/flux-get-started ?

Yes, but this speaks to my points about pulling the information from multiple different sources of information 🙂 There's some information in the documentation, some in this GitHub repo, some in the repo you pointed out, and some in the helm-operator getting started repo too.