Flux: Flux does not update image from purely numerical tags
Our Docker images are tagged with a simple numeric value by our build system. I've a HelmRelease defined with annotations as follows:
fluxcd.io/automated: true
filter.fluxcd.io/extractors: regexp:^\d+$
flux has successfully deployed this Helm release, Kubernetes has pulled the image and started the container succesfully. Flux has also identified that there are newer versions of the image available:
WORKLOAD CONTAINER IMAGE CREATED
dev:deployment/extractors extractors our.azurecr.io/extractors
| 154 19 Dec 19 15:48 UTC
| 153 10 Dec 19 12:06 UTC
| 152 09 Dec 19 09:45 UTC
| 151 28 Nov 19 14:17 UTC
'-> 150 27 Nov 19 09:04 UTC
149 21 Nov 19 17:49 UTC
b2bf9fa 21 Nov 19 13:14 UTC
However, the deployed image is, as indicated in that output, still v150, rather than the newest v154. Based on a comment from @stefanprodan, I have also tried specifying a filter of regexp:^[0-9]*$ with the same result.
I am able to provide logs with a bit of guidance on what components to focus on,e tc.
Additional context
Add any other context about the problem here, e.g
- Flux version: 1.16
- Helm Operator version: 1.0.0-rc4
- Kubernetes version: 1.15.4-gke.22
- Git provider: GitHub (SaaS)
- Container registry provider: Azure Container Registry
alastairs
All 13 comments
However, the deployed image is, as indicated in that output, still v150, rather than the newest v154
Can you show us the creation time and creation labels of both images?
2opremio
on 20 Dec 2019
Also, can you show us the full HelRelease? (with the full annotations and values)
2opremio
on 20 Dec 2019
@2opremio Thanks for your quick reply. I'm not sure what you mean by "creation label", sorry. Is the "created" column in the issue description different from what you mean by "creation time"?
The HelmRelease is as follows:
---
apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
name: extractors
namespace: dev
annotations:
fluxcd.io/automated: "true"
# Tell Flux which repository to watch
repository.fluxcd.io/extractors: extractors
# Tell Flux how to update the Docker tag
filter.fluxcd.io/extractors: regexp:^[0-9]*$
spec:
releaseName: vehement-bunny
chart:
git: [email protected]:OurOrg/our-repo
path: charts/extractors
ref: deploy/dev
rollback:
enable: true
values:
# The number of instances of the service to run.
replicaCount: 1
# Secret to reference for credentials, etc.
secretName: vehement-bunny-extractors
# The number of revisions of the deployment to keep. Each `helm upgrade` counts
# as a deployment revision.
revisionHistoryLimit: 2
image:
# The registry containing the our images. The image name is hard-
# coded into this chart.
registry: our.azurecr.io
# The version of the image to pull from the registry. The image name is hard-
# coded into this chart.
tag: 150
# Under what circumstances Kubernetes will pull the image for updates. If not
# set here, will default to IfNotPresent.
pullPolicy: IfNotPresent
# The secret(s) containing the credentials for authenticating with the Docker
# registry when pulling images.
pullSecrets:
- docker-registry
rabbitmq:
# The URI of a RabbitMQ message broker.
uri: #redacted
environment:
#redacted
persistence:
#redacted
# Override resource requests by uncommenting the below lines and adjusting them
# as necessary, and remove the curly braces after 'resources:'.
resources:
limits:
cpu: 500m
memory: 750Mi
requests:
cpu: 100m
memory: 500Mi
# See https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
# This will be copied wholesale into the Deployment resource for this chart.
nodeSelector: {}
# See https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
# This will be copied wholesale into the Deployment resource for this chart.
tolerations:
- key: "dedicated"
operator: "Equal"
value: "worker"
effect: "NoSchedule"
# See https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
# This will be copied wholesale into the Deployment resource for this chart.
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 90
preference:
matchExpressions:
- key: "role"
operator: "In"
values:
- "worker"
Replace filter.fluxcd.io/extractors: regexp:^[0-9]*$ with filter.fluxcd.io/chart-image: regexp:^[0-9]*$ and please read the docs to see how automation works for HelmReleases http://docs.fluxcd.io/en/1.17.0/references/helm-operator-integration.html
stefanprodan
on 20 Dec 2019
In addition to @stefanprodan his comment, the repository.fluxcd.io/extractors annotation can also be removed.
hiddeco
on 20 Dec 2019
Replace
filter.fluxcd.io/extractors: regexp:^[0-9]*$withfilter.fluxcd.io/chart-image: regexp:^[0-9]*$and please read the docs to see how automation works for HelmReleases http://docs.fluxcd.io/en/1.17.0/references/helm-operator-integration.html
Thanks @stefanprodan. I'm finding this area _extremely_ confusing; in fact, in #2663 you yourself seemed to recommend changing from a chart-image annotation to an alias annotation.
In addition to @stefanprodan his comment, the
repository.fluxcd.io/extractorsannotation can also be removed.
Thanks @hiddeco. I'm further confused, because in creating #2670, it was indicated that the repository.fluxcd.io/<alias> annotation was _required_ in order for the equivalent registry/tag/filter annotations to take effect (now documented as a result of that PR). Annoyingly I can't now find the reference for that, either here on GitHub or in Slack.
alastairs
on 20 Dec 2019
@alastairs the annotations should match the HelmRelease values structure, not the deployment one, the problem with your values is that the image is not specified so Flux can't update it.
values:
registry:
image:
tag:
So based on http://docs.fluxcd.io/en/1.17.0/references/helm-operator-integration.html#automated-image-detection, I think the structure should be:
values:
image:
registry: our.azurecr.io
repository: extractors # not image
tag: 150
Does that look right?
alastairs
on 20 Dec 2019
Yes that looks ok to me
stefanprodan
on 20 Dec 2019
Ok, still no luck. Here's the current state of play:
HelmRelease
apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
name: extractors
namespace: dev
annotations:
fluxcd.io/automated: "true"
# Tell Flux how to update the Docker tag
filter.fluxcd.io/chart-image: regexp:^[0-9]*$
spec:
releaseName: vehement-bunny
chart:
git: [email protected]:OurOrg/our-repo
path: charts/extractors
ref: deploy/dev
rollback:
enable: true
values:
# The number of instances of the service to run.
replicaCount: 1
# Secret to reference for credentials, etc.
secretName: vehement-bunny-extractors
# The number of revisions of the deployment to keep. Each `helm upgrade` counts
# as a deployment revision.
revisionHistoryLimit: 2
image:
# The registry containing our images. The image name is hard-coded into this chart.
registry: our.azurecr.io
# The version of the image to pull from the registry. The image name is hard-
# coded into this chart.
tag: 150
# Used by flux only, to find newer tags for deployment. The image name is hard-
# coded into this chart.
repository: extractors
# Under what circumstances Kubernetes will pull the image for updates. If not
# set here, will default to IfNotPresent.
pullPolicy: IfNotPresent
# The secret(s) containing the credentials for authenticating with the Docker
# registry when pulling images.
pullSecrets:
- docker-registry
rabbitmq:
# The URI of a RabbitMQ message broker.
uri: # redacted
environment:
# redacted
persistence:
# redacted
# Override resource requests by uncommenting the below lines and adjusting them
# as necessary, and remove the curly braces after 'resources:'.
resources:
limits:
cpu: 500m
memory: 750Mi
requests:
cpu: 100m
memory: 500Mi
# See https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
# This will be copied wholesale into the Deployment resource for this chart.
nodeSelector: {}
# See https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
# This will be copied wholesale into the Deployment resource for this chart.
tolerations:
- key: "dedicated"
operator: "Equal"
value: "worker"
effect: "NoSchedule"
# See https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
# This will be copied wholesale into the Deployment resource for this chart.
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 90
preference:
matchExpressions:
- key: "role"
operator: "In"
values:
- "worker"
fluxctl list-workloads -ndev
WORKLOAD CONTAINER IMAGE RELEASE POLICY
dev:deployment/vehement-bunny-extractors extractors ch360-on.azurecr.io/extractors:150 ready
dev:helmrelease/extractors chart-image ch360-on.azurecr.io/extractors DEPLOYED automated
fluxctl list-images -w dev:deployment/vehement-bunny-extractors
WORKLOAD CONTAINER IMAGE CREATED
dev:deployment/vehement-bunny-extractors extractors our.azurecr.io/extractors
| 155 20 Dec 19 10:39 UTC
| 154 19 Dec 19 15:48 UTC
| 153 10 Dec 19 12:06 UTC
| 152 09 Dec 19 09:45 UTC
| 151 28 Nov 19 14:17 UTC
'-> 150 27 Nov 19 09:04 UTC
149 21 Nov 19 17:49 UTC
fluxctl list-images -w dev:helmrelease/extractors
WORKLOAD CONTAINER IMAGE CREATED
dev:helmrelease/extractors chart-image
| 155 20 Dec 19 10:39 UTC
| 154 19 Dec 19 15:48 UTC
| 153 10 Dec 19 12:06 UTC
| 152 09 Dec 19 09:45 UTC
| 151 28 Nov 19 14:17 UTC
| 150 27 Nov 19 09:04 UTC
| 149 21 Nov 19 17:49 UTC
'-> (untagged) ?
Does the chart in charts/extractors also define entries with the same structure as the following?
values:
image:
# The registry containing our images. The image name is hard-coded into this chart.
registry: our.azurecr.io
# The version of the image to pull from the registry. The image name is hard-
# coded into this chart.
tag: 150
# Used by flux only, to find newer tags for deployment. The image name is hard-
# coded into this chart.
repository: extractors
In YAML tag: 150 makes the value an integer and I think it breaks the parser, should be tag: "150"
stefanprodan
on 8 Jan 2020
Thank you @stefanprodan and @2opremio for your replies. I suspect it is the YAML parsing issue, good spot. I'm not able to verify this fix, unfortunately, so please assume this is the correct resolution 馃槃
alastairs
on 17 Jan 2020
Related issues
alexhumphreys
路
4Comments
astraldragon
路
3Comments
phoppe93
路
4Comments
pmarkus
路
4Comments
kuburoman
路
3Comments