Flux: Automate helm release with multiple images
Describe the bug
I would like to automate the prometheus operator helm release. My config looks like this:
---
apiVersion: flux.weave.works/v1beta1
kind: HelmRelease
metadata:
name: prometheus-operator
namespace: monitoring
annotations:
flux.weave.works/automated: 'true'
flux.weave.works/tag.prometheus: semver:~2.10
flux.weave.works/tag.alertmanager: semver:~0.16
flux.weave.works/tag.grafana: semver:~6.2
Unfortunately nothing gets updated automatically. I would expect prometheus to update to 2.10 and grafana to 6.2.4
To Reproduce
Deploy prometheus operator with my sample config.
Expected behavior
Flux should use the newer images.
Logs
$ fluxctl list-workloads -n monitoring
WORKLOAD CONTAINER IMAGE RELEASE POLICY
monitoring:daemonset/prometheus-prometheus-node-exporter node-exporter quay.io/prometheus/node-exporter:v0.17.0 ready
monitoring:deployment/prometheus-grafana grafana-sc-dashboard kiwigrid/k8s-sidecar:0.0.16 ready
grafana grafana/grafana:6.2.0
grafana-sc-datasources kiwigrid/k8s-sidecar:0.0.16
monitoring:deployment/prometheus-kube-state-metrics kube-state-metrics k8s.gcr.io/kube-state-metrics:v1.5.0 ready
monitoring:deployment/prometheus-prometheus-oper-operator prometheus-operator quay.io/coreos/prometheus-operator:v0.29.0 ready
monitoring:helmrelease/prometheus-operator DEPLOYED automated
monitoring:statefulset/alertmanager-prometheus-prometheus-oper-alertmanager alertmanager quay.io/prometheus/alertmanager:v0.17.0 ready
config-reloader quay.io/coreos/configmap-reload:v0.0.1
monitoring:statefulset/prometheus-prometheus-prometheus-oper-prometheus prometheus quay.io/prometheus/prometheus:v2.9.1 ready
prometheus-config-reloader quay.io/coreos/prometheus-config-reloader:v0.29.0
rules-configmap-reloader quay.io/coreos/configmap-reload:v0.0.1
$ fluxctl list-images -n monitoring
E0619 12:21:12.810749 14294 portforward.go:385] error copying from local connection to remote stream: read tcp4 127.0.0.1:39659->127.0.0.1:46076: read: connection reset by peer
WORKLOAD CONTAINER IMAGE CREATED
monitoring:daemonset/prometheus-prometheus-node-exporter node-exporter quay.io/prometheus/node-exporter
| master 12 Jun 19 18:59 UTC
| latest 04 Jun 19 16:51 UTC
| v0.18.1 04 Jun 19 16:51 UTC
| v0.18.0 09 May 19 23:23 UTC
'-> v0.17.0 30 Nov 18 15:59 UTC
v0.17.0-rc.0 20 Oct 18 06:36 UTC
v0.16.0 15 May 18 15:58 UTC
v0.16.0-rc.3 27 Apr 18 15:56 UTC
v0.16.0-rc.2 17 Apr 18 14:12 UTC
v0.15.2 05 Dec 17 14:55 UTC
monitoring:deployment/prometheus-grafana grafana-sc-dashboard kiwigrid/k8s-sidecar
| latest 11 Jun 19 11:20 UTC
'-> 0.0.16 13 Mar 19 06:55 UTC
0.0.15 13 Mar 19 06:47 UTC
0.0.14 13 Mar 19 06:39 UTC
0.0.13 12 Mar 19 07:47 UTC
0.0.12 04 Mar 19 06:36 UTC
0.0.11 19 Feb 19 07:39 UTC
0.0.10 14 Feb 19 07:23 UTC
0.0.9 14 Feb 19 07:20 UTC
0.0.8 14 Feb 19 07:16 UTC
grafana grafana/grafana
| master 18 Jun 19 12:52 UTC
| 6.2.4 18 Jun 19 11:10 UTC
| 6.2.2 05 Jun 19 13:25 UTC
| latest 05 Jun 19 13:25 UTC
| 6.2.1 27 May 19 13:08 UTC
'-> 6.2.0 22 May 19 11:34 UTC
6.2.0-beta2 15 May 19 11:49 UTC
6.2.0-beta1 07 May 19 14:33 UTC
6.1.6 29 Apr 19 14:04 UTC
5.4.4 29 Apr 19 12:18 UTC
grafana-sc-datasources kiwigrid/k8s-sidecar
| latest 11 Jun 19 11:20 UTC
'-> 0.0.16 13 Mar 19 06:55 UTC
0.0.15 13 Mar 19 06:47 UTC
0.0.14 13 Mar 19 06:39 UTC
0.0.13 12 Mar 19 07:47 UTC
0.0.12 04 Mar 19 06:36 UTC
0.0.11 19 Feb 19 07:39 UTC
0.0.10 14 Feb 19 07:23 UTC
0.0.9 14 Feb 19 07:20 UTC
0.0.8 14 Feb 19 07:16 UTC
monitoring:deployment/prometheus-kube-state-metrics kube-state-metrics k8s.gcr.io/kube-state-metrics image data not available
'-> v1.5.0 ?
monitoring:deployment/prometheus-prometheus-oper-operator prometheus-operator quay.io/coreos/prometheus-operator
| master 07 Jun 19 12:19 UTC
| v0.30.1 31 May 19 20:47 UTC
| v0.30.0 10 May 19 20:42 UTC
'-> v0.29.0 20 Feb 19 11:11 UTC
v0.28.0 29 Jan 19 14:45 UTC
v0.27.0 08 Jan 19 16:25 UTC
v0.26.0 30 Nov 18 12:55 UTC
v0.25.0 24 Oct 18 13:25 UTC
v0.24.0 11 Oct 18 12:05 UTC
v0.23.2 23 Aug 18 13:36 UTC
monitoring:helmrelease/prometheus-operator
monitoring:statefulset/alertmanager-prometheus-prometheus-oper-alertmanager alertmanager quay.io/prometheus/alertmanager
| master 14 Jun 19 13:17 UTC
| latest 03 May 19 09:48 UTC
'-> v0.17.0 03 May 19 09:48 UTC
v0.16.2 05 Apr 19 12:56 UTC
v0.16.1 31 Jan 19 15:34 UTC
v0.16.0 21 Jan 19 13:25 UTC
v0.16.0-beta.0 15 Jan 19 20:22 UTC
v0.16.0-alpha.0 30 Nov 18 22:41 UTC
v0.15.3 09 Nov 18 16:02 UTC
v0.15.2 14 Aug 18 11:13 UTC
config-reloader quay.io/coreos/configmap-reload
| latest 02 Jan 17 23:21 UTC
'-> v0.0.1 02 Jan 17 23:21 UTC
monitoring:statefulset/prometheus-prometheus-prometheus-oper-prometheus prometheus quay.io/prometheus/prometheus
| master 17 Jun 19 19:40 UTC
| latest 25 May 19 13:59 UTC
| v2.10.0 25 May 19 13:59 UTC
| v2.10.0-rc.0 22 May 19 13:52 UTC
| v2.9.2 24 Apr 19 16:59 UTC
'-> v2.9.1 16 Apr 19 19:18 UTC
v2.9.0 15 Apr 19 10:38 UTC
v2.9.0-rc.0 10 Apr 19 21:50 UTC
v2.8.1 28 Mar 19 19:21 UTC
v2.8.0 12 Mar 19 09:05 UTC
prometheus-config-reloader quay.io/coreos/prometheus-config-reloader
| master 07 Jun 19 12:19 UTC
| v0.30.1 31 May 19 20:47 UTC
| v0.30.0 10 May 19 20:42 UTC
'-> v0.29.0 20 Feb 19 11:11 UTC
v0.28.0 29 Jan 19 14:46 UTC
v0.27.0 08 Jan 19 16:25 UTC
v0.26.0 30 Nov 18 12:55 UTC
v0.25.0 24 Oct 18 13:25 UTC
v0.24.0 11 Oct 18 12:05 UTC
v0.23.2 23 Aug 18 13:36 UTC
rules-configmap-reloader quay.io/coreos/configmap-reload
| latest 02 Jan 17 23:21 UTC
'-> v0.0.1 02 Jan 17 23:21 UTC
Additional context
Add any other context about the problem here, e.g
- Flux version: 1.13.0
- Helm Operator version: 0.9.2
- Kubernetes version: 1.12
- Git provider: github
- Container registry provider: dockerhub / other
runningman84
All 12 comments
We need a (redacted) copy of your HelmRelease (including the values field) to confirm if this is a bug.
hiddeco
on 19 Jun 2019
Does this help?
$ kubectl describe helmrelease prometheus-operator -n monitoring
Name: prometheus-operator
Namespace: monitoring
Labels: flux.weave.works/sync-gc-mark=sha256.WmJXjMVidRbo08dV5CYwOf5Wa00UQECRwnK1FChHsRo
Annotations: flux.weave.works/automated: true
flux.weave.works/sync-checksum: c00e156ad15c82f240fc52be222147c4d58e42c3
flux.weave.works/tag.alertmanager: semver:~0.16
flux.weave.works/tag.grafana: semver:~6.2
flux.weave.works/tag.prometheus: semver:~2.10
kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"flux.weave.works/v1beta1","kind":"HelmRelease","metadata":{"annotations":{"flux.weave.works/automated":"true","flux.weave.w...
API Version: flux.weave.works/v1beta1
Kind: HelmRelease
Metadata:
Creation Timestamp: 2019-06-18T15:06:19Z
Generation: 4
Resource Version: 14404959
Self Link: /apis/flux.weave.works/v1beta1/namespaces/monitoring/helmreleases/prometheus-operator
UID: a0bbad80-91da-11e9-96c8-02a228bf1dc8
Spec:
Chart:
Name: prometheus-operator
Repository: https://kubernetes-charts.storage.googleapis.com/
Version: 5.10.5
Release Name: prometheus
Values:
Alertmanager:
Enabled: true
You need to specify the images within the .spec.values for Flux to be able to update the image refs.
hiddeco
on 19 Jun 2019
ah I thought the default values would be used from the chart... manually specifying them worked for me.
runningman84
on 19 Jun 2019
We recently added a note to the documentation about why this is required: https://github.com/weaveworks/flux/commit/1c1bf10c5db4e0c2a4aeed652300b58433d94ab5
hiddeco
on 19 Jun 2019
How does it work with the traefik chart, it uses the paramater imageTag: 1.7.12
What would be the correct annotation?
runningman84
on 19 Jun 2019
This does not work yet, I am working on a feature which makes it possible to configure YAML dot notation paths for such cases, but before I am able to land this the separation for registry keys (#2149) needs to be in place first.
hiddeco
on 19 Jun 2019
What about updating helm charts instead of images, are you going to implement that too? This would make updating helm applications quite easy.
runningman84
on 19 Jun 2019
This is already semi-supported by using charts from git sources, we have no plans to implement this for 'published' charts from repositories, as we can not guarantee those charts do not contain any breaking changes that would corrupt the (existing) release in a non-revertible way.
hiddeco
on 19 Jun 2019
yes I already use it for git based charts. I would be great to have a way to use this also for published charts. If you could configure only minor updates I think this would not be a big deal.
I also miss some info how to handle cases where only the latest tag is published. Is flux able to recognize that the image behind the latest tag changed? Which would be the correct annotation?
runningman84
on 19 Jun 2019
Flux does not support latest tag or any other mutable tag.
stefanprodan
on 20 Jun 2019
as we can not guarantee those charts do not contain any breaking changes that would corrupt the (existing) release in a non-revertible way
@hiddeco Just out of curiosity: The Charts pulled from git repos can contain arbitrary changes, and that’s (unintentionally?) supported? While automated updates from Helm repos are ruled out for that same reason?
Please don’t take this as backhanded criticism – I just genuinely don’t understand. Personally, I’d love to be able to automate Helm Chart upgrades.
tjanson
on 18 Sep 2019
Related issues
javefang
·
4Comments
alexhumphreys
·
4Comments
brantb
·
3Comments
mar1n3r0
·
4Comments
jstevans
·
4Comments
Most helpful comment
@hiddeco Just out of curiosity: The Charts pulled from git repos can contain arbitrary changes, and that’s (unintentionally?) supported? While automated updates from Helm repos are ruled out for that same reason?
Please don’t take this as backhanded criticism – I just genuinely don’t understand. Personally, I’d love to be able to automate Helm Chart upgrades.