Helm-operator: Pods not being recreated after ConfigMap/Secret checksum change

[NidhalRouissi]

Describe the bug

Helm checksum annotation doesn't seem to be working when using flux & helm-operator. In a HelmRelease, when values that are linked to a ConfigMap/Secret change, the pod is not recreated even though the checksum has changed.

To Reproduce

Steps to reproduce the behaviour:

1. Install a Flux & Helm Operator (Configure Flux with a git repo)
2. In the git repo, create a HelmRelease to install a helm chart (the chart must have a checksum annotation)
3. Make a change to a ConfigMap/Secret value through the HelmRelease values, and synchronize Flux

Expected behavior

The pod should be recreated after the change of the checksum value in the deployment resource.

Logs

ts=2020-04-06T15:32:42.221995545Z caller=release.go:342 component=release release=mysql-server targetNamespace=mysql-server resource=mysql-server:helmrelease/mysql-server helmVersion=v2 info="performing dry-run upgrade to see if release has diverged"
ts=2020-04-06T15:32:42.595443923Z caller=release.go:377 component=release release=mysql-server targetNamespace=mysql-server resource=mysql-server:helmrelease/mysql-server helmVersion=v2 info="no changes" action=skip

Additional context

• Helm Operator version: 1.0.0-rc8
• Flux version: 1.17.1
• Helm version: v2.14.3
• Kubernetes version: v1.14.1

[gitsto]

I really need this in my project

[hiddeco]

Does this only happen for charts sourced from a Git source, or for any .spec.values change to a HelmRelease making use of a Git or Helm repository source?

[NidhalRouissi]

Does this only happen for charts sourced from a Git source, or for any .spec.values change to a HelmRelease making use of a Git or Helm repository source?

None actually ! Our git repository holds only HelmRelease files and Namespaces. Our charts are hosted on a Helm Repository

Here is an example :

mysql_hr.yaml (HelmRelease, hosted on git)

apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: mysql-server
  namespace: mysql-server
  annotations:
    fluxcd.io/automated: "false"
spec:
  releaseName: mysql-server
  forceUpgrade: true
  chart:
    repository: https://[HELM_REPO_URL]/
    name: mysql-server
    version: 1.2.20200330051700
  values:
    mysqlRootPassword: FooBarPwd
    mysqlPassword: FooBarPwd

values.yaml

...
mysqlRootPassword: ""
mysqlPassword: ""
...

secret.yaml

{{- if not .Values.existingSecret }}
apiVersion: v1
kind: Secret
metadata:
  name: {{ template "mysql.fullname" . }}
  namespace: {{ .Release.Namespace }}
  labels:
    app: {{ template "mysql.fullname" . }}
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
type: Opaque
data:
  {{ if .Values.mysqlRootPassword }}
  mysql-root-password:  {{ .Values.mysqlRootPassword | b64enc | quote }}
  {{ else }}
  mysql-root-password: {{ randAlphaNum 10 | b64enc | quote }}
  {{ end }}
  {{ if .Values.mysqlPassword }}
  mysql-password:  {{ .Values.mysqlPassword | b64enc | quote }}
  {{ else }}
  mysql-password: {{ randAlphaNum 10 | b64enc | quote }}
  {{ end }}
{{- if .Values.ssl.enabled }}
{{ if .Values.ssl.certificates }}
{{- range .Values.ssl.certificates }}

deployement.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: {{.Release.Name }}-mysql-server
  namespace: {{ .Release.Namespace }}
  labels:
    app: {{ template "mysql.fullname" . }}
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
  annotations:
    checksum/config1: {{ include (print $.Template.BasePath "/configurationFiles-configmap.yaml") . | sha256sum }}
    checksum/config2: {{ include (print $.Template.BasePath "/docker-config.yaml") . | sha256sum }}
    checksum/config3: {{ include (print $.Template.BasePath "/initializationFiles-configmap.yaml") . | sha256sum }}
    checksum/config4: {{ include (print $.Template.BasePath "/secret.yaml") . | sha256sum }}
spec:
...

In this example, if if change the value of one of the passwords in the helm release, the checksum of the secret should change and force the pod de be recreated. But it's like the helm operator doesn't even perform an upgrade.

Any ideas ?

[hiddeco]

@NidhalRouissi are you by any chance aware of a lightweight publicly available Helm chart with a likewise structure that I could use to replicate this?

[NidhalRouissi]

@NidhalRouissi are you by any chance aware of a lightweight publicly available Helm chart with a likewise structure that I could use to replicate this?

@hiddeco sorry for this long delay.

I could find a simple lightweight helm chart indeed. Unbound, a caching DNS resolver.

This is how to reproduce:

Let's suppose you'll install the chart via helm:
helm upgrade --install unbound --version 1.1.2 stable/unbound --namespace unbound

Then you check that the pod is running
kubectl -n unbound get pods

If you run an upgrade with the same parameters nothing will change, the pods will keep running, but if you change a certain value :
helm upgrade --install unbound --version 1.1.2 stable/unbound --namespace unbound --set unbound.numThreads=2

This will trigger a pod recreation because that value is linked to a configmap, and there is a checksum on that configmap itself in the deployment manifest
https://github.com/helm/charts/blob/master/stable/unbound/templates/deployment.yaml#L28-L29

annotations:
  checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}

But, if you change that same value directly via kubectl for example, nothing will happen:
kubectl -n unbound edit cm unbound-unbound

I am assuming that HelmOperator is not running a "helm upgrade" command whenever i change a value in my HelmRelease manifest, this is why my pods are not recreated. If it's the case, HelmOperator is breaking a natively supported helm feature i guess..

[NidhalRouissi]

@hiddeco were you able to reproduce the issue ? Please let me know if you need more details

[kammous]

@hiddeco @stefanprodan
Could you please give us an update about this issue? Were you able to reproduce it? Any idea about root cause or a workaround waiting a fix?
Thanks a lot

[erpel]

Not sure if this is the point here, but putting the annotation on the deployment itself (instead of in the podSpec template of the deployment) might not cause pods to be recreated.
The proper place for the checksum annotations can be seen here: https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments

Might be completely missing the point but it caught my eye when browsing past...

[kammous]

I agree with @erpel. I put the example provided by @NidhalRouissi into a HelmRelease file, configured fluxcd and helm-operator to synchronize a git repo but wasn't able to reproduce the problem. It works as expected, meaning, each time a ConfigMap changes, pods are restarted.
@hiddeco @stefanprodan, this issue can be closed. Thanks