Helm-operator: Attempts to update immutable field

[steven-sheehy]

Describe the bug

When attempting to deploy a v2 chart with the only changes being an image tag change and a replica count change, Helm Operator will fail with an error indicating it is updating an immutable field that I did not change. I don't even expose the service.clusterIP in the templated service.yaml, so it can't be me changing it.

ts=2020-05-15T05:36:50.809472317Z caller=release.go:287 component=release release=fluxcd targetNamespace=dev resource=dev:helmrelease/demo helmVersion=v3 error="upgrade failed: failed to replace object: Service \"fluxcd-fluxcd-demo\" is invalid: spec.clusterIP: Invalid value: \"\": field is immutable" action=upgrade

To Reproduce
flux.yaml

clusterRole:
  create: true
git:
  branch: deploy
  pollInterval: 1m
  readonly: true
  url: "https://github.com/steven-sheehy/fluxcd-demo.git"
memcached:
  enabled: false
registry:
  disableScanning: true
syncGarbageCollection:
  enabled: true
sync:
  interval: 1m

flux-helm.yaml

chartsSyncInterval: 30s
git:
  pollInterval: 1m
helm:
  versions: v3
logReleaseDiffs: true
updateChartDeps: false

Can fork my fluxcd-demo project. Steps to reproduce the behaviour:

1. helm upgrade -i --wait -n flux -f flux.yaml flux fluxcd/flux
2. helm upgrade -i --wait -n flux -f flux-helm.yaml flux-helm fluxcd/helm-operator
3. https://github.com/steven-sheehy/fluxcd-demo/blob/deploy/dev/helmrelease.yaml
4. Make multiple replica and image tag version. Usually takes 4-5+ commits/deploys to reproduce
5. Post the HelmRelease status, you can get this by running kubectl describe helmrelease <name>

Status:
  Conditions:
    Last Transition Time:  2020-05-15T05:36:50Z
    Last Update Time:      2020-05-15T05:36:50Z
    Message:               Release failed for Helm release 'fluxcd' in 'dev'.
    Reason:                Failed
    Status:                False
    Type:                  Released
    Last Transition Time:  2020-05-15T05:37:03Z
    Last Update Time:      2020-05-15T05:37:03Z
    Message:               Rolled back Helm release 'fluxcd' in 'dev'.
    Reason:                RolledBack
    Status:                True
    Type:                  RolledBack
    Last Transition Time:  2020-05-15T05:04:44Z
    Last Update Time:      2020-05-15T05:54:13Z
    Message:               Chart fetch was successful for Helm release 'fluxcd' in 'dev'.
    Reason:                ChartFetched
    Status:                True
    Type:                  ChartFetched
  Observed Generation:     6
  Phase:                   ChartFetched
  Release Name:            fluxcd
  Release Status:          deployed
  Revision:                80e7be5f400b34abcb90712d6d8dcd5c6739872f
  Rollback Count:          1
Events:
  Type     Reason             Age                  From           Message
  ----     ------             ----                 ----           -------
  Warning  FailedReleaseSync  51m                  helm-operator  synchronization of release 'fluxcd' in namespace 'dev' failed: failed to prepare chart for release: chart not ready: git repo has not been cloned yet
  Normal   ReleaseSynced      98s (x106 over 51m)  helm-operator  managed release 'fluxcd' in namespace 'dev' sychronized

Expected behavior

A clear and concise description of what you expected to happen.

Logs

ts=2020-05-15T05:36:50.132029474Z caller=release.go:75 component=release release=fluxcd targetNamespace=dev resource=dev:helmrelease/demo helmVersion=v3 info="starting sync run"
ts=2020-05-15T05:36:50.377899829Z caller=release.go:284 component=release release=fluxcd targetNamespace=dev resource=dev:helmrelease/demo helmVersion=v3 info="running upgrade" action=upgrade
ts=2020-05-15T05:36:50.414591543Z caller=helm.go:69 component=helm version=v3 info="preparing upgrade for fluxcd" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:36:50.419825334Z caller=helm.go:69 component=helm version=v3 info="resetting values to the chart's original version" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:36:50.621845929Z caller=helm.go:69 component=helm version=v3 info="performing update for fluxcd" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:36:50.636213764Z caller=helm.go:69 component=helm version=v3 info="creating upgraded release for fluxcd" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:36:50.651593131Z caller=helm.go:69 component=helm version=v3 info="checking 3 resources for changes" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:36:50.664771901Z caller=helm.go:69 component=helm version=v3 info="Replaced \"fluxcd-fluxcd-demo\" with kind ServiceAccount for kind ServiceAccount\n" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:36:50.681541213Z caller=helm.go:69 component=helm version=v3 info="error updating the resource \"fluxcd-fluxcd-demo\":\n\t failed to replace object: Service \"fluxcd-fluxcd-demo\" is invalid: spec.clusterIP: Invalid value: \"\": field is immutable" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:36:50.702025736Z caller=helm.go:69 component=helm version=v3 info="Replaced \"fluxcd-fluxcd-demo\" with kind Deployment for kind Deployment\n" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:36:50.72134769Z caller=helm.go:69 component=helm version=v3 info="warning: Upgrade \"fluxcd\" failed: failed to replace object: Service \"fluxcd-fluxcd-demo\" is invalid: spec.clusterIP: Invalid value: \"\": field is immutable" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:36:50.809472317Z caller=release.go:287 component=release release=fluxcd targetNamespace=dev resource=dev:helmrelease/demo helmVersion=v3 error="upgrade failed: failed to replace object: Service \"fluxcd-fluxcd-demo\" is invalid: spec.clusterIP: Invalid value: \"\": field is immutable" action=upgrade
ts=2020-05-15T05:36:50.841592677Z caller=release.go:309 component=release release=fluxcd targetNamespace=dev resource=dev:helmrelease/demo helmVersion=v3 info="running rollback" phase=rollback
ts=2020-05-15T05:36:50.879620772Z caller=helm.go:69 component=helm version=v3 info="preparing rollback of fluxcd" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:36:50.892585363Z caller=helm.go:69 component=helm version=v3 info="rolling back fluxcd (current: v9, target: v8)" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:36:50.896988576Z caller=helm.go:69 component=helm version=v3 info="creating rolled back release for fluxcd" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:36:50.910749008Z caller=helm.go:69 component=helm version=v3 info="performing rollback of fluxcd" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:36:50.924990246Z caller=helm.go:69 component=helm version=v3 info="checking 3 resources for changes" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:36:50.943824865Z caller=helm.go:69 component=helm version=v3 info="Looks like there are no changes for Service \"fluxcd-fluxcd-demo\"" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:36:50.968468787Z caller=helm.go:69 component=helm version=v3 info="beginning wait for 3 resources with timeout of 8m20s" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:36:52.980171456Z caller=helm.go:69 component=helm version=v3 info="Deployment is not ready: dev/fluxcd-fluxcd-demo. 1 out of 2 expected pods are ready" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:36:54.983047005Z caller=helm.go:69 component=helm version=v3 info="Deployment is not ready: dev/fluxcd-fluxcd-demo. 1 out of 2 expected pods are ready" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:36:56.978817041Z caller=helm.go:69 component=helm version=v3 info="Deployment is not ready: dev/fluxcd-fluxcd-demo. 1 out of 2 expected pods are ready" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:36:58.975585126Z caller=helm.go:69 component=helm version=v3 info="Deployment is not ready: dev/fluxcd-fluxcd-demo. 1 out of 2 expected pods are ready" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:37:00.975891295Z caller=helm.go:69 component=helm version=v3 info="Deployment is not ready: dev/fluxcd-fluxcd-demo. 1 out of 2 expected pods are ready" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:37:02.980576067Z caller=helm.go:69 component=helm version=v3 info="superseding previous deployment 8" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:37:02.990809452Z caller=helm.go:69 component=helm version=v3 info="updating status for rolled back release for fluxcd" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:37:03.041438977Z caller=release.go:315 component=release release=fluxcd targetNamespace=dev resource=dev:helmrelease/demo helmVersion=v3 info="rollback succeeded" phase=rollback
ts=2020-05-15T05:37:13.619763377Z caller=release.go:75 component=release release=fluxcd targetNamespace=dev resource=dev:helmrelease/demo helmVersion=v3 info="starting sync run"
ts=2020-05-15T05:37:13.619781935Z caller=release.go:75 component=release release=fluxcd targetNamespace=staging resource=staging:helmrelease/demo helmVersion=v3 info="starting sync run"
ts=2020-05-15T05:37:13.735800507Z caller=release.go:249 component=release release=fluxcd targetNamespace=staging resource=staging:helmrelease/demo helmVersion=v3 info="running dry-run upgrade to compare with release version '1'" action=dry-run-compare
ts=2020-05-15T05:37:13.738160142Z caller=helm.go:69 component=helm version=v3 info="preparing upgrade for fluxcd" targetNamespace=staging release=fluxcd
ts=2020-05-15T05:37:13.741820732Z caller=helm.go:69 component=helm version=v3 info="resetting values to the chart's original version" targetNamespace=staging release=fluxcd
ts=2020-05-15T05:37:13.750994349Z caller=helm.go:69 component=helm version=v3 info="getting history for release fluxcd" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:37:13.769913558Z caller=release.go:249 component=release release=fluxcd targetNamespace=dev resource=dev:helmrelease/demo helmVersion=v3 info="running dry-run upgrade to compare with release version '9'" action=dry-run-compare
ts=2020-05-15T05:37:13.773554912Z caller=helm.go:69 component=helm version=v3 info="preparing upgrade for fluxcd" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:37:13.779208095Z caller=helm.go:69 component=helm version=v3 info="resetting values to the chart's original version" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:37:13.952446725Z caller=helm.go:69 component=helm version=v3 info="performing update for fluxcd" targetNamespace=staging release=fluxcd
ts=2020-05-15T05:37:13.964656578Z caller=helm.go:69 component=helm version=v3 info="dry run for fluxcd" targetNamespace=staging release=fluxcd
ts=2020-05-15T05:37:13.977953827Z caller=helm.go:69 component=helm version=v3 info="performing update for fluxcd" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:37:13.987492588Z caller=helm.go:69 component=helm version=v3 info="dry run for fluxcd" targetNamespace=dev release=fluxcd
ts=2020-05-15T05:37:13.994357614Z caller=release.go:268 component=release release=fluxcd targetNamespace=staging resource=staging:helmrelease/demo helmVersion=v3 info="no changes" phase=dry-run-compare
ts=2020-05-15T05:37:14.016495527Z caller=release.go:268 component=release release=fluxcd targetNamespace=dev resource=dev:helmrelease/demo helmVersion=v3 info="no changes" phase=dry-run-compare

Additional context

• Helm Operator version: 1.0.1
• Kubernetes version: v1.18.0
• Git provider: GitHub
• Helm repository provider: GitHub

[steven-sheehy]

service.clusterIP is a special field that is generally empty in charts, but when applied to the cluster it gets filled in automatically. I'm guessing that later when Helm Operator does the diff between git state and live cluster state it notices that that field has been changed from empty in the chart to populated and tries to reconcile that? Unfortunately, that field can't be changed as it's immutable.

It's weird that it doesn't occur every time though. Maybe it has something to do with my use of resetValues=true? I've never had this issue with the helm upgrade command, so whatever mechanism they're using to keep this from happening could be utilized here.

[steven-sheehy]

It's also strange that logReleaseDiffs: true did not log the diff. I saw it log the diff for the first few releases after operator startup, then it inexplicably stopped showing up. It's output was a bit hard to read, being a collapsed diff of Go objects and not multi-line YAML.

[steven-sheehy]

This turned out to be due using the forceUpgrade: true and how Helm 3 changed its behavior for that flag. It no longer deletes and recreates, but attempts to do a replace which will fail for resources that have immutable fields. They're working on a new recreate flag, which would be good to add to Helm Operator when it becomes available.

[acarsercan]

We are hitting this with concourse helmchart + flux v1 helm-operator-1.2.0

@steven-sheehy can you clarify what the solution was?

Regards