Charts: [stable/external-dns] invalid value for spec.clusterIP on 0.5.3 upgrade

[cknowles]

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

Version of Helm and Kubernetes:
Helm 2.8.1 and k8s 1.9.6/1.7.3 (same error, different clusters)

Which chart:
external-dns

What happened:
Without changing any user defined values, upgrading from 0.5.2 to 0.5.3 fails with this when the service.clusterIP setting is not customised:

Error: UPGRADE FAILED: Service "external-dns" is invalid: spec.clusterIP: Invalid value: "": field is immutable

What you expected to happen:
Upgrade from 0.5.2 to 0.5.3 to succeed when values set by the user have not changed.

How to reproduce it (as minimally and precisely as possible):

1. Install external-dns chart 0.5.2 without customising service.clusterIP
2. Try to upgrade to 0.5.3 with the same options, upgrade fails

Anything else we need to know:
v0.5.3 changed this in https://github.com/kubernetes/charts/pull/4377. I think that PR might resolve going from a set value back to default (empty string) but something might be validating a different situation of empty string to not set?

Workaround:
Pin version to 0.5.2 for now.

[cknowles]

@hobti01 for now I pinned the chart version but if you have any ideas we could resolve this slightly different situation we could fix the chart for the next version. I suspect the solution might involve changing the default, was just checking what other charts do.

[hobti01]

Very strange. I used the same method as several other stable charts, but I see the same issue. Many different approaches have been used with zero issues reported.

For example, using the same approach as the stable grafana chart fails.

It's worth noting that with the value unset in values.yaml this succeeds:

clusterIP: "{{ .Values.service.clusterIP }}"

While this fails:

clusterIP: {{ .Values.service.clusterIP | quote }}

And this succeeds:

clusterIP: {{ .Values.service.clusterIP | default "" | quote }}

It should also be noted that --dry-run succeeds but actual deployment fails:

# grep clusterIP external-dns-0.5.3/templates/service.yaml
  clusterIP: {{ .Values.service.clusterIP | quote }}
# helm upgrade dns ./external-dns-0.5.3 --dry-run
Release "dns" has been upgraded. Happy Helming!
...
# helm upgrade dns ./external-dns-0.5.3
Error: UPGRADE FAILED: Service "dns-external-dns" is invalid: spec.clusterIP: Invalid value: "": field is immutable

I cannot find where the fault lies or how to reproduce the error from #4377.

[cknowles]

Yeah it's definitely odd, I wonder which is the best option to go with. I suspect dry run might be using the kubectl style validation which doesn't validate absolutely everything, just a guess though.

On the notes about | quote, the middle one is probably expected to fail given the templating code.

[monotek]

I have the same issue with chart version 0.6.0

[monotek]

And 0.6.1

[hobti01]

We continue to have this issue as well. We cannot update external-dns and must purge and re-deploy.

[cknowles]

Getting this again upgrading from chart v0.5.2 to v1.0.2. I do not have anything set for service.clusterIP, k8s v1.11.4 and Helm 2.9.1.

[cknowles]

Tried with Helm 2.11.0 but same result. A rollback also fails with the same error now it's in that state. The only way to get out of that situation appears to be running helm upgrade with the --force flag.

[cknowles]

Also weirdly after successfully upgrading with --force, running the exact same upgrade command again but without the force fails with the spec.clusterIP error.

That led me to think perhaps the Tiller state was somewhat inconsistent or causing the failure. So I deleted the install with helm delete external-dns-master --purge and then re-installed. Now the same upgrade command as before consistently succeeds. So it seems that theory about the Tiller store state affecting this was accurate.

Summary - if you are having this problem I recommend to delete/purge this chart install and reinstall.