Charts: [stable/prometheus-operator] Failed to upgrade to 8.10.0 due to immutable field changed

[dogzzdogzz]

Describe the bug
Failed to upgrade to 8.10.0 due to immutable field changed

Version of Helm and Kubernetes:
helm v3.1.1
k8s: 1.14

What happened:
When upgrade release from 8.7.0 to 8.10.0, got below error due to selector field changed

# helm upgrade --installl prometheus-operator .
Error: UPGRADE FAILED: cannot patch "prometheus-operator-grafana" with kind Deployment: Deployment.apps "prometheus-operator-grafana" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"prometheus-operator", "app.kubernetes.io/name":"grafana"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

[onedr0p]

I am also getting this error. I had to rollback to 8.9.2

[helm-operator-77cb687cc7-nn4pz] ts=2020-03-10T21:40:21.248468381Z caller=helm.go:69 component=helm version=v3 info="error updating the resource \"prometheus-operator-grafana\":\n\t cannot patch \"prometheus-operator-grafana\" with kind Deployment: Deployment.apps \"prometheus-operator-grafana\" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{\"app.kubernetes.io/instance\":\"prometheus-operator\", \"app.kubernetes.io/name\":\"grafana\"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable" targetNamespace=monitoring release=prometheus-operator
[helm-operator-77cb687cc7-nn4pz] ts=2020-03-10T21:40:21.32830952Z caller=helm.go:69 component=helm version=v3 info="Created a new Ingress called \"prometheus-operator-grafana\" in monitoring\n" targetNamespace=monitoring release=prometheus-operator
[helm-operator-77cb687cc7-nn4pz] ts=2020-03-10T21:40:23.968396896Z caller=helm.go:69 component=helm version=v3 info="warning: Upgrade \"prometheus-operator\" failed: cannot patch \"prometheus-operator-grafana\" with kind Deployment: Deployment.apps \"prometheus-operator-grafana\" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{\"app.kubernetes.io/instance\":\"prometheus-operator\", \"app.kubernetes.io/name\":\"grafana\"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable" targetNamespace=monitoring release=prometheus-operator
[helm-operator-77cb687cc7-nn4pz] ts=2020-03-10T21:40:24.572927983Z caller=release.go:216 component=release release=prometheus-operator targetNamespace=monitoring resource=monitoring:helmrelease/prometheus-operator helmVersion=v3 error="Helm release failed" revision=8.12.1 err="failed to upgrade chart for release [prometheus-operator]: cannot patch \"prometheus-operator-grafana\" with kind Deployment: Deployment.apps \"prometheus-operator-grafana\" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{\"app.kubernetes.io/instance\":\"prometheus-operator\", \"app.kubernetes.io/name\":\"grafana\"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable"

[onedr0p]

I was able to upgrade after doing the following:

# get latest successful deployed revision number
helm history prometheus-operator -n monitoring
# rollback to that revision number
helm rollback prometheus-operator <revision number> -n monitoring

# delete grafana deployment and ingress
kubectl delete deployment.apps/prometheus-operator-grafana -n monitoring
kubectl delete ingress/prometheus-operator-grafana -n monitoring

Now you should be able to update with your helm command

[stefanandres]

This also affects the grafana pvc

$  helm upgrade --install prom --namespace monitoring -f values.yaml stable/prometheus-operator --version 8.12.1
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
Error: UPGRADE FAILED: failed to replace object: PersistentVolumeClaim "prom-grafana" is invalid: spec: Forbidden: is immutable after creation except resources.requests for bound claims && failed to replace object: Service "prom-grafana" is invalid: spec.clusterIP: Invalid value: "": field is immutable && failed to replace object: Deployment.apps "prom-grafana" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"prom", "app.kubernetes.io/name":"grafana"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

This I would not want to delete and recreate.

[kam1kaze]

@stefanandres it's due to #17806 PR. You can just set persistence.storageClassName with default value in HELM values to bypass the error.

[stefanandres]

@kam1kaze I already have grafana.persitence.storegeClassName set to workaround the #17806 issue. This here seems to be another issue. The grafana chart uses new label and helm tries to patch them which it not allowed because the labelSelector are immutable on the PVC object.

[stefanandres]

I've now used a hacky workaround for this issue to upgrade:

# Backup old pvc data to local
# Change pod name
kubectl cp -c grafana prom-grafana-65977b8676-hxc74:/var/lib/grafana/ grafana

# Delete immutable parent objects
$ kubectl delete deployments.apps  prom-grafana
kudeployment.apps "prom-grafana" deleted
$ kubectl delete ingress prom-grafana
kubeingress.extensions "prom-grafana" deleted
$ kubectl delete pvc prom-grafana
persistentvolumeclaim "prom-grafana" deleted

$ helm upgrade --install prom --namespace monitoring -f values.yaml stable/prometheus-operator --version 8.12.2

# Restore grafana pvc
# Change pod name
kubectl cp grafana/ -c grafana prom-grafana-588f59668b-p9rrq:/var/lib/

# Restart grafana to read in restored data
kubectl rollout  restart prom-grafana

[Arabus]

Just ran into the same issue.
I would very much like this to be resolved in a proper manner since this is a breaking change. Forcing user to delete/restore their stateful data for an upgrade is not something I'd like my users to encounter in a minor release.

[iMacX]

Same error here, trying to update from 8.9.2 to 8.10.0.

I don't want to play with those hacky workaround, official fix is more than welcome. Thank you

[bpehling]

In the Grafana chart, the workaround is using the --force flag when doing the helm upgrade to force the label change https://github.com/helm/charts/tree/master/stable/grafana#to-500
I haven't tried the workaround yet though

[Arabus]

Unfortunately --force appears to be just a wrapper for kubectl replace --force i.e., delete and recreate. Which is the same as the workaround above does.

[bpehling]

@Arabus I just tried the --force flag and the pvc was not deleted and recreated. If it were, wouldn't I see the Age reset?

[bpehling]

The labels were updated as well

[limod]

Hello!
I have the prometheus-operator in my helmfile.yaml and tried to upgrade from 8.9.3 -> 8.10.0
For the prometheus-operator release i set the optionforce: true, but i still have problems with the PersistentVolumeClaim:

in k8s/helmfile.yaml: failed processing release prometheus: helm exited with status 1:
  manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
  manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
  manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
  manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
  manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
  manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
  Error: UPGRADE FAILED: failed to replace object: PersistentVolumeClaim "prometheus-grafana" is invalid: spec: Forbidden: is immutable after creation except resources.requests for bound claims && failed to replace object: Service "prometheus-grafana" is invalid: spec.clusterIP: Invalid value: "": field is immutable && failed to replace object: Service "prometheus-kube-state-metrics" is invalid: spec.clusterIP: Invalid value: "": field is immutable && failed to replace object: Service "prometheus-prometheus-node-exporter" is invalid: spec.clusterIP: Invalid value: "": field is immutable && failed to replace object: Service "prometheus-prometheus-oper-alertmanager" is invalid: spec.clusterIP: Invalid value: "": field is immutable && failed to replace object: Service "prometheus-prometheus-oper-operator" is invalid: spec.clusterIP: Invalid value: "": field is immutable && failed to replace object: Service "prometheus-prometheus-oper-prometheus" is invalid: spec.clusterIP: Invalid value: "": field is immutable && failed to replace object: Deployment.apps "prometheus-grafana" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"prometheus", "app.kubernetes.io/name":"grafana"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

[stefanandres]

--force does not seem to do the trick with helm-3. The only workaround I found is: https://github.com/helm/charts/issues/21200#issuecomment-600023504

[infa-kparida]

Any work around ? or The hack is the only way out ?

[Morriz]

Please look at helm 3.2.0's new adoption method, it might be of help: https://github.com/helm/helm/issues/7697#issuecomment-613535044

[alexises]

Hello,
I was ennoyed with the same issue to upgrade from 7.4 to 8.13.8 as a workaround I had to do :

kubectl delete role prometheus-operator-alertmanager -n monitoring
kubectl delete clusterrole prometheus-operator-alertmanager
kubectl delete clusterrolebindings prometheus-operator-alertmanager -n monitoring
kubectl delete ingress/prometheus-operator-grafana -n monitoring
kubectl delete deployment.apps/prometheus-operator-grafana -n monitoring
kubectl delete rolebindings -n monitoring prometheus-operator-alertmanager
kubectl delete prometheusrules -n monitoring prometheus-operator-kubernetes-absent
kubectl delete prometheusrules -n monitoring prometheus-operator-alertmanager.rules
kubectl delete prometheusrules -n monitoring prometheus-operator-etcd
kubectl delete prometheusrules -n monitoring prometheus-operator-general.rules
kubectl delete prometheusrules -n monitoring prometheus-operator-k8s.rules
kubectl delete prometheusrules -n monitoring prometheus-operator-kube-apiserver-slos
kubectl delete prometheusrules -n monitoring prometheus-operator-kube-apiserver.rules
kubectl delete prometheusrules -n monitoring prometheus-operator-kube-prometheus-general.rules
kubectl delete prometheusrules -n monitoring prometheus-operator-kube-prometheus-node-recording.rules
kubectl delete prometheusrules -n monitoring prometheus-operator-kube-scheduler.rules
kubectl delete prometheusrules -n monitoring prometheus-operator-kube-state-metrics
kubectl delete prometheusrules -n monitoring prometheus-operator-kubelet.rules
kubectl delete prometheusrules -n monitoring prometheus-operator-kubernetes-apps
kubectl delete prometheusrules -n monitoring prometheus-operator-kubernetes-resources
kubectl delete prometheusrules -n monitoring prometheus-operator-kubernetes-storage
kubectl delete prometheusrules -n monitoring prometheus-operator-kubernetes-system
kubectl delete prometheusrules -n monitoring prometheus-operator-kubernetes-system-apiserver
kubectl delete prometheusrules -n monitoring prometheus-operator-kubernetes-system-controller-manager
kubectl delete prometheusrules -n monitoring prometheus-operator-kubernetes-system-kubelet
kubectl delete prometheusrules -n monitoring prometheus-operator-kubernetes-system-scheduler
kubectl delete prometheusrules -n monitoring prometheus-operator-node-exporter
kubectl delete prometheusrules -n monitoring prometheus-operator-node-exporter.rules
kubectl delete prometheusrules -n monitoring prometheus-operator-node-network
kubectl delete prometheusrules -n monitoring prometheus-operator-node.rules
kubectl delete prometheusrules -n monitoring prometheus-operator-prometheus
kubectl delete prometheusrules -n monitoring prometheus-operator-prometheus-operator

following with a rollback, another row of previous command and an upgrade.

please note by sometime the rollback can fail, in this case, just relaunch it and it.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[jwalton]

This is still an issue.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.

[bugok]

Just hit this as well.

[saixiaohui]

Just hit this as well.

[morremeyer]

@saixiaohui This won’t be fixed in this chart as helm/stable is not being maintained any more.

You can use the hack by @stefanandres to update to a newer version, but you should also switch to kube-prometheus-stack if you can.