Charts: [stable/nginx-ingress] Cannot upgrade release with Helm 3

[MartinodF]

Describe the bug
The dreaded "spec.clusterIP: Invalid value" error (fixed in #13646) is back because of Helm 3.
Once the chart has been installed, it is no longer possible to upgrade it regardless of whether omitClusterIP has been set.

Version of Helm and Kubernetes:

• version.BuildInfo{Version:"v3.0.0", GitCommit:"e29ce2a54e96cd02ccfce88bee4f58bb6e2a28b6", GitTreeState:"clean", GoVersion:"go1.13.4"}
• Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.8-gke.12", GitCommit:"188432a69210ca32cafded81b4dd1c063720cac0", GitTreeState:"clean", BuildDate:"2019-11-07T19:27:01Z", GoVersion:"go1.12.11b4", Compiler:"gc", Platform:"linux/amd64"}

Which chart:
stable/nginx-ingress

What happened:

Error: UPGRADE FAILED: cannot patch "nginx-nginx-ingress-controller-metrics" with kind Service: Service "nginx-nginx-ingress-controller-metrics" is invalid: spec.clusterIP: Invalid value: "": field is immutable && cannot patch "nginx-nginx-ingress-controller" with kind Service: Service "nginx-nginx-ingress-controller" is invalid: spec.clusterIP: Invalid value: "": field is immutable && cannot patch "nginx-nginx-ingress-default-backend" with kind Service: Service "nginx-nginx-ingress-default-backend" is invalid: spec.clusterIP: Invalid value: "": field is immutable

What you expected to happen:
I would expect the release to be upgradable after the first deploy.

How to reproduce it (as minimally and precisely as possible):
Our values.yml:

controller:
  config:
    hsts-include-subdomains: 'false'
    http-redirect-code: '301'
    proxy-body-size: 512m
    server-name-hash-bucket-size: '256'
  controller:
    minAvailable: 1
    replicaCount: 3
  publishService:
    enabled: true
    pathOverride: 'nginx/nginx-nginx-ingress-controller'
  extraArgs:
    annotations-prefix: ingress.kubernetes.io
  service:
    externalTrafficPolicy: Local
    loadBalancerIP: XXX.XXX.XXX.XXX
    healthCheckNodePort: 32619
    omitClusterIP: true
  metrics:
    enabled: true
    service:
      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/port: 10254
tcp:
  2222: 'storage/sftp-app:2222'

# helm -n nginx install nginx stable/nginx-ingress --values values.yml --wait
NAME: nginx
LAST DEPLOYED: Thu Nov 21 13:04:31 2019
NAMESPACE: nginx
STATUS: deployed
REVISION: 1
TEST SUITE: None
# helm -n nginx upgrade nginx stable/nginx-ingress --values values.yml --wait
Error: UPGRADE FAILED: cannot patch "nginx-nginx-ingress-controller-metrics" with kind Service: Service "nginx-nginx-ingress-controller-metrics" is invalid: spec.clusterIP: Invalid value: "": field is immutable && cannot patch "nginx-nginx-ingress-controller" with kind Service: Service "nginx-nginx-ingress-controller" is invalid: spec.clusterIP: Invalid value: "": field is immutable && cannot patch "nginx-nginx-ingress-default-backend" with kind Service: Service "nginx-nginx-ingress-default-backend" is invalid: spec.clusterIP: Invalid value: "": field is immutable

Anything else we need to know:
There have already been a few reports in the previous pull request: #13646

[deini]

Some more context: https://github.com/helm/helm/issues/6378

[naseemkullah]

Same thing happened to me, migrated releases to helm3 and now the clusterIp error is back in full force, and it's too late for omitclusterIP ... @ChiefAlexander @taharah can we completely remove setting of clusterIP in services (and consequently omitclusterIP which was a commendable workaround for helm2)? Does anybody actually need this?

[naseemkullah]

https://github.com/helm/helm/issues/6378#issuecomment-557746499

[naseemkullah]

Sorry, this is not fixed. Should still be open.

I guess there are two solutions:

1) Uninstall and re install nginx ingress (there will be a brief downtime), obviously no clusterIP should be in the values file.

2) Hardcode the clusterIPs in your values file, until you find an opportunity to reinstall, remove the hardcoded clusterIPs in values file before redeploying though

[davidham]

This is indeed still broken. I went to GREAT LENGTHS to upgrade my chart from 1.3.1 to 1.29.6, and the only way I was able to do it was to hardcode the clusterIPs into my values.yaml. But now, when I try to remove the clusterIP values, I still get the error. Setting omitClusterIP has no effect. How can I remove these settings from my values.yaml? Open to any kind of workaround.

[alex-sainer]

same problem here and hardcoding the clusterIP shouldn't be the real solution...
any other workarounds out there?

[cforce]

same here

[FrankGiesecke]

The problem seems to be still there...

Error: UPGRADE FAILED: an error occurred while rolling back the release. original upgrade error: cannot patch "ingress-nginx-ingress-controller" with kind Service: Service "ingress-nginx-ingress-controller" is invalid: spec.clusterIP: Invalid value: "": field is immutable && cannot patch "ingress-nginx-ingress-default-backend" with kind Service: Service "ingress-nginx-ingress-default-backend" is invalid: spec.clusterIP: Invalid value: "": field is immutable: cannot patch "ingress-nginx-ingress-controller" with kind Service: Service "ingress-nginx-ingress-controller" is invalid: spec.clusterIP: Invalid value: "": field is immutable && cannot patch "ingress-nginx-ingress-default-backend" with kind Service: Service "ingress-nginx-ingress-default-backend" is invalid: spec.clusterIP: Invalid value: "": field is immutable

I've tried the following commands:

helm upgrade \
      ingress \
      nginx-ingress \
      --install \
      --atomic \
      --namespace=ingress \
      --repo https://kubernetes-charts.storage.googleapis.com/ \
      --set rbac.create=true,controller.service.omitClusterIP=true,defaultBackend.service.omitClusterIP=true,controller.admissionWebhooks.service.omitClusterIP=true \
      --version 0.30.0

helm upgrade \
      ingress \
      nginx-ingress \
      --install \
      --atomic \
      --namespace=ingress \
      --repo https://kubernetes-charts.storage.googleapis.com/ \
      --set rbac.create=true,controller.service.omitClusterIP=true,defaultBackend.service.omitClusterIP=true,controller.admissionWebhooks.service.omitClusterIP=true \
      --reuse-values \
      --version 0.30.0

helm upgrade \
      ingress \
      nginx-ingress \
      --install \
      --atomic \
      --namespace=ingress \
      --repo https://kubernetes-charts.storage.googleapis.com/ \
      --set rbac.create=true \
      --reuse-values \
      --version 0.30.0

[haghabozorgi]

can we have this issue re-opened. As other have noted, this is still an issue.

[rajivreddy]

We have similar issue

cannot patch "nginx-ingress-controller" with kind Service: Service "nginx-ingress-controller" is invalid: spec.clusterIP: Invalid value: "": field is immutable 
cannot patch "nginx-ingress-default-backend" with kind Service: Service "nginx-ingress-default-backend" is invalid: spec.clusterIP: Invalid value: "": field is immutable

[kluevandrew]

Same issue for me, and where is nothing about ingress in my diff

in ./helmfile.yaml: in .helmfiles[3]: in helmfiles/70-service-layer-apps.yaml: failed processing release nginx-ingress: helm exited with status 1: Error: UPGRADE FAILED: cannot patch "nginx-ingress-controller-metrics" with kind Service: Service "nginx-ingress-controller-metrics" is invalid: spec.clusterIP: Invalid value: "": field is immutable && cannot patch "nginx-ingress-controller" with kind Service: Service "nginx-ingress-controller" is invalid: spec.clusterIP: Invalid value: "": field is immutable && cannot patch "nginx-ingress-default-backend" with kind Service: Service "nginx-ingress-default-backend" is invalid: spec.clusterIP: Invalid value: "": field is immutable ERROR: Job failed: exit code 1