Ingress-nginx: After upgrade from 0.24.1 to 0.31 ... remote error: tls: bad certificate

[sadortun]

NGINX Ingress controller version:
0.31
Kubernetes version (use kubectl version):
Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.11-gke.9", GitCommit:"e1af17fd873e15a48769e2c7b9851405f89e3d0d", GitTreeState:"clean", BuildDate:"2020-04-06T20:56:54Z", GoVersion:"go1.12.17b4", Compiler:"gc", Platform:"linux/amd64"}
Environment:
GKE
Linux cs-6000-devshell-vm-601a26fc-e33d-4c40-a844-c24cbbc51f18 4.19.112+ #1 SMP Sat Apr 4 00:12:45 PDT 2020 x86_64 GNU/Linux

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What happened:
We upgraded from 0.24.1 to 0.31 and we are no longer able to create ingresses

kubectl apply -f tmp/apps-ingress.yml
Error from server (InternalError): error when creating "tmp/apps-ingress.yml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s: x509: certificate signed by unknown authority

In the logs, we see

http: TLS handshake error from 10.162.0.60:35350: remote error: tls: bad certificate

What you expected to happen:

How to reproduce it:

No idea ! 😢

Anything else we need to know:

/kind bug

[manobi]

I'm having the same error after de upgrade and the loadbalancer from cloud provider (Oracle cloud) is with status critical.

[sadortun]

Little update, i ended up deleting everything in the ingress-nginx namespace, incl webhooks, and re-install from fresh using helm.

Unfortunatly, the same issue is still there.

[aledbf]

@sadortun I cannot reproduce the issue

# Use 1.15
kind create cluster --image=kindest/node:v1.15.7

# create a namespace
k create ns ing

# Add chart repository
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install my-release ingress-nginx/ingress-nginx \
  --namespace ing \
  --set controller.service.type=NodePort

# Install example app
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/docs/examples/http-svc.yaml

echo "
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
      - path: /
        backend:
          serviceName: http-svc
          servicePort: 80
" | kubectl apply -f -

How are you installing the ingress controller?

[sadortun]

Hi @aledbf Have you tried to install first version 0.24.1, then upgrade it to latest ? I am sure there is no issues with only installing latest release.

Yes, i installed/re-installed the ingress controller.

[sadortun]

Maybe related to https://github.com/kubernetes/ingress-nginx/issues/5401

[aledbf]

Maybe related to #5401

That issue is fixed in 0.31.1.

[sadortun]

New release fixed the issue.

Thanks,
Samuel