Ingress-nginx: Regression in 0.27: Ingress address is deleted

[Neki]

NGINX Ingress controller version: 0.27.0

Kubernetes version (use kubectl version): v1.13.12-eks-eb1860

Environment:

• Cloud provider or hardware configuration: AWS EKS
• Others: I'm using the helm chart at https://github.com/helm/charts/tree/master/stable/nginx-ingress, version 1.29.1.

I use an AWS ELB for SSL termination, with proxy protocol enabled.

$ kubectl get svc nginx-ingress-controller
NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP                                         PORT(S)                      AGE
nginx-ingress-controller   LoadBalancer   10.100.253.7   <some random name>.eu-central-1.elb.amazonaws.com   80:31708/TCP,443:31589/TCP   215d

What happened:

When upgrading ingress-nginx from 0.26.2 to 0.27.0, all the ingresses lost their address (in the output of kubectl get ing). The nginx-ingress-controller service external IP did not change.

In other words, in the output of kubectl get ingress -o yaml, this block:

  status:
    loadBalancer:
      ingress:
      - hostname: <some random name>.eu-central-1.elb.amazonaws.com

became

  status:
    loadBalancer: {}

This also applied to new ingresses, created after the upgrade.

A rollback to 0.26.2 repopulated the addresses for all ingresses.

The controller arguments are

    Args:
      /nginx-ingress-controller
      --default-backend-service=ingress-nginx/nginx-ingress-default-backend
      --publish-service=ingress-nginx/nginx-ingress-controller
      --election-id=ingress-controller-leader
      --ingress-class=nginx
      --configmap=ingress-nginx/nginx-ingress-controller

I don't see any (relevant) error message in the ingress controller logs, even when increasing the verbosity to --v=3, nor in the Kubernetes apiserver logs. I haven't changed anything in my environment except from the controller image.

What you expected to happen:

The existing ingress addresses should not have been modified.
When creating an ingress, its address should be correctly populated.

How to reproduce it:

• Deploy a working ingress controller on version 0.26.2 on a AWS EKS cluster, using the Helm chart linked above, with the following values

controller:
  config:
    http-snippet: |
      map true $pass_access_scheme {
        default "https";
      }
      map true $pass_port {
        default 443;nginx-ingress-controller
      }

      server {
        listen 8080 proxy_protocol;
        return 308 https://$host$request_uri;
      }
    proxy-real-ip-cidr: 0.0.0.0/0
    server-tokens: "false"
    ssl-redirect: "false"
    use-forwarded-headers: "true"
    use-proxy-protocol: "true"
  image:
    tag: 0.26.2
  kind: DaemonSet
  metrics:
    enabled: true
    serviceMonitor:
      enabled: true
  publishService:
    enabled: true
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
      service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
      service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <arn of a AWS ACM ssl certificate>
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
    externalTrafficPolicy: Cluster
    targetPorts:
      http: 8080
      https: http
  updateStrategy:
    type: RollingUpdate
defaultBackend:
  replicaCount: 2
podSecurityPolicy:
  enabled: true

You'll need to manually kubectl edit ds nginx-ingress-controller to change runAsUser: 101 to runAsUser: 33 because of the switch to alpine.
Create an ingress, check that kubectl get ing list an address.

• Update the image in these values to 0.27.0 (don't change the chart version), check that kubectl get ing no longer gives an address.

I'll try to work on a smaller reproduction, but I figured I could make an early report.

Anything else we need to know:

Please let me know if you need more information (such as log extracts). Thanks!

/kind bug

[aledbf]

Closing. Please check https://kubernetes.github.io/ingress-nginx/user-guide/cli-arguments/
(--update-status-on-shutdown)

[Neki]

Thanks a lot for this very quick answer! It did solve my issue, although I don't understand why from the documentation (new ingresses created after the controller upgrade did not get an address either).

[Neki]

Actually I spoke too soon. With this flag set to false, existing ingresses do not lose their addresses, but new ingresses don't get an address.

Edit: since I don't find any other reference to such an issue, I suspect something must be off with the RBAC rules. I'll investigate further, thanks for your help.

Edit 2: I haven't found the root cause, but upgrading the EKS cluster to 1.14 solved the issue.

[abayer]

We’ve seen the same problem with new ingresses not getting addresses on GKE with k8s 1.13 - is 1.14 now the minimum required version?

[bas2k]

We've faced the same problem. And looks like it isn't related to --update-status-on-shutdown flag.
I think this issue should be re-opened.

[catalinpan]

I'm getting the same issue in kubernetes 1.12 after ingress upgrade. I don't see any breaking changes mentioned in the release 0.27.0 but I see this PR which seems to be merged since 0.27.0.
https://github.com/kubernetes/ingress-nginx/pull/4827/commits/0dce5be743ae012dc5b22c1cf4b91dcdb8b64d88#diff-faf7dbbe290f73987c249a9c5dc15bcc

The ingress apiVersion has been changed to support only networking.k8s.io/.... which is available only from kubernetes v1.14.

@aledbf can we have please some help with this issue? Or if this was planned will be nice if the documentation and the release notes will be updated.

[aledbf]

The ingress apiVersion has been changed to support only networking.k8s.io/.... which is available only from kubernetes v1.14.

This is not true. The ingress controller supports both versions.
The reason why you see the docs networking.k8s.io is just to avoid users using k8s >= 1.16 copy something it will not work (extensions/beta1 throws an error when you create a new ingress.)

@aledbf can we have please some help with this issue?

I am more than happy to see what's wrong if someone can help with a self-contained way to reproduce this. Something like this would be great https://github.com/kubernetes/ingress-nginx/issues/4979#issuecomment-579439387

[catalinpan]

@aledbf Sorry for not being more specific. This is the part which breaks the ingress status update for Kubernetes clusters older than 1.14 due to the fact that the ingress kind was not part of the networking.k8s.io apiVersion prior 1.14, click here for more details. More exactly the lines 268 and 281 on that file. I'm not a golang expert but if nobody will have the time to resolve this I will try a PR.

Thanks for all your contributions and for making this project great!

[aledbf]

More exactly the lines 268 and 281 on that file. I'm not a golang expert but if nobody will have the time to resolve this I will try a PR.

@catalinpan thank you for finding the issue. This was my fault when I added the logic for the networking api.
Please use quay.io/kubernetes-ingress-controller/nginx-ingress-controller-amd64:fix-status
Such image contains PR #4996

[catalinpan]

@aledbf there is nobody's fault. Many thanks again for all your help!

[gb-kpatel]

Spent couple hours debugging for this. Glad I found this issue here.

@aledbf any idea when would be the next release?