Ingress-nginx: https redirect doesn't work on AWS EKS (error 308 too many redirects)

[adriano-fonseca]

I would like to start saying that I have tried all suggestions that have worked in the past for the same issue without success, seems like a regression for me.

NGINX Ingress controller version:
quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.22.0

Kubernetes version (use kubectl version):
1.11

Environment:

I am deploying applications on AWS EKS, my ingress-nginx is configured as follows:

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-west-2:XXXXXXXX:certificate/XXXXXX-XXXXXXX-XXXXXXX-XXXXXXXX"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  ports:
    - name: http
      port: 80
      targetPort: http
    - name: https
      port: 443
      targetPort: http

My application ingress is using sel-sighed certificates stored as a secret and it is configured as follows:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/ssl-redirect: "false"
  name: wordpress
  namespace: mrcartr
spec:
  rules:
    - host: blog2.mrcartr.com
      http:
        paths:
          - backend:
              serviceName:  wordpress-mrcartr
              servicePort: 80
            path: /
# This section is only required if TLS is to be enabled for the Ingress
  tls:
        - hosts:
            - blog2.mrcartr.com
          secretName: example-tls

I already have the a CNAME (blog2.mrcartr.com) that points to the public ELB provisioned for the service ingress-nginx, posted above.

My service is defined as bellow:

apiVersion: v1
kind: Service
metadata:
  name: wordpress-mrcartr
  labels:
    app: wordpress-mrcartr
spec:
  ports:
    - port: 80
  selector:
    app: wordpress
    product: mrcartr

The service points for a statefulset configured as follows:

apiVersion: apps/v1 
kind: StatefulSet
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  selector:
    matchLabels:
      app: wordpress
      product: mrcartr
  serviceName: wordpress
  replicas: 1
  template:
    metadata:
      labels:
        app: wordpress
        product: mrcartr
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - image: wordpress:5.0.3-php7.2-apache-qelp-base
        name: wordpress
...

What happened:
Once I try to access the application I receive the 308 error (ERR_TOO_MANY_REDIRECTS) even with ssl-redirects disable. I have try different configurations to implement https in the applications running in a EKS cluster. However, so far none of those configurations worked.

What you expected to happen:
I expect to have http redirect to https.

How to reproduce it (as minimally and precisely as possible):

Following the installation guide and deploying any application (standard wordpres image, for instance) and trying to access using http and https.

[ElvinEfendi]

@adriano-fonseca have you noticed the breaking change in 0.22.0 release notes?

By default do not trust any client to extract true client IP address from X-Forwarded-For header using realip module (use-forwarded-headers: "false")

Please make sure you adjust it.

[njtman]

Downgrading to version 0.21.0 fixed the issue for us.

[adriano-fonseca]

Downgrading to version 0.21 also fixes the issue for me. The only remaining issue was that the annotation ingress.kubernetes.io/ssl-redirect: "true" was not proxying my traffic to https when I try to access the application over http protocol. This was achieved using the nginx.ingress.kubernetes.io/force-ssl-redirect: "true" annotation. This was the configuration that worked for me.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"

[ElvinEfendi]

@adriano-fonseca I recommend you try latest version with use-forwarded-headers set to "true" in the configmap. I'd also advise you to adjust proxy-real-ip-cidr in the configmap so that you don't trust any client and trust only ELB.

ingress.kubernetes.io/ssl-redirect: "true" is a default behaviour unless you changed it in configmap: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#server-side-https-enforcement-through-redirect.

[adriano-fonseca]

Something like this right, @ElvinEfendi ?

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
data:
  use-proxy-protocol: "false"
  use-forwarded-headers: "true"
  proxy-real-ip-cidr: "0.0.0.0/0" # restrict this to the IP addresses of ELB

Is it possible to use the public DNS in order to white list the ELB with the nginx.ingress.kubernetes.io/whitelist-source-range annotation?
I am asking that because the ELB IPs can change, as I read here.

I also realized that with this configuration the browser is complaining that is not a private connection. Does someone know if it is possible to have https end to end using the AWS public certificates from Certificate Manager? (In a Positive case an example to achieve this would be much appreciated)

[phisco]

For future reference, setting use-forwarded-headers: "true" in the configmap, solved the issue for me too, as suggested by @ElvinEfendi and shown here