Ingress-nginx: 308 redirect loop when setting "force-ssl-redirect" to "true"

[danielfm]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.): No

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.): force ssl redirect loop 308

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug report

NGINX Ingress controller version: 0.10.0

Kubernetes version (use kubectl version): v1.8.5

Environment:

• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): Container Linux by CoreOS 1576.5.0 (Ladybug)
• Kernel (e.g. uname -a): 4.14.11-coreos
• Install tools: kube-aws
• Others:

What happened:

The force-ssl-redirect flag seems not to be working anymore.

What you expected to happen:

This is what happened before I updated the nginx ingress controller (this is from version 0.9.0-beta.19):

$ kubectl -n kube-system port-forward hissing-wasp-nginx-ingress-controller-75b88b9b55-9vq8h 8080:80
Forwarding from 127.0.0.1:8080 -> 80

# In another terminal, redirect is expected
$ curl -I -H "Host: <ingress host>" -H "X-Forwarded-Proto: http" http://localhost:8080
HTTP/1.1 301 Moved Permanently
Server: nginx/1.13.7
Date: Tue, 23 Jan 2018 18:48:26 GMT
Content-Type: text/html
Content-Length: 185
Connection: keep-alive
Location: https://<ingress host>/

# Redirect is NOT expected
$ curl -I -H "Host: <ingress host>" -H "X-Forwarded-Proto: https" -H "X-Forwarded-Port: 443" http://localhost:8080
HTTP/1.1 200 OK
Server: nginx/1.13.7
Date: Tue, 23 Jan 2018 18:48:31 GMT
Content-Type: text/html
Content-Length: 79850
Connection: keep-alive
Vary: Accept-Encoding
X-Powered-By: Next.js 4.2.1

This is what happens in 0.10.0:

# Redirect is expected
$ curl -I -H "Host: <ingress host>" -H "X-Forwarded-Proto: http" http://localhost:8080
HTTP/1.1 308 Permanent Redirect
Server: nginx/1.13.8
Date: Tue, 23 Jan 2018 18:47:04 GMT
Content-Type: text/html
Content-Length: 187
Connection: keep-alive
Location: https://<ingress host>/

# Redirect is NOT expected
$ curl -I -H "Host: <ingress host>" -H "X-Forwarded-Proto: https" -H "X-Forwarded-Port: 443" http://localhost:8080
HTTP/1.1 308 Permanent Redirect
Server: nginx/1.13.8
Date: Tue, 23 Jan 2018 18:47:19 GMT
Content-Type: text/html
Content-Length: 187
Connection: keep-alive
Location: https://<ingress host>/

[grebois]

Also happening with;

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kubernetes-dashboard
  namespace: kube-system
  annotations:
    ingress.kubernetes.io/ssl-passthrough: "true"
    ingress.kubernetes.io/secure-backends: "true"
    kubernetes.io/ingress.allow-http: "false"
    kubernetes.io/ingress.class: "nginx"

spec:
  rules:
  - host: dashboard.<my_domain>
    http:
      paths:
      - path: /
        backend:
          serviceName: kubernetes-dashboard
          servicePort: 443

[fyuvb]

The same thing happens to me with config

nginx.ingress.kubernetes.io/force-ssl-redirect: "true"

on version 0.10.0

[JoelSpeed]

Also seeing this problem. I believe it was introduced in this PR https://github.com/kubernetes/ingress-nginx/pull/1854/files

We have ELBs in front of our ingress which terminate the TLS connection and connect to the ingress controllers over http. Therefore $scheme=http and $pass_access_scheme=https so the rule is matched and $redirect_to_https is set to true.

I believe this to be a perfectly valid setup and I believe the intention of this change was to stop the case where an upstream load balancer was receiving an http connection but issuing https connections to the ingress controller, in which case the rule would be https:http not http:https

If the upstream proto is https then I don't think it should ever perform the redirect.

@aledbf Would you mind confirming my suspicions of the intention?

[aledbf]

@danielfm @JoelSpeed please use quay.io/aledbf/nginx-ingress-controller:0.318

[JoelSpeed]

@aledbf Can I ask what you've changed?

[danielfm]

@aledbf this image fixed the issue for me.

[aledbf]

@JoelSpeed you are right, that PR checks the variables $scheme and $pass_access_scheme
This is the table that shows when is required a redirect:

Scheme | X-Forwarded-Proto | Redirect?
---------|--------------------|-----------
http | http | yes
http | https | yes
https | http | no
https | https | no

[aledbf]

Can I ask what you've changed?

I removed the change in #1854

Edit: everything started with this issue https://github.com/kubernetes/ingress-nginx/issues/1841

Edit: with this change the table is:

Scheme | X-Forwarded-Proto | Redirect?
---------|--------------------|-----------
http | http | yes
http | https | no
https | http | no
https | https | no

[JoelSpeed]

@aledbf I think lines two and three of that table should be swapped.

If X-Forwarded-Proto is http that means the connection to the upstream load balancer was made by http which would be when we want to upgrade, but line three suggests otherwise?

Similarly if X-Forwarded-Proto is https the original connection to the upstream load balancer was https and then we don't care if the connection to nginx was http, the original client connected by https which is the desired effect right?

[JoelSpeed]

@aledbf quay.io/aledbf/nginx-ingress-controller:0.318 works for me

[aledbf]

If X-Forwarded-Proto is HTTP that means the connection to the upstream load balancer was made by HTTP which would be when we want to upgrade, but line three suggests otherwise?

Please keep in mind we cannot differentiate a connection from a client or a load balancer. With that in mind if you make a request to NGINX using https we should not redirect (again) to https.

[JoelSpeed]

Please keep in mind we cannot differentiate a connection from a client or a load balancer. With that in mind if you make a request to NGINX using https we should not redirect (again) to https.

I don't think I follow here.

If a client connects directly to the ingress controller, then the X-Forwarded-Proto header won't exist and so $pass_access_scheme=$scheme

https://github.com/kubernetes/ingress-nginx/blob/3bc2c6827debbd95ae97861b042511b4d42d25af/rootfs/etc/nginx/template/nginx.tmpl#L197-L200

The only case when you would get a https:http for your table is when there is a load balancer between the client and the ingress controller and the client made the connection to the load balancer by http?

If that isn't the case could you quickly explain a scenario where you'd get https:http from a client directly?

[aledbf]

If that isn't the case could you quickly explain a scenario where you'd get https:http from a client directly?

The load balancer uses HTTPS to connect to NGINX or a client sends the X-Forwarded-Proto header

[JoelSpeed]

The load balancer uses HTTPS to connect to NGINX

That's what I was suggesting, sorry, I didn't make that too clear

Client -- HTTP -- > LB -- HTTPS --> NGINX

Ends up with X-Forwarded-Proto=http therefore $pass_access_scheme=http but $scheme=https so the table entry is https:http and we should redirect to get

Client -- HTTPS -- > LB -- HTTPS --> NGINX

[aledbf]

Ends up with X-Forwarded-Proto=http therefore $pass_access_scheme=http but $scheme=https so the table entry is https:http and we should redirect to get

What's the difference handling a connection between a load balancer and direct client?
If we add a redirect for https:http then you can force a redirect using an HTTPS connection setting X-Forwarded-Proto=http which lead to a redirect loop.

[aledbf]

@JoelSpeed just to be clear, I think you are right and the table should be

    map "$scheme:$pass_access_scheme" $redirect_to_https {
        default          0;
        "http:http"      1;
        "https:http"     1;
    }

but this issue has already bit me too many times.

[aledbf]

Please test quay.io/aledbf/nginx-ingress-controller:0.319. It contains ^^

[JoelSpeed]

I'm getting Image pull failures, looks like that image didn't push to quay

[erwbgy]

The quay.io/aledbf/nginx-ingress-controller:0.318 image also fixed a problem I was having where:

nginx.ingress.kubernetes.io/ssl-passthrough: "true"

was being ignored.

[aledbf]

I'm getting Image pull failures, looks like that image didn't push to quay

Please try again

[JoelSpeed]

Works for me 👍

[aledbf]

@JoelSpeed are you ok with the change in the redirect map?

[JoelSpeed]

Yeah, I think this is a good solution

[discostur]

Hi,

just had a similary problem ... trying to configure a catch all ingress like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    releaseTime: 2018-03-28 12:14:07.305364616 +0000 UTC
  creationTimestamp: 2018-03-27T20:40:51Z
  generation: 11
  labels:
    heritage: Tiller
    release: int
  name: redirection-v1
  namespace: integration
spec:
  rules:
  - host: '*.de'
    http:
      paths:
      - backend:
          serviceName: redirection-v1
          servicePort: 8080

I'm trying to reach the service via plain HTTP and there should be NO redirect, but:

$curl -I -H "X-Forwarded-Proto: https" http://redirection.integration.de/?domain=xyz
HTTP/1.1 200 OK
Server: nginx/1.13.9
Date: Wed, 28 Mar 2018 12:51:58 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding

$curl -I -H "X-Forwarded-Proto: http" http://redirection.integration.de/?domain=xyz
HTTP/1.1 308 Permanent Redirect
Server: nginx/1.13.9
Date: Wed, 28 Mar 2018 12:51:54 GMT
Content-Type: text/html
Content-Length: 187
Connection: keep-alive
Location: https://redirection.integration.de/?domain=xyz

I don't unterstand why i get a redirect if i'm connecting via HTTP ... it i set x-forwarded-for to https the request is plain http and i get an answer from my service. Is there anything else i have to configure?

[adriancislo]

Based on https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/rewrite/README.md
I set (changing default value)
nginx.ingress.kubernetes.io/ssl-redirect: "false" in ingress annotations and it started to behave as I expected. Call service via http returns 200 not 308

[trueinviso]

It appears that this may be an issue again.

I am using kubernetes on AWS and this is what my ingress looks like:

```apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-service
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
rules:

• host: "example.com"
  http:
  paths:
  
  
  
  • path: /
    
    backend:
    
    serviceName: app-cluster-ip-service
    
    servicePort: 80
    
    ```
  
  

I am terminating TLS at the load balancer and I was getting the infinite redirect loop when I turned on force-ssl-redirect="true", if i set it to false the loop stops but I don't get a redirect to https. When I update the image in the ingress-nginx-controller deployment file to the previous version then all is well. So in case anyone else is unfamiliar with this like I was:

kubectl edit deployment -n ingress-nginx nginx-ingress-controller

Look for:

image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:<version>

The newest version right now is 0.22.0:

quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.22.0

and I changed it to:

quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0

It appears that this was fixed in the previous version, and then some new changes broke it again.

[bfin]

@trueinviso , in my case – also using the controller behind an AWS ELB – the infinite redirect loop was caused by a (documented) breaking change introduced in v0.22.0 regarding forwarded headers. Setting use-forwarded-headers: "true" in the configmap (docs) fixed the issue for me.

[trueinviso]

@bfin Thanks! That fixes it. I'm not sure I would have been able to figure that out on my own haha.

[pauldutheil]

Thanks @bfin ! works for me on 0.22.0 behind an AWS ELB

[joshbranham]

Thanks @bfin this fixed our issues as well

[gdelgado]

Tried the above use-forwarded-headers: "true" within the configmap and it did not work. Using nginx version 0.24.1. Any one else having the same issue with 0.24.1

[juniorrocha]

Tried the above use-forwarded-headers: "true" within the configmap and it did not work. Using nginx version 0.24.1. Any one else having the same issue with 0.24.1

I had the same problem and fixed with use-forwaded-headers: "true", within 0.24.1

[BobbyJohansen]

It would seem that when using:

http-snippet: |
    map true $pass_access_scheme {
      default "https";
    }
    map true $pass_port {
      default 443;
    }

    server {
      listen 8080 proxy_protocol;
      return 308 https://$host$request_uri;
    }

I do not get X-forwarded-for headers even when I have use-forwaded-headers: "true" set I am also using quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.24.1

Does anyone know how to get the X-Forwarded-For headers to propagate over the http to https redirect?