Ingress-nginx: X-Forwarded-Proto not propagated when using auth-url with load balancer

[morganchristiansson]

NGINX Ingress controller version: 0.24.1

Kubernetes version client: v1.14.1 server: v1.12.6-eks-d69f1b

Environment:

• Cloud provider or hardware configuration: AWS EKS
• OS (e.g. from /etc/os-release): Default EKS worker AMI
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What happened:
Application is receiving incorrectly receiving X-Forwarded-Proto: http for https connections.

AWS load balancer is correctly sending X-Forwarded-Proto: https that nginx then forwards as X-Forwarded-Proto: http

I use ingress annotations for authentication. Removing these annotations restores correct behaviour.

  annotations:
    nginx.ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth"
    nginx.ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start?rd=$escaped_request_uri"

What you expected to happen:
To receive X-Forwarded-Proto: https for https connections

How to reproduce it (as minimally and precisely as possible):
Add annotations above.

Anything else we need to know:

I run oauth2_proxy helm chart configured with Azure auth provider. It's has ingress configured for same host: with path: /oauth2 similar to suggested at https://github.com/kubernetes/ingress-nginx/tree/master/docs/examples/auth/oauth-external-auth

[morganchristiansson]

May be cause of #3389 PR #3405 which unsets the X-Forwarded-Proto header and doesn't re-set it for the actual request.

I'm not a contributor or Go developer or I would write a test case. I can provide a minimum setup to reproduce if needed.

And there's a few workarounds I can try.

[aledbf]

I can provide a minimum setup to reproduce if needed.

That would be great

[morganchristiansson]

The following kubernetes resources reproduce the problem.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: echoserver0
spec:
  rules:
  - host: echo0.127.0.0.1.xip.io
    http:
      paths:
      - path: /
        backend:
          serviceName: echoserver
          servicePort: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: echoserver1
  annotations:
    nginx.ingress.kubernetes.io/auth-url: https://httpstat.us/200
spec:
  rules:
  - host: echo1.127.0.0.1.xip.io
    http:
      paths:
      - path: /
        backend:
          serviceName: echoserver
          servicePort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: echoserver
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
  selector:
    app: echoserver
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: echoserver
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: echoserver
    spec:
      containers:
      - image: gcr.io/google_containers/echoserver:1.4
        imagePullPolicy: Always
        name: echoserver
        ports:
        - containerPort: 8080

change echo0.127.0.0.1.xip.io and echo1.127.0.0.1.xip.io to hosts that work for your ingress.

save as echobug.yaml and run kubectl apply -f echobug.yaml

echo0 responds with:
```CLIENT VALUES:
client_address=internal.network.ip.address
command=GET
real path=/
query=nil
request_version=1.1
request_uri=http://echo0.127.0.0.1.xip.io:8080/

SERVER VALUES:
server_version=nginx: 1.10.0 - lua: 10001

HEADERS RECEIVED:
accept=text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
accept-encoding=gzip, deflate, br
accept-language=en-US,en;q=0.5
host=echo0.127.0.0.1.xip.io
upgrade-insecure-requests=1
user-agent=Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0
x-forwarded-for=my.public.ip.address
x-forwarded-host=echo0.127.0.0.1.xip.io
x-forwarded-port=443
x-forwarded-proto=https
x-original-forwarded-for=my.public.ip.address
x-original-uri=/
x-real-ip=my.public.ip.address
x-request-id=f25d5f8d60038440d9043c66b3f3f957
x-scheme=https
BODY:
-no body in request-

echo1 responds with:

CLIENT VALUES:
client_address=internal.network.ip.address
command=GET
real path=/
query=nil
request_version=1.1
request_uri=http://echo1.127.0.0.1.xip.io:8080/

SERVER VALUES:
server_version=nginx: 1.10.0 - lua: 10001

HEADERS RECEIVED:
accept=text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
accept-encoding=gzip, deflate, br
accept-language=en-US,en;q=0.5
host=echo1.127.0.0.1.xip.io
upgrade-insecure-requests=1
user-agent=Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0
x-forwarded-for=my.public.ip.address
x-forwarded-host=echo1.127.0.0.1.xip.io
x-forwarded-port=80
x-forwarded-proto=http
x-original-forwarded-for=my.public.ip.address
x-original-uri=/
x-real-ip=my.public.ip.address
x-request-id=97a02a239034423c4090451108b863ce
x-scheme=http
BODY:
-no body in request-
```

notice: x-forwarded-proto and x-scheme are http from echo1 aswell as x-forwarded-port being set to 80

they were both https requests, only the number in the url was changed.

You need a load balancer that sends X-Forwarded-Proto: https header to nginx-ingress and set use-forwarded-headers: "true" option in nginx configmap

[morganchristiansson]

I just applied valid workaround following steps in https://github.com/kubernetes/ingress-nginx/issues/3481#issuecomment-485938466

Since my traffic is always https this works for me.

It is not related to issue #3389 as I suspected above.

$pass_port and $pass_access_scheme have incorrect values when auth-url annotations is used.

[morganchristiansson]

I found a smaller workaround.

I use nginx stable helm chart.

controller:
  service:
    type: LoadBalancer

    annotations:
      external-dns.alpha.kubernetes.io/hostname: your.route53.dns.name
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:xxxxxxxxx
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    targetPorts:
      http: 8080 # Send http traffic to port 8080 in config.http-snippet below
      https: http

  config:
    use-forwarded-headers: "true"
    http-snippet: |
      server {
        listen 8080;
        return 308 https://$host$request_uri;
      }

    server-snippet: |
      set $pass_access_scheme "https";
      set $pass_port 443;

This implements global https redirect (which is default behaviour without https offloading load balancer). And forces https values to variables that are used in nginx.tmpl

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[kissgyorgy]

I can reproduce this. I'm using the image quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.25.1 (from https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml)

My setup is the following:
global nginx with SSL ---> ingress-nginx (no SSL configured) ---> pod nginx (also no SSL)

Here is the first config:

 server {
     listen 192.168.1.2:443 ssl http2;
     server_name *.example.com;

     location / { 
         allow 192.168.1.0/24;
         deny all;

         # ingress-nginx service mapped NodePort 80
         proxy_pass http://127.0.0.1:30080;
         include proxy_params;
         proxy_http_version 1.1;
         proxy_set_header Upgrade $http_upgrade;
         proxy_set_header Connection $connection_upgrade;

     }   
 }

here is the Ingress manifest:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myapp
spec:
  rules:
  - host: sub.example.com
    http:
      paths:
      - backend:
          serviceName: myapp
          servicePort: http

When I send a request to the Pod, it is clear that the X-Forwarded-Proto is not set properly:

2019/09/16 00:29:32 [debug] 409#409: *79 http header: "X-Request-ID: 3957c04259ad76f34e18cc1f22538882"
2019/09/16 00:29:32 [debug] 409#409: *79 http header: "X-Real-IP: 192.168.1.2"
2019/09/16 00:29:32 [debug] 409#409: *79 http header: "X-Forwarded-For: 192.168.1.2"
2019/09/16 00:29:32 [debug] 409#409: *79 http header: "X-Forwarded-Host: sub.example.com"
2019/09/16 00:29:32 [debug] 409#409: *79 http header: "X-Forwarded-Port: 80"
2019/09/16 00:29:32 [debug] 409#409: *79 http header: "X-Forwarded-Proto: http"
2019/09/16 00:29:32 [debug] 409#409: *79 http header: "X-Original-URI: /images/default/logo.png"
2019/09/16 00:29:32 [debug] 409#409: *79 http header: "X-Scheme: http"
2019/09/16 00:29:32 [debug] 409#409: *79 http header: "X-Original-Forwarded-For: 192.168.1.4"
2019/09/16 00:29:32 [debug] 409#409: *79 http header: "sec-fetch-mode: no-cors"
2019/09/16 00:29:32 [debug] 409#409: *79 http header: "if-none-match: "5d7ed4e5-a07c""
2019/09/16 00:29:32 [debug] 409#409: *79 http header: "if-modified-since: Mon, 16 Sep 2019 00:18:45 GMT"
...
2019/09/16 00:29:32 [debug] 409#409: *79 http header: "referer: https://sub.example.com/cmd.php?server_id=1&redirect=true"

[kissgyorgy]

I think I misunderstood the problem, setting

  config:
    use-forwarded-headers: "true"

solves my problem. Sorry about the noise.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[techs2resolve]

This configmap is working for me. you need to patch it.
nginx-controller-version :- 0.30.0
Image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0

vim nginx-patch-configmap-l7.yaml

kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-configuration
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
data:
use-proxy-protocol: "false"
use-forwarded-headers: "true"

[techs2resolve]

`This configmap is working for me. you need to patch it.
nginx-controller-version :- 0.30.0
Image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0

vim nginx-patch-configmap-l7.yaml

kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-configuration
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
data:
use-proxy-protocol: "false"
use-forwarded-headers: "true"
`

[teochenglim]

@techs2resolve your solution saved me =)