Ingress-nginx: proxy-set-headers appends instead of overwriting

Created on 28 Nov 2018 · 21Comments · Source: kubernetes/ingress-nginx

Is this a request for help? No

What keywords did you search in NGINX Ingress controller issues before filing this one? proxy-set-headers, overwrite, override, add-headers

Is this a BUG REPORT or FEATURE REQUEST? Bug Report

NGINX Ingress controller version: 0.21.0

Kubernetes version (use kubectl version): 1.10.9

Environment:

Cloud provider: AWS, can provide specifics but I assume they're not relevant

What happened:
My instance of ingress-nginx is running behind an AWS ELB that terminates the TLS connection. Thus all requests show up with the "X-Forwarded-*" headers indicating a plain old HTTP connection. However my backend services generate user-facing URLs from the values supplied from these headers. Therefore, I must change them as if the ingress-nginx had received an HTTPS request so it returns the proper URLs.

In order to do this I have configured the configuration option "proxy-set-headers" to point to a ConfigMap with the following data:

X-Forwarded-Port: "443"
X-Forwarded-Proto: https

But when I checked on the requests that arrive at an echo server (gcr.io/google_containers/echoserver:1.4) this is what the backend would get: (Values get appended instead of set)

x-forwarded-port=80443
x-forwarded-proto=httphttps

What you expected to happen:
The backend should receive (Values should get replaced)

x-forwarded-port=443
x-forwarded-proto=https

How to reproduce it (as minimally and precisely as possible): See "What happened"

Anything else we need to know: Can't think of anything, but will answer any follow-up questions

lifecyclrotten

Source

xcq1

👍3

Most helpful comment

Running into this on our ingress which runs on port 80 behind an SSL proxy that incorrectly sends us an X-Forwarded-Port: 80 header.
I cannot override this with the correct value since proxy-set-headers configmap values get appended.

fdlk on 23 Apr 2019

👍4

All 21 comments

@xcq1 what are the relevant configmap, annotations configuration snippets? Can you paste them verbatim?

dcherniv on 30 Nov 2018

@dcherniv I think these would be the relevant bits:

kind: ConfigMap
apiVersion: v1
metadata:
  name: ingress-nginx
  labels:
    k8s-addon: ingress-nginx.addons.k8s.io
data:
  use-proxy-protocol: "true"
  proxy-body-size: "100m"
  proxy-read-timeout: "900"
  proxy-send-timeout: "900"
  proxy-set-headers: "infrastructure/ingress-nginx-headers"
  custom-http-errors: "404,502,503,504"
  log-format-upstream: '{"time": "$msec", "format": "access", "message": "$request", "request_id": "$request_id", "request_status": "$status", "request_method": "$request_method", "request_host": "$host", "request_uri": "$request_uri", "request_length": "$request_length", "request_time": "$request_time", "request_body_bytes_sent": "$body_bytes_sent", "request_referrer": "$http_referer", "request_user_agent": "$http_user_agent", "request_remote_addr": "$proxy_protocol_addr", "request_remote_user": "$remote_user", "request_forwarded_for": "$proxy_add_x_forwarded_for", "request_upstream_address": "$upstream_addr", "request_upstream_response_length": "$upstream_response_length", "request_upstream_response_time": "$upstream_response_time", "request_upstream_status": "$upstream_status" }'
  http-snippet: |
    server {
      listen 8000 proxy_protocol;
      server_tokens off;
      return 301 https://$host$request_uri;
    }

  hide-headers: "Server"
  hide-headers: "X-Application-Context"
  hide-headers: "X-Powered-By"
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: ingress-nginx-headers
  labels:
    k8s-addon: ingress-nginx.addons.k8s.io
data:
  Request-Id: "$request_id"
  Strict-Transport-Security: "max-age=31536000; includeSubDomains; preload"
  Content-Security-Policy: "frame-ancestors 'self' https://*.example1.com https://*.example2.com https://*.example3.com"
  X-Content-Type-Options: "nosniff"
  X-XSS-Protection: "1; mode=block"
  Referrer-Policy: "strict-origin-when-cross-origin"
  X-Forwarded-Port: "443"
  X-Forwarded-Proto: https
---
kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  labels:
    k8s-addon: ingress-nginx.addons.k8s.io
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: '$((elb-certificate-arn))'
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: '443'
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '900'
spec:
  type: LoadBalancer
  selector:
    app: ingress-nginx
  ports:
  - name: http
    port: 80
    targetPort: 8000
  - name: https
    port: 443
    targetPort: 80

Port 8000 has also been opened on the deployment.
Right now I'm guessing in my case it would be preferable if I just switched the ELB to HTTPS backend requests, but I figured the behavior is a little bit odd nonetheless.

xcq1 on 30 Nov 2018

I think this is the default proxy_set_header behavior in nginx.
To overwrite headers we might need to use headers_more.
https://www.nginx.com/resources/wiki/modules/headers_more/
Or use lua maybe.
My apologies i couldn't be of much help. This does appear to be a valid concern however. We should be able to override headers passed to upstream.

dcherniv on 30 Nov 2018

Just found out add-headers is guilty of a similar problem:
With the above ConfigMaps, when a backend returns X-XSS-Protection: 1; mode=block the client receives those headers like this:

...
X-XSS-Protection: 1; mode=block
X-XSS-Protection: 1; mode=block
...

Which in this case leads to problems because the appended headers are not valid: Error parsing header X-XSS-Protection: 1; mode=block, 1; mode=block: expected semicolon at character position 13. The default protections will be applied.

I got around this for now by replacing add_headers in line 219 of the nginx.tmpl with your suggestion more_set_headers. However it's not exactly clear to me where I would replace proxy_set_header with more_set_headers to solve it for the proxy-set-header config option (or if that would break other things).

xcq1 on 5 Dec 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 5 Mar 2019

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot on 4 Apr 2019

fdlk on 23 Apr 2019

👍4

/remove-lifecycle rotten

@fdlk In case it might help you: We ended up workarounding this by hard patching the default nginx.tmpl and including it as a separate config map as described in https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/custom-template/

It works if you have no HTTP-only connections, however it isn't going to win any beauty awards. And you have to accept the fact that any version upgrade can theoretically break your config.

This is the patch:

--- nginx.tmpl.orig 2019-01-15 00:15:21.000000000 +0100
+++ nginx.tmpl  2019-02-12 13:09:05.069068796 +0100
@@ -1278,8 +1278,8 @@
             {{ $proxySetHeader }} X-Forwarded-For        $the_real_ip;
             {{ end }}
             {{ $proxySetHeader }} X-Forwarded-Host       $best_http_host;
-            {{ $proxySetHeader }} X-Forwarded-Port       $pass_port;
-            {{ $proxySetHeader }} X-Forwarded-Proto      $pass_access_scheme;
+            {{ $proxySetHeader }} X-Forwarded-Port       "443";
+            {{ $proxySetHeader }} X-Forwarded-Proto      "https";
             {{ if $all.Cfg.ProxyAddOriginalURIHeader }}
             {{ $proxySetHeader }} X-Original-URI         $request_uri;
             {{ end }}

Which is then deployed like so:
kubectl create cm nginx-template -n infrastructure --from-file=controller/service/nginx.tmpl --dry-run -o yaml | kubectl apply -f -
And then included as a volume as described in the link to the user guide.

xcq1 on 23 Apr 2019

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 22 Jul 2019

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot on 21 Aug 2019

/remove-lifecycle rotten
To whom it may concern: The workaround in the referenced issue https://github.com/kubernetes/ingress-nginx/issues/4096#issuecomment-494181706 is even simpler, avoids the hard template dependency and works just as well (under the same assumptions).

xcq1 on 23 Aug 2019

It looks like this has been fixed in https://github.com/kubernetes/ingress-nginx/commit/a44b5cf3f3e3817c01f979dcd7190cc62d435f7b, Version 0.26.0 via #4463

Edit: The referenced fix is for set-headers not proxy-set-headers. Unfortunately proxy-set-headers continues to append values, not overwrite v0.41.0

hobti01 on 3 Oct 2019

👍2

The upgrade in a44b5cf did not work for me. What did work was adding the header as so:

more_set_input_headers "X-Forwarded-Proto: https";

gdmello on 4 Dec 2019

👍2

Ok my problem with X-Forwarded-Proto was not caused by ingress-nginx but by the AWS ALB in front of it.
It seems the ALB always overrides this header.

owengo on 17 Dec 2019

How did you fix it in ALB?
I am running my stack using EKS with ALB at the front and I am getting X-Forwarded-Proto as HTTP. I want to change it to https.

prcongithub on 30 Dec 2019

@prcongithub
You can't fix ALB, it will trash any incoming X-Forwarded-Proto and set its own. X-F-P is badly designed: opposed to X-F-F this is not a list of incoming protocols, so intermediate proxies must choose between passing it as is ( which is a security issue, among other things ) or setting it to the protocol with which they received the request. It seems unlikely that AWS will give the possibility to pass the header.
If you know that your traffic will always be https at the edge, you can hack your ingress with something like this:

    annotations:
       nginx.ingress.kubernetes.io/configuration-snippet: |
           more_set_input_headers "X-Forwarded-Proto: https";

It makes sense, especially if you force a http => https redirect somewhere between your edge and ingress-nginx ( both included ).
Alternatively, if you can set your own header at the edge you can set your own "My-Forwarded-Proto: .." ( or use a header setup by CloudFront or your edge servers ) and do some hacking after the alb..

owengo on 30 Dec 2019

❤3

I just fixed it in kong. Had to add trusted_ips in kong configuration to allow X-Forwarded-* headers to pass through. I am using https everywhere. ALB redirects http to https. SSL Terminates at ALB and connects with Kong using HTTP. But if I enable trusted_ips, kong passes X-Forwarded-* headers with correct values.

prcongithub on 30 Dec 2019

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 29 Mar 2020

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot on 28 Apr 2020

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

fejta-bot on 28 May 2020

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.