Ingress-nginx: client IP is incorrect when using CloudFront -> ELB -> Nginx and proxy protocol

[noqcks]

NGINX Ingress controller version: 0.30.0

Kubernetes version (use kubectl version): 1.15.9

Environment: Kops on AWS

• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): Debian 9
• Kernel (e.g. uname -a): Linux ip-172-20-1-51 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20) x86_64 GNU/Linux
• Install tools: Helm
• Others: N/A

What happened:

We are attempting to setup a our ingress so that we have traffic like so

CloudFront -> ELB -> Nginx Ingress

Previously it was ELB -> Nginx Ingress and everything worked. However, once we switched to add CloudFront our IP whitelisting no longer works. The reason for this I believe is that the real_ip is being set incorrectly.

This is what our logs look like

Mar 10 12:45:23 v2-nginx-ingress-controller-7884c7cd8b-hsngz nginx-ingress-controller { 
"@timestamp": "2020-03-10T16:45:23+00:00", "remote_addr": "70.132.33.156", "x-forward-for": 
"18.232.45.xxx", "request_id": "20db865d9855b088de361d652cec83f7", "remote_user": "-", 
"bytes_sent": 428, "request_time": 0.000, "status": 403, "vhost": "REDACTED", "request_proto": 
"HTTP/1.1", "path": "/api", "request_query": "-", "request_length": 1019, "duration": 0.000, "method": "GET", 
"http_referrer": "-", "http_user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) 
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" }

My Client/Computer IP = 18.232.45.xxx

The IP that's listed under remote_addr (70.132.33.156) is using $proxy_protocol_addr and is from the CloudFront IP address range. This is also used by real_ip module here

configmap
(the proxy-real-ip-cidr list contains all the CloudFront IP addresses as well as our VPC CIDR (172.20.0.0/16))

  compute-full-forwarded-for: "true"
  enable-vts-status: "true"
  limit-req-status-code: "429"
  log-format-upstream: '{ "@timestamp": "$time_iso8601", "remote_addr": "$proxy_protocol_addr",
    "x-forward-for": "$http_x_forwarded_for", "request_id": "$request_id", "remote_user":
    "$remote_user", "bytes_sent": $bytes_sent, "request_time": $request_time, "status":
    $status, "vhost": "$host", "request_proto": "$server_protocol", "path": "$uri",
    "request_query": "$args", "request_length": $request_length, "duration": $request_time,
    "method": "$request_method", "http_referrer": "$http_referer", "http_user_agent":
    "$http_user_agent" }'
  proxy-body-size: 64m
  proxy-next-upstream: error timeout invalid_header http_502 http_503 http_504
  proxy-real-ip-cidr: 172.20.0.0/16,144.220.0.0/16,52.124.128.0/17,54.230.0.0/16,54.239.128.0/18,52.82.128.0/19,99.84.0.0/16,204.246.172.0/24,54.239.192.0/19,70.132.0.0/18,13.32.0.0/15,205.251.208.0/20,13.224.0.0/14,13.35.0.0/16,204.246.164.0/22,204.246.168.0/22,71.152.0.0/17,216.137.32.0/19,205.251.249.0/24,99.86.0.0/16,52.46.0.0/18,52.84.0.0/15,204.246.173.0/24,130.176.0.0/16,205.251.200.0/21,204.246.174.0/23,64.252.128.0/18,205.251.254.0/24,143.204.0.0/16,205.251.252.0/23,204.246.176.0/20,13.249.0.0/16,54.240.128.0/18,205.251.250.0/23,52.222.128.0/17,54.182.0.0/16,54.192.0.0/16,13.124.199.0/24,34.226.14.0/24,52.15.127.128/26,35.158.136.0/24,52.57.254.0/24,18.216.170.128/25,13.52.204.0/23,13.54.63.128/26,13.59.250.0/26,13.210.67.128/26,35.167.191.128/26,52.47.139.0/24,52.199.127.192/26,52.212.248.0/26,52.66.194.128/26,13.113.203.0/24,99.79.168.0/23,34.195.252.0/24,35.162.63.192/26,34.223.12.224/27,52.56.127.0/25,34.223.80.192/26,13.228.69.0/24,34.216.51.0/25,3.231.2.0/25,54.233.255.128/26,18.200.212.0/23,64.252.64.0/18,52.52.191.128/26,3.234.232.224/27,52.78.247.128/26,52.220.191.0/26,34.232.163.208/29
  server-tokens: "False"
  use-forwarded-headers: "true"
  use-proxy-protocol: "true"

And a simple ingress i'm using to test whitelisting

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx-ingress-v2"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/whitelist-source-range: 18.232.45.xxx
spec:
  tls:
    - secretName: tls-secret
      hosts:
        - "test.<redacted>"
  rules:
    - host: "test.<redacted>"
      http:
        paths:
          - path: /(.*)
            backend:
              serviceName: api
              servicePort: 80

What you expected to happen:

I expected whitelisting to work correctly based on the correct IP (18.232.45.xxx), but it seems to checking whitelist against the CloudFront IP (70.132.33.156) instead. If I update my upstream whitelist to allow 70.132.33.156 I no longer get a 403 error.

How to reproduce it:

1. Setup a CloudFront Origin pointing to an ELB. See screenshots below for exact settings.
   
   

2. Setup a new ingress that has IP whitelisting for your IP address.

Supplementary Links:

• https://medium.com/@pahud/get-real-ip-coming-via-aws-cloudfront-and-elb-to-nginx-d418e5070976
• AWS IP Ranges
• The discussion I had with @aledbf in Kubernetes Slack about this

Similar Issues:

• https://github.com/kubernetes/ingress-nginx/issues/4429
• https://github.com/kubernetes/ingress-nginx/issues/3172

/kind bug

[noqcks]

/assign @aledbf

[aledbf]

use-forwarded-headers: "true"
use-proxy-protocol: "true"

These two annotations don't work together. If you enable the two, only the use-proxy-protocol will be used.

Please add the annotation

nginx.ingress.kubernetes.io/server-snippet: |
  real_ip_header X-Forwarded-For;

[aledbf]

@noqcks it works?

[noqcks]

Yes! It worked! Thank you!

Tried on 0.21.0 (an older version we're running). If you had any idea what might make it work also on 0.21.0 that would be very helpful too.

[aledbf]

If you had any idea what might make it work also on 0.21.0 that would be very helpful too.

Unfortunately, in versions older than 0.26.0 is not that easy. You should try to update to a more recent version

[noqcks]

Yep, I figured that was the case. Thank you.