Kubernetes-ingress: HTTP > HTTPS not working with NLB with TLS termination and Proxy Protocol V2

[sgasquet]

Describe the bug
Hi there.
We are facing a weird issue regarding HTTPS redirection with this ingress on this context:

• Load-Balancer : NLB with TLS termination / Certificate in AWS cert manager
• Deployment via HELM Chart
• ProxyProtocol V2 enable in chart and target groups

To Reproduce
_Our configuration in values.yaml file :_

controller:
kind: daemonset
config:
entries:
hsts: "True"
proxy-protocol: "True"
real-ip-header: proxy_protocol
redirect-to-https: "True"
server-tokens: "False"
set-real-ip-from: 0.0.0.0/0
service:
type: LoadBalancer
externalTrafficPolicy: Cluster
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
service.beta.kubernetes.io/aws-load-balancer-ssl-cert-ports: https
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: "ELBSecurityPolicy-TLS-1-2-2017-01"
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:xxxxxxxxxxxxxxxxxx
httpPort:
enable: true
port: 80
targetPort: http
httpsPort:
enable: true
port: 443
targetPort: http

Expected behavior
In ingress of a web backend application, HTTPS redirect should occur with this kind of annotations :
nginx.org/redirect-to-https
ingress.kubernetes.io/ssl-redirect: "True"
But none of this works actually.

Debug
To debug this behavior, we played with tcpdump inside a POD and see what Headers are saw.
First thing, the header X-FORWARDED-PROTO is always the same, reaching the endpoint in HTTP or HTTPS won't change that. And still no redirection is done.
Here are the headers we can see :
X-Real-IP: X.X.X.X OK
X-Forwarded-For: X.X.X.X OK
X-Forwarded-Host: host.blabla OK
X-Forwarded-Port: 80 (always)
X-Forwarded-Proto: HTTPS (always)

So all forwarded headers are good but not X-Forwarded-Port and X-Forwarded-Proto which never change.

Is anything we are doing wrong in the configuration ? Is it related to the ProxyProtocol which forwards the protocol defined here "aws-load-balancer-backend-protocol: http" and not the real protocol used by client ?
Any help appreciated :)

Thanks for your help !

[pleshakov]

Hi @sgasquet

It should be possible to support an SSL redirect based on $proxy_protocol_server_port variable, assuming that when NLB terminates SSL, the value is 443, and 80 otherwise.

Would it be possible if you try use a custom Ingress template for the Ingress Controller with the following change and let us know if it works?

    if ($http_x_forwarded_proto = 'http') {
        return 301 https://$host$request_uri;
    }

->

    if ($proxy_protocol_server_port != '443') {
        return 301 https://$host$request_uri;
    }

In the ConfigMap, ssl-redirect should be "false" and redirect-to-https - "true".

[sgasquet]

Hey @pleshakov !

It's working !! The second problem you solve in two days :D.
So the fix in my helm configuration / values.yaml is "simply":

    ingress-template: |
        {{- if $server.RedirectToHTTPS}}
        if ($proxy_protocol_server_port != '443') {
          return 301 https://$host$request_uri;
        }
        {{- end}}

Now everything is redirected smoothly thanks again for your help !
As TLS termination & Proxy Protocol V2 are very new with NLB i guess this will be useful for others soon or later.

Have a nice day !

[sgasquet]

Hi again !

So ! The settings are working only on reload with nginx, which keeps the entire nginx.ingress template in memory.
But after reboot the entire template file is reduced to the short code described in my last post (and obviously Nginx wont start).

Is there a way to have this incremental change in config ? Or do we need to apply the whole NGINX INGRESS TEMPLATE file ?

Thanks again for your help :).

[pleshakov]

Hi @sgasquet

Yep, the whole template needs to be applied. Unfortunately, it is not possible to modify a part of the template.

[sgasquet]

Thanks for your help :)

[niraj1234567890]

@sgasquet Hey I am facing the same issue as you were facing on the redirection from http to https where I am making ssl termination at the load balancer level. I tried to add the custom ingress template in the config-map as seen below:

apiVersion: networking.k8s.io/v1beta1
kind: IngressClass
metadata:
  name: nginx
spec:
  controller: nginx.org/ingress-controller
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-config
  namespace: nginx-ingress
data:
  proxy-protocol: "True"
  real-ip-header: "proxy_protocol"
  set-real-ip-from: "0.0.0.0/0"
  redirect-to-https: "True"
  ingress-template: |
        {{- if $server.RedirectToHTTPS}}
        if ($proxy_protocol_server_port != '443') {
          return 301 https://$host$request_uri;
        }
        {{- end}}

My other config's are same as your's (BUT not using helm-chart). When I am implementing the above configmap yaml, I am getting this error in the logs:
Error: variable server is not defiend
I suppose there must be something on the custom ingress template I am missing. Can you please suggest if there is anything you can think of on the above error.

FYI: Http and Https, both are working fine.

[pleshakov]

@niraj1234567890
this is not obvious, but the entire template needs to be added in the ingress-template, not just the excerpt.

[niraj1234567890]

@pleshakov It worked. It came out that I was not passing the entire nginx-ingress.teml file into the configmap which l did it eventually and now it's working like a charm.