Ingress-nginx: Load Balancer (or external) SSL termination and HTTP -> HTTPS redirect support.

[hsyed]

We are using AWS ELB's and have SSL termination occur on the ELB's.

The ELB is requisitioned with the following service resource:

apiVersion: v1
kind: Service
metadata:
  name: ingress-external
  namespace: services
  annotations:
  # https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/aws.go
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm.......
spec:
  type: LoadBalancer
  selector:
    app: ingress.controller
  ports:
    - name: http
      port: 80
      targetPort: 80
    - name: https
      port: 443
      targetPort: 80

So HTTP and HTTPS (terminated as HTTP) go to port 80.

An ingress resources need to be TLS enabled for the redirects to work with an ingress controller -- but a valid TLS definition requires setting a certificate and we don't want the ingress controller doing SSL termination.

I can't find a way of performing a HTTP -> HTTPS redirect without providing a custom template. If I go for a custom template it will be a global configuration and ideally we'd like to have some resources respond to HTTP requests.

So I propose an annotation for the ingress resource that expresses the intent that a particular ingress resource is expecting traffic from an external SSL terminating source. Perhaps something along the lines of .../proxy-ssl-termination-redirect.

When this annotation is present I expect the following template to be used in the location blocks:

if ($http_x_forwarded_proto = 'http') {
    return 301 https://$host$request_uri;
}

Related Issues:

General purpose redirect proposal

[bprashanth]

This is useful when tiering loadbalancers, it's also why we have this image: https://github.com/kubernetes/contrib/blob/master/ingress/echoheaders-redirect/nginx.conf (but I guess you want the existing nginx ingress to adopt that config).

I think the general purpose redirect will surface as the outcome of discussion on both https://github.com/kubernetes/kubernetes/issues/28443 and https://groups.google.com/forum/#!topic/kubernetes-sig-network/tPOvADUSoqc

I believe disucssion on those will converge in the start of 2017. Can you wait till then?

[hsyed]

@bprashanth Yes 2017 is fine. I'll get a custom template going for now. I will give your image a look and see if I can adapt it. Thanks.

[hsyed]

The following worked the first time around without using the proxy protocol, I did have to use the tmpl file from the 0.8.3 image as the contrib repo did not have release tags for 0.8.3 and the tmpl found there was incompatible.

...
{{- range $location := $server.Locations }}
...
if ($http_x_forwarded_proto = "http") {
    return 301 https://$host$request_uri;
}
...

[voor]

We're currently using a slightly different configuration to support web sockets:

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  labels:
    k8s-addon: ingress-nginx.addons.k8s.io
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:...."
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: '443'
    # See https://github.com/kubernetes/kubernetes/issues/36845
spec:
  type: LoadBalancer
  selector:
    app: ingress-nginx
  ports:
  - name: http
    port: 80
    targetPort: http
  - name: https
    port: 443
    targetPort: http

And would also be interested in a best practices for how to handle a redirect from the ingress controller that has support for web sockets.

[voor]

Also wanted to note that with proxy protocol on, the $http_x_forwarded_proto value is just - and therefore not usable in this scenario.

[SomeoneWeird]

Any update on this?

[od0]

@SomeoneWeird the latest image (gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.2) works great!

[asmith60]

@od0 how does this work in the 0.9.0-beta.2 image? Was there an annotation/config added for per ingress resource HTTP --> HTTPS redirects? I cannot seem to find any mention of it in the changelog

[od0]

@asmith60 no annotation needed, I believe it defaults to https redirect now if TLS is defined for the ingress. For example, using kube-lego, something like this would work:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: foo
  annotations:
    kubernetes.io/tls-acme: "true"
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
  - secretName: foo-tls
    hosts:
    - foo.bar.com
  rules:
  - host: foo.bar.com
    http:
      paths:
      - path: /
        backend:
          serviceName: foo
          servicePort: 4444

[gregtap]

I am trying to use that on Google Cloud Engine.
Since everything seems fine from a conf perspective could it be an issue with outdated Ingress version on GCE?

Somehow as soon as I add kubernetes.io/ingress.class: "nginx" to my Ingress config it will stop working and routing properly giving 404s.

Working conf.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: chefclub-tools
  annotations:
    kubernetes.io/ingress.global-static-ip-name: "test-ip"
    ingress.kubernetes.io/ssl-redirect: "true" # Not working
spec:
  tls:
  - secretName: chefclub-tools-tls-secret
    hosts:
    - api.chefclub.tools
    - auto-liker.chefclub.tools
  rules:
  - host: api.chefclub.tools
    http:
      paths:
      - path: /*
        backend:
          serviceName: facebook-service
          servicePort: 80

Router not working

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: chefclub-tools
  annotations:
    kubernetes.io/ingress.class: "nginx" ############  added this line
    kubernetes.io/ingress.global-static-ip-name: "test-ip"
    ingress.kubernetes.io/ssl-redirect: "true"
spec:
  tls:
  - secretName: chefclub-tools-tls-secret
    hosts:
    - api.chefclub.tools
    - auto-liker.chefclub.tools
  rules:
  - host: api.chefclub.tools
    http:
      paths:
      - path: /*
        backend:
          serviceName: facebook-service
          servicePort: 80

> kubectl describe ing
Name:           chefclub-tools
Namespace:      default
Address:
Default backend:    default-http-backend:80 (10.16.2.21:8080)
TLS:
  chefclub-tools-tls-secret terminates api.chefclub.tools,auto-liker.chefclub.tools
Rules:
  Host              Path    Backends
  ----              ----    --------
  api.chefclub.tools
                    /*  facebook-service:80 (<none>)

Annotations:
  ssl-redirect: true
No events.

[aledbf]

@coulix the annotation ingress.kubernetes.io/ssl-redirect is not supported in the GCE ingress controller and the path /* should be / in the nginx ingress controller.

[aledbf]

@hsyed since 0.9-beta.3 is possible to add additional configuration to nginx using annotations.
In this case you just need to add:

  annotations:
    ingress.kubernetes.io/configuration-snippet: |
      if ($http_x_forwarded_proto != 'https') { 
        return 301 https://$host$request_uri;
      }

to the ingress rule.

[eduardobaitello]

@voor Have you found a solution in how to handle ssl-redirects with L4 ELB and proxy-protocol enabled?
I need to use websockets too, and I don't know how to handle this.

[voor]

Yes,

It's not pretty, though.

I stand up a very simple nginx configuration on another port (8081 or whatever) and just do a typical nginx SSL redirect:

    # redirect port, just redirects to same endpoint on HTTPS, used since AWS ELB is always on HTTP.
    server {
      listen 8888 proxy_protocol;
      return 301 https://$host$request_uri;
    }

Just override the typical nginx.tmpl for the ingress configuration

kubectl --namespace ${NAMESPACE} create configmap ingress-nginx-template --from-file=nginx.tmpl=${WHEREEVER}/nginx.tmpl

[eduardobaitello]

@voor
Thanks, that worked!

I just made a little different, appending the new _server_ inside the _http_ context using the http-snippet.

Something like this:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app: ingress-nginx
  name: nginx-ingress-configuration
  namespace: ingresns
data:
  ssl-redirect: "false"
  hsts: "true"
  server-tokens: "false"
  http-snippet: |
    server {
      listen 8000 proxy_protocol;
      server_tokens off;
      return 301 https://$host$request_uri;
    }

This prevents overriding the original nginx.tmpl, ensuring more compatibility in case of upgrade the Nginx Ingress version.

[marwatk]

For anyone arriving here from Google like I did, in current versions (I'm using 0.19) simply adding nginx.ingress.kubernetes.io/force-ssl-redirect: "true" to your annotations will enable redirect even with --enable-ssl-passthrough and nginx.ingress.kubernetes.io/ssl-passthrough: "true".

[voor]

Nice, but will that handle the use case of proxy_protocol as well?

[dthomason]

I second voor's question. In order to enable websockets the controller must be set to L4 and this breaks the force ssl redirect. Please let me know if anyone has come to a solution for this.