Ingress-nginx: SSL redirect broken when default-ssl-certificate is used

[bashofmann]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.): no

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.): tls redirect, ssl redirect

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

NGINX Ingress controller version: >= 0.26.0

Kubernetes version (use kubectl version): 1.15.3

Environment:

• Cloud provider or hardware configuration: https://metakube.syseleven.de/ on OpenStack
• OS (e.g. from /etc/os-release): Ubuntu 18.04.3 LTS
• Kernel (e.g. uname -a): 4.15.0-64-generic
• Install tools: Helm with stable/nginx-ingress
• Others:

What happened:
When you configure ingress-nginx to use a default-ssl-certificate for ingresses that do not contain a secretName in their TLS configuration, the certificate is used correctly but http requests are not redirect to https anymore.
This is still working on version 0.25.1.

What you expected to happen:
By default http requests to this ingress are redirected to https.

How to reproduce it (as minimally and precisely as possible):
Add an argument:

--default-ssl-certificate=namespace/wildcard-certificate

to the ingress nginx controller.

Create an ingress like:

kind: Ingress
apiVersion: networking.k8s.io/v1beta1
metadata:
  name: ingress-name
spec:
  tls:
    - hosts:
        - ingress.example.com
  rules:
    - host: ingress.example.com
      http:
        paths:
          - path: /
            backend:
              serviceName: backend-service
              servicePort: 80

TLS works then correctly but redirects do not.

If you force tls redirects with an annotation, the redirect works:

kind: Ingress
apiVersion: networking.k8s.io/v1beta1
metadata:
  name: ingress-name
  annotations:
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
  tls:
    - hosts:
        - ingress.example.com
  rules:
    - host: ingress.example.com
      http:
        paths:
          - path: /
            backend:
              serviceName: backend-service
              servicePort: 80

Anything else we need to know:

[mikehollinger]

I seem to be seeing the same thing on 0.26.1. When I roll back to an older ingress container (0.23) we see a proper redirect to https in our particular environment.

In my case, I have...

    nginx.ingress.kubernetes.io/proxy-redirect-from: ~*^https?://[^/]*(.*)$ $scheme://$host:$server_port$1
    nginx.ingress.kubernetes.io/ssl-redirect: "true"

If I add nginx.ingress.kubernetes.io/force-ssl-redirect: "true" to our ingress, things work as I expect. I'm passing --default-ssl-certificate=whatever to the ingress-controller on startup.

[kdomanski]

Bisecting between the versions shows that the redirect was broken in commit 80bd481abbedb2e4a44f47f66c312b37d385a0bb

[ElvinEfendi]

I think the problem is at line https://github.com/kubernetes/ingress-nginx/commit/80bd481abbedb2e4a44f47f66c312b37d385a0bb#diff-86ac9ff9d75a0c5005c116e766a6127dL1095. In that commit we deleted the line that was assigning default certificate to the server in case it missed one. @aledbf do you know why that line was deleted? Maybe we assumed that it is taken care of during TLS handshake but we forget to consider effect of that change on redirect. I'm not sure if simply bringing them back is good enough tho. We want redirection to happen only when the custom default certificate covers the domain right? But we don't check this when picking certificate for doing TLS handshake. So maybe bringing those lines back are just sufficient after all. What will happen is even if there's no custom certificate as soon as tls section is defined on the ingress we will redirect to HTTPs - to me that sounds clear enough.

[kdomanski]

TBH removing this looks like a debug measure that made it to the final commit, together with https://github.com/kubernetes/ingress-nginx/commit/80bd481abbedb2e4a44f47f66c312b37d385a0bb#diff-652deabba49a2a87d4c5a79700dfa1e3R1180

4816 fixes the redirect on my environments.

[aledbf]

do you know why that line was deleted?

I think it was to avoid duplication of the SSL certificate.

[ElvinEfendi]

I think it was to avoid duplication of the SSL certificate.

Duplication is not a concern anymore right because certificates are kept separately and indexed by UUID rather than hostname?

[olivierboudet]

I always have this issue after upgrade to 0.27.1, am I the only one ?

This is the ingress controller container spec :

    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --configmap=integration/nginx-configuration
        - --tcp-services-configmap=integration/tcp-services-internal
        - --udp-services-configmap=integration/udp-services-internal
        - --publish-service=$(POD_NAMESPACE)/ingress-nginx-integration-internal
        - --annotations-prefix=nginx.ingress.kubernetes.io
        - --ingress-class=nginx-integration-internal
        - --default-ssl-certificate=default/my-secret-tls
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.27.1

and this is my ingress :

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.allow-http: "false"
    kubernetes.io/ingress.class: nginx-integration-internal
    kubernetes.io/ingress.global-static-ip-name: integration-internal
    nginx.ingress.kubernetes.io/configuration-snippet: |
      more_set_input_headers "X-Correlation-ID: $request_id";
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  name: ingress
  namespace: integration
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: passwebv2-service
          servicePort: 80
        path: /(.*)
  tls:
  - hosts:
    - integration.mydomain.io

[kdomanski]

~The official image for v0.27.1 isn't built from the code tagged as v0.27.1 and thus doesn't contain the fix.~

EDIT: My mistake. What I wrote above is the case for v0.26.1

[tehho]

Is this still a thing?
We're running 0.30.0 in AKS, 1.15.8, and don't get ssl-redirect to work.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    certmanager.k8s.io/acme-dns01-provider: dns
    certmanager.k8s.io/cluster-issuer: letsencrypt-dns
    ingress.kubernetes.io/force-ssl-redirect: 'true'
    ingress.kubernetes.io/ssl-redirect: 'true'
    kubernetes.io/ingress.class: nginx-external
    kubernetes.io/tls-acme: 'true'
    nginx.ingress.kubernetes.io/force-ssl-redirect: 'true'
    nginx.ingress.kubernetes.io/server-snippet: |-
      http2_max_field_size 16k;
      http2_max_header_size 128k;
    nginx.ingress.kubernetes.io/ssl-redirect: 'true'
  labels:
    app.kubernetes.io/instance: applicaion
    app.kubernetes.io/name: applicaion
    team: team
    name: applicaion
  name: applicaion
  namespace: team-application
spec:
  rules:
    - host: applicaion.domain.com
      http:
        paths:
          - backend:
              serviceName: applicaion
              servicePort: 80
            path: /
  tls:
    - hosts:
        - applicaion.domain.com
      secretName: domain-tls

[jsalatiel]

Facing same problem with quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0

[prageethw]

jee after some trial got it working on latest, note this config does SSL offloading at ELB and redirect HTTP to https

helm install  stable/nginx-ingress --name nginx-ingress --namespace nginx-ingress \
            --version 1.36.0 \
            --set controller.publishService.enabled=true \
            --set controller.service.targetPorts.https=http \
            --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-proxy-protocol"=* \
            --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-backend-protocol"=http \
            --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-connection-idle-timeout"=3600 \
            --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-ssl-cert"=$AWS_SSL_CERT_ARN \
            --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-ssl-ports"=https \
            --set-string controller.config.ssl-redirect=true \
            --set-string controller.config.use-forwarded-headers=true \
            --set-string controller.config.use-proxy-protocol=false \
            --set-string controller.config.force-ssl-redirect=true \
            --set-string controller.config.hsts=true
kubectl -n nginx-ingress  rollout status deployment nginx-ingress-controller

[dguyhasnoname]

I was able to redirect after doing below settings in stable/nginx-ingress helm chart version 1.36.3:

  config:
    ssl-redirect: "true"
    use-forwarded-headers: "true"
    use-proxy-protocol: "false"
    hsts: "true"
  services:
      elbSslCert: "arn:aws:acm:us-east-1:xxxxxxxxx:certificate/xxxxxxxxx"
      annotations:
        nvt.mckinsey.cloud/firewall-operator-should-ignore: "true"
        service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
        service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
        service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '3600'
      targetPorts:
        https: http
        http: http

SSL termination is happening at ELB as suggested here:
https://github.com/helm/charts/tree/master/stable/nginx-ingress#aws-l7-elb-with-ssl-termination

In ingress-controller created ELB I configured 2 listeners:

1. HTTP(port 80) >> HTTP(nodeport) ACM not needed
2. HTTPS(port 443) >> HTTP(nodeport) with ACM

My service port and annotations looked like below after helm deployment:

    ports:
    - name: https
      nodePort: 31788
      port: 443
      protocol: TCP
      targetPort: 80
    - name: http
      nodePort: 31749
      port: 80
      protocol: TCP
      targetPort: http

    annotations:
      dns.alpha.kubernetes.io/external: kube-nginx-grafana.cloud
      external-dns.alpha.kubernetes.io/hostname: kube-nginx-grafana.cloud
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
      service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled: "true"
      service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
      service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-east-1:xxxxxxxx:certificate/xxxxxxxxxxxxxxxxxx
      service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: ELBSecurityPolicy-TLS-1-2-2017-01
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https



md5-1502009c677cb7e99ed36ccd5973bd91



apiVersion: v1
data:
  hsts: "true"
  proxy-buffer-size: 16k
  ssl-redirect: "true"
  use-forwarded-headers: "true"
  use-proxy-protocol: "false"
kind: ConfigMap