Ingress-nginx: permanent redirect with nginx and lets encrypt

Created on 9 Dec 2017 · 18Comments · Source: kubernetes/ingress-nginx

NGINX Ingress controller version:
quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0

Kubernetes version (use kubectl version):
1.8.4

Environment:

Cloud provider or hardware configuration:AWS
OS (e.g. from /etc/os-release):Debian GNU/Linux 8 (jessie)

What happened:
Created cluster using kops

Created nginx ingress controller and lets encrypt. Created simple echo-headers service with ingress.

What you expected to happen:
Redirect to https and return response from pod

How to reproduce it (as minimally and precisely as possible):
Create nginx ingress
Create kube-lego
Create deployment and ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: echo-tls
  annotations:
    kubernetes.io/ingress.class: "nginx"
    kubernetes.io/tls-acme: "true"
    ingress.kubernetes.io/configuration-snippet: |
     more_set_headers "Request-Id: $request_id";
    ingress.kubernetes.io/ssl-redirect: "true"
    ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
  tls:
   - hosts:
     - echo-tls.shanelee.xyz
     secretName: echoserver-tls
  rules:
  - host: echo-tls.shanelee.xyz
    http:
      paths:
      - path: /
        backend:
          serviceName: http-svc
          servicePort: 80

Deployment

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: http-svc
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: http-svc
    spec:
      containers:
      - name: http-svc
        image: gcr.io/google_containers/echoserver:1.8
        ports:
        - containerPort: 8080

---

apiVersion: v1
kind: Service
metadata:
  name: http-svc
  labels:
    app: http-svc
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
    name: http
  selector:
    app: http-svc

Logs kube-lego

time="2017-12-09T10:46:16Z" level=info msg="Attempting to create new secret" context=secret name=kube-lego-account namespace=kube-lego 
time="2017-12-09T10:46:16Z" level=info msg="Secret successfully stored" context=secret name=kube-lego-account namespace=kube-lego 
time="2017-12-09T10:47:22Z" level=info msg="authorization successful" context=acme domain=echo-tls.shanelee.xyz 
time="2017-12-09T10:47:24Z" level=info msg="successfully got certificate: domains=[echo-tls.shanelee.xyz]

Logs nginx

13.210.28.247 - [13.210.28.247] - - [09/Dec/2017:10:47:20 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 200 16 "-" "Go-http-client/1.1" 241 0.003 [kube-lego-kube-lego-nginx-8080] 100.96.4.5:8080 16 0.003 200
66.133.109.36 - [66.133.109.36] - - [09/Dec/2017:10:47:21 +0000] "GET /.well-known/acme-challenge/xxx HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 357 0.003 [kube-lego-kube-lego-nginx-8080] 100.96.4.5:8080 87 0.003 200
I1209 10:47:29.455844       5 backend_ssl.go:64] adding secret default/echoserver-tls to the local store
I1209 10:47:29.456296       5 controller.go:211] backend reload required
I1209 10:47:29.631193       5 controller.go:220] ingress backend successfully reloaded...
I1209 10:48:30.645007       5 backend_ssl.go:160] updating local copy of ssl certificate default/echoserver-tls with missing intermediate CA certs
I1209 10:48:30.645478       5 controller.go:211] backend reload required
I1209 10:48:30.777330       5 controller.go:220] ingress backend successfully reloaded...

$ curl echo-tls.shanelee.xyz
<html>
<head><title>308 Permanent Redirect</title></head>
<body bgcolor="white">
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx/1.13.7</center>
</body>
</html>

Anything else we need to know:
I have done this many times before with aws cert manager. First time using lets encrypt and is something im missing.

Thanks,
Shane.

Source

shavo007

Most helpful comment

Hey @shavo007, did you by any chance solve this already?

I am having the exact same problem. I configured kube lego to get me the cert -> which it did successfully. The Cert is there as a secret, it is configured in the ingress config properly.
However, I do not get these valid certificates back from nginx. (0.9.0) And I see this line in the ingress logs:
backend_ssl.go:64] adding secret namespace/my-cert-acme to the local store (...) backend_ssl.go:160] updating local copy of ssl certificate namespace/my-cert-acme with missing intermediate CA certs

DocValerian on 14 Feb 2018

👍4

All 18 comments

Your ingress annotations namespace is wrong. After 0.9.0-beta.18 this changed from ingress.kubernetes.io to nginx.ingress.kubernetes.io.

macropin on 11 Dec 2017

👍3

thanks for response @macropin

unfortunately, that did not work either
ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: echo-tls
  annotations:
    kubernetes.io/ingress.class: "nginx"
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/configuration-snippet: |
     more_set_headers "Request-Id: $request_id";
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
  tls:
   - hosts:
     - echo-tls.shanelee.xyz
     secretName: echoserver-tls
  rules:
  - host: echo-tls.shanelee.xyz
    http:
      paths:
      - path: /
        backend:
          serviceName: http-svc
          servicePort: 80

logs

60.241.90.69 - [60.241.90.69] - - [11/Dec/2017:11:09:20 +0000] "GET / HTTP/1.1" 308 187 "" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" 250 0.000 [default-http-svc-80]

no idea whats wrong here.

shavo007 on 11 Dec 2017

nginx deployment

Name:                   nginx-ingress-controller
Namespace:              ingress-nginx
CreationTimestamp:      Mon, 11 Dec 2017 21:40:57 +1100
Labels:                 app=ingress-nginx
Annotations:            deployment.kubernetes.io/revision=2
                        kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"extensions/v1beta1","kind":"Deployment","metadata":{"annotations":{},"name":"nginx-ingress-controller","namespace":"ingress-nginx"},"spe...
Selector:               app=ingress-nginx
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 1 max surge
Pod Template:
  Labels:  app=ingress-nginx
  Containers:
   nginx-ingress-controller:
    Image:  quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0
    Ports:  80/TCP, 443/TCP
    Args:
      /nginx-ingress-controller
      --default-backend-service=$(POD_NAMESPACE)/default-http-backend
      --configmap=$(POD_NAMESPACE)/nginx-configuration
      --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
      --udp-services-configmap=$(POD_NAMESPACE)/udp-services
      --publish-service=$(POD_NAMESPACE)/ingress-nginx
      --annotations-prefix=nginx.ingress.kubernetes.io
    Environment:
      POD_NAME:        (v1:metadata.name)
      POD_NAMESPACE:   (v1:metadata.namespace)
    Mounts:           <none>
  Volumes:            <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   nginx-ingress-controller-dd8d87997 (1/1 replicas created)

shavo007 on 11 Dec 2017

Isn't it redirecting because the echo-tls Ingress resource has been configured in the yaml to make sure redirects are forced?

    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"

Shouldn't they be set to false if you don't want the redirect?

My problem was I still had ingress.kubernetes.io/ssl-redirect: "false" rather than nginx.ingress.kubernetes.io/ssl-redirect: "false". Once I made that change, the redirect to https stopped

markfermor on 11 Dec 2017

I have used these annotations before with aws cert manager and worked fine.

This just redirects to https from http. TLS termination still happens.

I dont believe this is the problem.

shavo007 on 12 Dec 2017

Closing. You cannot use nginx.ingress.kubernetes.io/force-ssl-redirect: "true" when there's no SSL certificate configured becuase it will be redirected to the default backend.
The annotation nginx.ingress.kubernetes.io/ssl-redirect: "true" is not required because the default behavior is to redirect to SSL when a certificate is configured.

aledbf on 2 Jan 2018

308 redirection is the default behavior but you can change the redirection code to whatever make sense to your stack.
I know that some of our clients are still using IE9-11 and the 308 redirection seems to not be supported here (maybe your problem :v: ).

Justkant on 2 Jan 2018

@aledbf a bit trigger happy closing issues mate!

I'm using let's encrypt with kube Lego as stated above. You can see in the logs it has successfully received the certificate. So there is a certificate configured. Why am I getting redirect then?

shavo007 on 2 Jan 2018

@shavo007 what do you mean?

If you use nginx.ingress.kubernetes.io/force-ssl-redirect: "true" and the certificate is loaded in nginx any HTTP request will be redirected to HTTPS.
https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/tls.md#server-side-https-enforcement

aledbf on 2 Jan 2018

I understand the annotation...

My question is I have a certificate configured with let's encrypt.

It should redirect from http to https but it's not. I'm always getting redirect.

It could simply be an error on my part with the configuration. That is why I want to confirm.

shavo007 on 2 Jan 2018

@shavo007 have you only tried with curl using the non https url ?

Justkant on 2 Jan 2018

@shavo007 please use curl -v -k -L echo-tls.shanelee.xyz to see the response from nginx

aledbf on 2 Jan 2018

@justkant nah I tried in browser aswell.

I'm on holidays at mo. What I might try and do next time is copy the nginx.conf and show you.

Apart from that the manifests are like the above.

shavo007 on 2 Jan 2018

Hey @shavo007, did you by any chance solve this already?

DocValerian on 14 Feb 2018

👍4

Hey @DocValerian, I'm running into the problem you described. Wondering there is any solution found?

suman-ganta on 14 Jun 2018

Hi @suman-ganta, sorry, but I have no solution.
I think, back then, we worked around it by copy/pasting the still valid certs from our old blue/green cluster and the issue disappeared. So maybe kube-lego related... However, we are in the process of migrating to the cert-manager project anyways and haven't seen this issue in the meantime.

DocValerian on 14 Jun 2018

same issue here; do we have any documents about the requirements of the .crt and key?

jack4it on 16 Jun 2018

Same Issue here. I was able to run cert-manager (letsencrypt-prod) this 2 moths back. This version previously worked for me when I installed with helm. nginx-ingress-0.23.0 0.15.0 (app-version). The current version is throwing the above mentioned errors.