Ingress-nginx: New TLS certificates are not reloaded by nginx-ingress-controller

Created on 11 Jul 2017  路  23Comments  路  Source: kubernetes/ingress-nginx

I'll be glad if #879 is reopened, since the issue persists in our infrastructure -- still following up on https://github.com/kubernetes/ingress/issues/879

Using kube-lego:0.1.5 and nginx-ingress-controller:0.9-beta.10.

Given it's a brand a new installation, kube-lego was able to create the first certificate for ingresses. Even though certificates were successfully issued, nginx-ingress-controller kept presenting the Kubernetes' default fake Acme certs.

172.31.134.112 - [172.31.134.112] - - [11/Jul/2017:14:48:40 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 358 0.000 [-] - - - -
172.31.134.112 - [172.31.134.112] - - [11/Jul/2017:14:48:40 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 129 0.001 [default-kube-lego-nginx-8080] 127.0.0.1:8181 213 0.001 503
172.31.134.112 - [172.31.134.112] - - [11/Jul/2017:14:48:41 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 360 0.000 [-] - - - -
172.31.134.112 - [172.31.134.112] - - [11/Jul/2017:14:48:41 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 130 0.000 [default-kube-lego-nginx-8080] 127.0.0.1:8181 213 0.000 503
172.31.134.112 - [172.31.134.112] - - [11/Jul/2017:14:48:41 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 358 0.000 [-] - - - -
172.31.134.112 - [172.31.134.112] - - [11/Jul/2017:14:48:41 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 129 0.000 [default-kube-lego-nginx-8080] 127.0.0.1:8181 213 0.000 503
172.31.134.112 - [172.31.134.112] - - [11/Jul/2017:14:48:43 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 360 0.000 [-] - - - -
172.31.134.112 - [172.31.134.112] - - [11/Jul/2017:14:48:43 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 130 0.000 [default-kube-lego-nginx-8080] 127.0.0.1:8181 213 0.000 503
W0711 14:48:44.381157      13 controller.go:826] error obtaining service endpoints: service default/servicoc does not exist
I0711 14:48:44.381428      13 controller.go:1060] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0711 14:48:44.381447      13 controller.go:1060] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0711 14:48:44.382332      13 controller.go:427] backend reload required
I0711 14:48:44.434462      13 controller.go:437] ingress backend successfully reloaded...
172.31.134.112 - [172.31.134.112] - - [11/Jul/2017:14:48:44 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 200 16 "-" "Go-http-client/1.1" 129 0.002 [default-kube-lego-nginx-8080] 10.40.0.3:8080 16 0.002 200
66.133.109.36 - [66.133.109.36] - - [11/Jul/2017:14:48:46 +0000] "GET /.well-known/acme-challenge/mvu1i1XKz4WGv8ZRi35H2FPJY9Py7A9lW07YeYhav2Y HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 264 0.001 [default-kube-lego-nginx-8080] 10.40.0.3:8080 87 0.001 200
66.133.109.36 - [66.133.109.36] - - [11/Jul/2017:14:48:46 +0000] "GET /.well-known/acme-challenge/nId5mR6oyV9_EZh1WclOl61clL8RtYlFyxQ4wEJiBoE HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 265 0.000 [default-kube-lego-nginx-8080] 10.40.0.3:8080 87 0.000 200

Only after a restart of the nginx pods, the new certificate was loaded.

As a side note, this is reproducible using LE's staging environment. Every time we clean up and fire up related pods and services, nginx-ingress-controller does not reload the newly created cert.

I0711 15:10:19.073608      13 controller.go:1060] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0711 15:10:19.073625      13 controller.go:1060] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
W0711 15:10:22.999008      13 controller.go:826] error obtaining service endpoints: service default/servicoc does not exist
I0711 15:10:22.999277      13 controller.go:1060] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0711 15:10:22.999289      13 controller.go:1060] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0711 15:10:23.000199      13 controller.go:427] backend reload required
I0711 15:10:23.052018      13 controller.go:437] ingress backend successfully reloaded...
172.31.134.111 - [172.31.134.111] - - [11/Jul/2017:15:10:24 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 200 16 "-" "Go-http-client/1.1" 129 0.002 [default-kube-lego-nginx-8080] 10.46.0.5:8080 16 0.002 200
66.133.109.36 - [66.133.109.36] - - [11/Jul/2017:15:10:25 +0000] "GET /.well-known/acme-challenge/t6GU4Wpl5WxionquBGLxS9K3nj9Dr2BOQ-FLUNaT3nY HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 264 0.001 [default-kube-lego-nginx-8080] 10.46.0.5:8080 87 0.001 200
W0711 15:10:26.086186      13 backend_ssl.go:46] error obtaining PEM from secret default/tls.lab.tjpr.net: secret named default/tls.lab.tjpr.net does not exist
I0711 15:10:36.087222      13 backend_ssl.go:63] adding secret default/tls.lab.tjpr.net to the local store

My steps to reproduce this are:

  • Clean up the environment: delete nginx-ingress-controller pods, kube-lego pods, and any kube-lego secrets that were automatically created.
  • Fire up nginx-ingress-controller
  • Fire up kube-lego
picture juliohm1978

Most helpful comment

Following up.

We are able to force the nginx-ingress-controller to reload by applying a dummy patch to related ingresses.

kubectl patch ingress myingress -p '{"metadata":{"labels":{"dummy":"some_unique_new_value"}}}'

For now, we might be able to workaround this by having a job that applies this patch every week or so.

picture juliohm1978 on 11 Jul 2017
🎉4

All 23 comments

Following up.

We are able to force the nginx-ingress-controller to reload by applying a dummy patch to related ingresses.

kubectl patch ingress myingress -p '{"metadata":{"labels":{"dummy":"some_unique_new_value"}}}'

For now, we might be able to workaround this by having a job that applies this patch every week or so.

juliohm1978 picture juliohm1978 on 11 Jul 2017
🎉4

Hi, maybe it's related to the issue I have reported yesterday, nginx is not reloaded after a new ingress is created - https://github.com/kubernetes/ingress/issues/945.

Do you see the same problem, or creating and updating ingress resources works for you and nginx is always properly reloaded after these events?

stibi picture stibi on 11 Jul 2017

@stibi: I don't have the same problem as you have with ingress updates. For that matter, any updates to the ingress causes nginx-ingress-controller to reload.

In my case, I might be able to schedule that patch command to run every week to force ingress reloads. This works around letsencrypt certs not being reloaded.

juliohm1978 picture juliohm1978 on 11 Jul 2017

ok, thanks for the info. I wonder what I聽have wrong on my side, in case it works for you.

stibi picture stibi on 11 Jul 2017

@juliohm1978 please update the image to gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.11

aledbf picture aledbf on 12 Jul 2017

Closing. Please reopen if the issue persists in 0.9.0-beta.11

aledbf picture aledbf on 16 Jul 2017

This issue still exists.
I used 0.9.0-beta.11.
First I create an Ingress wich need a secret. Suppose its domain is www.mytest.com
Wait 5 seconds ...
When I curl https://www.mytest.com, I get 'Default backent - 404'
Wait for another 5 seconds ...
I create the secret which contains the key/certs for www.mytest.com .
Wait for 1 minitute, still get 'Default backent - 404'

jerryjxj picture jerryjxj on 18 Jul 2017

@jerryjxj please post the ingress logs

aledbf picture aledbf on 18 Jul 2017

@aledbf , this issue can't be produced everytime. I just tested again with more steps
Steps:

  • Create ingress, then create secret. Checked the desired domain [https://vivianxh.club], status 200, OK
  • Delete secret. The domain falled back to default service with status 404. This correct.
  • Recreate the secret. wait for a long time. Still default service status 404. NOT the desired status 200.
I0718 14:57:31.607514       7 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"za", Name:"vivianxh.club", UID:"6ccd1500-6bc9-11e7-b195-000c2956f9bb", APIVersion:"extensions", ResourceVersion:"298467", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress za/vivianxh.club
I0718 14:57:31.614180       7 controller.go:1052] ssl certificate "za/d82503ee519963b57653fb6d642470181434af37b31f587a7ba7190829124719" does not exist in local store
I0718 14:57:31.628783       7 controller.go:428] backend reload required
W0718 14:57:31.637173       7 backend_ssl.go:46] error obtaining PEM from secret za/d82503ee519963b57653fb6d642470181434af37b31f587a7ba7190829124719: secret named za/d82503ee519963b57653fb6d642470181434af37b31f587a7ba7190829124719 does not exist
I0718 14:57:31.723162       7 controller.go:438] ingress backend successfully reloaded...
W0718 14:57:41.643779       7 backend_ssl.go:46] error obtaining PEM from secret za/d82503ee519963b57653fb6d642470181434af37b31f587a7ba7190829124719: secret named za/d82503ee519963b57653fb6d642470181434af37b31f587a7ba7190829124719 does not exist
W0718 14:57:51.650473       7 backend_ssl.go:46] error obtaining PEM from secret za/d82503ee519963b57653fb6d642470181434af37b31f587a7ba7190829124719: secret named za/d82503ee519963b57653fb6d642470181434af37b31f587a7ba7190829124719 does not exist
I0718 14:57:57.686288       7 status.go:310] updating Ingress za/vivianxh.club status to [{192.168.30.20 }]
I0718 14:57:57.689805       7 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"za", Name:"vivianxh.club", UID:"6ccd1500-6bc9-11e7-b195-000c2956f9bb", APIVersion:"extensions", ResourceVersion:"298524", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress za/vivianxh.club
I0718 14:57:57.694116       7 controller.go:1052] ssl certificate "za/d82503ee519963b57653fb6d642470181434af37b31f587a7ba7190829124719" does not exist in local store
W0718 14:58:01.656416       7 backend_ssl.go:46] error obtaining PEM from secret za/d82503ee519963b57653fb6d642470181434af37b31f587a7ba7190829124719: secret named za/d82503ee519963b57653fb6d642470181434af37b31f587a7ba7190829124719 does not exist
I0718 14:58:08.441369       7 controller.go:1052] ssl certificate "za/d82503ee519963b57653fb6d642470181434af37b31f587a7ba7190829124719" does not exist in local store
I0718 14:58:11.664422       7 backend_ssl.go:64] adding secret za/d82503ee519963b57653fb6d642470181434af37b31f587a7ba7190829124719 to the local store
I0718 14:58:34.750608       7 controller.go:428] backend reload required
I0718 14:58:34.845970       7 controller.go:438] ingress backend successfully reloaded...
127.0.0.1 - [127.0.0.1] - - [18/Jul/2017:14:58:38 +0000] "GET / HTTP/1.1" 200 51566 "-" "curl/7.35.0" 77 0.361 [sticky-za-eshop-443] 172.16.16.15:443 51477 0.361 200
I0718 14:59:07.289235       7 controller.go:1052] ssl certificate "za/d82503ee519963b57653fb6d642470181434af37b31f587a7ba7190829124719" does not exist in local store
I0718 14:59:07.302350       7 controller.go:428] backend reload required
I0718 14:59:07.393042       7 controller.go:438] ingress backend successfully reloaded...
127.0.0.1 - [127.0.0.1] - - [18/Jul/2017:14:59:22 +0000] "GET / HTTP/1.1" 404 21 "-" "curl/7.35.0" 77 0.002 [upstream-default-backend] 172.16.101.2:8080 21 0.002 404
I0718 14:59:34.727094       7 controller.go:1052] ssl certificate "za/d82503ee519963b57653fb6d642470181434af37b31f587a7ba7190829124719" does not exist in local store
I0718 15:00:04.732507       7 controller.go:1052] ssl certificate "za/d82503ee519963b57653fb6d642470181434af37b31f587a7ba7190829124719" does not exist in local store
I0718 15:00:37.288009       7 controller.go:1052] ssl certificate "za/d82503ee519963b57653fb6d642470181434af37b31f587a7ba7190829124719" does not exist in local store
I0718 15:01:07.341200       7 controller.go:1052] ssl certificate "za/d82503ee519963b57653fb6d642470181434af37b31f587a7ba7190829124719" does not exist in local store
I0718 15:03:07.789636       7 controller.go:1052] ssl certificate "za/d82503ee519963b57653fb6d642470181434af37b31f587a7ba7190829124719" does not exist in local store
I0718 15:04:05.387438       7 controller.go:1052] ssl certificate "za/d82503ee519963b57653fb6d642470181434af37b31f587a7ba7190829124719" does not exist in local store
I0718 15:04:37.289995       7 controller.go:1052] ssl certificate "za/d82503ee519963b57653fb6d642470181434af37b31f587a7ba7190829124719" does not exist in local store
I0718 15:05:07.540572       7 controller.go:1052] ssl certificate "za/d82503ee519963b57653fb6d642470181434af37b31f587a7ba7190829124719" does not exist in local store
jerryjxj picture jerryjxj on 18 Jul 2017

@jerryjxj from the logs there's no secret with name za/d82503ee519963b57653fb6d642470181434af37b31f587a7ba7190829124719

W0718 14:58:01.656416 7 backend_ssl.go:46] error obtaining PEM from secret za/d82503ee519963b57653fb6d642470181434af37b31f587a7ba7190829124719: secret named za/d82503ee519963b57653fb6d642470181434af37b31f587a7ba7190829124719 does not exist

aledbf picture aledbf on 18 Jul 2017

@aledbf
As stated in step 3, I created the secret again. But it did not take effect.
BTW.
I'm studying the code of ingress controller. I found periodical check is used. For example, using endless loop to check secret every 10 seconds. Why do not rely on the K8S events watching API, which already used in Ingress controller?

jerryjxj picture jerryjxj on 18 Jul 2017
👍2

Hello @jerryjxj, maybe you are dealing with a similar problem as I had, take a look on my patch, maybe the problem is in similar place (https://github.com/kubernetes/ingress/pull/973)

stibi picture stibi on 18 Jul 2017

Why do not rely on the K8S events watching API, which already used in Ingress controller?

@jerryjxj we are using the watch from k8s. The periodic check is to dump (to a file) just the secrets that are referenced in ingress rules to disk and not ALL the secrets being watched

aledbf picture aledbf on 18 Jul 2017

@stibi your patch is already included in beta.11

aledbf picture aledbf on 18 Jul 2017

@aledbf yes I know, I have deployed it already on my cluster鈥ut I thought that maybe there is a similar problem with checking if the configuration has changed鈥s I'm looking on the logs provided here, there is a reload triggered, so the problem is something else most probably, sorry for confusion

stibi picture stibi on 18 Jul 2017

@jerryjxj please update the image to quay.io/aledbf/nginx-ingress-controller:0.169

aledbf picture aledbf on 18 Jul 2017

I was finally able to test this again.

0.9.0-beta.11 does not reload, same issue.

I0718 21:47:01.045403      13 launch.go:105] &{NGINX 0.9.0-beta.11 git-a3131c5 https://github.com/kubernetes/ingress}
I0718 21:47:01.045436      13 launch.go:108] Watching for ingress class: nginx
I0718 21:47:01.045587      13 launch.go:262] Creating API server client for https://10.96.0.1:443
I0718 21:47:01.055536      13 launch.go:124] validated default/default-http-backend as the default backend
I0718 21:47:01.061068      13 controller.go:1190] starting Ingress controller
I0718 21:47:01.063727      13 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"myingress", UID:"5910a23d-6678-11e7-8847-005056a64cce", APIVersion:"extensions", ResourceVersion:"827637", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/myingress
I0718 21:47:01.063762      13 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"kube-lego-nginx", UID:"226d6ecc-6600-11e7-8847-005056a64cce", APIVersion:"extensions", ResourceVersion:"827629", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/kube-lego-nginx
I0718 21:47:01.161461      13 leaderelection.go:203] attempting to acquire leader lease...
W0718 21:47:01.161521      13 backend_ssl.go:46] error obtaining PEM from secret default/tls.lab.tjpr.net: secret named default/tls.lab.tjpr.net does not exist
W0718 21:47:01.161757      13 controller.go:886] service default/kube-lego-nginx does not have any active endpoints
I0718 21:47:01.428599      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:01.428682      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:01.429683      13 controller.go:428] backend reload required
I0718 21:47:01.429809      13 metrics.go:34] changing prometheus collector from  to default
I0718 21:47:01.808989      13 controller.go:438] ingress backend successfully reloaded...
W0718 21:47:04.389400      13 controller.go:886] service default/kube-lego-nginx does not have any active endpoints
I0718 21:47:04.389747      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:04.389765      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
W0718 21:47:07.722759      13 controller.go:886] service default/kube-lego-nginx does not have any active endpoints
I0718 21:47:07.723099      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:07.723118      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
W0718 21:47:11.056056      13 controller.go:886] service default/kube-lego-nginx does not have any active endpoints
I0718 21:47:11.056414      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:11.056431      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
W0718 21:47:11.161688      13 backend_ssl.go:46] error obtaining PEM from secret default/tls.lab.tjpr.net: secret named default/tls.lab.tjpr.net does not exist
W0718 21:47:14.389381      13 controller.go:886] service default/kube-lego-nginx does not have any active endpoints
I0718 21:47:14.389709      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:14.389723      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
W0718 21:47:17.722726      13 controller.go:886] service default/kube-lego-nginx does not have any active endpoints
I0718 21:47:17.723112      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:17.723129      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
W0718 21:47:21.056072      13 controller.go:886] service default/kube-lego-nginx does not have any active endpoints
I0718 21:47:21.056458      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:21.056473      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
W0718 21:47:21.161835      13 backend_ssl.go:46] error obtaining PEM from secret default/tls.lab.tjpr.net: secret named default/tls.lab.tjpr.net does not exist
W0718 21:47:24.389370      13 controller.go:886] service default/kube-lego-nginx does not have any active endpoints
I0718 21:47:24.389704      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:24.389723      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
W0718 21:47:27.722784      13 controller.go:886] service default/kube-lego-nginx does not have any active endpoints
I0718 21:47:27.723255      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:27.723282      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
W0718 21:47:31.056108      13 controller.go:886] service default/kube-lego-nginx does not have any active endpoints
I0718 21:47:31.056590      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:31.056609      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
W0718 21:47:31.161981      13 backend_ssl.go:46] error obtaining PEM from secret default/tls.lab.tjpr.net: secret named default/tls.lab.tjpr.net does not exist
W0718 21:47:34.389529      13 controller.go:886] service default/kube-lego-nginx does not have any active endpoints
I0718 21:47:34.390016      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:34.390038      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:40.039576      13 leaderelection.go:213] successfully acquired lease default/ingress-controller-leader-nginx
W0718 21:47:41.162154      13 backend_ssl.go:46] error obtaining PEM from secret default/tls.lab.tjpr.net: secret named default/tls.lab.tjpr.net does not exist
W0718 21:47:51.162299      13 backend_ssl.go:46] error obtaining PEM from secret default/tls.lab.tjpr.net: secret named default/tls.lab.tjpr.net does not exist
W0718 21:47:59.158749      13 controller.go:886] service default/kube-lego-nginx does not have any active endpoints
I0718 21:47:59.160353      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:59.160373      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
unexpected stream type ""root@infra00-lab:~# k logs -f nginx-ingress-controller-vqwdg
[dumb-init] Unable to detach from controlling tty (errno=25 Inappropriate ioctl for device).
[dumb-init] Child spawned with PID 13.
[dumb-init] Unable to attach to controlling tty (errno=25 Inappropriate ioctl for device).
[dumb-init] setsid complete.
I0718 21:47:01.045403      13 launch.go:105] &{NGINX 0.9.0-beta.11 git-a3131c5 https://github.com/kubernetes/ingress}
I0718 21:47:01.045436      13 launch.go:108] Watching for ingress class: nginx
I0718 21:47:01.045587      13 launch.go:262] Creating API server client for https://10.96.0.1:443
I0718 21:47:01.055536      13 launch.go:124] validated default/default-http-backend as the default backend
I0718 21:47:01.061068      13 controller.go:1190] starting Ingress controller
I0718 21:47:01.063727      13 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"myingress", UID:"5910a23d-6678-11e7-8847-005056a64cce", APIVersion:"extensions", ResourceVersion:"827637", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/myingress
I0718 21:47:01.063762      13 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"kube-lego-nginx", UID:"226d6ecc-6600-11e7-8847-005056a64cce", APIVersion:"extensions", ResourceVersion:"827629", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/kube-lego-nginx
I0718 21:47:01.161461      13 leaderelection.go:203] attempting to acquire leader lease...
W0718 21:47:01.161521      13 backend_ssl.go:46] error obtaining PEM from secret default/tls.lab.tjpr.net: secret named default/tls.lab.tjpr.net does not exist
W0718 21:47:01.161757      13 controller.go:886] service default/kube-lego-nginx does not have any active endpoints
I0718 21:47:01.428599      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:01.428682      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:01.429683      13 controller.go:428] backend reload required
I0718 21:47:01.429809      13 metrics.go:34] changing prometheus collector from  to default
I0718 21:47:01.808989      13 controller.go:438] ingress backend successfully reloaded...
W0718 21:47:04.389400      13 controller.go:886] service default/kube-lego-nginx does not have any active endpoints
I0718 21:47:04.389747      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:04.389765      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
W0718 21:47:07.722759      13 controller.go:886] service default/kube-lego-nginx does not have any active endpoints
I0718 21:47:07.723099      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:07.723118      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
W0718 21:47:11.056056      13 controller.go:886] service default/kube-lego-nginx does not have any active endpoints
I0718 21:47:11.056414      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:11.056431      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
W0718 21:47:11.161688      13 backend_ssl.go:46] error obtaining PEM from secret default/tls.lab.tjpr.net: secret named default/tls.lab.tjpr.net does not exist
W0718 21:47:14.389381      13 controller.go:886] service default/kube-lego-nginx does not have any active endpoints
I0718 21:47:14.389709      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:14.389723      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
W0718 21:47:17.722726      13 controller.go:886] service default/kube-lego-nginx does not have any active endpoints
I0718 21:47:17.723112      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:17.723129      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
W0718 21:47:21.056072      13 controller.go:886] service default/kube-lego-nginx does not have any active endpoints
I0718 21:47:21.056458      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:21.056473      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
W0718 21:47:21.161835      13 backend_ssl.go:46] error obtaining PEM from secret default/tls.lab.tjpr.net: secret named default/tls.lab.tjpr.net does not exist
W0718 21:47:24.389370      13 controller.go:886] service default/kube-lego-nginx does not have any active endpoints
I0718 21:47:24.389704      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:24.389723      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
W0718 21:47:27.722784      13 controller.go:886] service default/kube-lego-nginx does not have any active endpoints
I0718 21:47:27.723255      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:27.723282      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
W0718 21:47:31.056108      13 controller.go:886] service default/kube-lego-nginx does not have any active endpoints
I0718 21:47:31.056590      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:31.056609      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
W0718 21:47:31.161981      13 backend_ssl.go:46] error obtaining PEM from secret default/tls.lab.tjpr.net: secret named default/tls.lab.tjpr.net does not exist
W0718 21:47:34.389529      13 controller.go:886] service default/kube-lego-nginx does not have any active endpoints
I0718 21:47:34.390016      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:34.390038      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:40.039576      13 leaderelection.go:213] successfully acquired lease default/ingress-controller-leader-nginx
W0718 21:47:41.162154      13 backend_ssl.go:46] error obtaining PEM from secret default/tls.lab.tjpr.net: secret named default/tls.lab.tjpr.net does not exist
W0718 21:47:51.162299      13 backend_ssl.go:46] error obtaining PEM from secret default/tls.lab.tjpr.net: secret named default/tls.lab.tjpr.net does not exist
W0718 21:47:59.158749      13 controller.go:886] service default/kube-lego-nginx does not have any active endpoints
I0718 21:47:59.160353      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:47:59.160373      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
W0718 21:48:01.162444      13 backend_ssl.go:46] error obtaining PEM from secret default/tls.lab.tjpr.net: secret named default/tls.lab.tjpr.net does not exist
10.46.0.1 - [10.46.0.1] - - [18/Jul/2017:21:48:01 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 348 0.000 [-] - - - -
10.46.0.1 - [10.46.0.1] - - [18/Jul/2017:21:48:01 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 129 0.000 [default-kube-lego-nginx-8080] 127.0.0.1:8181 213 0.000 503
10.46.0.1 - [10.46.0.1] - - [18/Jul/2017:21:48:02 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 350 0.000 [-] - - - -
10.46.0.1 - [10.46.0.1] - - [18/Jul/2017:21:48:02 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 130 0.000 [default-kube-lego-nginx-8080] 127.0.0.1:8181 213 0.000 503
10.46.0.1 - [10.46.0.1] - - [18/Jul/2017:21:48:03 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 348 0.000 [-] - - - -
10.46.0.1 - [10.46.0.1] - - [18/Jul/2017:21:48:03 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 129 0.000 [default-kube-lego-nginx-8080] 127.0.0.1:8181 213 0.000 503
10.46.0.1 - [10.46.0.1] - - [18/Jul/2017:21:48:04 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 350 0.000 [-] - - - -
10.46.0.1 - [10.46.0.1] - - [18/Jul/2017:21:48:04 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 130 0.000 [default-kube-lego-nginx-8080] 127.0.0.1:8181 213 0.000 503
I0718 21:48:04.720133      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:48:04.720154      13 controller.go:1052] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:48:04.721215      13 controller.go:428] backend reload required
I0718 21:48:04.816612      13 controller.go:438] ingress backend successfully reloaded...
66.133.109.36 - [66.133.109.36] - - [18/Jul/2017:21:48:05 +0000] "GET /.well-known/acme-challenge/fnWSWuMmGYpyUz7-xvedsTv3IdT8fkEmoY831Rbw9dc HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 264 0.000 [default-kube-lego-nginx-8080] 10.46.0.1:8080 87 0.000 200
I0718 21:48:11.163454      13 backend_ssl.go:64] adding secret default/tls.lab.tjpr.net to the local store

quay.io/aledbf/nginx-ingress-controller:0.169 does not even recognize a new TLS secret was added by kube-lego.

I0718 21:55:10.144232      13 launch.go:108] &{NGINX 0.9.0-beta.11 git-05ef427a https://github.com/aledbf/ingress}
I0718 21:55:10.144258      13 launch.go:111] Watching for ingress class: nginx
I0718 21:55:10.144394      13 launch.go:266] Creating API server client for https://10.96.0.1:443
I0718 21:55:10.153252      13 launch.go:127] validated default/default-http-backend as the default backend
I0718 21:55:10.159318      13 controller.go:1191] starting Ingress controller
W0718 21:55:10.161864      13 backend_ssl.go:46] error obtaining PEM from secret default/tls.lab.tjpr.net: secret named default/tls.lab.tjpr.net does not exist
I0718 21:55:10.162058      13 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"myingress", UID:"5910a23d-6678-11e7-8847-005056a64cce", APIVersion:"extensions", ResourceVersion:"827637", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/myingress
I0718 21:55:10.162077      13 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"kube-lego-nginx", UID:"226d6ecc-6600-11e7-8847-005056a64cce", APIVersion:"extensions", ResourceVersion:"827629", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/kube-lego-nginx
I0718 21:55:10.259494      13 leaderelection.go:203] attempting to acquire leader lease...
W0718 21:55:10.259718      13 controller.go:882] service default/kube-lego-nginx does not have any active endpoints
I0718 21:55:10.366249      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:55:10.366274      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:55:10.367172      13 controller.go:421] backend reload required
I0718 21:55:10.367264      13 metrics.go:34] changing prometheus collector from  to default
I0718 21:55:10.419730      13 controller.go:431] ingress backend successfully reloaded...
W0718 21:55:13.487166      13 controller.go:882] service default/kube-lego-nginx does not have any active endpoints
I0718 21:55:13.487520      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:55:13.487536      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
W0718 21:55:16.844182      13 controller.go:882] service default/kube-lego-nginx does not have any active endpoints
I0718 21:55:16.844547      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:55:16.844566      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
W0718 21:55:20.153861      13 controller.go:882] service default/kube-lego-nginx does not have any active endpoints
I0718 21:55:20.154235      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:55:20.154258      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
W0718 21:55:23.527551      13 controller.go:882] service default/kube-lego-nginx does not have any active endpoints
I0718 21:55:23.527926      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:55:23.527938      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
W0718 21:55:26.820466      13 controller.go:882] service default/kube-lego-nginx does not have any active endpoints
I0718 21:55:26.820843      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:55:26.820861      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
10.46.0.1 - [10.46.0.1] - - [18/Jul/2017:21:55:29 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 348 0.000 [-] - - - -
10.46.0.1 - [10.46.0.1] - - [18/Jul/2017:21:55:29 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 129 0.000 [default-kube-lego-nginx-8080] 127.0.0.1:8181 213 0.000 503
10.46.0.1 - [10.46.0.1] - - [18/Jul/2017:21:55:29 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 350 0.000 [-] - - - -
10.46.0.1 - [10.46.0.1] - - [18/Jul/2017:21:55:29 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 130 0.000 [default-kube-lego-nginx-8080] 127.0.0.1:8181 213 0.000 503
W0718 21:55:30.153858      13 controller.go:882] service default/kube-lego-nginx does not have any active endpoints
I0718 21:55:30.154259      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:55:30.154274      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
10.46.0.1 - [10.46.0.1] - - [18/Jul/2017:21:55:31 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 350 0.000 [-] - - - -
10.46.0.1 - [10.46.0.1] - - [18/Jul/2017:21:55:31 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 348 0.000 [-] - - - -
10.46.0.1 - [10.46.0.1] - - [18/Jul/2017:21:55:31 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 129 0.000 [default-kube-lego-nginx-8080] 127.0.0.1:8181 213 0.000 503
10.46.0.1 - [10.46.0.1] - - [18/Jul/2017:21:55:31 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 130 0.000 [default-kube-lego-nginx-8080] 127.0.0.1:8181 213 0.000 503
W0718 21:55:33.487252      13 controller.go:882] service default/kube-lego-nginx does not have any active endpoints
I0718 21:55:33.487760      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:55:33.487800      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
10.46.0.1 - [10.46.0.1] - - [18/Jul/2017:21:55:34 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 350 0.000 [-] - - - -
10.46.0.1 - [10.46.0.1] - - [18/Jul/2017:21:55:34 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 130 0.000 [default-kube-lego-nginx-8080] 127.0.0.1:8181 213 0.000 503
10.46.0.1 - [10.46.0.1] - - [18/Jul/2017:21:55:34 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 348 0.000 [-] - - - -
10.46.0.1 - [10.46.0.1] - - [18/Jul/2017:21:55:34 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 129 0.000 [default-kube-lego-nginx-8080] 127.0.0.1:8181 213 0.000 503
W0718 21:55:36.820514      13 controller.go:882] service default/kube-lego-nginx does not have any active endpoints
I0718 21:55:36.821116      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:55:36.821146      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
W0718 21:55:40.153830      13 controller.go:882] service default/kube-lego-nginx does not have any active endpoints
I0718 21:55:40.154162      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:55:40.154174      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
10.46.0.1 - [10.46.0.1] - - [18/Jul/2017:21:55:40 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 348 0.000 [-] - - - -
10.46.0.1 - [10.46.0.1] - - [18/Jul/2017:21:55:40 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 129 0.001 [default-kube-lego-nginx-8080] 127.0.0.1:8181 213 0.001 503
10.46.0.1 - [10.46.0.1] - - [18/Jul/2017:21:55:40 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 350 0.000 [-] - - - -
10.46.0.1 - [10.46.0.1] - - [18/Jul/2017:21:55:40 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 130 0.000 [default-kube-lego-nginx-8080] 127.0.0.1:8181 213 0.000 503
I0718 21:55:43.487496      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:55:43.487519      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:55:43.488382      13 controller.go:421] backend reload required
I0718 21:55:43.634780      13 controller.go:431] ingress backend successfully reloaded...
I0718 21:55:46.820841      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:55:46.820869      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
66.133.109.36 - [66.133.109.36] - - [18/Jul/2017:21:55:47 +0000] "GET /.well-known/acme-challenge/G_rLZKy8nyoGEHnA9P5lchjJstJvufUffUDqL6CHPXY HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 265 0.001 [default-kube-lego-nginx-8080] 10.46.0.1:8080 87 0.001 200
I0718 21:55:50.154118      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:55:50.154153      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:55:53.487460      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:55:53.487491      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:55:56.820908      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:55:56.820930      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:00.154249      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:00.154275      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:03.487419      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:03.487445      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:06.820797      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:06.820824      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:10.154221      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:10.154256      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:13.487475      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:13.487502      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:16.820853      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:16.820892      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:20.154439      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:20.154486      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:23.487458      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:23.487480      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:26.820831      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:26.820857      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:30.154109      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:30.154134      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:33.487447      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:33.487468      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:36.820750      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:36.820773      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:40.154093      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:40.154120      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:43.487441      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
I0718 21:56:43.487462      13 controller.go:1053] ssl certificate "default/tls.lab.tjpr.net" does not exist in local store
[dumb-init] Received signal 15.
[dumb-init] Forwarded signal 15 to children.
I0718 21:56:46.467294      13 main.go:49] Received SIGTERM, shutting down
I0718 21:56:46.467317      13 controller.go:1177] shutting down controller queues
I0718 21:56:46.467353      13 main.go:57] Exiting with 0
[dumb-init] Received signal 17.
[dumb-init] A child with PID 13 exited with exit status 0.
[dumb-init] Forwarded signal 15 to children.
[dumb-init] Child exited with status 0. Goodbye.

I noticed the last few lines of 0.9.0-beta.11 are interesting.

I0718 21:48:04.721215      13 controller.go:428] backend reload required
I0718 21:48:04.816612      13 controller.go:438] ingress backend successfully reloaded...
66.133.109.36 - [66.133.109.36] - - [18/Jul/2017:21:48:05 +0000] "GET /.well-known/acme-challenge/fnWSWuMmGYpyUz7-xvedsTv3IdT8fkEmoY831Rbw9dc HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 264 0.000 [default-kube-lego-nginx-8080] 10.46.0.1:8080 87 0.000 200
I0718 21:48:11.163454      13 backend_ssl.go:64] adding secret default/tls.lab.tjpr.net to the local store

Notice that the tls secret event is received AFTER nginx config is reloaded. Certainly doesn't make sense.

juliohm1978 picture juliohm1978 on 19 Jul 2017

Notice that the tls secret event is received AFTER nginx config is reloaded. Certainly doesn't make sense.

All the processes are sync, nginx can be reloaded by a change in the endpoints, configmap, secrets and ingress

aledbf picture aledbf on 19 Jul 2017

All the processes are sync, nginx can be reloaded by a change in the endpoints, configmap, secrets and ingress

The secret was changed. A new cert was issued. Shouldn't it have reloaded? This is still not working as expected.

juliohm1978 picture juliohm1978 on 19 Jul 2017

I believe I'm seeing the same issue with gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.13.

petergardfjall picture petergardfjall on 28 Sep 2017

Looks like it's when I refresh the ingress.

When it gets reloaded due to a new ingress being added or an old ingress being removed:

I0928 18:12:29.136769       6 controller.go:428] backend reload required
I0928 18:12:29.221133       6 controller.go:438] ingress backend successfully reloaded...
I0928 18:12:32.462580       6 controller.go:1052] ssl certificate "default/booking-production-tls" does not exist in local store

and if I look at secrets

NAME                                 TYPE                                  DATA      AGE
booking-production-tls               kubernetes.io/tls                     2         21h

The current solution for me is to just use the kubectl patch function that was mentioned near the top of the issue.

Spittal picture Spittal on 28 Sep 2017

@aledbf I'm also seeing the HTTP 503 status problem, using kube-lego and kubernetes-nginx-ingress on AWS. Not sure if this is the right issue for that but it's the only one I found that mentions the same log errors as I'm seeing.

Logs from the nginx pod:

W1116 14:34:20.128648       5 controller.go:869] service infra/kube-lego-nginx does not have any active endpoints
W1116 14:34:20.128719       5 controller.go:1100] ssl certificate "default/echoserver-ingress-tls" does not exist in local store
172.20.42.71 - [172.20.42.71] - - [16/Nov/2017:14:34:20 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 138 0.000 [] - - - -

Logs from the lego pod contain a whole lot of lines like these:

time="2017-11-16T14:34:28Z" level=debug msg="testing reachability of http://echo.syntaxis.systems/.well-known/acme-challenge/_selftest" context=acme domain=echo.syntaxis.systems 
time="2017-11-16T14:34:28Z" level=debug msg="error while authorizing: reachability test failed: wrong status code '503'" context=acme domain=echo.syntaxis.systems

I'm able to inspect the NGINX config with kubectl -n infra exec nginx-1248418661-f20j4 cat /etc/nginx/nginx.conf, which shows this:

upstream default-echoserver-8080 {
  server 100.96.2.10:8080 max_fails=0 fail_timeout=0;
  server 100.96.1.9:8080 max_fails=0 fail_timeout=0;
  [...]
}

server {
  server_name echo.syntaxis.systems ;
  listen 80;

  location /.well-known/acme-challenge {
    # No endpoints available for the request
    return 503;
    }
}

Seems like kube-lego isn't properly configured as backend in the NGINX config? I would expect the location ./well-known/acme-challenge block to have a proxy_pass to a kube-lego backend, but neither the backend nor the proxy-pass are there.

There is an Ingress for kube-lego-nginx:

ingresses - kubernetes dashboard 2017-11-16 15-48-56

And there's a Service with no External endpoints:

services - kubernetes dashboard 2017-11-16 15-49-53

Versions

  • nginx-ingress-controller: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0-beta.17
  • kube-lego: jetstack/kube-lego:0.1.5
marceldegraaf picture marceldegraaf on 16 Nov 2017

It seems this is caused by the lego Pod disappearing from the kube-lego-nginx Service. I've opened an issue on the lego tracker, here.

marceldegraaf picture marceldegraaf on 16 Nov 2017
Was this page helpful?
0 / 5 - 0 ratings

Related issues

oilbeater picture oilbeater  路  3Comments
lachlancooper picture lachlancooper  路  3Comments
vdavidoff picture vdavidoff  路  3Comments
juliohm1978 picture juliohm1978  路  3Comments
smeruelo picture smeruelo  路  3Comments