Ingress-nginx: Question about dynamic reloads feature

[rikatz]

Is this a request for help? No

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.):

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Feature request / feature explanation

NGINX Ingress controller version: 0.15

Kubernetes version (use kubectl version): 1.7.15

Environment: CoreOS

What happened: We've finally upgraded ingress to 0.15 (again, thanks for the great work guys), and we're using the dynamic updates.

Everything seems to work fine, except that at each 1 minute or less it still issues a 'backend reload':

May 18 14:11:18  I0518 17:11:18.590457       7 controller.go:177] ingress backend successfully reloaded...
May 18 14:11:22  I0518 17:11:22.970907       7 controller.go:168] backend reload required
May 18 14:11:25  I0518 17:11:25.266393       7 controller.go:177] ingress backend successfully reloaded...
May 18 14:11:39  I0518 17:11:39.640257       7 controller.go:168] backend reload required
May 18 14:11:41  I0518 17:11:41.897076       7 controller.go:177] ingress backend successfully reloaded...
May 18 14:11:42  I0518 17:11:42.997075       7 controller.go:168] backend reload required
May 18 14:11:45  I0518 17:11:45.246361       7 controller.go:177] ingress backend successfully reloaded...
May 18 14:11:46  I0518 17:11:46.305488       7 controller.go:168] backend reload required
May 18 14:11:48  I0518 17:11:48.566609       7 controller.go:177] ingress backend successfully reloaded...
May 18 14:11:49  I0518 17:11:49.638432       7 controller.go:168] backend reload required
May 18 14:11:51  I0518 17:11:51.904147       7 controller.go:177] ingress backend successfully reloaded...
May 18 14:11:52  I0518 17:11:52.971443       7 controller.go:168] backend reload required
May 18 14:11:55  I0518 17:11:55.255585       7 controller.go:177] ingress backend successfully reloaded...
May 18 14:11:56  I0518 17:11:56.305789       7 controller.go:168] backend reload required
May 18 14:11:58  I0518 17:11:58.595199       7 controller.go:177] ingress backend successfully reloaded...
May 18 14:11:59  I0518 17:11:59.639205       7 controller.go:166] skipping reload

In the last line, you can see the dynamic reload happening. I've supressed other lines from the log (keeping only the lines containing 'reload' word).

So, my question here is: when ingress chooses to reload and when it uses the dynamic reload? Is this some kind of problem with secrets syncronization?

We have actually 844 ingresses objects, and each of them with some secrets being used as CA, TLS/HTTPs or even using Let's Encrypt/kube-lego here.

Thanks!

[aledbf]

So, my question here is: when ingress chooses to reload and when it uses the dynamic reload? Is this some kind of problem with secrets syncronization?

The dynamic reload feature works for changes in the endpoints (pods).

[aledbf]

@rikatz the next step is to avoid reloads for change in secrets

[aledbf]

ping @ElvinEfendi

[rikatz]

I'm just wondering how to reload NGINX secrets without reloading it :)

[aledbf]

I'm just wondering how to reload NGINX secrets without reloading it :)

Lua :wink:

Edit: this also means there is no files for secrets in the running container

[rikatz]

@aledbf nice. I'm trying to understand this better. I've seen that there's no files anymore for balancers, and I'm wondering also how this impacts in memory consumption.

This is a hash table loaded into memory by Lua, right?

[aledbf]

@rikatz you should see an increase of ~10MB of ram in nginx https://github.com/kubernetes/ingress-nginx/blob/2baad5d0066dd2d6e7f9ff196d42ab227b55d8de/internal/ingress/controller/template/template.go#L191-L199

[rikatz]

@aledbf Thanks! I'm pretty away from the project, will take some time in sunday to take a look into this new feature to see if I can be helpful in this part :)

Also, I'll close this issue, as the question is solved.

Thanks!

[ElvinEfendi]

I'm just wondering how to reload NGINX secrets without reloading it :)

@rikatz the idea is that instead of creating file in the container the controller will post certificates into a Lua endpoint where Lua will store in a shared dictionary inside Nginx. Then using Lua we will dynamically serve certificates from the memory.