Ingress-nginx: Question: Performance at scale

[TheBrigandier]

Hey ingress-nginx team,

Ran across this page the other day: https://itnext.io/kubernetes-ingress-controllers-how-to-choose-the-right-one-part-1-41d3554978d2 - and it seems to really talk down NGINX ingress performance:

Do not share Nginx Ingress for multiple environments
After abusing a shared ingress controller by 30+ environments, the Nginx config file got humongous and very slow to reload. POD IPs got stale and we started to see 5xx errors. Keep scaling up the same ingress controller deployment did not seem to solve the problem.
Since then, we started to use dedicated ingress controllers for each environment.

Considering ingress-nginx by default installs to handle all ingresses as default controller, and considering I can't find much out there on the net about this besides this one article, I am inclined to think it's just this guy was having issues; however, I would also like some peace of mind on this subject before I go deploying it as our standard.

Is there any information out there on how well a single ingress-nginx controller scales (say, how many ingress ruels?), or recommended layout suggestions for large clusters?

Thanks!

[aledbf]

@VileBrigandier some comments:

After abusing a shared ingress controller by 30+ environments, the Nginx config file got humongous and very slow to reload.

Since 0.18.0 we have dynamic reloads for pod changes, i.e., there are no upstreams in the generated nginx.conf file. This means we don't need to reload when your applications scale up/down or a pod is restarted.
For this, it seems the version used is a bit old (at least August of last year).

Keep scaling up the same ingress controller deployment did not seem to solve the problem.

For what I just mentioned, doing this won't help.

Is there any information out there on how well a single ingress-nginx controller scales (say, how many ingress rules?), or recommended layout suggestions for large clusters?

First, define what "large cluster" is for you. There are several users dealing with thousands of ingresses with services with more than one replica. Personally, I've used one ingress-nginx controller with more than three thousand ingresses without issues.

recommended layout suggestions for large clusters?

You can go gradually from this list:

• Multiple replicas using affinity rules
• Dedicated nodes for the controller
• Multiple ingress controller deployments:
  
  
  
  • Using ingress.class annotation
  
  
  • watching a particular namespace
  
  

[TheBrigandier]

@aledbf Thank you for this response. I am sorry for being so vague. My company is just now starting with Kubernetes so we're very low on experience, but very eager to do things right from the start. :)

You've answered my questions perfectly. The article was written pretty badly, so we had some people assuming by "30+ environments" they actually meant "30+ ingresses", and it was causing some anxiety over our initial choice of ingress-nginx. Hearing that you've used a single ingress-nginx controller with more than 3,000 ingresses puts that to rest I think.

The next issue I was worried about was the config reloads, but you've quelled fears on that point as well.

Thanks!