Ingress-nginx: Nginx Ingress Controller - Specify load balancing method

[silasbw]

The nginx ingress controller hard codes least_conn as the load balancing method.

It would be helpful to specify the load balancing method per ingress resource. For example, via an annotation on the ingress resource.

Not all workloads perform well under least_conn, and the nginx documentation recommends experimenting with different load balancing methods. The ability to specify the method per ingress resource would facilitate this experimentation.

[silasbw]

Related: is least_conn a good default in general?

Consider the case of multiple nginx controller pods (_e.g._, using a Deployment of DaemonSet). Assuming each Pod has the same nginx config (_i.e._, the upstream servers are listed in the same order), least_conn might have poor load balancing since each nginx controller Pod will route it's first connection to the first upstream server in the list. The second connections will all go to the second upstream server in the list, etc.

I think our current cluster setup is bumping up against this imbalance behavior.

A few half-baked proposals to get better load balancing:

• change the default load balancing method to round-robin; or
• randomize the lists of upstream servers in the nginx config; or
• ...

[silasbw]

Chatted with @aledbf. He points out:

• A good option for changing load balancing behavior globally is via nginx ConfigMap customization.
• Randomizing upstream servers can be problematic with scale ups/downs. Also https://github.com/Nordstrom/kubernetes-contrib/commit/53c6509466f7c745ff5e51c3176eabf7d365665b.
• Adding load balancing methods per Ingress resource is the most complicated and there's some concerns around making it portable between ingress controllers.

[aledbf]

fixes in #673