Application-gateway-kubernetes-ingress: Recommendation of different options for AKS?

[sowsan]

We have a great recommendation of different network considerations here https://docs.microsoft.com/en-us/azure/aks/operator-best-practices-network#secure-traffic-with-a-web-application-firewall-waf,

Since App Gateway Ingress is another option available now, what would be the recommendation for picking it compared to regular Ingress Controllers or Application Gateway?

[asridharan]

@sowsan thanks for bringing up this question. We should probably have a page highlighting the performance benefits of AG ingress controller vs nginx ingress controller. Going to list a few benefits here:
a) Having AG as the ingress frees up compute on the AKS cluster for ingress and SSL off-loading. This removes the burden of the operator from planning for compute on the AKS cluster. NOTE data-plane of AG ingress is running outside AKS.
b) AG v2 comes with auto-scaling and it has always had WAF, both these properties take the head-acle of elasticity and security out of the domain of the AKS admin/developer.
c) AG ingress controller add the pods directly as the backend to AG. This probably is one of the most important benefits. This removes the need for an SLB or node-port. More importantly, this removes any D-NAT and hence conntrack usage within the kubelets making the system perform a lot better. Removing D-NAT improves latency, and removing reliance on conntrack for load-balancing, especially for ingress, removes any non-determinism that can result from connections being dropped due to conntrack exhaustion.

There are definitely a few limitations, we hope to remove them sooner rather than later:
a) AG ingress works only with advanced networking. We hope to make it work with basic networking as well.
b) AG updates currently take around 30-40 seconds. This is obviously higher than nginx ingress. We hope to reduce this update interval to a second or so to better adapt to high frequency scale out/in events in AKS.

[sowsan]

Great. Thank you so much for the details @asridharan

[lastcoolnameleft]

@asridharan Thanks for the info. I assumed that advanced networking would be a hard requirement for this given that with Basic Networking, the Pods are not exposed as 1st class citizens to the Vnet. Without giving too much away, how will that limitation be removed?

[asridharan]

no magic here. Basically with basic networking we would need the UDR (User Defined Route) created by AKS for the pod subnet to be associated with the AG subnet. This would allow the AG to directly route to the pod IP, even though they are not an ARM resource. The catch being that the pod subnet needs to be from the VNET. The limitation, currently in doing this is that AG v2 doesn't allow any UDR to be associated with AG subnet. There are some technical reasons for this limitation which we should be able to remove sooner rather than later, but due to this limitation we can't use AG v2 with basic networking in AKS.

[lastcoolnameleft]

Thanks for the explanation @asridharan! And thanks for your work on this.

[asridharan]

Looks like the question has been addressed so closing this out for the time being. Feel free to open it in case there are any other questions.