Application-gateway-kubernetes-ingress: Major drawbacks of using Application Gateway Ingress Controller

[landro]

From the announcement at https://azure.microsoft.com/nb-no/blog/application-gateway-ingress-controller-for-azure-kubernetes-service/ and documentation at https://docs.microsoft.com/en-us/azure/application-gateway/application-gateway-faq#configuration---ingress-controller-for-aks one can get the impression that using the azure application gateway ingress controller in order to configure an azure application gateway using standard kubernetes ingress resources deployed to azure kubernetes service, is ready for prime time. However, our experience from using this ingress controller in production for a couple days, clearly indicates that you should not do so if you're looking for zero-downtime during deployments and other events in your cluster that might impact application gateway configuration (scaling operations, pod crashes etc etc).

It turns out that it can take several minutes for the ingress controller to perform config changes in azure application gateway. This is also backed by the FAQ found at https://docs.microsoft.com/en-us/azure/application-gateway/application-gateway-faq#how-long-does-it-take-to-deploy-an-application-gateway-will-my-application-gateway-work-while-its-being-updated where one can read:

_Question: "Will my application gateway work while it's being updated?"
Answer: "Most deployments that use the v2 SKU take around 6 minutes to provision. However it can take longer depending on the type of deployment. For example, deployments across multiple Availability Zones with many instances can take more than 6 minutes."_

Our experience shows that configuration changes sometimes can take significantly longer (we've experienced 10 minutes in two different clusters/application gateways). Other users have seen even longer delays: https://feedback.azure.com/forums/281804-azure-resource-manager/suggestions/19119910-application-gateway-management-operations-are-agon

The impact of this is that your app gw backend pool might point to ips of pods that are long gone, leading to users getting 502s.

I only see only possible fix to this issue: configuration changes in application gateway have to be way faster and more reliable. For instance being able to perform config changes in less than 30s , would help a lot.

Right now I would not recommend using this ingress controller to anyone.
Azure, you're loosing business.

Azure support ticket: 120012822002716

[RobLSDev]

This exactly mirrors the findings my company had with AGIC, as well as other technical limitations and stability issues with the App Gateways themselves. The AGIC seems to function reasonably well enough, but the AG itself seems terribly implemented.

For instance, during deployments we allow up to 1 pod to be unavailable, and the count can burst by 1. This means we'll see an initial request to remove a pod from the backend pool. Then, when the two new pods come up each will trigger it's own request to add it, soon to be followed up by one or two requests to remove other pods from the previous deployment, and so on until all of the pods are replaced. We've observed that these requests sometimes preempt, and sometimes they just fail. The results are inconsistent, and always led to users seeing 502s or timeouts.

[mscatyao]

Hey @landro,

We’re sorry to hear that you had a bad experience with AGIC. The expected configuration update time for Application Gateway v2 is on the order of ~40 seconds.

We looked into the logs for the Application Gateway you provided in the support ticket and the other Application Gateway that experienced slow update times, and acknowledge that two of the updates to the Application Gateways took over 10 minutes to complete. On the same day for those two Application Gateways, 38 other configuration updates completed in ~15 seconds and 4 configuration updates took ~30 seconds.

The feedback piece you mentioned was closed once fast update was released with AppGW v2. However, if fast update fails for any reason, we take a slower update path which can take longer to complete.

On January 27th there was a platform issue in a few regions including West Europe. This affected a limited set of gateways and caused a failure in the fast update path between the management plane and Application Gateway instances.

The FAQ documentation section you linked to talks about provisioning (Gateway creation) time, which is on the order of 5-6 minutes. Configuration updates to Application Gateway v2 should take 40 seconds or less.

We hear your feedback and are currently working on further reducing the update time, as well as making the channel between our management plane and Application Gateway more resilient.

[sdesmond46]

We ran into this problem as well except the updates just completely failed and timed out after 15 minutes. The gateway was then stuck in an updating state so we couldn't make any changes to it at all.

I spent the afternoon switching to use nginx-ingress instead 😞

Our ticket number is 120013121001561 if you wanna look at the logs

[landro]

Hey @mscatyao,

Thanks for getting back to me.

I have a few more questions that I would like you to comment on:

Expected performance

The expected configuration update time for Application Gateway v2 is on the order of ~40 seconds.

I have found no official sources documenting this.
It would be great if you could provide us with detailed response time metrics - I'm sure you're actively tracking this number across all app gw instances that you manage. 90, 95, 99, 99.9 percentiles would be great as a reference.

Incidents

You call what happened to us on January 27 _a platform issue_. I can find no trace of this issue on https://status.azure.com/en-us/status/history/ - why is that?
How often do these platform issues occur? What's your track record here? Please, be sincere.

SLA

Does a delay like this count towards an SLA (to us the app gw was in practice unavailable)? If you're confident the app gw config changes normally complete within 40s, why not put it in the SLA explicitly?

FAQ

The FAQ should be updated - it says "Will my application gateway work while it's being updated?" which might lead people to think that this applies to config changes.
In addition, the support agent referred to this number during a phone call I had with him.

Timeline

When do you plan to fix the underlying issue here?

[hajdukd]

@landro You may consider using an explicit lifeCycleHook on your deployments to wait X amount of time (depending on your draining configuration) and only after that time is finished start removing the pod - it will be switched to terminating status and AGIC should reconfigure AG to remove it but let requests finish.

I hoped for being able to run RollingUpdates for my deployments swapping pods 1b1 but unfortunately for w/e reason AG backend healthchecks can fail against completely fine and working pods :(
Sometimes leaving you with no backends to handle the traffic and causing thousends of 502s responses to the client.

[landro]

We're already doing what you're suggesting @hajdukd - that won't help much however if app gw spends 10 minutes performing config changes.

[mscatyao]

Hey @sdesmond46 - thanks for reaching out. We're sorry for the delays; we'll reach out to you shortly.

Hey @landro - these are all great questions. We understand your frustrations; we've reached out via the support case you opened and will be following up there. In terms of the FAQ, I'll change the wording to be more clear and specify that the answer refers to provisioning/create time and not update time.

[mscatyao]

Resolved via support ticket internally.

[landro]

I see no reason for closing this ticket. The issue still remains - for workloads that require true HA, using app gw ingress controller is not adequate. Please keep this issue open until the underlying issue of config change propagation times is fixed, @mscatyao

[DenisBiondic]

It seems that the AGIC does not respect the readiness probes of the pods when issuing health probes? Can this be true? We observe downtime when using AGIC on every deployment, although the pods have readiness checks.

EDIT: nevermind, found explanation here: https://github.com/Azure/application-gateway-kubernetes-ingress/blob/master/docs/how-tos/minimize-downtime-during-deployments.md#minimizing-downtime-during-deployments

[atrauzzi]

What if the timeout was paired with an AGIC proxy sidecar that forwarded calls to new pods as they ready-up during a rollover?

[Baklap4]

Can this be true?

@DenisBiondic
This sure is true. It uses its own healthcheck mechanics and doesn't respect the settings provided by kubernetes. This is confirmed in https://github.com/Azure/application-gateway-kubernetes-ingress/issues/796

[iamNoah1]

Out of the different people describing those incidents and having support cases, is there any general information available on how to solve this problem? Or do I have to open a support case as well?

[Baklap4]

Haven't checked in lately but the general consensus is that it isn't fixed and you can somehow 'bypass' it using this method: https://github.com/Azure/application-gateway-kubernetes-ingress/blob/master/docs/how-tos/minimize-downtime-during-deployments.md#minimizing-downtime-during-deployments however true zero-downtime is not a thing as much as im aware of.

[Segflow]

I0221 18:29:19.646282       1 controller.go:151] Completed last event loop run in: 7m50.758823353s

I think this issue should be opened.