Aks-engine: Basic LB + large VMSS clusters can't reconcile

[jackfrancis]

When adding a basic LoadBalancer to a cluster that includes a non-trivial number of VMSS instances as nodes (repro'd easily w/ 50 node clusters) the LB reconciliation generates _n_ number of calls to the Azure network APIs per instance, resulting in subscription throttling to those APIs. In practice the LB is never able to reconcile.

controller-manager will show errors like the following:

LoadBalancerUpdateFailed.....Error updating load balancer with new hosts map[<lots o' vmss instances>]....EnsureHostInPool....failed to ensure host in pool: "compute.VirtualMachineScaleSetVMsClient#Update: Failure sending request: StatusCode=0 -- Original Error: Code=\"RetryableError\" Message=\"A retryable error occurred.\" Details=[{\"code\":\"RetryableErrorDueToAnotherOperation\",\"message\":\"Operation ValidateVMScaleSetOperation....The call can be retried in 13 seconds.\"}]"

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mariojacobo]

ran into the same issue and after 10-15 min it finally went through. @jackfrancis did you every find a solution to this issue ? in my case, this seems to happen only when the Ingress IP is precreated and part of another resource group. if I deploy and let AKS create the IP it runs just fine.

[ragoragino]

We are having exactly the same issue as described by @mariojacobo, although in our case it takes around 20-30m for services to be assigned external IP addresses (also from another resource group than AKS resources). We are getting Terraform Kubernetes service resource failures due to that. That's why we would love to hear if there is any new development regarding the issue.

[zhiweiv]

Will https://github.com/kubernetes/kubernetes/pull/88699 help?

[ragoragino]

Yes, it seems that it could be the root cause. We will be waiting for the 1.18 AKS version to become GA.

[jackfrancis]

Hi folks, the following Kubernetes versions improve on the stated symptoms in this issue:

• 1.15.11
• 1.16.8
• 1.17.4

Is anyone running those versions and having these symptoms?

As @zhiweiv and @ragoragino suggest, 1.18.0 has further improvements to LoadBalancer reconciliation, but there is a regression in 1.18.0 (orthogonally related to the issues being described in this thread) that will ship with 1.18.1.

tl;dr I think for most folks the above 3 Kubernetes versions can be targeted in the immediate term to improve LoadBalancer performance. And we definitely recommend running Standard LB! (Although largely the same symptoms and guidance applies, use the new Kubernetes versions for improved behaviors.)

[mariojacobo]

with 1.15.10 I have seen this occurrence just once, from a total of 20+ deployments. have not tried the other ones.