Cluster-api-provider-azure: MachinePool has more worker nodes than the desired value

[chewong]

/kind bug

What steps did you take and what happened:
[A clear and concise description of what the bug is.]

https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/capz-azure-file-machinepool-1-19/1388742406299455488

Waiting for 1 control plane machine(s) and 2 worker machine(s) to become Ready
node/capz-h8u4r3-control-plane-5m8mq condition met
node/capz-h8u4r3-mp-0000000 condition met
node/capz-h8u4r3-mp-0000002 condition met
NAME                              STATUS     ROLES    AGE     VERSION                           INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
capz-h8u4r3-control-plane-5m8mq   Ready      master   3m59s   v1.19.11-rc.0.23+3ed2eb74c47e2f   10.0.0.4      <none>        Ubuntu 18.04.5 LTS   5.3.0-1034-azure   containerd://1.3.4
capz-h8u4r3-mp-0000000            Ready      <none>   37s     v1.19.11-rc.0.23+3ed2eb74c47e2f   10.1.0.4      <none>        Ubuntu 18.04.5 LTS   5.3.0-1034-azure   containerd://1.3.4
capz-h8u4r3-mp-0000001            NotReady   <none>   8s      v1.19.11-rc.0.23+3ed2eb74c47e2f   10.1.0.5      <none>        Ubuntu 18.04.5 LTS   5.3.0-1034-azure   containerd://1.3.4
capz-h8u4r3-mp-0000002            Ready      <none>   20s     v1.19.11-rc.0.23+3ed2eb74c47e2f   10.1.0.6      <none>        Ubuntu 18.04.5 LTS   5.3.0-1034-azure   containerd://1.3.4

Eventually, the NodeReady node was removed.

What did you expect to happen:

Have no more than the desired MachinePool replicas at any given moment

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• cluster-api-provider-azure version: master
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[CecileRobertMichon]

I believe this is by design because VMSS overprovisioning is enabled by default. The scale set will provision more instances than desired replicas and only keep the first n instances that become ready.

cc @devigned

[devigned]

I believe this is by design because VMSS overprovisioning is enabled by default. The scale set will provision more instances than desired replicas and only keep the first n instances that become ready.

That's exactly what is happening. How would you like this to behave. It is nice to use over-provisioning to increase overall speed and success rate of VMSS provisioning.

[jsturtevant]

I think it is fine from user standpoint. We found this issue in the e2e tests. In e2e.test there is a check to verify all the nodes are ready, which is causing it to hang while waiting to run. One possibility would to validate that we have the expected number of nodes and run e2e.test with --allowed-not-ready-nodes=1 which would allow the tests to proceed.