Cluster-api-provider-azure: Windows VMSS E2E flakes

[CecileRobertMichon]

/kind bug

What steps did you take and what happened:
[A clear and concise description of what the bug is.]

https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api-provider-azure/1178/pull-cluster-api-provider-azure-e2e-windows/1362121987211661312

E0217 19:56:17.869588       1 azuremachinepool_controller.go:290] controllers/AzureMachinePool "msg"="failed to reconcile AzureMachinePool" "error"="failed to create scale set: failed to get VMSS win-p-win after create or update: failed to get result from future: reconcile error occurred that can be recovered. Object will be requeued after 15s The actual error is: operation type PATCH on Azure resource capz-e2e-odyqry/win-p-win is not done" "AzureCluster"="capz-e2e-odyqry" "azureMachinePool"="capz-e2e-odyqry-mp-win" "cluster"="capz-e2e-odyqry" "machinePool"="capz-e2e-odyqry-mp-win" "namespace"="create-workload-cluster-heu6os" "name"="win-p-win"

job history: https://prow.k8s.io/job-history/gs/kubernetes-jenkins/pr-logs/directory/pull-cluster-api-provider-azure-e2e-windows

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• cluster-api-provider-azure version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[devigned]

I believe this is fixed in #1105

[CecileRobertMichon]

I believe this is fixed in #1105

Which part of #1105 fixes it? 😄 Can we pull the fix into a smaller / quick merging PR or is it part of the refactor?

[devigned]

I think the issue is the bootstrap data secret doesn’t get picked up until 10 mins after it is created. We can either requeue or set a watch.

For brevity, I just requeued.

[CecileRobertMichon]

how come we weren't running into this before? Did it only become relevant with #1067 ?

[devigned]

I did find that node ref names weren’t aligned with expected machine names since Windows names machine pools differently, but I don’t know if that also affected #1067.

Might have to dig a little deeper on this one.

[CecileRobertMichon]

/assign @jsturtevant

[jsturtevant]

I was able to reproduce this using the main branch with the tilt machinepool-windows flavor:

azuremachinepool_controller.go:290] controllers/AzureMachinePool "msg"="failed to reconcile AzureMachinePool" "error"="failed 
to create scale set: failed to get VMSS win-p-win: failed to get result from future: reconcile error occurred that can be recovered. 
Object will be requeued after 15s The actual error is: operation type PATCH on Azure resource capi-e2e/win-p-win is not done" 
"AzureCluster"="machinepool-win-template" "azureMachinePool"="machinepool-win-template-mp-win" "cluster"="machinepool-
win-template" "machinePool"="machinepool-win-template-mp-win" "namespace"="default" "name"="win-p-win"

[jsturtevant]

2 of the four nodes came online:

k get nodes
NAME                                           STATUS     ROLES    AGE     VERSION
machinepool-win-template-control-plane-r4rjv   Ready      master   9m39s   v1.19.7
machinepool-win-template-mp-0000000            Ready      <none>   7m27s   v1.19.7
machinepool-win-template-mp-0000002            Ready      <none>   7m35s   v1.19.7
win-p-win000000                                NotReady   <none>   6m2s    v1.19.7
win-p-win000001                                NotReady   <none>   6m12s   v1.19.7
win-p-win000002                                Ready      <none>   6m14s   v1.19.7
win-p-win000003                                Ready      <none>   6m34s   v1.19.7

[jsturtevant]

But there should only be 4 worker nodes total across windows and linux according to my config:

apiVersion: exp.cluster.x-k8s.io/v1alpha3
kind: MachinePool
metadata:
  name: machinepool-win-template-mp-0
  namespace: default
spec:
  clusterName: machinepool-win-template
  replicas: 2

apiVersion: exp.cluster.x-k8s.io/v1alpha3
kind: MachinePool
metadata:
  name: machinepool-win-template-mp-win
  namespace: default
spec:
  clusterName: machinepool-win-template
  replicas: 2

and my portal only shows 2 vms in the windows pool:

 az vmss list -o table
Name                           ResourceGroup          Location    Zones    Capacity    Overprovision    UpgradePolicy
-----------------------------  ---------------------  ----------  -------  ----------  ---------------  ---------------
machinepool-win-template-mp-0  CAPI-E2E               eastus               2           True             Manual
win-p-win                      CAPI-E2E               eastus               2           True             Manual

[CecileRobertMichon]

Could this be related to VMSS overprovisioning ?

[devigned]

I like your thought about overprovisioning, @CecileRobertMichon. I think it might be that and the difference in pool naming / instance naming in Windows.

@jsturtevant If you get the below error continuously, then the PATCH to update the VMSS has never completed, or it keeps re-running. The controller is seeing that it needs to patch the VMSS again. The error below is a transient error and is why it requeues the controller to run again. It happens when the VMSS is in a long running operation like a PUT or PATCH.

azuremachinepool_controller.go:290] controllers/AzureMachinePool "msg"="failed to reconcile AzureMachinePool" "error"="failed 
to create scale set: failed to get VMSS win-p-win: failed to get result from future: reconcile error occurred that can be recovered. 
Object will be requeued after 15s The actual error is: operation type PATCH on Azure resource capi-e2e/win-p-win is not done" 
"AzureCluster"="machinepool-win-template" "azureMachinePool"="machinepool-win-template-mp-win" "cluster"="machinepool-
win-template" "machinePool"="machinepool-win-template-mp-win" "namespace"="default" "name"="win-p-win"

[jsturtevant]

I am seeing it run over and over again. Adding a few more logs interesting logs that keep repeating:

[manager] I0219 19:57:11.294970       8 azuremachinepool_controller.go:250] controllers/AzureMachinePool "msg"="Reconciling AzureMachinePool" "AzureCluster"="machinepool-win-template" "azureMachinePool"="machinepool-win-template-mp-win" "cluster"="machinepool-win-template" "machinePool"="machinepool-win-template-mp-win" "namespace"="default" 

[manager] I0219 19:57:11.710087       8 machinepool.go:337] controllers/AzureMachinePool "msg"="No image specified for machine, using default Windows Image" "AzureCluster"="machinepool-win-template" "azureMachinePool"="machinepool-win-template-mp-win" "cluster"="machinepool-win-template" "machinePool"="machinepool-win-template-mp-win" "namespace"="default" "machine"="machinepool-win-template-mp-win"

[manager] E0219 19:57:12.662873       8 azuremachinepool_controller.go:290] controllers/AzureMachinePool "msg"="failed to reconcile AzureMachinePool" "error"="failed to create scale set: failed to get VMSS win-p-win after create or update: failed to get result from future: reconcile error occurred that can be recovered. Object will be requeued after 15s The actual error is: operation type PATCH on Azure resource capi-e2e/win-p-win is not done" "AzureCluster"="machinepool-win-template" "azureMachinePool"="machinepool-win-template-mp-win" "cluster"="machinepool-win-template" "machinePool"="machinepool-win-template-mp-win" "namespace"="default" "name"="win-p-win"





[manager] I0219 19:57:12.684054       8 azuremachinepool_controller.go:250] controllers/AzureMachinePool "msg"="Reconciling AzureMachinePool" "AzureCluster"="machinepool-win-template" "azureMachinePool"="machinepool-win-template-mp-win" "cluster"="machinepool-win-template" "machinePool"="machinepool-win-template-mp-win" "namespace"="default" 

[manager] I0219 19:57:12.684974       8 helpers.go:319] controllers/AzureJSONMachinePool "msg"="returning early from json reconcile, no update needed" "azureMachinePool"="machinepool-win-template-mp-win" "cluster"="machinepool-win-template" "machinePool"="machinepool-win-template-mp-win" "namespace"="default" 

[manager] E0219 19:57:12.894983       8 azuremachinepool_controller.go:290] controllers/AzureMachinePool "msg"="failed to reconcile AzureMachinePool" "error"="failed to create scale set: failed to get VMSS win-p-win: failed to get result from future: reconcile error occurred that can be recovered. Object will be requeued after 15s The actual error is: operation type PATCH on Azure resource capi-e2e/win-p-win is not done" "AzureCluster"="machinepool-win-template" "azureMachinePool"="machinepool-win-template-mp-win" "cluster"="machinepool-win-template" "machinePool"="machinepool-win-template-mp-win" "namespace"="default" "name"="win-p-win"

[jsturtevant]

I think I've narrowed it down to the hash being generated is different each time due to the fact we have a generated password:

https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/b9c90e48157f63f9b3321a1cdccd0e5f2a3f21d0/azure/services/scalesets/scalesets.go#L198-L203

buildVMSSFromSpec eventually calls to create the OSprofile and which generates a new password:

https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/b9c90e48157f63f9b3321a1cdccd0e5f2a3f21d0/azure/services/scalesets/scalesets.go#L593

and causes the hash to be different