Aks-engine: SyncLoadBalancerFailed error because service controller can't find load balancer that was deleted

[GeneralKenobi]

Describe the bug

Cluster can't remove both service types (public, internal) - it fails when deleting the second one.

Let's assume the internal service is deleted first. Deleting the public service won't succeed because the service controller won't be able to find the internal load balancer that has already been deleted, see the output of kubectl describe attached. Another consequence is that the VMSS used by cluster will transition into 'Failed', see the error attached.

The same thing happens the other way round - when public service is deleted before the internal one.

This is certainly something new. I'm sure it worked a couple of months ago as I tested a scenario like this. Besides the version used to create this ticket that is given below I also reproduced this issue on k8s cluster 1.18.8 provisioned with AKS engine v0.55.0.

Steps To Reproduce

1. Deploy a cluster using the provided API model. The only important configuration is that a load balancer with public IP used for outbound connectivity is not created.
2. Deploy a public service to the cluster. Wait for the public load balancer to be created and for an IP to be assigned to it.
3. Deploy an internal service to the cluster. Wait for the internal load balancer to be created and for an IP to be assigned to it.
4. Delete the internal service from the cluster. Wait for the internal load balancer to be removed from Azure before proceeding.
5. Delete the public service from the cluster (note: use --wait=false as this operation won't finish).

Expected behavior
Public and internal services are successfully removed from the cluster. Public and internal load balancers are successfully removed from Azure.

AKS Engine version
Version: v0.59.0
GitCommit: 3eafba286
GitTreeState: clean

Kubernetes version
v1.20.1

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-14T05:14:17Z", GoVersion:"go1.15.6", Compiler:"gc", Platform:"darwin/amd64"}

Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.1", GitCommit:"c4d752765b3bbac2237bf87cf0b1c2e307844666", GitTreeState:"clean", BuildDate:"2020-12-18T16:06:29Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

Additional context

Zipped attachments:

• API model
• Yaml files used to create public/internal k8s services
• Output of kubectl describe with errors related to this issue.
• Error code and message related to this issue displayed on VMSS in Azure Portal

controller-manager-logs-snippet.txt

[welcome[bot]]

👋 Thanks for opening your first issue here! If you're reporting a 🐞 bug, please make sure you include steps to reproduce it.

[jackfrancis]

cc @CecileRobertMichon @devigned

[jackfrancis]

@GeneralKenobi, can you look through your controller-manager logs (kubectl logs ...) and paste any relevant messages around the service being deleted in your repro Step 5?

Thanks!

[devigned]

This sounds like what we are running into here: https://github.com/kubernetes-sigs/cluster-api-provider-azure/issues/1115

[GeneralKenobi]

@GeneralKenobi, can you look through your controller-manager logs (kubectl logs ...) and paste any relevant messages around the service being deleted in your repro Step 5?

Thanks!

Sure, I'm attaching snippet of controller-manager logs since private service deletion (step 4) to public service deletion (step 5). Deletion timestamp of public service is 2021-01-26T07:46:46Z. The last few lines (from event.go:291 to the formatted json error message) are repeating in the logs every 30 seconds.

controller-manager-logs-snippet.txt

[GeneralKenobi]

Fyi: I created a cluster using Azure Kubernetes Service and the same issue occurred.

[CecileRobertMichon]

From the logs you shared it definitely looks like the same issue as kubernetes-sigs/cluster-api-provider-azure#1115. We have root caused it to an underlying platform issue, fix is being rolled out and should be completely out in two weeks.

[dennis-benzinger-hybris]

Thanks for the ETA. Will Azure China also be fixed in two weeks?

[jackfrancis]

Do we have a functional workaround that can unblock folks like @GeneralKenobi in the meantime? Like, if the LB CRUD order of operations is moved around, or delays are added in between LB state transitions (create/delete), are we able to work around this?

[jackfrancis]

@dennis-benzinger-hybris Can you confirm that this issue is observed in Azure China?

[dennis-benzinger-hybris]

@jackfrancis No, I can't confirm. Neither that it happens nor that it does not happen. I just assumed that China runs on the same code base and therefore has the same issue.

@GeneralKenobi Could you please test in China? Thanks.

[GeneralKenobi]

I tested this and I can confirm that this bug also affects China.