Aad-pod-identity: Deleting azureassignedidentities.aadpodidentity.k8s.io resources takes very long

[BartVanBerkel]

Describe the bug
In the 1.6.1 version of aad-pod-identity it takes a very long time to manually remove azureassignedidentities. I had it running for more than an hour, but nothing happens. In some cases it does work, but it is definitely not very consistent.
In version 1.5.5 it worked flawless and took around 2 seconds to process.

Steps To Reproduce

kubectl -n <namespace> delete azureassignedidentities.aadpodidentity.k8s.io <azureassignedidentity name>

It outputs the line below and then the command hangs..

azureassignedidentity.aadpodidentity.k8s.io "<azureassignedidentity name>" deleted

Expected behavior
In version 1.5.5 of aad-pod-identity it took roughly 1 or 2 seconds to remove the resource.

AAD Pod Identity version
Version: 1.6.1. Build date: 2020-05-22-17:50

Kubernetes version
1.16.9

Additional context
n/a

[chewong]

Hey @BartVanBerkel. AzureAssignedIdentity isn't supposed to be deleted by the user, but by MIC. In what situation would you need to manually delete it? MIC will automatically delete it when it detects deletion of AzureIdentity, AzureIdentityBinding or the pod that is consuming the AzureIdentity.

With the addition of finalizer (introduced in https://github.com/Azure/aad-pod-identity/pull/593), AzureAssignedIdentity will be fully removed only when the user-assigned identity on Azure is un-assigned from the underlying VM / VMSS. That's why it hangs in your situation.

[BartVanBerkel]

Hi @chewong
Ah I see that the behavior has been changed.

I used it as mentioned in #633. I think you've also said that the MIC uses the AzureAssignedIdentity as a single source of truth, but wouldn't it be better if MIC would periodically verify if it's truth is still true? Maybe just to prevent troubles in the future.

[chewong]

Yea I think it will be beneficial to have a periodic check. Opened https://github.com/Azure/aad-pod-identity/issues/647 to track the feature.

[BartVanBerkel]

@chewong Thanks

[DiegoBallve]

I'm having a bit of problem with this one, I'm using the aad-pod-identity chart. I did an uninstall (to move it to default namespace because it is not working in a custom aad-pod-identity namespace), but now I cannot redeploy it because some resources already exists.

README says:
To uninstall/delete the last deployment:

$ helm ls
$ helm delete [last deployment] --purge

The command removes all the Kubernetes components associated with the chart and deletes the release.

The CRD created by the chart are not removed by default and should be manually cleaned up (if required)

kubectl delete crd azureassignedidentities.aadpodidentity.k8s.io
kubectl delete crd azureidentities.aadpodidentity.k8s.io
kubectl delete crd azureidentitybindings.aadpodidentity.k8s.io
kubectl delete crd azurepodidentityexceptions.aadpodidentity.k8s.io

But as BartVanBerkel reports, it gets stuck in deleting.

[chewong]

@DiegoBallve could you try manually un-assign all user-assigned identities from the underlying nodes on Azure before uninstalling?

[DiegoBallve]

Thank you for the tip, @chewong . Actually I just had to wait a couple of minutes for the identity to get deleted before I tried to delete crd, then there was no problem.

kubectl get azureassignedidentities.aadpodidentity.k8s.io --all-namespaces

In my case I was deleting the azure-ingress chart, which was creating the azureassignedidentities objects. Once that chart got deleted, the azureassignedidentities stayed behind for a couple of minutes.

[kcdevops46]

I am facing issues with deleting AzureAssignedIdentity.
I tried deleting with:
$kubectl delete AzureAssignedIdentities $name
kubectl delete crd azureassignedidentities.aadpodidentity.k8s.io

Thank you

[aramase]

@kcdevops46 We have added a finalizer to the AzureAssignedIdentity which means MIC pod will ensure the identities are removed from the underlying node and only then delete the AzureAssignedIdentity. If MIC is deleted, then the identities that were assigned to the node will be left behind. In cases where you want to delete MIC without the cleanup, you can run the following command to remove the finalizer -

kubectl get azureassignedidentity -A -o=json | jq '.items[].metadata.finalizers=null' | kubectl apply -f -

This above command will manually remove the finalizer, but please note since MIC was deleted the identities that were assigned will need to be manually cleaned up if not required.

[kcdevops46]

@aramase Thank you very much. It worked.