Aad-pod-identity: Pod Identity disabled on cluster upgrade

[markgould]

I went to upgrade my cluster from 1.18 to 1.19.3 and initially ran into a vcore quotea limit in my subscription - resolved that and then completed the upgrade. After that, none of my managed identities worked. My pod logs showed "identity not found" coming from the managed identity service. All of the CRDs had been removed.

I set this up initially using the new preview pod identity feature (https://docs.microsoft.com/en-us/azure/aks/use-azure-ad-pod-identity). I saw some previous issues where the managed identities get removed from the VMSS, but I checked and they were all intact.

I attempted to re-add my pod identity using the "az aks pod-identity add" command and received the error:

The pod identity addon is not enabled for this managed cluster yet.
To enable, run "az aks update --enable-pod-identity

After enabling the pod identity feature, and then re-running the "az aks pod-identity add" command, everything started working again.

I'm not sure if the initial vcore quota caused the issue, or if the cluster upgrade causes the pod identity feature to be removed.

[chewong]

Looks like you are not using the open-source version of aad-pod-identity. Could you open the same issue at https://github.com/Azure/AKS/issues? cc @bcho

[bcho]

@markgould could you file a customer support ticket with your cluster information in Azure portal? So we can check the related operations with your cluster information in our end. Thanks.

[bcho]

I also tried following setup:

1. create cluster with 1.18.10
2. assign identity to the cluster with az cli
3. check the cluster ✅
4. upgrade the cluster to 1.19.3 with az cli
5. check the cluster ✅

cannot repro this.

[markgould]

@bcho Yes, I'll file that now. I think it must have had something to do with hitting the subscription quota limit, the cluster was in a failed state as it was unable to provision a new node for the upgrade.