Aad-pod-identity: Update of an AKS Cluster is causing the User Assigned Identities to the VMSS to disappear

[BartVanBerkel]

Describe the bug
When you have a running instance of AKS and redeploy the AKS cluster it will remove the User Assigned Identities from the VMSS, which are assigned to the VMSS by the MIC. The MIC has already created the "azureassignedidentities.aadpodidentity.k8s.io" so for the pods, so it won't notice that it has to assign the identity to the VMSS, as it 'thinks' it already did that.

Steps To Reproduce

• Deploy AKS, with Managed Identity as per https://docs.microsoft.com/en-us/azure/aks/use-managed-identity.
• Deploy Pod Identity
• Deploy App which uses an identity
• Check in the Azure Portal if the identity is assigned to the VMSS
• Redeploy the AKS cluster. (in our case with ARM)
• Check again in the Azure Portal if the identity is assigned to the VMSS (in our case it isn't)
• If not assigned, delete one of the application pods.
• Check again in the Azure Portal if the identity is assigned to the VMSS, now it should be.

Expected behavior
I would expect that MIC check on a regular basis whether the Identity is still assigned to the VMSS. This way it would notice that there is an identity missing and add it to the VMSS.

AAD Pod Identity version
Version: 1.5.5. Build date: 2020-01-28-23:11

Kubernetes version
Kubernetes server version: v1.16.9 (but also on v1.16.7)

Additional context
n/a

[galvesribeiro]

Hello @BartVanBerkel

We are planning on deploy a new system using AKS and we currently use MSI on other Azure Services to access other Azure resources (i.e. App Configuration, KeyVault etc.) and we’re look to use MIC to support the same MSI as we have on Azure Functions and App Services but in kubernetes with pods.

Looking into the issues, the most critical we’ve found is yours... The issue title of this issue says “update”... Do you mean that if we update the cluster thru the portal the identities are missed? Or are you experiencing this when you _redeploy_ the cluster using ARM?

We don’t see a use case for us where we need to redeploy our cluster with ARM once it is out, so it should not be a problem for us. However, if when I upgrade the cluster version from the portal we have the identities cleaned up, that it definitively is a nogo for us with MIC.

Can you clarify those points? It would help us a lot!

Thank you!

[BartVanBerkel]

Hi @galvesribeiro

I've also noticed it when I scale the cluster with az aks scale .. Havn't tested an upgrade or the cluster autoscaling feature, but I assume that it will cause the same problem.

How I'm looking at it is that the AKS resource is updating the VMSS on behalf of me, so it isn't aware of the User Assigned Identities which are managed through Pod Identity. So after the update has finished the Identities are not connected to it anymore.

As I'm not sure whether it is a problem with Pod Identity or an Azure problem and therefor I have also opened a case with Azure. The engineer has been able to reproduce the issue and is opening a ticket at the AKS PG.

[galvesribeiro]

Thanks @BartVanBerkel! Have the support team gave you some feedback? Please keep us updated on your issue. This is total blocker. I believe that if it happens on the scale operation as well, that would happen with the case of failure of the node and start of a new one. If that basic primitive doesn't work, it makes the whole thing unusable in production.

@aramase / @chewong do you have any thoughts on this?

[chewong]

Hi @BartVanBerkel. MIC will only unassign user-assigned identities from the underlying node (VM/VMSS) if one of AzureIdentity, AzureIdentityBinding, or the pods that are consuming the user-assgined identity is deleted. It doesn't periodically check whether a user-assigned identity is actually assigned to the node on Azure because MIC uses AzureAssignedIdentity CRD as the single source of truth for the current state of identity assignment (as you mentioned above).

Could you share how you are redeploying your AKS cluster?

[BartVanBerkel]

Hi @BartVanBerkel. MIC will only unassign user-assigned identities from the underlying node (VM/VMSS) if one of AzureIdentity, AzureIdentityBinding, or the pods that are consuming the user-assgined identity is deleted. It doesn't periodically check whether a user-assigned identity is actually assigned to the node on Azure because MIC uses AzureAssignedIdentity CRD as the single source of truth for the current state of identity assignment (as you mentioned above).

Could you share how you are redeploying your AKS cluster?

Hi @chewong
Please have a look at the first post of this issue. Updating with ARM isn’t required, as an az aks scale also causes the issue. The MIC still asumes that all the Identities are assigned as it had created the AzureAssignedIdentity.

@galvesribeiro to resolve the issue you can safely remove all `AzureAssignedIdentities’ so MIC will reassign the Identities to the VMSS.

[aramase]

@BartVanBerkel That's right! MIC uses the k8s CRD as the state of world. When AzureAssignedIdentity is in Assigned state, the assumption is the identities have been successfully assigned by MIC and the lifecycle of identity is managed by pod-identity. There is no reconciliation for changes done outside of the k8s cluster which in this case is AKS removing the identities that it's not aware of from the VMSS model during upgrades/scale. We are also checking internally with the AKS team about this behavior.

[galvesribeiro]

@galvesribeiro to resolve the issue you can safely remove all `AzureAssignedIdentities’ so MIC will reassign the Identities to the VMSS.

@BartVanBerkel remove where?

[BartVanBerkel]

@galvesribeiro to resolve the issue you can safely remove all `AzureAssignedIdentities’ so MIC will reassign the Identities to the VMSS.

@BartVanBerkel remove where?

@galvesribeiro in K8s, so if you run kubectl get AzureAssignedIdentities -A -oyaml | kubectl delete -f - you should be good.

[galvesribeiro]

ahh ok, delete the CRDs

[aramase]

@BartVanBerkel I just tried out az aks scale to scale up/down my aks cluster and didn't see the identities being removed from the VMSS. Even when pods were moved around during scale down, the identities were still assigned to the VMSS and MIC tried re-assigning without any errors.

Could you provide any steps to repo the issue you observed during az aks scale?

[BartVanBerkel]

@BartVanBerkel I just tried out az aks scale to scale up/down my aks cluster and didn't see the identities being removed from the VMSS. Even when pods were moved around during scale down, the identities were still assigned to the VMSS and MIC tried re-assigning without any errors.

Could you provide any steps to repo the issue you observed during az aks scale?

@aramase The support engineer was able to repro the issue I observed, so I guess you are doing something wrong.

Important is that you deploy AKS with Managed Identity as per https://docs.microsoft.com/en-us/azure/aks/use-managed-identity.

1. First execute az vmss show -n <nodepoolname> -g <MC_resourcegroup> --query 'identity.userAssignedIdentities' | grep subscriptions you should (at least) see one identity. (<clustername>-agentpool which is used for kubelet)
2. Make sure that you set the right roles assignments for pod-identity
3. Deploy pod-identity
4. Create a User Assigned Identity in the portal or with az-cli, whatever you like. ;-) Make sure you put it in a ResourceGroup which you have set the right roles assignments for.
5. Deploy the AzureIdentity, AzureIdentityBinding and Pod or Deployment with the right selector to it.
6. execute az vmss show -n <nodepoolname> -g <MC_resourcegroup> --query 'identity.userAssignedIdentities' | grep subscriptions you should see (at least) two identities. One will be the <clustername>-agentpool the other one should be the identity you just deployed into the cluster.
7. then run az aks scale -c <int> so it adds or removes a node from the VMSS.
8. execute az vmss show -n <nodepoolname> -g <MC_resourcegroup> --query 'identity.userAssignedIdentities' | grep subscriptions you should be able to see that the Identity created in step 4 is removed from the VMSS.
9. Execute kubectl get AzureAssignedIdentities -A -oyaml | kubectl delete -f - wait some time so the MIC is able to add the AzureAssignedIdentities again.
10. Execute az vmss show -n <nodepoolname> -g <MC_resourcegroup> --query 'identity.userAssignedIdentities' | grep subscriptions you should be able to see that the Identity created in step 4 is added to the VMSS.

[aramase]

Important is that you deploy AKS with Managed Identity as per https://docs.microsoft.com/en-us/azure/aks/use-managed-identity.

@BartVanBerkel I was trying it out with a Service Principal cluster. I'll try it out with a Managed Identity cluster to confirm.

[BartVanBerkel]

@aramase I see that you have created a known issue. Thanks.

[galvesribeiro]

Nice! Following both your issue and AKS one.

I wonder if the fix will be something that will be applied to existent clusters or something that we require a re-deploy...

[BartVanBerkel]

This issue can be closed, as Managed Identity AKS clusters - intermittent removal of identities on upgrade/scale is fixed.

[elsesiy]

@aramase @chewong This should be re-opened as @TomGeske re-opened the linked issue and I can confirm it's not working yet.

[aramase]

@elsesiy The issue was re-opened for upgrade scenarios. During upgrade, MIC will re-assign the identities that were deleted by AKS during upgrade. Just checking to see if you're seeing that behavior?

[elsesiy]

I upgraded a cluster from 1.15.10 to 1.16.10 and it didn't work. The identities were still there but all calls to the respective Azure service failed with "identity not found". Only after refreshing the assigned identities manually it worked. IIRC I saw a lot of 404s in the NMI logs but could confirm that the identity exists in the AAD.