Aad-pod-identity: Missing Managed Identity is breaking other deployments/apps

[vickeyrihal1]

MI = Managed Identity

Describe the bug
The cluster has multiple client apps hosted with azure vmss using specific MIs assigned to resources in different subscription. Lately one of the client deleted the managed identity and missed to remove it from vmss.
After this, the clients that had other MIs mapped, started seeing missing MI errors. Although their actual identities were intact. Upon deletion of former MI from vmss, everything started working alright.

Issue interception
Client-A started seeing MI failures in their application.

• There were plenty of 404 errors in aad-pod-identity-nmi.
  
  
  
  • and against multiple namespaces/pods.
  
  
• The aad-pod-identity-mic deployment logs however pointed at specific (broken) Identity.
  
  
  
  • The problematic identity belonged to a separate client though (say client-B)
  
  

Steps To Reproduce
Attach multiple MIs to multiple vmss and pods. And then remove MI from azure, however don't cleanup the mapping from vmss. Then try to follow the app behaviour of all the apps.

Expected behavior

AAD Pod Identity version
1.5.4-rc1

Kubernetes version
1.16.4

Additional context

[aramase]

@vickeyrihal1 This behavior is expected with the current releases, because MIC gets the list of identities that's currently on the VMSS, appends/deletes any identities and then calls a VMSS CreateOrUpdate. During this CreateOrUpdate call if the old identities that are no longer in azure aren't cleaned up that'll result in an error.

[aramase]

@vickeyrihal1 I see you are using the release candidate image. I would recommend you use an official build release - https://github.com/Azure/aad-pod-identity/releases/tag/v1.5.5.

[vickeyrihal1]

@aramase - Thanks for the info.

During this CreateOrUpdate call if the old identities that are no longer in azure aren't cleaned up that'll result in an error.

Then this scenario could cause outage for other apps, no? It's not only about missing MI, but also if someone revokes its access.

I am coming from multi-tanent platform. I don't want to make a problematic user/app to have an impact on other clients/apps.

Would this be eligible for a Critical bug?

[vickeyrihal1]

• @kkmsft

[dharmab]

@aramase this behavior seems quite fragile, although I understand that the root issue is due to the createOrUpdate call within the management API refusing to update if any submitted identity does not exist.

Would it be possible or practical for AAD Pod Identity to check for 404'd identities and prune from from the list before submitting the VMSS update?

[aramase]

@dharmab That might be possible, but if there are multiple identities with issue, the error returned from the CreateOrUpdate call is only for the first identity with issue in the list of identities. We could think about potentially updating one identity at a time if there is any error, but that does not scale well.

[dharmab]

I don't think it'd have to do one at a time on error. If you have ten identities, and the API tells you that identities[3] is invalid, you could pop the third identity out of the list and try again with the remaining nine. Then if it tells you that identities[7] is invalid, you can pop again. That would scale the number of createOrUpdate attempts with the number of invalid identities instead of the total number of identities.

The worst case can get pretty bad, but the common case is usually going to be a small number of bad identities.

[dharmab]

@aramase, what are your thoughts about the above error handling approach?

We could also cap the number of retries to some configurable level to avoid retrying a createOrUpdate 100+ times in an extreme case.

[aramase]

@dharmab Retries with a cap does sound reasonable. I'll play around that and few other approaches this week to see what can be done.

[aramase]

This particular issue should be resolved with the PR to switch to patch update instead of doing CreateOrUpdate. The update call will only be made with the desired list of identities to be added/removed.

Adding a finalizer will ensure when the pods/namespace are removed from the cluster, the identities from the underlying vm/vmss are also cleaned up.

The issue with one wrong identity in the batch still messing with the entire operation exists. I'll add the retries with cap for such scenarios.

[vickeyrihal1]

Thanks @aramase. Assuming we are waiting for adding retry logic before cutting out a new release?

[aramase]

There are a couple of changes that have gone in to prevent most of error scenarios - validation of resourceID, switching to PATCH instead of PUT. The plan is to cut release 1.6 this week. The retry logic will probably make it in 1.6.1.

The PATCH will handle cases where an identity still exists on the VMSS, but has been deleted from azure.

[vickeyrihal1]

Perfect! thanks @aramase.

[vickeyrihal1]

@aramase - any news?

[aramase]

@vickeyrihal1 Just cut a new release - https://github.com/Azure/aad-pod-identity/releases/tag/v1.6.0. Please review the changes and let us know if you have any feedback after trying it out.

[vickeyrihal1]

@aramase QQ: let's suppose we add a new VMSS to the worker tier. And scheduler tries to schedule three pods with 3 different identities. Out of 3 identities, the first identity is invalid. Then how would MIC treat that? I mean, if the first one is invalid, would it jump over and attach the next to vmss or will error out?

Looking forward to get the retry logic shipped in next release.

[jaswanth-v]

@aramase Thank you for working on this. Is there an ETA for v1.6.1 release with retry logic ?

[aramase]

Out of 3 identities, the first identity is invalid. Then how would MIC treat that? I mean, if the first one is invalid, would it jump over and attach the next to vmss or will error out?

If there is an error with single identity, then the entire operation is rejected.

I'll work on it next week. We expect to release 1.6.1 mid May, so that should be some time in the next 2 weeks.

[alexcristi]

More findings: when having any identities with empty specs, the type upgrades are also failing, resulting in aad-pod-identity-mic not starting at all.

Have this AzureIdentity:

$ k -n ns-ethos-allphin11-dev get AzureIdentity allphin--scale-msi-test-dev-qbs-azureidentity -oyaml
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentity
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"aadpodidentity.k8s.io/v1","kind":"AzureIdentity","metadata":{"annotations":{},"labels":{"kd_containerid_container1":"true","kd_created":"true","kd_environment":"dev","kd_gitrepo":"allphin--scale-msi-test"},"name":"allphin--scale-msi-test-dev-qbs-azureidentity","namespace":"ns-ethos-allphin11-dev"},"spec":{"ClientID":{"client-id":null},"ResourceID":{"resource-id":null}}}
  creationTimestamp: "2019-08-14T21:30:51Z"
  generation: 2
  labels:
    kd_containerid_container1: "true"
    kd_created: "true"
    kd_environment: dev
    kd_gitrepo: allphin--scale-msi-test
  name: allphin--scale-msi-test-dev-qbs-azureidentity
  namespace: ns-ethos-allphin11-dev
  resourceVersion: "648845163"
  selfLink: /apis/aadpodidentity.k8s.io/v1/namespaces/ns-ethos-allphin11-dev/azureidentities/allphin--scale-msi-test-dev-qbs-azureidentity
  uid: f048ccf5-725b-44bb-be22-1fde458bf92b
spec:
  ClientID: {}
  ResourceID: {}

Install aad-pod-identity v1.6.0; It is failing to start with:

$ ks logs aad-pod-identity-mic-654844bb9c-9k67j -c mic
I0507 07:19:48.704474       1 main.go:95] Starting mic process. Version: 1.6.0. Build date: 2020-04-24-19:19
W0507 07:19:48.704535       1 main.go:100] --kubeconfig not passed will use InClusterConfig
I0507 07:19:48.704543       1 main.go:114] kubeconfig () cloudconfig (/etc/kubernetes/azure.yaml)
I0507 07:19:48.704820       1 main.go:122] Running MIC in namespaced mode: true
I0507 07:19:48.704845       1 main.go:126] Client QPS set to: 5. Burst to: 5
I0507 07:19:48.704861       1 mic.go:124] Starting to create the pod identity client. Version: 1.6.0. Build date: 2020-04-24-19:19
I0507 07:19:48.743159       1 mic.go:130] Kubernetes server version: v1.16.9
I0507 07:19:48.743846       1 probes.go:42] Initialized health probe on port 8080
I0507 07:19:48.743865       1 probes.go:45] Started health probe
I0507 07:19:48.743947       1 metrics.go:330] Registered views for metric
I0507 07:19:48.743999       1 prometheus_exporter.go:23] Starting Prometheus exporter
I0507 07:19:48.744013       1 metrics.go:337] Registered and exported metrics on port 8888
I0507 07:19:48.744018       1 mic.go:206] Initiating MIC Leader election
I0507 07:19:48.744033       1 leaderelection.go:242] attempting to acquire leader lease  default/aad-pod-identity-mic...
I0507 07:19:48.756304       1 leaderelection.go:252] successfully acquired lease default/aad-pod-identity-mic
I0507 07:19:48.761177       1 mic.go:279] Upgrading the types to work with case sensitive go-client
F0507 07:19:48.766335       1 mic.go:305] Type upgrade failed with error: type upgrade failed. error: unmarshal to object: *v1.AzureIdentityList, error: json: cannot unmarshal object into Go struct field AzureIdentitySpec.items.spec.clientID of type string

[vickeyrihal1]

@aramase - Based on previous comment, this could still result in outage sort of situation. Hopefully, that would be sorted with timeout introduction.

https://github.com/Azure/aad-pod-identity/issues/515#issuecomment-622511796

wonder if we could change the priority of this ticket to P1? Is there any keyword for us to add a label?

[aramase]

@vickeyrihal1 This is the PR for retries - https://github.com/Azure/aad-pod-identity/pull/601. Feel free to review and chime in on the PR if you have any concerns. We plan to merge this PR this week and include in 1.6.1.

[vickeyrihal1]

@aramase - #601 looks to have merged now. Do we have some timelines for v1.6.1?

[chewong]

We are aiming to release 1.6.1 by the end of this week.

[mhsh64]

we have upgraded to 1.6.1 but we still see this issue, pod identity is not able to remove the MIs removed from azure from vmss and we are keeping continuous errors :

Applying binding my_identityname node aks-agentpool-64646-vmss for pod mypod resulted in error compute.VirtualMachineScaleSetsClient#Update: Failure sending request: StatusCode=400 -- Original Error: Code="FailedIdentityOperation" Message="Identity operation for resource '/subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Compute/virtualMachineScaleSets/aks-agentpool-64646-vmss' failed with error 'Failed to perform resource identity operation. Status: 'BadRequest'. Response: '{\"error\":{\"code\":\"BadRequest\",\"message\":\"Resource '/subscriptions/xxx/resourcegroups/xxx/providers/microsoft.managedidentity/userassignedidentities/my_identityname' was not found.\"}}'.'."

[dharmab]

@mhsh64 did you set the --update-user-msi-max-retry flag on ~NMI~ MIC?

[mhsh64]

@dharmab I do not know this flag, in the documentation I did not find anything related to that. Can you please let me know what shoud i do?

[chewong]

Sorry for the lack of documentation. I will add it ASAP. Basically, --update-user-msi-max-retry is the number of attempts for the UpdateUserMSI function (the function that performs assigning / unassigning identities from the underlying VMSS).

In the meantime, could you provide some logs on what happened after that error occurred?

[mhsh64]

This is the entire log for each cycle mic is doing its tasks

I0605 14:17:41.798402 1 mic.go:1009] Processing node aks-agentpool-xxx-vmss, add [1], del [0], update [0] I0605 14:17:41.841498 1 cloudprovider.go:208] Updating user assigned MSIs on aks-agentpool-xxx-vmss, assign [1], unassign [0] E0605 14:17:42.082812 1 vmss.go:85] Failed to update VM with error compute.VirtualMachineScaleSetsClient#Update: Failure sending request: StatusCode=400 -- Original Error: Code="FailedIdentityOperation" Message="Identity operation for resource '/subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Compute/virtualMachineScaleSets/aks-agentpool-xxx-vmss' failed with error 'Failed to perform resource identity operation. Status: 'BadRequest'. Response: '{\"error\":{\"code\":\"BadRequest\",\"message\":\"Resource '/subscriptions/xxx/resourcegroups/xxx/providers/microsoft.managedidentity/userassignedidentities/my_identity' was not found.\"}}'.'." I0605 14:17:42.082994 1 cloudprovider.go:222] Removing /subscriptions/xxx/resourcegroups/xxx/providers/microsoft.managedidentity/userassignedidentities/my_identity from ID list since it is erroneous E0605 14:17:42.083012 1 mic.go:1080] Updating msis on node aks-agentpool-xxx-vmss, add [1], del [0], update[0] failed with error compute.VirtualMachineScaleSetsClient#Update: Failure sending request: StatusCode=400 -- Original Error: Code="FailedIdentityOperation" Message="Identity operation for resource '/subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Compute/virtualMachineScaleSets/aks-agentpool-xxx-vmss' failed with error 'Failed to perform resource identity operation. Status: 'BadRequest'. Response: '{\"error\":{\"code\":\"BadRequest\",\"message\":\"Resource '/subscriptions/xxx/resourcegroups/xxx/providers/microsoft.managedidentity/userassignedidentities/my_identity' was not found.\"}}'.'." E0605 14:17:42.282671 1 mic.go:1097] Applying binding my_identity node aks-agentpool-xxx-vmss000006 for pod my_pod resulted in error compute.VirtualMachineScaleSetsClient#Update: Failure sending request: StatusCode=400 -- Original Error: Code="FailedIdentityOperation" Message="Identity operation for resource '/subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Compute/virtualMachineScaleSets/aks-agentpool-xxx-vmss' failed with error 'Failed to perform resource identity operation. Status: 'BadRequest'. Response: '{\"error\":{\"code\":\"BadRequest\",\"message\":\"Resource '/subscriptions/xxx/resourcegroups/xxx/providers/microsoft.managedidentity/userassignedidentities/my_identity' was not found.\"}}'.'."

[mhsh64]

@chewong thanks. Where and how should I set that flag?

[chewong]

@mhsh64 based on your logs, it looks like the sync cycle is only assigning /subscriptions/xxx/resourcegroups/xxx/providers/microsoft.managedidentity/userassignedidentities/my_identity (based on assign [1], unassign [0]) and since MIC can't find the user-assigned identity on Azure, it removed it from the update list. The updated list is now empty and there is nothing that MIC could do.

If you are anticipating a lot of errors when assigning identities, you can add --update-user-msi-max-retry=<desired retry value> flag (default to 2) to the MIC args - https://github.com/Azure/aad-pod-identity/blob/master/deploy/infra/deployment-rbac.yaml#L235-L240, or through helm install - helm install aad-pod-identity charts/aad-pod-identity --set=mic.updateUserMSIMaxRetry=<desired retry value>

[mhsh64]

@chewong
Thanks
But I am getting the above error continuously.
If the default value for above flag is 2, do we know why I am getting that error all the time?

[mhsh64]

@chewong
I set that flag explicitly to 2, but still I am keep getting that error

[chewong]

@mhsh64 have you checked whether /subscriptions/xxx/resourcegroups/xxx/providers/microsoft.managedidentity/userassignedidentities/my_identity exists or not? The error is expected if the user-assigned identity does not exist.

[mhsh64]

@chewong the Id does not exist is Azure...
I thought in this new release, mic is able to recognize the deleted identity from azure, am I correct?

but it seems it is trying to assign a identity which doesn’t exist and since it cannot find it, it throws an error

[chewong]

MIC does not create or delete user-assigned identities on Azure, that's responsible by the users. It only assigns or unassigns existing user-assigned identities from the underlying VM / VMSS.

It also performs batch identity assignment and unassignement in a single sync cycle because of performance reasons. If there is an erroneous user-assigned identity in the batch (in your case, it does not exist on Azure), MIC removes it from the batch and attempts the assignment / unassignment again.

[mhsh64]

Correct, but do we expect MIC removes any deleted identity (from azure) from VMSS?
@chewong
That was the issue we expected to got resolved. On VMSS we have many Identities which have been deleted from Azure, but mic was not able to remove(unbind) them from VMSS
And this is the reason we are getting so many errors in mic logs

[mhsh64]

@chewong
The other thing, you are adding a configmap to record and confirm the successful type upgrade for existing Azureidentity and Azureidentitybinding.
We had these resources with capital letters in version 1.5 , now should I expect to see any confirmation in configmap that these existing resources fields have been upgraded to the correct format?
I do not see anything in there:

kubectl describe configmap aad-pod-identity-config
Name: aad-pod-identity-config
Namespace: podidentityns
Labels:
Annotations:

Data

type-upgrade-status:

1.6.1
Events:

[aramase]

Correct, but do we expect MIC removes any deleted identity (from azure) from VMSS?

@mhsh64 In this case as you can see in the logs, the identity is trying to be assigned as there is still a pod referencing the identity? MIC is trying to assign the identity and it fails at every sync cycle. If this identity was being unassigned then MIC would detect if the identity doesn't exist in the VMSS in the first place.

Updating user assigned MSIs on aks-agentpool-xxx-vmss, assign [1], unassign [0]

We had these resources with capital letters in version 1.5 , now should I expect to see any confirmation in configmap that these existing resources fields have been upgraded to the correct format?

type-upgrade-status 1.6.1

indicates the 1.6.1 release on the cluster was responsible for upgrading all the AzureIdentity and AzureIdentityBinding to the right case. MIC sets this key in the config map only after successfully upgrading all the current cluster resources. The next time MIC is deployed, if this value has already been set, then MIC will skip type upgrade as it's already complete.

[mhsh64]

Thanks for your response. So since we have this key, it confirms the resources have the correct format, which I already checked manually as well.

Regarding the deleted Identities, that is correct, it might there some pods referring to the deleted ids, so I guess if we delete that pod also, mic is able to remove that deleted id from vmss

Thanks

[aramase]

@vickeyrihal1 Hope the above enhancements and fixes have resolved the issues. Closing this issue now. Please feel free to reopen if you have any questions.