Aad-pod-identity: MIC hitting (un-)assignment limit of 150

[aranair]

Describe the bug

Whenever MIC needs to either assign or unassign >150 (in one cycle?) - it fails with the following message:

...
User Assigned Identities Number '165' Assign exceed limit '150' in one request
...

This is the same error message as the issue I've raised previously: https://github.com/Azure/aad-pod-identity/issues/637 before. We were on v1.5.4 previously but we upgraded to v1.6.3 last night and it ran into the same issue.

When this happens, no new assignments would go through.

The only way to "unstuck" it was by manual intervention. We removed some role assignments from the VMSS via the UI, to lower the number of un-assignments that it was trying to do.

Steps To Reproduce

Hypothetical steps that I think will reproduce it but I'm not 100% on how it arrived at this state atm:

• Stop MIC operations (e.g. scale it down first or something to simulate a CLBO)
• Manually assign 150 role assignments that are not needed onto the VMSS role assignment via the UI. These should be
• Start MIC
• I presume it would try and un-assign a list >150

Expected behavior

• If the number of assignment/un-assignments is greater than 150, it should potentially split it up into multiple cycles

AAD Pod Identity version
v1.6.3

Kubernetes version
1.16.10

Additional context

This is on role assignments to a VMSS not to individual nodes.

[chewong]

Thanks for raising the issue. Let me try to reproduce the issue and get back to you as soon as possible

[chewong]

To clarify, do I need to create 150 unique user-assigned identities to repro this issue? I have 150 random role assignments for 20 user-assignment identities to my VM but wasn't able to repro the issue

[aranair]

Hmmm, I do believe it needs to be 150 identities.
(Through the VMSS UI, we can only see the identities directly I believe)

Also, side question: how are you able to have 150 role assignments with only 20 identities if it's on the VMSS?

[chewong]

Got it, let me give it a try again. To answer your question, I assigned different roles to the same user-assigned identity (i.e. 8 role assignment for each user-assigned identity).

[chewong]

We confirmed with the internal team that it does have a limit of 150 user assigned identities in a single IdentityPut or IdentityPatch request. Let me put this issue to the v1.7.0 milestone.