Aad-pod-identity: Better documentation and automation scripts to set MIC permissions

[pascal-enz]

My team has been struggling with getting the role assignments to work multiple times and by looking at the issues list, it seems we are not the only ones. Better documentation as well as scripts to automate that step would be highly appreciated.

Issues:

• It seems the MIC pods need to be restarted for changed permissions to take effect. This is not documented and can lead to massive confusion and frustration. I believe in many cases, the permissions we set were actually correct, but because they didn't take effect, we spent a lot of time troubleshooting.
• It seems the required permissions are different for RBAC and non-RBAC enabled clusters. But I might be wrong and been confused by the previous issue. However, some clarification would be appreciated.
• Assigning permissions for every single identity does not seem practical in many scenarios. Is there a way to grant the cluster general access to assign identities?
• It's also not documented, how to troubleshoot issues (i.e look at the MIC logs).

Apart from improving the documentation, we would highly recommend and appreciate if a PowerShell or Bash script could be provided to reliably automate the assignment of permissions.

[pascal-enz]

One more thing, the documentation chapter "Set Permission for MIC" states: "This step is only required for user-assigned MSI."
What does that mean? Does the solution support system-assigned identities as well?

[erlacherl-city]

Hi @pascal-enz , I'm also struggling with this at the moment. As for system-assigned identities, have you seen the docs/readmes folder? It contains instructions for system-managed identities: https://github.com/Azure/aad-pod-identity/blob/master/docs/readmes/README.msi.md

EDIT: There is also debugging information in the wiki - https://github.com/Azure/aad-pod-identity/wiki/Debugging

It's all very fragmented and definitely not enough information, so I agree that it would be great if more work could be put into making the documentation more consistent and with more debugging information.

[Veraticus]

We've run into this a lot too -- especially providing MSI credentials for every identity is extremely fiddly. Shouldn't the cluster be able to look those up itself and provide them to pods and services as requested? The user experience between this project and the equivalent AWS project (https://github.com/uswitch/kiam) is really night-and-day.

I don't want to just be whiny about this, but I think a pass at stability and user experience would be really helpful here.

[aramase]

Thank you for opening this issue. I'm in the process of adding troubleshooting docs. Will also add the suggestions here in the docs.

@Veraticus The resourceID in AzureIdentity is required so MIC knows which user assign identity to assign to the underlying VM/VMSS. The clientID in AzureIdentity is used by NMI to match the identity when the request for token is made with clientID.

[chewong]

@pascal-enz We had added a significant amount of documentation on role assignment since the creation of this issue. To address your concerns:

It seems the MIC pods need to be restarted for changed permissions to take effect. This is not documented and can lead to massive confusion and frustration. I believe in many cases, the permissions we set were actually correct, but because they didn't take effect, we spent a lot of time troubleshooting.

It's suggested that you should perform all the necessary role assignments before you deploy aad-pod-identity. ~Also, I don't think a restart is necessary for the permission to take effect as long as you are assigning the appropriate roles to your cluster identity.~

After some validation, we do have to restart MIC for the new permission to take effect. That's because MIC doesn't refresh the token for its Azure client. I will look into this issue when working on https://github.com/Azure/aad-pod-identity/issues/647.

It seems the required permissions are different for RBAC and non-RBAC enabled clusters. But I might be wrong and been confused by the previous issue. However, some clarification would be appreciated.

RBAC and non-RBAC require the same role assignment. I can document this in our repo.

Assigning permissions for every single identity does not seem practical in many scenarios. Is there a way to grant the cluster general access to assign identities?

You could assign Managed Identity Operator role to your cluster identity / service principal at the resource group level, or even the subscription level. See https://github.com/Azure/aad-pod-identity/blob/master/docs/readmes/README.role-assignment.md#user-assigned-identities-that-are-not-within-the-cluster-resource-group for more details.

It's also not documented, how to troubleshoot issues (i.e look at the MIC logs).

We also added a list of common issues and how to troubleshoot them in https://github.com/Azure/aad-pod-identity/blob/master/docs/readmes/README.troubleshooting.md. Feel free to open an issue / PR if you are experiencing issues that aren't on the list.

I understand that currently, there is a lot of manual labors before users can start using aad-pod-identity so I am going create a script to automate these role assignments.