Aad-pod-identity: mic pod has insufficient permissions on AKS

[xtellurian]

Let me premise this by saying I followed the README and got into this position. The root cause may be a bug, but more likely is missing or misleading documentation.

ENV:

• AKS
• eastus
• 1 node

Deployed the Infra and the demo, plus the bindings, as described in the README.

Looks like the MIC pod is unable to assign the identity.

Cannot perform action Microsoft.ManagedIdentity/userAssignedIdentities/assign/action.

computer:~$ kubectl logs -f mic-64ddcf5f65-h4hft
I0606 05:34:42.676870       1 mic.go:28] kubeconfig (/etc/kubernetes/kubeconfig/kubeconfig) cloudconfig (/etc/kubernetes/azure.json)
I0606 05:34:42.678773       1 mic.go:36] Starting to create the pod identity client
I0606 05:34:42.682479       1 mic.go:82] Pod watcher started !!
I0606 05:34:42.682533       1 mic.go:84] CRD watcher started
I0606 05:34:42.682558       1 mic.go:42] AAD Pod identity controller initialized!!
I0606 05:34:42.682584       1 mic.go:140] Sync thread started
I0606 05:34:44.956703       1 crd.go:123] Got id podid to assign
I0606 05:34:44.956798       1 crd.go:142] Creating assigned Id: demo-5788d95785-pwk7g-default-podid
I0606 05:34:44.965067       1 cloudprovider.go:170] Find aks-nodepool1-15831963-0 in resource group: MC_k8s-test_clusterFrank_eastus
E0606 05:34:46.288109       1 cloudprovider.go:147] compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=403 -- Original Error: Code="LinkedAuthorizationFailed" Message="The client 'd6f4dd69-b093-45cd-833d-298be532685d' with object id 'd6f4dd69-b093-45cd-833d-298be532685d' has permission to perform action 'Microsoft.Compute/virtualMachines/write' on scope '/subscriptions/c5760548-23c2-4223-b41e-5d68a8320a0c/resourceGroups/MC_k8s-test_clusterFrank_eastus/providers/Microsoft.Compute/virtualMachines/aks-nodepool1-15831963-0'; however, it does not have permission to perform action 'Microsoft.ManagedIdentity/userAssignedIdentities/assign/action' on the linked scope(s) '/subscriptions/c5760548-23c2-4223-b41e-5d68a8320a0c/resourcegroups/k8s-test/providers/Microsoft.ManagedIdentity/userAssignedIdentities/podId'."
E0606 05:34:46.288154       1 mic.go:365] compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=403 -- Original Error: Code="LinkedAuthorizationFailed" Message="The client 'd6f4dd69-b093-45cd-833d-298be532685d' with object id 'd6f4dd69-b093-45cd-833d-298be532685d' has permission to perform action 'Microsoft.Compute/virtualMachines/write' on scope '/subscriptions/c5760548-23c2-4223-b41e-5d68a8320a0c/resourceGroups/MC_k8s-test_clusterFrank_eastus/providers/Microsoft.Compute/virtualMachines/aks-nodepool1-15831963-0'; however, it does not have permission to perform action 'Microsoft.ManagedIdentity/userAssignedIdentities/assign/action' on the linked scope(s) '/subscriptions/c5760548-23c2-4223-b41e-5d68a8320a0c/resourcegroups/k8s-test/providers/Microsoft.ManagedIdentity/userAssignedIdentities/podId'."
I0606 05:34:46.288231       1 mic.go:259] Sync took: 1.356004792s

Do these permissions need to be set somewhere?

[kkmsft]

Thank you, for trying out pod identity and reporting the issue.

I will try to repro this issue in house. Meanwhile, in order to rule out issues relating to identity being in a different resource group - Is it possible to retry this with an identity created with the same resource group as that of nodes - MC_k8s-test_clusterFrank_eastus ?

[xtellurian]

I recreated the Azure Identity 'podId' in the Resource Group 'MC_k8s-test_clusterFrank_eastus'.

az identity create --name podId --resource-group MC_k8s-test_clusterFrank_eastus

Side note: 'MC_k8s-test_clusterFrank_eastus' was automatically created when I provisioned an AKS cluster in Resource Group 'k8s-test'.

mic pod now successfully binds the identity.

computer:~$ kubectl logs -f mic-64ddcf5f65-h4hft
...
I0606 23:15:52.742629       1 mic.go:259] Sync took: 494.998359ms
I0606 23:19:45.867711       1 crd.go:123] Got id podid to assign
I0606 23:19:45.867829       1 crd.go:142] Creating assigned Id: demo-5788d95785-ghzwv-default-podid
I0606 23:19:45.874002       1 cloudprovider.go:170] Find aks-nodepool1-15831963-0 in resource group: MC_k8s-test_clusterFrank_eastus
I0606 23:20:11.051552       1 cloudprovider.go:162] Underlying cloud provider operation took 25.04421296s
I0606 23:20:11.051846       1 mic.go:259] Sync took: 25.220821436s
I0606 23:20:11.052905       1 event.go:218] Event(v1.ObjectReference{Kind:"AzureIdentityBinding", Namespace:"default", Name:"id-binding-name", UID:"19a07e0e-69e0-11e8-9e9f-4addade2df92", APIVersion:"aadpodidentity.k8s.io/v1", ResourceVersion:"89529", FieldPath:""}): type: 'Normal' reason: 'binding applied' Binding id-binding-name applied on node aks-nodepool1-15831963-0 for pod demo-5788d95785-ghzwv-default-podid

[xtellurian]

More info on the relationship between identities and resource groups.

The following identity (in the same RG as the AKS cluster) caused the error above. Obviously I'd prefer to create identities in the 'k8s-test' RG rather than the automatically provisioned RG 'MC_k8s-test_clusterFrank_eastus'.

computer:~$ az resource list --resource-group k8s-test
[
  {
    "id": "/subscriptions/c5760548-23c2-4223-b41e-5d68a8320a0c/resourceGroups/k8s-test/providers/Microsoft.ContainerService/managedClusters/clusterFrank",
    "identity": null,
    "kind": null,
    "location": "eastus",
    "managedBy": null,
    "name": "clusterFrank",
    "plan": null,
    "properties": null,
    "resourceGroup": "k8s-test",
    "sku": null,
    "tags": null,
    "type": "Microsoft.ContainerService/managedClusters"
  },
  {
    "id": "/subscriptions/c5760548-23c2-4223-b41e-5d68a8320a0c/resourceGroups/k8s-test/providers/Microsoft.ManagedIdentity/userAssignedIdentities/podId",
    "identity": null,
    "kind": null,
    "location": "eastus",
    "managedBy": null,
    "name": "podId",
    "plan": null,
    "properties": null,
    "resourceGroup": "k8s-test",
    "sku": null,
    "tags": {},
    "type": "Microsoft.ManagedIdentity/userAssignedIdentities"
  }
]

[xtellurian]

It took me a while to understand the documentation on how to use and deploy this repository.

I've made a tutorial: Azure CLI and kubectl to use aad-pod-identity in AKS

https://github.com/xtellurian/aad-pods

[khenidak]

@xtellurian This is fantastic! any chance you can PR it into this repo? Would be great to have it along with the rest of stuff like events checking the binding etc..

[xtellurian]

Good idea
https://github.com/Azure/aad-pod-identity/pull/42

[xtellurian]

@khenidak can you expand on what you mean by events checking the binding?

[uday31in]

I have tested user assigned identity in AKS RG (insetad of MC_* RG) and it is working fine so not sure what is the exact issue referred here.

[xtellurian]

Root cause could have been related to the service principal for AKS, or perhaps another permissions issue in Azure.

[AndreasM009]

I had the same issue. I tried to test the application-gateway-kubernetes-ingress. The project provides a detailed installation description and uses aadpodidentity as well. When creating the UserAssignedIdentity within the cluster resource group the assignment doesn't work. Only when the identity is created inside MC_*RG the assignment works.

[kkmsft]

@AndreasM009 - The steps to enable the scenario when the identity is in a different resource group was added to the Readme with #48 Can you please confirm that even after the running the steps here: https://github.com/Azure/aad-pod-identity/blob/master/README.md#providing-required-permissions-for-mic you are still having the issue.

[AndreasM009]

@kkmsft it works now, I had some problems with my service principal. thx.

[wuhanyumsft]

I have met the same issue. The identity needs to be the same resource group of MC-XXX should be stated explicitly in document https://github.com/Azure/aad-pod-identity/blob/master/README.md