Aad-pod-identity: mic crashes when VMSS managed identity is not authorized

[ritazh]

Describe the bug
When using pod identity in a cluster running VMSS with system assigned identity, mic pod crashes when the system assigned identity does not have the right permission to perform role assignment.

Steps To Reproduce

• Create a k8s v1.14 cluster with aks-engine with "useManagedIdentity": true
• Ensure the cluster is running keyvault flexvol in kube-system and image used is mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.14
• Install pod identity: kubectl apply -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment-rbac.yaml, currently mic is at 1.5.2.
• Create azureidentity

apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentity
metadata:
 name: ritademo
spec:
 type: 0
 ResourceID: /subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ritademo
 ClientID: xxx

• Create azureidentitybinding

apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentityBinding
metadata:
 name: demo1-azure-identity-binding
spec:
 AzureIdentity: ritademo
 Selector: demo

• Schedule pod deployment with keyvault flexvol in the cluster:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-flex-kv-podid
  labels:
    aadpodidbinding: demo
spec:
  containers:
  - name: nginx-flex-kv-podid
    image: nginx
    volumeMounts:
    - name: test
      mountPath: /kvmnt
      readOnly: true
  volumes:
  - name: test
    flexVolume:
      driver: "azure/kv"
      options:
        usepodidentity: "true" 
        keyvaultname: ritakv
        keyvaultobjectname: secret
        keyvaultobjecttype: secret 
        tenantid: xxx

Expected behavior
Pod creation should fail. MIC should log the error, but it should not be crashing.

AAD Pod Identity version
1.5.2

Kubernetes version
1.14.6

Additional context

Here is the log dump:

E1001 22:03:21.499568 1 mic.go:739] Updating msis on node k8s-agentpool1-39588698-vmss000001, add [1], del [0] failed with error compute.VirtualMachineScaleSetsClient#CreateOrUpdate: Failure sending request: StatusCode=403 -- Original Error: Code="AuthorizationFailed" Message="The client '1851cf62-9f0d-45f3-a9f8-ed2e26e34xxx' with object id '1851cf62-9f0d-45f3-a9f8-ed2e26e34xxx' does not have authorization to perform action 'Microsoft.Compute/virtualMachineScaleSets/write' over scope '/subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-39588698-vmss' or the scope is invalid. If access was recently granted, please refresh your credentials."
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xc57410]

goroutine 83 [running]:
github.com/Azure/aad-pod-identity/pkg/cloudprovider.(vmssIdentityInfo).GetUserIdentityList(0xc00000cfa8, 0x15ab8c0, 0xc00000cfa8, 0xc00000cfa0)
/go/src/github.com/Azure/aad-pod-identity/pkg/cloudprovider/vmss.go:132 +0x10
github.com/Azure/aad-pod-identity/pkg/cloudprovider.(Client).GetUserMSIs(0xc0003ebce0, 0xc0000ecd20, 0x1458687, 0x41, 0xc0004f2d98, 0x4, 0x4)
/go/src/github.com/Azure/aad-pod-identity/pkg/cloudprovider/cloudprovider.go:166 +0x114
github.com/Azure/aad-pod-identity/pkg/mic.(Client).getUserMSIListForNode(0xc000115830, 0xc0000ecd20, 0xc0004f2d98, 0x4, 0x4, 0x0, 0x0)
/go/src/github.com/Azure/aad-pod-identity/pkg/mic/mic.go:628 +0x3e
github.com/Azure/aad-pod-identity/pkg/mic.(Client).updateUserMSI(0xc000115830, 0xc00065ca80, 0x1, 0x1, 0xc00064f0e0, 0x22, 0xc000656c40, 0x1, 0x1, 0x0, ...)
/go/src/github.com/Azure/aad-pod-identity/pkg/mic/mic.go:740 +0x1487
created by github.com/Azure/aad-pod-identity/pkg/mic.(*Client).updateNodeAndDeps
/go/src/github.com/Azure/aad-pod-identity/pkg/mic/mic.go:701 +0x1c5

[thanasisk]

This is what I was describing as bug 2 in issue #185 at my last comment
:+1:

[ritazh]

In case you got here by searching the errors, this doc really helped me. https://github.com/Azure/aad-pod-identity/blob/master/docs/readmes/README.msi.md#pre-requisites---role-assignments

If you are using VMSS system assigned identity for the cluster, TL;DR you need to assign both the "contributor" role to the RG AND the "managed identity operator" role to the user provided user assigned identity.

[JohnGalt1717]

The doc doesn't exist anymore and I'm still getting this error. Where are the pre-requisites now?

[aramase]

The required role assignments are documented here - https://github.com/Azure/aad-pod-identity/blob/master/docs/readmes/README.role-assignment.md

[irons]

Now they're here: https://azure.github.io/aad-pod-identity/docs/getting-started/role-assignment/