Application-gateway-kubernetes-ingress: Ingress pod aren't starting

Created on 17 Sep 2020  ·  11Comments  ·  Source: Azure/application-gateway-kubernetes-ingress

Describe the bug
Ingress pod failed to start

AKS Kubernetes Version: 1.17.7
Helm Chart version: 1.2.0
Docker image: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.2.0

To Reproduce
Followed https://azure.github.io/application-gateway-kubernetes-ingress/setup/install-existing/

az aks show -g kubernetes-staging -n staging -o tsv --query "servicePrincipalProfile.clientId" return msi. Assigning roles to msi doesn't work.

I've found out that 

  "identityProfile": {
    "kubeletidentity": {
      "clientId": "7888c525-c3f0-492c-8f1e-c653074337bc",
      "objectId": "fffde46d-d474-42df-a46b-831dec0dc0d8",
      "resourceId": "/subscriptions/563cc198-abc3-48df-84b8-5bf23b3738a0/resourcegroups/MC_kubernetes-staging_staging_northeurope/providers/Microsoft.ManagedIdentity/userAssignedIdentities/staging-agentpool"
    }
  }

is used. So I gave it the following permissions

 az role assignment create `
     --role "Managed Identity Operator" `
     --assignee "7888c525-c3f0-492c-8f1e-c653074337bc"  `
     --scope "/subscriptions/563cc198-abc3-48df-84b8-5bf23b3738a0/resourcegroups/kubernetes-staging/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress-staging"

az role assignment create `
   --role Contributor `
   --assignee "7888c525-c3f0-492c-8f1e-c653074337bc" `
   --scope "/subscriptions/563cc198-abc3-48df-84b8-5bf23b3738a0/resourceGroups/MC_kubernetes-staging_staging_northeurope”



md5-8a7fb93072a65a09abf3e70158bf2b0c



Name:         ingress-azure-6b8bf5d5cb-l9mz9
Namespace:    default
Priority:     0
Node:         aks-nodepool1-31409485-vmss000000/10.240.0.4
Start Time:   Fri, 18 Sep 2020 00:29:58 +0700
Labels:       aadpodidbinding=ingress-azure
              app=ingress-azure
              pod-template-hash=6b8bf5d5cb
              release=ingress-azure
Annotations:  checksum/config: ddaae4224ba9553b721ff1bbebd29f71011871262fa8b23a0ebb3deb3d10ad9b
              prometheus.io/port: 8123
              prometheus.io/scrape: true
Status:       Running
IP:           10.244.0.48
IPs:
  IP:           10.244.0.48
Controlled By:  ReplicaSet/ingress-azure-6b8bf5d5cb
Containers:
  ingress-azure:
    Container ID:   docker://56ba7202951b2f8d6aca8589935f9746092c6233ad99a9b710bbaada5df26b95
    Image:          mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.2.0
    Image ID:       docker-pullable://mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:de458f962eab0cd2de19d23dfeb9a0e4bc2565a38f8c45cc98a74f3cda8b940c
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Fri, 18 Sep 2020 00:40:13 +0700
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Fri, 18 Sep 2020 00:29:59 +0700
      Finished:     Fri, 18 Sep 2020 00:40:12 +0700
    Ready:          False
    Restart Count:  1
    Liveness:       http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:      http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      ingress-azure  ConfigMap  Optional: false
    Environment:
      AZURE_CLOUD_PROVIDER_LOCATION:  /etc/appgw/azure.json
      AGIC_POD_NAME:                  ingress-azure-6b8bf5d5cb-l9mz9 (v1:metadata.name)
      AGIC_POD_NAMESPACE:             default (v1:metadata.namespace)
    Mounts:
      /etc/appgw/azure.json from azure (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from ingress-azure-token-hjpg5 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  azure:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/azure.json
    HostPathType:  File
  ingress-azure-token-hjpg5:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-azure-token-hjpg5
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                 From                                        Message
  ----     ------     ----                ----                                        -------
  Normal   Scheduled  <unknown>           default-scheduler                           Successfully assigned default/ingress-azure-6b8bf5d5cb-l9mz9 to aks-nodepool1-31409485-vmss000000
  Normal   Pulling    16m                 kubelet, aks-nodepool1-31409485-vmss000000  Pulling image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.2.0"
  Normal   Pulled     16m                 kubelet, aks-nodepool1-31409485-vmss000000  Successfully pulled image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.2.0"
  Normal   Created    16m                 kubelet, aks-nodepool1-31409485-vmss000000  Created container ingress-azure
  Normal   Started    16m                 kubelet, aks-nodepool1-31409485-vmss000000  Started container ingress-azure
  Warning  Unhealthy  58s (x90 over 15m)  kubelet, aks-nodepool1-31409485-vmss000000  Readiness probe failed: Get http://10.244.0.48:8123/health/ready: net/http: request canceled (Client.Timeout exceeded while awaiting headers)



md5-753e53f47ad1906da2385dafbda60e1c



ERROR: logging before flag.Parse: I0917 17:40:13.401209       1 utils.go:115] Using verbosity level 5 from environment variable APPGW_VERBOSITY_LEVEL
I0917 17:40:13.448050       1 environment.go:240] KUBERNETES_WATCHNAMESPACE is not set. Watching all available namespaces.
I0917 17:40:13.448181       1 main.go:128] Appication Gateway Details: Subscription="563cc198-abc3-48df-84b8-5bf23b3738a0" Resource Group="kubernetes-staging" Name="staging-ag"
I0917 17:40:13.448195       1 auth.go:46] Creating authorizer from Azure Managed Service Identity
I0917 17:40:13.448221       1 client.go:118] Getting Application Gateway configuration.
I0917 17:40:13.448266       1 httpserver.go:57] Starting API Server on :8123



md5-f5e5904e1d691cb34009192544733435



6s          Warning   binding apply error   v1.azureidentitybinding/ingress-azure       Applying binding ingress-azure node aks-nodepool1-31409485-vmss000000 for pod ingress-azure-6b8bf5d5cb-l9mz9-default-ingress-azure resulted in error compute.VirtualMachineScaleSetsClient#CreateOrUpdate: Failure sending request: StatusCode=403 -- Original Error: Code="LinkedAuthorizationFailed" Message="The client 'fffde46d-d474-42df-a46b-831dec0dc0d8' with object id 'fffde46d-d474-42df-a46b-831dec0dc0d8' has permission to perform action 'Microsoft.Compute/virtualMachineScaleSets/write' on scope '/subscriptions/563cc198-abc3-48df-84b8-5bf23b3738a0/resourceGroups/MC_kubernetes-staging_staging_northeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-nodepool1-31409485-vmss'; however, it does not have permission to perform action 'Microsoft.ManagedIdentity/userAssignedIdentities/assign/action' on the linked scope(s) '/subscriptions/563cc198-abc3-48df-84b8-5bf23b3738a0/resourcegroups/kubernetes-staging/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress-staging' or the linked scope(s) are invalid."
picture Boychenko
👍3

All 11 comments

I have the same issue, except I used kubernetes 1.18.8 and agic 1.2.0-rc3. And I followed Greenfield deployment guide

Uriil picture Uriil on 23 Sep 2020

@Boychenko
As shown in the error, your cluster's service principal is missing required permissions over the user assigned identity. This permission allowed the cluster to assign the identity to a Pod.
Use the following Az CLI command:

az role assignment \
--assignee fffde46d-d474-42df-a46b-831dec0dc0d8 \
--scope /subscriptions/563cc198-abc3-48df-84b8-5bf23b3738a0/resourcegroups/kubernetes-staging/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress-staging \
--role "Managed Identity Operator"
akshaysngupta picture akshaysngupta on 23 Sep 2020

@Uriil In greenfield docs, we apply the permissions in the ARM template while deploying the Azure resources.
Please share the error you are seeing.

akshaysngupta picture akshaysngupta on 23 Sep 2020

@akshaysngupta here it is, from all 4 pods:

Ingress controller: the only resource in NON READY state, stuck on starting:

D:AKS> kubectl logs ingress-azure-779bfbd4b6-jtlgx
ERROR: logging before flag.Parse: I0923 16:10:30.210251 1 utils.go:115] Using verbosity level 3 from environment variable APPGW_VERBOSITY_LEVEL
I0923 16:10:30.239959 1 environment.go:240] KUBERNETES_WATCHNAMESPACE is not set. Watching all available namespaces.
I0923 16:10:30.240102 1 main.go:128] Appication Gateway Details: Subscription="8f7f56b6-5a01-4adc-bf6b-c12a8f1fd0de" Resource Group="testApp" Name="applicationgatewaye4b6"
I0923 16:10:30.240120 1 auth.go:46] Creating authorizer from Azure Managed Service Identity
I0923 16:10:30.240121 1 httpserver.go:57] Starting API Server on :8123

kubectl logs mic-67bb68db85-5df96

I0923 16:06:09.072606 1 main.go:95] Starting mic process. Version: 1.6.0. Build date: 2020-04-24-19:19
W0923 16:06:09.072655 1 main.go:100] --kubeconfig not passed will use InClusterConfig
I0923 16:06:09.072663 1 main.go:114] kubeconfig () cloudconfig (/etc/kubernetes/azure.json)
I0923 16:06:09.072808 1 main.go:122] Running MIC in namespaced mode: false
I0923 16:06:09.072823 1 main.go:126] Client QPS set to: 5. Burst to: 5
I0923 16:06:09.072836 1 mic.go:124] Starting to create the pod identity client. Version: 1.6.0. Build date: 2020-04-24-19:19
I0923 16:06:09.178782 1 mic.go:130] Kubernetes server version: v1.18.8
I0923 16:06:09.179168 1 probes.go:42] Initialized health probe on port 8080
I0923 16:06:09.179183 1 probes.go:45] Started health probe
I0923 16:06:09.179243 1 metrics.go:330] Registered views for metric
I0923 16:06:09.179277 1 prometheus_exporter.go:23] Starting Prometheus exporter
I0923 16:06:09.179285 1 metrics.go:337] Registered and exported metrics on port 8888
I0923 16:06:09.179288 1 mic.go:206] Initiating MIC Leader election
I0923 16:06:09.179298 1 leaderelection.go:242] attempting to acquire leader lease default/aad-pod-identity-mic...

kubectl logs mic-67bb68db85-spxt5

I0923 16:06:09.085379 1 main.go:95] Starting mic process. Version: 1.6.0. Build date: 2020-04-24-19:19
W0923 16:06:09.085429 1 main.go:100] --kubeconfig not passed will use InClusterConfig
I0923 16:06:09.085436 1 main.go:114] kubeconfig () cloudconfig (/etc/kubernetes/azure.json)
I0923 16:06:09.085619 1 main.go:122] Running MIC in namespaced mode: false
I0923 16:06:09.085630 1 main.go:126] Client QPS set to: 5. Burst to: 5
I0923 16:06:09.085643 1 mic.go:124] Starting to create the pod identity client. Version: 1.6.0. Build date: 2020-04-24-19:19
I0923 16:06:09.146750 1 mic.go:130] Kubernetes server version: v1.18.8
I0923 16:06:09.147372 1 probes.go:42] Initialized health probe on port 8080
I0923 16:06:09.147393 1 probes.go:45] Started health probe
I0923 16:06:09.147498 1 metrics.go:330] Registered views for metric
I0923 16:06:09.147606 1 prometheus_exporter.go:23] Starting Prometheus exporter
I0923 16:06:09.147618 1 metrics.go:337] Registered and exported metrics on port 8888
I0923 16:06:09.147625 1 mic.go:206] Initiating MIC Leader election
I0923 16:06:09.147643 1 leaderelection.go:242] attempting to acquire leader lease default/aad-pod-identity-mic...
I0923 16:06:09.164660 1 leaderelection.go:252] successfully acquired lease default/aad-pod-identity-mic
I0923 16:06:09.353928 1 mic.go:279] Upgrading the types to work with case sensitive go-client
I0923 16:06:09.367983 1 mic.go:283] Type upgrade completed !!
I0923 16:06:09.853691 1 crd.go:451] CRD informers started
I0923 16:06:10.253667 1 pod.go:74] Pod cache synchronized. Took 500.310405ms
I0923 16:06:10.253704 1 pod.go:81] Pod watcher started !!
I0923 16:06:10.253739 1 mic.go:354] Sync thread started.
I0923 16:10:18.814720 1 mic.go:976] Processing node aks-agentpool-35064155-0, add [1], del [0], update [0]
I0923 16:10:18.814758 1 crd.go:494] Got assigned id ingress-azure-779bfbd4b6-jtlgx-default-ingress-azure to assign
I0923 16:10:19.681942 1 cloudprovider.go:194] Updating user assigned MSIs on aks-agentpool-35064155-0, assign [1], unassign [0]
E0923 16:10:19.849622 1 vm.go:97] Failed to update VM with error compute.VirtualMachinesClient#Update: Failure sending request: StatusCode=403 -- Original Error: Code="LinkedAuthorizationFailed" Message="The client '26786d09-53a1-4764-b444-ba5a5bc471b5' with object id '26786d09-53a1-4764-b444-ba5a5bc471b5' has permission to perform action 'Microsoft.Compute/virtualMachines/write' on scope '/subscriptions/8f7f56b6-5a01-4adc-bf6b-c12a8f1fd0de/resourceGroups/MC_testApp_akse4b6_germanywestcentral/providers/Microsoft.Compute/virtualMachines/aks-agentpool-35064155-0'; however, it does not have permission to perform action 'Microsoft.ManagedIdentity/userAssignedIdentities/assign/action' on the linked scope(s) '/subscriptions/8f7f56b6-5a01-4adc-bf6b-c12a8f1fd0de/resourcegroups/testApps/providers/Microsoft.ManagedIdentity/userAssignedIdentities/testaks' or the linked scope(s) are invalid."
E0923 16:10:19.849681 1 mic.go:1047] Updating msis on node aks-agentpool-35064155-0, add [1], del [0] failed with error compute.VirtualMachinesClient#Update: Failure sending request: StatusCode=403 -- Original Error: Code="LinkedAuthorizationFailed" Message="The client '26786d09-53a1-4764-b444-ba5a5bc471b5' with object id '26786d09-53a1-4764-b444-ba5a5bc471b5' has permission to perform action 'Microsoft.Compute/virtualMachines/write' on scope '/subscriptions/8f7f56b6-5a01-4adc-bf6b-c12a8f1fd0de/resourceGroups/MC_testApp_akse4b6_germanywestcentral/providers/Microsoft.Compute/virtualMachines/aks-agentpool-35064155-0'; however, it does not have permission to perform action 'Microsoft.ManagedIdentity/userAssignedIdentities/assign/action' on the linked scope(s) '/subscriptions/8f7f56b6-5a01-4adc-bf6b-c12a8f1fd0de/resourcegroups/testApps/providers/Microsoft.ManagedIdentity/userAssignedIdentities/testaks' or the linked scope(s) are invalid."
E0923 16:10:20.339980 1 mic.go:1050] Getting list of msis from node aks-agentpool-35064155-0 resulted in error identity info is nil
I0923 16:10:20.340011 1 mic.go:475] Work done: true. Found 1 pods, 1 ids, 1 bindings
I0923 16:10:20.340018 1 mic.go:476] Total work cycles: 15, out of which work was done in: 1
I0923 16:10:20.340026 1 stats.go:99] * Stats collected *
I0923 16:10:20.340031 1 stats.go:82] Pod listing: 15.6µs
I0923 16:10:20.340042 1 stats.go:82] ID listing: 2.5µs
I0923 16:10:20.340046 1 stats.go:82] Binding listing: 2.7µs
I0923 16:10:20.340050 1 stats.go:82] Assigned ID listing: 600ns
I0923 16:10:20.340053 1 stats.go:82] System: 68.401µs
I0923 16:10:20.340056 1 stats.go:82] CacheSync: 2.1µs
I0923 16:10:20.340060 1 stats.go:82] Cloud provider get: 1.345235365s
I0923 16:10:20.340064 1 stats.go:82] Cloud provider put: 0s
I0923 16:10:20.340067 1 stats.go:82] Assigned ID addition: 12.146127ms
I0923 16:10:20.340071 1 stats.go:82] Assigned ID deletion: 0s
I0923 16:10:20.340074 1 stats.go:89] Number of cloud provider PUT: 0
I0923 16:10:20.340077 1 stats.go:89] Number of cloud provider GET: 2
I0923 16:10:20.340080 1 stats.go:89] Number of assigned ids created in this sync cycle: 0
I0923 16:10:20.340084 1 stats.go:89] Number of assigned ids updated in this sync cycle: 0
I0923 16:10:20.340087 1 stats.go:89] Number of assigned ids deleted in this sync cycle: 0
I0923 16:10:20.340090 1 stats.go:82] Find assigned ids to create: 0s
I0923 16:10:20.340093 1 stats.go:82] Find assigned ids to delete: 0s
I0923 16:10:20.340096 1 stats.go:82] Total time to assign or remove IDs: 0s
I0923 16:10:20.340100 1 stats.go:82] Event recording: 0s
I0923 16:10:20.340103 1 stats.go:82] Total: 1.525477448s
I0923 16:10:20.340106 1 stats.go:129] *******
I0923 16:11:29.928420 1 mic.go:976] Processing node aks-agentpool-35064155-0, add [1], del [0], update [0]
I0923 16:11:29.960191 1 cloudprovider.go:194] Updating user assigned MSIs on aks-agentpool-35064155-0, assign [1], unassign [0]
E0923 16:11:30.126918 1 vm.go:97] Failed to update VM with error compute.VirtualMachinesClient#Update: Failure sending request: StatusCode=403 -- Original Error: Code="LinkedAuthorizationFailed" Message="The client '26786d09-53a1-4764-b444-ba5a5bc471b5' with object id '26786d09-53a1-4764-b444-ba5a5bc471b5' has permission to perform action 'Microsoft.Compute/virtualMachines/write' on scope '/subscriptions/8f7f56b6-5a01-4adc-bf6b-c12a8f1fd0de/resourceGroups/MC_testApp_akse4b6_germanywestcentral/providers/Microsoft.Compute/virtualMachines/aks-agentpool-35064155-0'; however, it does not have permission to perform action 'Microsoft.ManagedIdentity/userAssignedIdentities/assign/action' on the linked scope(s) '/subscriptions/8f7f56b6-5a01-4adc-bf6b-c12a8f1fd0de/resourcegroups/testApps/providers/Microsoft.ManagedIdentity/userAssignedIdentities/testaks' or the linked scope(s) are invalid."
E0923 16:11:30.126990 1 mic.go:1047] Updating msis on node aks-agentpool-35064155-0, add [1], del [0] failed with error compute.VirtualMachinesClient#Update: Failure sending request: StatusCode=403 -- Original Error: Code="LinkedAuthorizationFailed" Message="The client '26786d09-53a1-4764-b444-ba5a5bc471b5' with object id '26786d09-53a1-4764-b444-ba5a5bc471b5' has permission to perform action 'Microsoft.Compute/virtualMachines/write' on scope '/subscriptions/8f7f56b6-5a01-4adc-bf6b-c12a8f1fd0de/resourceGroups/MC_testApp_akse4b6_germanywestcentral/providers/Microsoft.Compute/virtualMachines/aks-agentpool-35064155-0'; however, it does not have permission to perform action 'Microsoft.ManagedIdentity/userAssignedIdentities/assign/action' on the linked scope(s) '/subscriptions/8f7f56b6-5a01-4adc-bf6b-c12a8f1fd0de/resourcegroups/testApps/providers/Microsoft.ManagedIdentity/userAssignedIdentities/testaks' or the linked scope(s) are invalid."

kubectl logs nmi-67px2

I0923 16:06:07.256671 1 main.go:62] Starting nmi process. Version: 1.6.0. Build date: 2020-04-24-19:19.
I0923 16:06:07.457814 1 crd.go:442] CRD lite informers started
I0923 16:06:07.457839 1 main.go:86] Running NMI in namespaced mode: false
I0923 16:06:07.457849 1 nmi.go:51] Initializing in standard mode
I0923 16:06:07.457861 1 probes.go:42] Initialized health probe on port 8080
I0923 16:06:07.457868 1 probes.go:45] Started health probe
I0923 16:06:07.457931 1 metrics.go:330] Registered views for metric
I0923 16:06:07.457984 1 prometheus_exporter.go:23] Starting Prometheus exporter
I0923 16:06:07.457991 1 metrics.go:337] Registered and exported metrics on port 9090
I0923 16:06:07.458002 1 server.go:96] Listening on port 2579
W0923 16:06:07.461174 1 iptables.go:105] Flushing iptables to add aad-metadata custom chains
E0923 16:10:30.240640 1 client.go:185] pod list empty
W0923 16:10:30.240664 1 client.go:208] List pod error: pod list empty. Retrying, attempt number: 1
E0923 16:12:10.548360 1 server.go:366] failed to get matching identities for pod: default/ingress-azure-779bfbd4b6-jtlgx, error: getting assigned identities for pod default/ingress-azure-779bfbd4b6-jtlgx in ASSIGNED state failed after 20 attempts, retry duration [5]s. Error:
I0923 16:12:10.548424 1 server.go:190] Status (404) took 100307825550 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=10.0.0.11
E0923 16:13:52.555545 1 server.go:366] failed to get matching identities for pod: default/ingress-azure-779bfbd4b6-jtlgx, error: getting assigned identities for pod default/ingress-azure-779bfbd4b6-jtlgx in ASSIGNED state failed after 20 attempts, retry duration [5]s. Error:
I0923 16:13:52.555628 1 server.go:190] Status (404) took 100006081819 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=10.0.0.11
E0923 16:15:38.563092 1 server.go:366] failed to get matching identities for pod: default/ingress-azure-779bfbd4b6-jtlgx, error: getting assigned identities for pod default/ingress-azure-779bfbd4b6-jtlgx in ASSIGNED state failed after 20 attempts, retry duration [5]s. Error:

Uriil picture Uriil on 23 Sep 2020

Based on the error message in MIC logs, try following RBAC assignment:

az role assignment create `
     --role "Managed Identity Operator" `
     --assignee "26786d09-53a1-4764-b444-ba5a5bc471b5"  `
     --scope "/subscriptions/8f7f56b6-5a01-4adc-bf6b-c12a8f1fd0de/resourcegroups/testApps/providers/Microsoft.ManagedIdentity/userAssignedIdentities/testaks'"
akshaysngupta picture akshaysngupta on 23 Sep 2020

@akshaysngupta thanks for the reply.
I've figured this out.
From my understanding
https://azure.github.io/application-gateway-kubernetes-ingress/setup/install-existing/ is incomplete and outdated in many places. It takes quite some time to find the correct set of steps to make things working if I can't use the greenfield deployment option.
I hope at some point it'll be fixed.

Boychenko picture Boychenko on 24 Sep 2020
👍1

@akshaysngupta thanks for the reply.
I've figured this out.
From my understanding
https://azure.github.io/application-gateway-kubernetes-ingress/setup/install-existing/ is incomplete and outdated in many places. It takes quite some time to find the correct set of steps to make things working if I can't use the greenfield deployment option.
I hope at some point it'll be fixed.

Hi, did you keep any notes on what you had to change to get it sorted?

I have a similar issue from the mic pod logs showing the user assigned identity as not having the permissions over the VM scale set. Despite that identity being added as an owner (as a test) to the resource group of the nodes.

I also did the brownfield deployment, but used Azure CLI to deploy the cluster as a VMSS

I see:

I0930 14:20:05.530721 1 mic.go:787] Processing node aks-nodepool1-35064155-vmss, add [1], del [0] I0930 14:20:05.724532 1 cloudprovider.go:199] Updating user assigned MSIs on aks-nodepool1-35064155-vmss E0930 14:20:05.754267 1 vmss.go:75] compute.VirtualMachineScaleSetsClient#CreateOrUpdate: Failure sending request: StatusCode=403 -- Original Error: Code="LinkedAuthorizationFailed" Message="The client 'e73bd452-a997-4823-8b31-142707f932bc' with object id 'e73bd452-a997-4823-8b31-142707f932bc' has permission to perform action 'Microsoft.Compute/virtualMachineScaleSets/write' on scope '/subscriptions/cbd252d1-a1d9-40c7-9e82-2deabc637a91/resourceGroups/AKSPoC-RG-Kubernetes-Nodes/providers/Microsoft.Compute/virtualMachineScaleSets/aks-nodepool1-35064155-vmss'; however, it does not have permission to perform action 'Microsoft.ManagedIdentity/userAssignedIdentities/assign/action' on the linked scope(s) '/subscriptions/cbd252d1-a1d9-40c7-9e82-2deabc637a91/resourcegroups/AKSPoC-RG-Kubernetes-Master/providers/Microsoft.ManagedIdentity/userAssignedIdentities/AGIC' or the linked scope(s) are invalid." E0930 14:20:05.754338 1 mic.go:831] Updating msis on node aks-nodepool1-35064155-vmss, add [1], del [0] failed with error compute.VirtualMachineScaleSetsClient#CreateOrUpdate: Failure sending request: StatusCode=403 -- Original Error: Code="LinkedAuthorizationFailed" Message="The client 'e73bd452-a997-4823-8b31-142707f932bc' with object id 'e73bd452-a997-4823-8b31-142707f932bc' has permission to perform action 'Microsoft.Compute/virtualMachineScaleSets/write' on scope '/subscriptions/cbd252d1-a1d9-40c7-9e82-2deabc637a91/resourceGroups/AKSPoC-RG-Kubernetes-Nodes/providers/Microsoft.Compute/virtualMachineScaleSets/aks-nodepool1-35064155-vmss'; however, it does not have permission to perform action 'Microsoft.ManagedIdentity/userAssignedIdentities/assign/action' on the linked scope(s) '/subscriptions/cbd252d1-a1d9-40c7-9e82-2deabc637a91/resourcegroups/AKSPoC-RG-Kubernetes-Master/providers/Microsoft.ManagedIdentity/userAssignedIdentities/AGIC' or the linked scope(s) are invalid." E0930 14:20:05.838506 1 mic.go:848] Applying binding ingress-azure node aks-nodepool1-35064155-vmss000000 for pod ingress-azure-6bd5c956f4-ggl5z-default-ingress-azure resulted in error compute.VirtualMachineScaleSetsClient#CreateOrUpdate: Failure sending request: StatusCode=403 -- Original Error: Code="LinkedAuthorizationFailed" Message="The client 'e73bd452-a997-4823-8b31-142707f932bc' with object id 'e73bd452-a997-4823-8b31-142707f932bc' has permission to perform action 'Microsoft.Compute/virtualMachineScaleSets/write' on scope '/subscriptions/cbd252d1-a1d9-40c7-9e82-2deabc637a91/resourceGroups/AKSPoC-RG-Kubernetes-Nodes/providers/Microsoft.Compute/virtualMachineScaleSets/aks-nodepool1-35064155-vmss'; however, it does not have permission to perform action 'Microsoft.ManagedIdentity/userAssignedIdentities/assign/action' on the linked scope(s) '/subscriptions/cbd252d1-a1d9-40c7-9e82-2deabc637a91/resourcegroups/AKSPoC-RG-Kubernetes-Master/providers/Microsoft.ManagedIdentity/userAssignedIdentities/AGIC' or the linked scope(s) are invalid." I0930 14:20:05.838761 1 mic.go:367] Work done: true. Found 1 pods, 1 ids, 1 bindings

djh82uk picture djh82uk on 30 Sep 2020

@djh82uk
I've combined the guide and aad-pod-identity guides. Also, keep in mind that permissions are propagated not immediately sometimes.
The most hidden part is here
https://github.com/Azure/aad-pod-identity/blob/master/docs/readmes/README.role-assignment.md

  1. Some permissions need to be given BEFORE add pod identity deployment. Check the hack script
  2. I haven't seen the following mentioned in the guide. Spent a lot of time figuring this out

Cluster configuration | Command
-- | --
AKS cluster with managed identity | az aks show -g -n --query identityProfile.kubeletidentity.clientId -otsv

Boychenko picture Boychenko on 30 Sep 2020

@djh82uk
I've combined the guide and aad-pod-identity guides. Also, keep in mind that permissions are propagated not immediately sometimes.
The most hidden part is here
https://github.com/Azure/aad-pod-identity/blob/master/docs/readmes/README.role-assignment.md

  1. Some permissions need to be given BEFORE add pod identity deployment. Check the hack script
  2. I haven't seen the following mentioned in the guide. Spent a lot of time figuring this out

Cluster configuration Command
AKS cluster with managed identity az aks show -g -n --query identityProfile.kubeletidentity.clientId -otsv

That got me a lot closer, Thank You

I no longer have those identity errors 👍 Now I just need to figure out why the AppGW is not being re-configured by AKS

djh82uk picture djh82uk on 30 Sep 2020
🚀1

@Boychenko I will close this issue.

Please feel free to make any contributions that may help other users.

akshaysngupta picture akshaysngupta on 1 Oct 2020

Looks like repo was restructured and new link is
https://github.com/Azure/aad-pod-identity/blob/master/website/content/en/docs/Getting%20started/role-assignment.md

SychevIgor picture SychevIgor on 4 Dec 2020
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

Yiyito11 picture Yiyito11  ·  4Comments
pbloigu picture pbloigu  ·  6Comments
jasonchester picture jasonchester  ·  7Comments
Joon picture Joon  ·  3Comments
niteenkole picture niteenkole  ·  3Comments