Aad-pod-identity: Identity successfully assigned, but getting "identity not found" error on token refresh

[gapict]

Hi there

I have been using the aad pod identity in a four node cluster for several months now. Recently two nodes in the cluster have had their pods be unable to reach aad -- this prevents the pods from starting up correctly.

Adding new nodes to the cluster will add the nmi pod correctly to them and pods running there start up correctly.

Is there something specific I should look for on the two nodes having trouble?

Thanks!

[aramase]

@gapict What is the error you see in the logs for NMI pods running on the 2 nodes?

[gapict]

Hi @aramase

first node:
time="2020-05-11T15:56:50Z" level=info msg="Status (403) took 592115786 ns" req.method=GET req.path=/metadata/identity/oauth2/token req.remote=10.244.1.123
time="2020-05-11T16:01:59Z" level=info msg="matched identityType:0 clientid:dec4##### REDACTED #####a599 resource:https://vault.azure.net" req.method=GET req.path=/metadata/identity/oauth2/token req.remote=10.244.1.123
time="2020-05-11T16:01:59Z" level=error msg="failed to get service principal token for pod:azureml-/-5967ccb586-88xht, adal: Refresh request failed. Status Code = '400'. Response body: {\"error\":\"invalid_request\",\"error_description\":\"Identity not found\"}" req.method=GET req.path=/metadata/identity/oauth2/token req.remote=10.244.1.123
time="2020-05-11T16:01:59Z" level=info msg="Status (403) took 462795173 ns" req.method=GET req.path=/metadata/identity/oauth2/token req.remote=10.244.1.123

second node:
time="2020-05-11T16:09:01Z" level=info msg="Status (403) took 553759338 ns" req.method=GET req.path=/metadata/identity/oauth2/token req.remote=10.244.0.126
time="2020-05-11T16:14:15Z" level=info msg="matched identityType:0 clientid:dec4##### REDACTED #####a599 resource:https://vault.azure.net" req.method=GET req.path=/metadata/identity/oauth2/token req.remote=10.244.0.126
time="2020-05-11T16:14:16Z" level=error msg="failed to get service principal token for pod:azureml-/-5967ccb586-hwbtv, adal: Refresh request failed. Status Code = '400'. Response body: {\"error\":\"invalid_request\",\"error_description\":\"Identity not found\"}" req.method=GET req.path=/metadata/identity/oauth2/token req.remote=10.244.0.126
time="2020-05-11T16:14:16Z" level=info msg="Status (403) took 370701832 ns" req.method=GET req.path=/metadata/identity/oauth2/token req.remote=10.244.0.126

Thanks!

[aramase]

@gapict Thank you for the logs. This is a caching issue with IMDS which is being tracked by the Managed identity team internally. They have a fix and the deployment is still ongoing. If this issue is a blocker or is affecting you in other cluster, Please open a support ticket and I'll forward to the team that's handling the fix.

[gapict]

Thanks for the info @aramase -- not a blocker right now and only seems to be affecting a single cluster.

I'll hold tight for now. Thanks.

[aelmanaa]

@aramase what's the workaround when this issue occurs? can we just restart aad pods?

[aramase]

@aelmanaa Unfortunately, this is not a pod-identity issue. So restarting the aad-pod-identity pods will not resolve the issue. The fix is being rolled out for the issue. If you encounter this, please open a support ticket with reference to the issue and we can forward it to the team that owns Azure Metadata Service.

[shpathak-msft]

@aelmanaa : I saw this twice last week, I drained and restarted the node which mitigated the issue.

[asubmani]

would this issue affect "pod identity managed mode"?
I am seeing inconsistencies where AzurePodIdentityException is not being honored by a few nodes as per NMI logs.

I was thinking of moving to "managed mode" as it would make "pod identity" troubleshooting easier. All I do is get all "UserAssignedIdentities; foreach; az vmss assign %" every 10 mins.

[aramase]

@asubmani This issue with IMDS returning a token for existing identity. Because of the bug, IMDS returns "identity not found" even when identity exists on the VM/VMSS. This issue is independent of aad-pod-identity mode.

I am seeing inconsistencies where AzurePodIdentityException is not being honored by a few nodes as per NMI logs.

Can you please elaborate? AzurePodIdentityException is used to opt out pods from going through pod-identity validation. The AzurePodIdentityException needs to be in the same namespace as the application pod that wants to be excepted.

[mboersma]

Two Azure customers have reported errors similar to this recently:
Error code: Status Code = '400'. Response body: {'error':'invalid_request','error_description':'Identity not found'}

@aramase do you know if the IMDS team has fixed the caching issue yet?

[aramase]

Checking internally with the IMDS team on the status of the rollout. Will update here once we find out.

[aramase]

Quick things to check before confirming its an IMDS issue is -

• Check if the identity that the pod is requesting a token for is in-fact assigned to the vm/vmss by running

az vm/vmss identity show -g <resource group> -n <vm/vmss name>

Check the resourceID, clientID and confirm the same clientID is set in the AzureIdentity.

• If the identity does exist and error still is "identity not found", then please open a support ticket with reference to the issue and we can forward it to the team that owns Azure Metadata Service

[aramase]

Closing this issue and tracking in #681