Aad-pod-identity: IMDS endpoint availability is delayed

[PSanetra]

The IMDS endpoint, which is created by the NMI, is not always directly available after Pod startup. This causes critical issues with libraries, which expect the endpoint to be always available: https://github.com/Azure/azure-sdk-for-net/issues/18312

There is currently one generic workaround available by adding an initContainer to the pods, which are using the IMDS endpoint. The container should poll the IMDS endpoint until it becomes available, so that the actual containers can start regularly.

As far as I have understood, the developers of those libraries do not want to change the assumption, that the IMDS endpoint is always available. So the only option I see is, that the aad-pod-identity project needs to ensure that the IMDS endpoint is on all Pods, which are using the IMDS endpoint, available at startup time. This may be generically achievable with a mutating admission webhook, which adds an additional initContainer to all Pods, using the aadpodidbinding label. (See https://github.com/Azure/azure-sdk-for-net/issues/18312#issuecomment-781729340)

Steps To Reproduce

• See https://github.com/Azure/azure-sdk-for-net/issues/18312

Expected behavior

• The IMDS endpoint must be available when Pod containers start, which are using the IMDS endpoint.

AAD Pod Identity version

• AAD Pod Identity 1.7.1

Kubernetes version

• AKS 1.17.11

Related issues

• https://github.com/Azure/aad-pod-identity/issues/846
• https://github.com/Azure/azure-sdk-for-net/issues/18312

[chewong]

I think the delay is caused by the underlying identity assignment to the node described in https://azure.github.io/aad-pod-identity/docs/best-practices/#retry-on-token-retrieval. One workaround, as you mentioned, is to implement a init container that probes the IMDS endpoint until it's available. You can also look into managed mode (https://azure.github.io/aad-pod-identity/docs/configure/pod_identity_in_managed_mode/), where you assign the identity to the node manually before deploying the workload to bypass the delay.

[PSanetra]

@chewong anyway, I think this is a problem which must be handled by Microsoft. Either by making sure that the libraries (in this case the Azure .NET SDK) are fault tolerant against these types of errors (which is currently not the case) or by providing some out-of-the-box solution like a mutating admission webhook in Kubernetes to apply that workaround to all affected pods by default.