Aad-pod-identity: Getting assigned identities for pod dev/bikes in CREATED state failed after 16 attempts

[amsoedal]

Describe the bug
Our product's integration with Pod Identity is currently broken (it stopped working earlier this week). With a live repro, I see the NMI logs showing:

failed to get matching identities for pod: dev/bikes-78cfcbcb54-bbxgd, error: getting assigned identities for pod dev/bikes-78cfcbcb54-bbxgd in CREATED state failed after 16 attempts, retry duration [5]s, error: <nil>. Check MIC pod logs for identity assignment errors

However, when I check the MIC logs, everything appears fine:

I0409 15:16:31.321609       1 main.go:105] starting mic process. Version: v1.6.3. Build date: 2020-09-15-21:09
W0409 15:16:31.321649       1 main.go:110] --kubeconfig not passed will use InClusterConfig
I0409 15:16:31.321657       1 main.go:127] kubeconfig () cloudconfig (/etc/kubernetes/azure.json)
I0409 15:16:31.321759       1 main.go:135] running MIC in namespaced mode: false
I0409 15:16:31.321773       1 main.go:139] client QPS set to: 5. Burst to: 5
I0409 15:16:31.321786       1 mic.go:138] starting to create the pod identity client.  Version: v1.6.3. Build date: 2020-09-15-21:09
I0409 15:16:31.357673       1 mic.go:144] Kubernetes server version: v1.18.14
I0409 15:16:31.357960       1 cloudprovider.go:123] MIC using user assigned identity: 123b##### REDACTED #####9058 for authentication.
I0409 15:16:31.359491       1 probes.go:41] initialized health probe on port 8080
I0409 15:16:31.359502       1 probes.go:44] started health probe
I0409 15:16:31.359579       1 metrics.go:341] registered views for metric
I0409 15:16:31.359628       1 prometheus_exporter.go:21] starting Prometheus exporter
I0409 15:16:31.359644       1 metrics.go:347] registered and exported metrics on port 8888
I0409 15:16:31.359648       1 mic.go:239] initiating MIC Leader election
I0409 15:16:31.359659       1 leaderelection.go:242] attempting to acquire leader lease  default/aad-pod-identity-mic...
I0409 15:16:31.395166       1 leaderelection.go:252] successfully acquired lease default/aad-pod-identity-mic
I0409 15:16:31.566890       1 mic.go:312] upgrading the types to work with case sensitive go-client
I0409 15:16:31.777208       1 mic.go:316] type upgrade completed !!
I0409 15:16:32.466339       1 crd.go:455] CRD informers started
I0409 15:16:32.466339       1 pod.go:72] pod cache synchronized. Took 500.117401ms
I0409 15:16:32.466396       1 pod.go:79] pod watcher started !!
I0409 15:16:32.466418       1 mic.go:397] sync thread started.

AAD Pod Identity version
Version: v1.6.3.

Kubernetes version
1.18.14

Any help would be appreciated!

[daniv-msft]

I'm in the same team as @amsoedal. I reproduce the same issue with v1.7.5 (after enabling kubenet).

Could this be related to something missing in how we configure pod identity in our cluster?
As of today, we just deployed:

kubectl apply -f https://raw.githubusercontent.com/Azure/aad-pod-identity/v1.7.5/deploy/infra/deployment-rbac.yaml
kubectl apply -f https://raw.githubusercontent.com/Azure/aad-pod-identity/v1.7.5/deploy/infra/mic-exception.yaml

[chewong]

@amsoedal I don't see any errors in MIC. Could you share the AzureIdentity, AzureIdentityBinding and your workload spec?
@daniv-msft we discontinued supporting kubenet since v1.7.0 release. To enable it for v1.7.x, see https://azure.github.io/aad-pod-identity/docs/configure/aad_pod_identity_on_kubenet/#how-to-run-aad-pod-identity-on-clusters-with-kubenet.

[aramase]

@daniv-msft @amsoedal Are you upgrading pod identity from v1.5.x to v1.6.x or v1.7.x? There were breaking changes introduced in v1.6.0 release that have been documented here: https://azure.github.io/aad-pod-identity/docs/#v16x-breaking-change. No action is required for existing AzureIdentity or AzureIdentityBinding as MIC handles conversion but for new custom resource instances, they are case-sensitive and using an uppercase will not work.

[amsoedal]

@chewong @aramase thanks so much for your help. Our customer got back to us and the camelCase breaking change between 1.6.x and 1.7.x was likely the culprit of the issues he was seeing. Will close the issue since he seems to be unblocked.