Application-gateway-kubernetes-ingress: Affected by change on AAD-POD-Identity

[evmimagina]

Hi guys,

Since last aad-pod-identity release, 1.6.0, it is not posible to deploy AGIC following the default instructions.

Deploy AAD POD Identity with RBAC
kubectl create -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment-rbac.yaml

Deploy AAD POD Identity without RBAC
kubectl create -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment.yaml

Trying to deploy AGIC on AKS, it used to work with AAD-POD-identity 1.5.5 but since deployment of versión 1.6 it fails with the following error.

E0428 16:57:55.669130       1 client.go:132] Possible reasons: AKS Service Principal requires 'Managed Identity Operator' access on Controller Identity; 'identityResourceID' and/or 'identityClientID' are incorrect in the Helm config; AGIC Identity requires 'Contributor' access on Application Gateway and 'Reader' access on Application Gateway's Resource Group;
E0428 16:57:55.669160       1 client.go:145] Unexpected ARM status code on GET existing App Gateway config: 403
E0428 16:57:55.669167       1 client.go:148] Failed fetching config for App Gateway instance. Will retry in 10s. Error: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/4c4aee1a-cfd4-4e7a-abe3-*******/resourceGroups/RG-NAME-DEV/providers/Microsoft.Network/applicationGateways/AG-NAME-DEV?api-version=2019-09-01: StatusCode=403 -- Original Error: adal: Refresh request failed. Status Code = '403'. Response body: getting assigned identities for pod default/agile-opossum-ingress-azure-579cbb6b89-sldr5 in CREATED state failed after 16 attempts, retry duration [5]s. Error: <nil>

If I check the mic pod log, it shows following message;

"Ignoring azure identity default/agile-opossum-azid-ingress-azure, error: Invalid resource id: "", must match /subscriptions/<subid>/resourcegroups/<resourcegroup>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<name>"

I can tell that when using the "kubectl describe azureidentity ****-ingress-azure"I get the correct Resource ID.

/subscriptions/4c4aee1a-cfd4-4e7a-abe3-*********/resourcegroups/RESOURCEGROUPNAME-NODES/providers/Microsoft.ManagedIdentity/userAssignedIdentities/MANAGEDIDENTITYNAME

wich matches the expected format

/subscriptions/<subid>/resourcegroups/<resourcegroup>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<name>

The reason behind this ->
https://github.com/Azure/aad-pod-identity/tree/v1.6.0#v160-breaking-change

Just keep this in mind and apply fixes if proceed.

Many thanks and best regards,

[evmimagina]

I've just realised you already got it on … https://github.com/Azure/application-gateway-kubernetes-ingress/pull/825

and that the fix will be included on 1.2.0-rc2 … but, when it will it be released? a quick fix is needed ASAP!

Many thanks,

[evmimagina]

Hi @akshaysngupta ,

As you have merged it to the master branch, would you mind to merge it to the 1.2.0-rc1 as well?

Thanks in advance!

[akshaysngupta]

@evmimagina here is a work around.
https://github.com/Azure/application-gateway-kubernetes-ingress/issues/823#issuecomment-620180220

[akshaysngupta]

This has been fixed in the repo and will be in upcoming 1.2.0-rc2 of AGIC.

A workaround for the above mentioned issue is to use an older AAD Pod identity deployment yaml.
Non-RBAC clusters:

kubectl apply -f https://raw.githubusercontent.com/Azure/aad-pod-identity/v1.5.5/deploy/infra/deployment.yaml

RBAC clusters:

kubectl apply -f https://raw.githubusercontent.com/Azure/aad-pod-identity/v1.5.5/deploy/infra/deployment-rbac.yaml

[fstugren]

Hi folks - is there an ETA for a fix so that installing the latest master works again?

kubectl apply -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment-rbac.yaml

I set up a new cluster today and the managed identity still failed to apply.

Applying https://raw.githubusercontent.com/Azure/aad-pod-identity/v1.5.5/deploy/infra/deployment-rbac.yaml works.

[sudheerreddykoppula]

Hi, any ETA on fix ?

Applying https://raw.githubusercontent.com/Azure/aad-pod-identity/v1.5.5/deploy/infra/deployment-rbac.yaml works.

[akshaysngupta]

We have created the 1.2.0-rc2 release that addresses this issue.
Please try it out and let us know if you run into any issues.

[fstugren]

Is there a document I could read to understand how long after marking an issue as closed the fix becomes available in master? Is there a published schedule I can follow?

[vzburke]

I mean, duh, How do things like this happen? "Let's suddenly make this case sensitive!" - MS

[akshaysngupta]

@vzburke Actually, the background story is that AAD Pod identity project updated the kubernetes client to 1.16 in their latest release. This new client is case-sensitive to properties in the CRD.
https://github.com/kubernetes/kubernetes/issues/64612

[baptistepattyn]

it took me two days but finally made it work... thank you for the support... maybe a notice in the tutorial would be nice :)

[max77p]

Has this been fixed? or still same issue

[akshaysngupta]

@max77p yes, this is fixed. Please use AGIC 1.4.0 along with latest AAD pod identity.