Aks-engine: Failed to refresh token for kms-agent on every restart of master

[sylus]

Describe the bug

Having the following problem every single time we restart the cluster and it takes about 20 minutes for this to resolve. Have no idea what is going on and really looking for some assistance with this.

kms-agent:
2019/10/17 16:08:19 doDecrypt failed
2019/10/17 16:08:19 Processing DecryptRequest:
2019/10/17 16:08:19 doDecrypt failed
failed to decrypt, error:  failed to get vault, error: failed to get vault, error: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/subscription/resourceGroups/k8s-cancentral-01-dev-rg/providers/Microsoft.KeyVault/vaults/kvbd3jwvxcmzwau?api-version=2016-10-01: StatusCode=429 -- Original Error: adal: Refresh request failed. Status Code = '429'. Response body: { "error": "Too many requests" }

controller-manager:
E1017 16:04:06.271012       1 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/adp/adp-audit-storage-secret": error while decrypting key: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
F1017 16:04:09.129201       1 client_builder.go:238] unable to get token for service account: timed out waiting for the condition

Steps To Reproduce

Install an AKS Engine cluster with msi, and simply just try to restart the master node.

Expected behavior

Can actually restart the master node without issue

AKS Engine version:

v0.41.3

Kubernetes version

v1.15.3

Additional context

[devigned]

This is coming from the Azure SDK for Golang. It seems that the error is caused by Azure Active Directory responding with a 429 when the SDK Authorizer tries to fetch a refreshed token.

@jhendrixMSFT do you have any insight into this error?

[jhendrixMSFT]

It's a bug in our adal package, apparently there's no retry logic for token refresh on transient errors like this. :( I'll work on a fix.

[jhendrixMSFT]

Sorry I missed that this was for MSI. We do have logic to retry failed MSI authorization requests which includes 429. We follow the guidelines for retrying failed MSI authorization requests as put forth here. Assuming this is working correctly it would appear that the max number of retries (default is 5) has been exceeded. I'm afraid I don't know how to diagnose further than this.

[devigned]

Thank you for the quick response, @jhendrixMSFT! That helps a ton.

It sounds like we need to check the caching behavior in MSI or our usage.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.