Appconfiguration: Misleading Exception on MSI RBAC failure

[syndicatedshannon]

The following exception does not distinguish between network and authentication failures (it is raised from MSI RBAC failures, and possibly from other authorization failure scenarios):

Description: The process was terminated due to an unhandled exception.
Exception Info: System.AggregateException: One or more errors occurred. (The configuration store with the endpoint 'https://contoso-stg-configuration.azconfig.io/' could not be found.
Parameter name: endpoint) ---> System.ArgumentException: The configuration store with the endpoint 'https://contoso-stg-configuration.azconfig.io/' could not be found.
Parameter name: endpoint
   at Microsoft.Azure.AppConfiguration.ManagedIdentityConnector.ManagedIdentityConnector.GetConnectionString(Uri endpoint, Permissions permissions, RequestOptions options, CancellationToken cancellationToken)
   at Microsoft.Azure.AppConfiguration.ManagedIdentityConnector.AzconfigClientFactory.CreateClient(Uri endpoint, AzconfigClientFactoryOptions options, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationOptions.ConnectWithManagedIdentity(String endpoint)

This is important because network errors are often hard to diagnose in the cloud, and setting a person on the wrong track of solving a network problem could lead to a great deal of lost time.

[jimmyca15]

That exception occurs when the managed identity that is being used to connect to Azure App Configuration is successfully authenticated, but doesn't have any access to a configuration store with the specified endpoint. This could be because the store doesn't exist, or it could be because the store doesn't belong to any of the subscriptions that the identity is capable of reading from. I agree that the error message could be a bit more informative of what is happening and we can improve this in the next release.

[syndicatedshannon]

This could be because the store doesn't exist, or it could be because the store doesn't belong to any of the subscriptions that the identity is capable of reading from.

I verified it was specifically because the identity did not have RBAC grant, so technically it was neither of the above.

Actually, it had RBAC grant, but via group, so the application did not present a token with group membership. This takes 8 hours to catch up. Diagnosing this with the added confusion of the endpoint not found error is extremely difficult if you aren't expecting it.

Consequently, more accurate error communication is even more relevant. Getting one step closer by being informed that the App Configuration store did not believe the App Service was due access would have saved me hours. The better part of a very long day, to be more precise.

[syndicatedshannon]

At any rate, very glad it will be improved. It was frustrating (although the MSI group propagation delay was a significant part of that experience).

[jimmyca15]

@syndicatedshannon I apologize for the lost time trying to diagnose the failure to authenticate via MSI. We will take the opportunity to use this feedback so that others can be saved from a similar experience.

I am thinking of an error message which suggests the steps to alleviate this failure to connect rather than just describing a problem.

"The identity '[email protected]' does not have access to a configuration store with the endpoint 'https://example.azconfig.io'. Please verify that the configuration store exists and that the identity '[email protected]' has contributor access to the configuration store resource. If the permission has just been granted the application may need to be restarted to pick up the new changes."

You mentioned that there was an 8 hour catch-up window. I was not aware of this. I suppose that the AppService cached the token, but will need to verify. A restart might be required to fix that issue.

[syndicatedshannon]

I was not aware either, when I started troubleshooting.
Understanding Azure MSI (Managed Service Identity) tokens & caching

Troubleshooting for this issue also may be related to new configuration capabilities

[syndicatedshannon]

Note that restarting the application does not fix the problem, according to the above document.

[jimmyca15]

We have released an update to the Microsoft.Extensions.Configuration.AzureAppConfiguration package which now has the enhanced error message I suggested.

Release notes here