Azure-sdk-for-net: [BUG] DefaultAzureCredential does not fall back to other credential types unless ManagedIdentityCredential is disabled

[zhenlan]

Describe the bug
The DefaultAzureCredential is supposed to fall back to other credential types when ManagedIdentityCredential is failed, but it doesn't.

To Reproduce
This code fails with the exception (see call stack in the later section).

new DefaultAzureCredential()

However, it will succeed if I disable the ManagedIdentityCredential.

new DefaultAzureCredential(new DefaultAzureCredentialOptions() { ExcludeManagedIdentityCredential = true })

Exception or Stack Trace

Azure.Identity.AuthenticationFailedException
  HResult=0x80131500
  Message=The DefaultAzureCredential failed due to an unhandled exception:  ManagedIdentityCredential failed with unhandled exception The authentication request failed due to an unhandled exception.  See inner exception for details..
  EnvironmentCredential is unavailable Environment variables not fully configured. AZURE_TENANT_ID and AZURE_CLIENT_ID must be set, along with either AZURE_CLIENT_SECRET or AZURE_USERNAME and AZURE_PASSWORD. Currently set variables [  ].
  ManagedIdentityCredential failed with The authentication request failed due to an unhandled exception.  See inner exception for details..
See inner exception for more detail.
  Source=Azure.Identity
  StackTrace:
   at Azure.Identity.DefaultAzureCredential.<GetTokenAsync>d__10.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Azure.Identity.DefaultAzureCredential.<GetTokenAsync>d__9.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.<ProcessAsync>d__8.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable.ConfiguredValueTaskAwaiter.GetResult()
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable.ConfiguredValueTaskAwaiter.GetResult()
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable.ConfiguredValueTaskAwaiter.GetResult()
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable.ConfiguredValueTaskAwaiter.GetResult()
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable.ConfiguredValueTaskAwaiter.GetResult()
   at Azure.Core.Pipeline.HttpPipeline.<SendRequestAsync>d__10.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at Azure.Data.AppConfiguration.ConfigurationClient.<GetConfigurationSettingsPageAsync>d__44.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Azure.Core.PageResponseEnumerator.FuncAsyncPageable`1.<AsPages>d__2.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16 token)
   at Azure.Core.PageResponseEnumerator.FuncAsyncPageable`1.<AsPages>d__2.System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult(Int16 token)
   at Azure.AsyncPageable`1.<GetAsyncEnumerator>d__6.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Azure.AsyncPageable`1.<GetAsyncEnumerator>d__6.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16 token)
   at Azure.AsyncPageable`1.<GetAsyncEnumerator>d__6.System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult(Int16 token)
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.<>c__DisplayClass14_3.<<LoadAll>b__4>d.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.<>c__DisplayClass14_3.<<LoadAll>b__4>d.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult()
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.TracingUtils.<CallWithRequestTracing>d__4.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult()
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.<CallWithRequestTracing>d__21.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.<LoadAll>d__14.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult()
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.Load()
   at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at Microsoft.Extensions.Hosting.HostBuilder.BuildAppConfiguration()
   at Microsoft.Extensions.Hosting.HostBuilder.Build()
   at WebApplication5.Program.Main(String[] args)

Inner Exception 1:
AggregateException: The DefaultAzureCredential failed due to an unhandled exception:  ManagedIdentityCredential failed with unhandled exception The authentication request failed due to an unhandled exception.  See inner exception for details.. (Environment variables not fully configured. AZURE_TENANT_ID and AZURE_CLIENT_ID must be set, along with either AZURE_CLIENT_SECRET or AZURE_USERNAME and AZURE_PASSWORD. Currently set variables [  ]) (The authentication request failed due to an unhandled exception.  See inner exception for details.)

Inner Exception 2:
CredentialUnavailableException: Environment variables not fully configured. AZURE_TENANT_ID and AZURE_CLIENT_ID must be set, along with either AZURE_CLIENT_SECRET or AZURE_USERNAME and AZURE_PASSWORD. Currently set variables [  ]

Setup (please complete the following information):

• OS: Win10 v1909
• IDE : VS2019 16.5.0 Preview 1.0
• Version of the Library used: Azure.Identity v1.1.0

[schaabs]

@zhenlan Thanks for filing this issue. Sorry you're running into this trouble. We're currently looking into this issue, but unfortunately the exception details thrown from the DefaultAzureCredential are less than optimal and don't have the information we need to diagnose the issue you're encountering. We're working on reproducing your issue, but if you have a reliable repro you could debug and could provide a bit more info, it would help us out to get this issue fixed.

Can you enable Break on all exceptions (https://docs.microsoft.com/en-us/visualstudio/debugger/managing-exceptions-with-the-debugger?view=vs-2019) and copy the exception details for the exception that gets thrown when ManagedIdentityCredential.GetToken is on the stack.

Alternatively, if you catch the AuthenticationFailedException and print the inner exceptions corresponding to the ManagedIdentityCredential as below this would also give us the information needed.

~~~ c#
try{
...
{
catch(AuthenticationFailedException e)
{
foreach(var inner in ((AggregateException)e.InnerException).InnerExceptions)
{
if(inner.Message.Contains("ManagedIdentityCredential"))
{
Console.WriteLine(inner.ToString());

        Console.WriteLine(inner.InnerException.ToString());
   }
}

}
~~~

[zhenlan]

Thanks for looking into this, @schaabs.

I enabled all CLR exceptions and disabled just-my-code debugging.

I got Azure.RequestFailedException first. Here are the detailed exception and call stack.

After pressing F5 (continue), I then got Azure.Identity.AuthenticationFailedException. Here are the detailed exception and call stack. The call stacks are very similar. I attached both in case I miss anything.

I also want to mention, when I ran the app in debug mode, occasionally (1 out of ~5 times) after I got a bunch of TaskCanceledException (exception and call stack) the app succeeded in the end. The app never succeeded when it's run without debugger attached (ctrl+F5).

[schaabs]

Looking at the error message I'm seeing a response back from the Managed Identity endpoint:

~~~
Azure.RequestFailedException
HResult=0x80131500
Message=Service request failed.
Status: 400 (Bad Request)

Content:
{"error":"invalid_request","error_description":"Identity not found"}
~~~

Given this error and the fact that we got a response from the managed identity endpoint it seems like your running in an Azure hosted environment which supports Managed Identity. Are you sure that the host your running on has managed identity enabled?

[schaabs]

Or perhaps you're using an AzureVM as a dev machine? If this is the case, the DefaultAzureCredential doesn't currently support falling back to other credential types in this scenario without explicitly excluding the ManagedIdentityCredential

[zhenlan]

I’m using a VM from DevTest Lab, so this probably explained it. Thanks.

Should this be considered a bug? It's not uncommon for customers using VMs from Azure.

Do we know why it worked In debug mode sometimes? This made it even worse as it becomes indeterministic.

[schaabs]

I agree that this should be an scenario we support. I'm currently investigating the best solution for this and we'll try to include support for this as part of our planning for our next release.

As to why this worked in debug mode in some cases this is probably because the detection of IMDS (VM MSI) availability is timing sensitive. We try to connect to the endpoint but set a very short timeout for this connection (200ms I believe) as to not wait for the default timeout of 30 seconds before trying other credentials. So if you were stepping through with a debugger it's quite possible this timeout would be reached before the connection was established.

[schaabs]

@zhenlan I believe changes merged in PR #9025 should address your issue. The update handles 400 responses from the managed identity endpoint as indicating that no identity is available on the current resource.

This fix will be included in the next release of Azure.Identity which is currently scheduled for mid February. Please feel free to reopen this issue if you still have problems with this fix in place.

[balukantu]

Hi @schaabs .
I am usning the latest Idenityt (1.1.1) but facing the same issue. can you please provide whether the fix is available in the mentioned version.

[schaabs]

@balukantu the fix for this issue is in 1.1.1. Can you provide me with the exception details for the issue you're hitting?

[balukantu]

Here is the exception it is throwing

Azure.Identity.CredentialUnavailableException: 'No managed identity endpoint found.'

Code
string azureAppConfigEndPoint = "https://balukantuconfig.azconfig.io"; builder.AddAzureAppConfiguration(opt => opt.Connect(new Uri(azureAppConfigEndPoint), new ManagedIdentityCredential()));

FYI...

• The issue is coming when I try to debug in my local environment. In Dev, it working without any issues.

[balukantu]

Hi @schaabs , Do you have any update on this?

[schaabs]

@balukantu, to authenticate when debugging in your local environment you should be using the DefaultAzureCredential. This allows authenticating with managed idententy when available, and other mechanisms when running in a local environment. In the code sample you provided you are using new ManagedIdentityCredential() which will only authenticate when running in an azure hosted environment which supports managed identity.

[gbelzile]

I just converted a project from 2.2 to 3.1 using Azure.Identity 1.1.1 and I'm getting the same error: ManagedIdentityCredential authentication unavailable, no managed identity endpoint found.
Am I missing something? Worked well in 2.2.

Here is the code:

configurationBuilder.AddAzureAppConfiguration(options =>
            {
                options.Connect(new Uri(configEndpoint), new DefaultAzureCredential())
                    .ConfigureRefresh(refresh =>
                {
                    refresh.SetCacheExpiration(TimeSpan.FromSeconds(60));
                }).ConfigureKeyVault(kv =>
                    kv.SetCredential(new DefaultAzureCredential(new DefaultAzureCredentialOptions())
                ));
            });

Exception is:

Azure.Identity.CredentialUnavailableException: ManagedIdentityCredential authentication unavailable, no managed identity endpoint found.
   at Azure.Identity.ExtendedAccessToken.GetTokenOrThrow()
   at Azure.Identity.ManagedIdentityCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.ProcessAsync(HttpM
essage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.HttpPipeline.SendRequestAsync(Request request, CancellationToken cancellationToken)
   at Azure.Data.AppConfiguration.ConfigurationClient.GetConfigurationSettingsPageAsync(SettingSelector selector, String pageLink, CancellationToken cancellationToken)
   at Azure.Core.PageResponseEnumerator.FuncAsyncPageable`1.AsPages(String continuationToken, Nullable`1 pageSizeHint)+MoveNext()
   at Azure.Core.PageResponseEnumerator.FuncAsyncPageable`1.AsPages(String continuationToken, Nullable`1 pageSizeHint)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at Azure.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()
   at Azure.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()
   at Azure.AsyncPageable`1.GetAsyncEnumerator(C
ancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.<>c__DisplayClass12_3.<<LoadAll>b__4>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.<>c__DisplayClass12_3.<<LoadAll>b__4>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.TracingUtils.CallWithRequestTracing(Boolean tracingEnabled, RequestType requestType, HostType hostType, Func`1 clientCall)
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.CallWithRequestTracing(Func`1 clientCall)
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.LoadAll()
   at Microsoft.Extension
s.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.Load()
   at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
   at Hydro.HQConnect.Automation.Api.Program.Main(String[] args) in C:\\Users\\AdminGB\\source\\repos\\hqconnect_service_automation\\Hydro.HQConnect.Automation.Api\\Program.cs:line 13"

[schaabs]

@gbelzile sorry you're running into this issue. Just to clarify a bit. First, I'm not sure which library you're referring to when you say you've converted from 2.2 to 3.1. Secondly, from looking at you're callstack it seems that you're not authenticating with the DefaultAzureCredential but rather directly with the ManagedIdentityCredential

~
'Azure.Identity.CredentialUnavailableException: ManagedIdentityCredential authentication unavailable, no managed identity endpoint found.
at Azure.Identity.ExtendedAccessToken.GetTokenOrThrow()
at Azure.Identity.ManagedIdentityCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
...
~

If the authentication were using the DefaultAzureCredential I would expect this to be on the callstack between BearerTokenAuthenticationPolicy.ProcessAsync and ManagedIdentityCredential.GetTokenAsync. Perhaps the code in your snippet is out of sync with the callstack you provided. Or possibly the AzureAppConfigurationProvider is using a different credential under the covers I'm not sure. Adding @pakrym who might know more about this.

Could you also provide us iwth the version of Microsoft.Extensions.Configuration.AzureAppConfiguration and Azure.Data.AppConfiguration you're referencing?

[gbelzile]

@schaabs You were right. I had a wrong package reference in my project. Code wasn't up to date. DefaultAzureCredential works fine now. Sorry about that. Thanks for taking the time to take a look.

[schaabs]

@gbelzile No problem, glad you were able to get it working.

[schaabs]

The specific bug detailed in this issue has been resolved. If you are experiencing an issue authenticating with the DefaultAzureCredential please open a new issue, so we can track it appropriately.