Microsoft-authentication-library-for-dotnet: [Bug] AADSTS50196: The server terminated an operation because it encountered a loop while processing a request

[rfcdejong]

Which Version of MSAL are you using ?
MSAL 4.12.0

Platform
net462

What authentication flow has the issue?

• Desktop / Mobile
  
  
  
  • [x] Interactive
  
  
  • [ ] Integrated Windows Auth
  
  
  • [ ] Username Password
  
  
  • [ ] Device code flow (browserless)
  
  
• Web App
  
  
  
  • [ ] Authorization code
  
  
  • [ ] OBO
  
  
• Web API
  
  
  
  • [ ] OBO
  
  

Other? - please describe;

Is this a new or existing app?
c. This is a new app or experiment

Repro
Each requests from the desktop application to the backend a token is requested. This is happening from multiple threads.

        public IPublicClientApplication App {get;set;}
        ...
        public async Task<AuthenticationResult> AcquireTokenAsync(IAccount account) 
        {
                return await App.AcquireTokenSilent(Scopes, account)
                    .WithAuthority(AadAuthorityAudience.AzureAdAndPersonalMicrosoftAccount)
                    .ExecuteAsync();
        }

Expected behavior
I thought that App would cache the tokens and only acquire a new token when it is (nearly) expired?

Actual behavior
Exception is thrown. AADSTS50196: The server terminated an operation because it encountered a loop while processing a request

Possible Solution
Should I cache the token myself or lock the App instance for single use only?

Additional context/ Logs / Screenshots
Seems the Javascript MSAL library had the same issue
https://github.com/AzureAD/microsoft-authentication-library-for-js/issues/547

[jmprieur]

@rfcdejong can you please check if you see the same behavior with MSAL.NET 4.11.0 ?
(just to confirm an intuition)

[rfcdejong]

@rfcdejong can you please check if you see the same behavior with MSAL.NET 4.11.0 ?
(just to confirm an intuition)

I had the exception using 4.11.0 and updated to 4.12.0 didn't fix it.

[rfcdejong]

To verify with 4.12.0

Microsoft.Identity.Client.MsalUiRequiredException: 'AADSTS50196: The server terminated an operation because it encountered a client request loop. Please contact your app vendor.
Trace ID: 988e6c34-56a7-4888-ab51-860838141100
Correlation ID: c2d4883f-084e-4835-a368-b70be2153872
Timestamp: 2020-04-28 08:45:18Z'

This exception was originally thrown at this call stack:
Microsoft.Identity.Client.OAuth2.OAuth2Client.ThrowServerException(Microsoft.Identity.Client.Http.HttpResponse, Microsoft.Identity.Client.Core.RequestContext)
Microsoft.Identity.Client.OAuth2.OAuth2Client.ExecuteRequestAsync(System.Uri, System.Net.Http.HttpMethod, Microsoft.Identity.Client.Core.RequestContext, bool)
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Identity.Client.OAuth2.OAuth2Client.GetTokenAsync(System.Uri, Microsoft.Identity.Client.Core.RequestContext)
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Identity.Client.OAuth2.TokenClient.SendHttpAndClearTelemetryAsync(string)
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
...
[Call Stack Truncated]

[jmprieur]

@rfcdejong : thanks for confirming. Can you please tell us a bit more about the context of your code above. Is it called from several threads? do you have repro steps that we could use?

[jmprieur]

cc: @bgavrilMS

[rfcdejong]

@rfcdejong : thanks for confirming. Can you please tell us a bit more about the context of your code above. Is it called from several threads? do you have repro steps that we could use?

Well, it seems the code calling it is not async. We used MSAL v1 1.1.0-preview before for years in our application. I'm just migrating it to MSAL v2 but the code is not shareable anyway.

To overcome race conditions and the async await pattern and instead of ConfigureAwait in combination with GetAwaiter my co-workers wrote code like this

var task = Task<AuthenticationResult>.Factory.StartNew(() => Federation.AcquireTokenSilentAsync(user).Result);

I just refactored the code a bit and it seems I might have fixed it on our side. Perhaps token caching inside calls like above code do not work.

[jmprieur]

how did you refactor your code? @rfcdejong ?

[rfcdejong]

how did you refactor your code? @rfcdejong ?

By overriding the HttpClient implementing a custom MessageHandler. Then overriding the SendAsync method which is already async and adding the access token as header there instead of at construction of the HttpClient which is inside a non async method.

[bgavrilMS]

@rfcdejong - the error you are getting seems to indicate that the server is throttling you. This can occur if a variety of situations:

• you call AAD too often with same request
• AAD is having problems and is asking you to back off (but more likely is the 1st issue)

It may be that your app is retrying the same request too often, maybe there is a bug in your async handling or in some retry policy?

Some thoughts on high availability that might help: https://github.com/AzureAD/microsoft-authentication-library-for-dotnet/wiki/High-availability

@jmprieur - client side throttling was not released in 4.12, it will be available in 4.13. The server behavior when it detects a loop is to break the loop by sending an "invalid_grant" suberror, which MSALs interpret as "UI Required Exception". I am not sure if a retry-after header is sent, probably not. At some point, we want the server to send HTTP 429 / Retry-After X messages instead, but discussions are still ongoing. Without this signal, client side throttling will not work. Oh, and for now, client-side throttling will not be enabled on confidential client flows. So probably 4.14 if we decide it's a good idea?

[rfcdejong]

Closed as I fixed it locally. Perhaps it would be a solution to implement an Execute function next to ExecuteAsync so that code which isn't async will also work without workarounds on the awaiter.

[nmushovic]

I'm getting the same error. Just to test I ran AcquireTokenSilent in a loop sleeping 2 seconds with Fiddler running. When I startup, I see the initial call to http://login.microsoftonline.com:443, it runs 19 times and then fails with AADSTS50196. Doesn't look like it's throttled by the server because I don't see any more calls to http://login.microsoftonline.com:443, just that initial one.

I thought I could pull from the cache at will and it would only make the call when the token was close to expiring.

[jmprieur]

@nmushovic : what type of app do you have? this is probably the symptom that you are not handling conditional access properly. if you are building an ASP.NET Core web app, please see https://github.com/AzureAD/microsoft-identity-web/wiki/Managing-incremental-consent-and-conditional-access

[nmushovic]

It's a Winforms desktop app. A few call's are made to a secured web api over a short period of time. I call AcquireTokenSilent before each web api, but if I get to 19 calls, then I get the loop error (verified using a console app and just looping calls to AcquireTokenSilent). Checking fiddler though, it only ever hit http://login.microsoftonline.com:443 once.

Is AcquireTokenSilent not meant to be used a cache that can be frequently accessed?

[TheJoeFin]

I am seeing this issue on my UWP app when I update the identity package beyond 4.18. I am using this in an application where I am uploading many little files. Is there a new way to access the token frequently?

[bgavrilMS]

@nmushovic @TheJoeFin - could you please open new work items and reference this one? Closed items tend to get less attention.

It is possible there is a bug in the library or on the server if the token cache is bypassed by AcquireTokenSilent. This can happen if you request scopes X, Y, Z and the server reports scopes X, Y only.

Could please detail exactly the scopes you request, if you serialize the cache (not required on UWP) and if you follow the pattern

try 
   AcquireTokenSilent
catch UIRequiredException
   AcquireTokenInteractive