Microsoft-authentication-library-for-dotnet: [Bug] ArgumentNullException / "A device attached to the system is not functioning" in RSACryptoServiceProvider under heavy load

[alexzaitzev]

Which Version of MSAL are you using ?
Microsoft.Identity.Client 4.22.0

Platform
net461

What authentication flow has the issue?

• Desktop / Mobile
  
  
  
  • [ ] Interactive
  
  
  • [ ] Integrated Windows Auth
  
  
  • [ ] Username Password
  
  
  • [ ] Device code flow (browserless)
  
  
• Web App
  
  
  
  • [x] Authorization code
  
  
  • [ ] OBO
  
  
• Web API
  
  
  
  • [ ] OBO
  
  

Is this a new or existing app?
The app was in production with the library Microsoft.IdentityModel.Clients.ActiveDirectory 5.2.8. I haven't upgraded MSAL, but sometimes started to observe an exception: The parameter is incorrect. Exception type: CryptographicException. I've updated to Microsoft.Identity.Client 4.22.0 but still continue receiving exceptions with the similar stack trace: Value cannot be null. Parameter name: keyBlob Exception type: ArgumentNullException.

Repro
It's hard to reproduce, issue began to appear when request rate was increased. It usually happens after swap production & staging slots during deployment in Azure Cloud service (Classic) or after scaling out. Restarting problem VM fixes the problem.

This code initializes KeyVaultClient(AuthenticationCallback authenticationCallback, params DelegatingHandler[] handlers) in AuthenticationCallback delegate:

private async Task<string> GetToken(string authority, string resource, string scope)
{
    IConfidentialClientApplication clientApplication = ConfidentialClientApplicationBuilder
        .Create(this.clientId)
        .WithAuthority(new Uri(authority), validateAuthority: true)
        .WithCertificate(this.clientCertificate)
        .Build();

    AuthenticationResult result = await clientApplication
        .AcquireTokenForClient(new[] { scope }) // exception throws here
        .ExecuteAsync()
        .ConfigureAwait(false);

    return result.AccessToken;
}

Expected behavior
Acquire authentication token without an exception.

Actual behavior
Sometimes exception is being thrown:

Value cannot be null.
Parameter name: keyBlob Exception type: ArgumentNullException 
Stack trace:
   at System.Security.Cryptography.RSACryptoServiceProvider.IsPublic(Byte[] keyBlob)
   at System.Security.Cryptography.X509Certificates.PublicKey.get_Key()
   at Microsoft.Identity.Client.Platforms.net45.NetDesktopCryptographyManager.SignWithCertificate(String message, X509Certificate2 certificate)
   at Microsoft.Identity.Client.Internal.JsonWebToken.Sign(ClientCredentialWrapper credential, Boolean sendCertificate)
   at Microsoft.Identity.Client.Internal.Requests.ClientCredentialHelper.CreateClientCredentialBodyParameters(ICoreLogger logger, ICryptographyManager cryptographyManager, ClientCredentialWrapper clientCredential, String clientId, AuthorityEndpoints endpoints, Boolean sendX5C)
   at Microsoft.Identity.Client.OAuth2.TokenClient.AddBodyParamsAndHeaders(IDictionary`2 additionalBodyParameters, String scopes)
   at Microsoft.Identity.Client.OAuth2.TokenClient.<SendTokenRequestAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Identity.Client.Internal.Requests.RequestBase.<SendTokenRequestAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.<FetchNewAccessTokenAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.<ExecuteAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Identity.Client.Internal.Requests.RequestBase.<RunAsync>d__13.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Identity.Client.ApiConfig.Executors.ConfidentialClientExecutor.<ExecuteAsync>d__3.MoveNext()

Possible Solution
None.

Additional context/ Logs / Screenshots
Stack trace of the exception in Microsoft.IdentityModel.Clients.ActiveDirectory 5.2.8 was:

The parameter is incorrect.
Exception type: CryptographicException 
Stack trace:
   at System.Security.Cryptography.CryptographicException.ThrowCryptographicException(Int32 hr)
   at System.Security.Cryptography.Utils.SignValue(SafeKeyHandle hKey, Int32 keyNumber, Int32 calgKey, Int32 calgHash, Byte[] hash, Int32 cbHash, ObjectHandleOnStack retSignature)
   at System.Security.Cryptography.Utils.SignValue(SafeKeyHandle hKey, Int32 keyNumber, Int32 calgKey, Int32 calgHash, Byte[] hash)
   at System.Security.Cryptography.RSACryptoServiceProvider.SignHash(Byte[] rgbHash, Int32 calgHash)
   at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Platform.SigningHelper.SignWithCertificate(String message, X509Certificate2 certificate)
   at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.ClientCreds.JsonWebToken.Sign(IClientAssertionCertificate credential, Boolean sendX5c)
   at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.ClientCreds.ClientKey.AddToParameters(IDictionary`2 parameters)
   at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Flows.AcquireTokenHandlerBase.<SendTokenRequestAsync>d__70.MoveNext()

[bgavrilMS]

Thanks for raising this issue @alexzaitzev; We are actively working on a similar issue reported by an internal partner and I'll update this thread with the result. Restarting VMs for now is a the only known mitigation.

As an alternative, you can do the signing yourself, MSAL allows this - see https://github.com/AzureAD/microsoft-authentication-library-for-dotnet/wiki/Client-Assertions#signed-assertions (read below)

Details

Tests have shown that when a large number or RSA objects are created between 1 in 15 - 30 K will start failing when a new RSA object is created before the signing. When the same RSA object is used, there are no errors. However it is not simple to reuse / cache this object. This is likely a bug in the crypto stack, and fixing that is not practical.

The code that we use to sign with a certificate is here - see the #else branch as you target .net461.

The issue we are investigating has a very similar stack trace but different message.

System.Security.Cryptography.CryptographicException: A device attached to the system is not functioning.
at System.Security.Cryptography.CryptographicException.ThrowCryptographicException(Int32 hr) at System.Security.Cryptography.Utils.SignValue(SafeKeyHandle hKey, Int32 keyNumber, Int32 calgKey, Int32 calgHash, Byte[] hash, Int32 cbHash, ObjectHandleOnStack retSignature) at System.Security.Cryptography.Utils.SignValue(SafeKeyHandle hKey, Int32 keyNumber, Int32 calgKey, Int32 calgHash, Byte[] hash) at System.Security.Cryptography.RSACryptoServiceProvider.SignHash(Byte[] rgbHash, Int32 calgHash) at Microsoft.Identity.Client.Platforms.net45.NetDesktopCryptographyManager.SignWithCertificate(String message, X509Certificate2 certificate) at Microsoft.Identity.Client.Internal.JsonWebToken.Sign(ClientCredentialWrapper credential, Boolean sendCertificate) at Microsoft.Identity.Client.Internal.Requests.ClientCredentialHelper.CreateClientCredentialBodyParameters(ICoreLogger logger, ICryptographyManager cryptographyManager, ClientCredentialWrapper clientCredential, String clientId, AuthorityEndpoints endpoints, Boolean sendX5C) at Microsoft.Identity.Client.OAuth2.TokenClient.AddBodyParamsAndHeaders(IDictionary2 additionalBodyParameters, String scopes) at Microsoft.Identity.Client.OAuth2.TokenClient.<SendTokenRequestAsync>d__5.MoveNext()

[bgavrilMS]

Just an update that we're still working on this.

[henrik-me]

Current progress: https://github.com/AzureAD/microsoft-authentication-library-for-dotnet/pull/2276

[alexzaitzev]

Thanks for the updates. Btw workaround with self-signing helped. Waiting for your PR be merged.

[pmaytak]

This is included in MSAL 4.25.0 release.

cc: @alexzaitzev