Runtime: Deadlock between HttpClientHandler and HttpWebRequest

[jahmai]

Classic out of order lock. This issue exists in 4.1.1.1 but not the .net Framework version (4.0.0.0).
Seems to be introduced by https://github.com/dotnet/corefx/commit/d24ddedc5edb10d4ed385a1eb8bfb2351a117cd5.
Which is related to dotnet/runtime#18280

Thread 1:

     [Managed to Native Transition]    
     System.dll!System.Net.Connection.CloseOnIdle()    !!! lock (this) !!!
     System.dll!System.Net.ConnectionGroup.PruneExcesiveConnections()    
     System.dll!System.Net.ServicePoint.ResolveConnectionLimit()    
     System.dll!System.Net.ServicePoint.ConnectionLimit.set(int value)    !!! lock (this) !!!
     System.Net.Http.dll!System.Net.Http.HttpClientHandler.CreateAndPrepareWebRequest(System.Net.Http.HttpRequestMessage request)    
     System.Net.Http.dll!System.Net.Http.HttpClientHandler.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken)    
     System.Net.Http.dll!System.Net.Http.HttpMessageInvoker.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken)    
     System.Net.Http.dll!System.Net.Http.HttpClient.SendAsync(System.Net.Http.HttpRequestMessage request, System.Net.Http.HttpCompletionOption completionOption, System.Threading.CancellationToken cancellationToken)    
     System.Net.Http.dll!System.Net.Http.HttpClient.SendAsync(System.Net.Http.HttpRequestMessage request)    

Thread 2:

     [Managed to Native Transition]    
     System.dll!System.Net.ServicePoint.IncrementConnection()      !!! lock (this) !!!
     System.dll!System.Net.Connection.CheckNonIdle()    
     System.dll!System.Net.Connection.StartRequest(System.Net.HttpWebRequest request, bool canPollRead)    
     System.dll!System.Net.Connection.SubmitRequest(System.Net.HttpWebRequest request, bool forcedsubmit)    !!! lock (this) !!!
     System.dll!System.Net.ServicePoint.SubmitRequest(System.Net.HttpWebRequest request, string connName)    
     System.dll!System.Net.HttpWebRequest.SubmitRequest(System.Net.ServicePoint servicePoint)    
     System.dll!System.Net.HttpWebRequest.GetResponse()    

[jahmai]

A (possible) workaround is to ensure that your ServicePoint.ConnectionLimit and ServicePointManager.DefaultConnectionLimit values are the same before using HttpClientHandler - this avoids the lock inside ServicePoint.ConnectionLimit.

[davidsh]

Thank you for reporting this issue. This issue you are reporting in about the .NET Framework System.Net.Http. .NET Core uses a different HTTP stack than .NET Framework.

The commit you reference above was about us reverting the implementation in .NET Core so that when the package installs on .NET Framework, it will use the original HTTP stack in .NET Framework. Thus, the stack traces you show are about .NET Framework behavior.

Do you have a repro (Visual Studio solution)? If so, we can use that as I then will move this bug to a different bug database.

[jahmai]

corefx21796.zip

Here's the repro.

[kohner86]

I seem to be running into the same issue inside of the AWS Dynamo DB client. I have a few dozen worker threads which seem to slowly lockup, all of which have stack traces which look like this:
[Managed to Native Transition]
System.dll!System.Net.ConnectionGroup.IncrementConnection() Unknown

System.dll!System.Net.Connection.CheckNonIdle() Unknown
System.dll!System.Net.Connection.StartRequest(System.Net.HttpWebRequest request, bool canPollRead) Unknown
System.dll!System.Net.Connection.SubmitRequest(System.Net.HttpWebRequest request, bool forcedsubmit) Unknown
System.dll!System.Net.ServicePoint.SubmitRequest(System.Net.HttpWebRequest request, string connName) Unknown
System.dll!System.Net.HttpWebRequest.SubmitRequest(System.Net.ServicePoint servicePoint) Unknown
System.dll!System.Net.HttpWebRequest.GetRequestStream(out System.Net.TransportContext context) Unknown
System.dll!System.Net.HttpWebRequest.GetRequestStream() Unknown
AWSSDK.Core.dll!Amazon.Runtime.Internal.HttpHandler.InvokeSync(Amazon.Runtime.IExecutionContext executionContext) Unknown
AWSSDK.Core.dll!Amazon.Runtime.Internal.Unmarshaller.InvokeSync(Amazon.Runtime.IExecutionContext executionContext) Unknown
AWSSDK.Core.dll!Amazon.Runtime.Internal.ErrorHandler.InvokeSync(Amazon.Runtime.IExecutionContext executionContext) Unknown
AWSSDK.Core.dll!Amazon.Runtime.Internal.CallbackHandler.InvokeSync(Amazon.Runtime.IExecutionContext executionContext) Unknown
AWSSDK.Core.dll!Amazon.Runtime.Internal.RetryHandler.InvokeSync(Amazon.Runtime.IExecutionContext executionContext) Unknown
AWSSDK.Core.dll!Amazon.Runtime.Internal.CallbackHandler.InvokeSync(Amazon.Runtime.IExecutionContext executionContext) Unknown
AWSSDK.Core.dll!Amazon.Runtime.Internal.CallbackHandler.InvokeSync(Amazon.Runtime.IExecutionContext executionContext) Unknown
AWSSDK.Core.dll!Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeSync(Amazon.Runtime.IExecutionContext executionContext) Unknown
AWSSDK.Core.dll!Amazon.Runtime.Internal.MetricsHandler.InvokeSync(Amazon.Runtime.IExecutionContext executionContext) Unknown
AWSSDK.Core.dll!Amazon.Runtime.Internal.RuntimePipeline.InvokeSync(Amazon.Runtime.IExecutionContext executionContext) Unknown
AWSSDK.Core.dll!Amazon.Runtime.AmazonServiceClient.Invoke(Amazon.DynamoDBv2.Model.GetItemRequest request, Amazon.Runtime.Internal.Transform.IMarshaller marshaller, Amazon.Runtime.Internal.Transform.ResponseUnmarshaller unmarshaller) Unknown
AWSSDK.DynamoDBv2.dll!Amazon.DynamoDBv2.AmazonDynamoDBClient.GetItem(Amazon.DynamoDBv2.Model.GetItemRequest request) Unknown

[jahmai]

@davidsh I've attached the repro.

[davidsh]

@davidsh I've attached the repro.

Thanks! We'll take the information and create a .NET Framework bug report from it.

[davidsh]

Thank you for the bug report. I was able to repro the deadlock. This is a bug in the .NET Framework and still repro's even with the latest .NET 4.7.1.

I have moved this bug to the .NET Framework bug database. You can track the progress here:
https://developercommunity.visualstudio.com/content/problem/154698/deadlock-between-httpclienthandler-and-httpwebrequ.html

[jahmai]

4 months to confirm a bug with an attached repro?!

[Plasma]

Just hit this bug in production NET FX 4.6.1

[Plasma]

@jahmai @kohner86 I have a potential mitigation:

// Configure default endpoint limit
// This is the value used by AWS SDK, but by explicitly setting it here
// we are hoping to work around a deadlock bug in .NET FX being encountered
// See https://github.com/dotnet/corefx/issues/21796
ServicePointManager.DefaultConnectionLimit = 50;

If you are using AWS SDK, try setting this during application startup.

The deadlock occurs when the ServicePoint limit is changed (which the SDK does for each request), so there is a high chance of deadlocks.

By setting the DefaultConnectionLimit to match what the Amazon SDK keeps setting for each request, the locks are skipped, because the new value equals the (above) default value.