Runtime: HttpClient: OperationCanceledException hides previous exceptions

[Frank591]

Hello.

We use System.Net.Http.HttpClientfor calls between microservices inside k8s.
OS Linux(inside docker)
dotnet TargetFramework: netcoreapp2.1
server: Kestrel
protocol: http

Few days ago we noticed very strange behaviour of http calls:
some calls between microservice(near 2-3%) failed with error

System.OperationCanceledException: The operation was canceled.
   at System.Net.Http.HttpClient.HandleFinishSendAsyncError(Exception e, CancellationTokenSource cts)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at OurCode.HttpServiceClient.GetStringResponseAsync(String methodUri, HttpMethod httpMethod)
   ...another our code...

after our timeout for http calls(it is 3 sec). But there was no logs about call inside callee service.
We enabled packetbeat for tracing http requests and also noticed, that no any http requests from
caller service to callee service was executed.
CPU, memory and network for this services was OK in this time.
Simplified version of our code for http calls looks like:
```c#
public async Task GetStringResponseAsync(String methodUri, HttpMethod httpMethod)
{
int timeoutInMilliseconds = 3000;
var tokenSource = new CancellationTokenSource();
var rm = new HttpRequestMessage(httpMethod, methodUri);
Task httpTask = HttpClient.SendAsync(rm, tokenSource.Token);
tokenSource.CancelAfter(timeoutInMilliseconds);
HttpResponseMessage response = await httpTask;
await EnsureSuccessStatusCode(response);
return await response.Content.ReadAsStringAsync();
}

I opened `HttpClient `code and saw that initial exception can be hidden by `OperationCanceledException`
```c#
private void HandleFinishSendAsyncError(Exception e, CancellationTokenSource cts)
{
    if (NetEventSource.IsEnabled) NetEventSource.Error(this, e);

    // If the cancellation token was canceled, we consider the exception to be caused by the
    // cancellation (e.g. WebException when reading from canceled response stream).
    if (cts.IsCancellationRequested && e is HttpRequestException)
    {
        if (NetEventSource.IsEnabled) NetEventSource.Error(this, $"Canceled");
        throw new OperationCanceledException(cts.Token);
    }
}

Rigth now i totally can't understand what problem can cause this strange behaviour without http request throught network.
My main goal rigth now is uderestand problem reason and solve this problem. Do you have any ideas what problem can cause that behaviour?

And my second goal: pls fix HandleFinishSendAsyncErrorfor stopping hide initial exception
For example
c# if (cts.IsCancellationRequested && e is HttpRequestException) { if (NetEventSource.IsEnabled) NetEventSource.Error(this, $"Canceled"); throw new OperationCanceledException(e,cts.Token); }
Current behavior of HandleFinishSendAsyncError deprives a chance to understand the cause of the problem.

PS Sorry for my bad English

[NArnott]

I'm seeing this as well when calling HttpClient.GetAsync, but only occasionally:

System.OperationCanceledException:
   at System.Net.Http.HttpClient.HandleFinishSendAsyncError (System.Net.Http, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Net.Http.HttpClient+<FinishSendAsyncBuffered>d__58.MoveNext (System.Net.Http, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)

[Gr3yR0n1n]

Has this been resolved yet? I'm seeing it in netcoreapp2.1 and netcoreapp2.2, but only when I request HTTPS. HTTP requests seem to work fine.
Oddly, I too only see this on linux. On OS X it works fine.

Thank you,

[karelz]

The issue is opened, so no, it was not resolved yet.
This will be likely non-trivial to chase down and to root cause. If there is a repro, that would speed up things

[Gr3yR0n1n]

Further research shows that most of the issues may not be specific to corefx. I was able to get other requests working with a much simpler example. I'm unable to share the projects due to the nature of the environment I'm working in and it's security requirements. I can say the only area I'm now having an issue with is using client side x509certificates. Those request I'm unable to get working on linux, but they are working fine in OS X

[NodeAdam]

@karelz

Sorry I may not be understanding the issue but isn't the problem that the initial exception isn't being wrapped in the rethrow? Can't we just call the OperationCanceledException constructor that takes an exception as argument and the problem is solved?

[karelz]

@NodeAdam if you think you have a fix, please submit a PR. Having a repro to validate (even if it is just in specific environment) would be great.

[manums]

I'm seeing OperationCanceledException exception from HttpClient often from

httpClient.SendAsync(httpRequest, HttpCompletionOption.ResponseHeadersRead)

.NET Core 2.1

Exception Trace:

System.OperationCanceledException: The operation was canceled.
at System.Net.Http.HttpClient.HandleFinishSendAsyncError(Exception e, CancellationTokenSource cts)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task"1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)

We have catch block to handle TaskCanceledException (as client timeout) and gracefully retrying SendAsync again.

@karelz Since we do not pass cancellation token to SendAsync in our code, should we treat OperationCanceledException from SendAsync similar to TaskCanceledException (as client timeout probably)? What does OperationCanceledException exception mean in our case where we are not explicitly passing cancellation token?

[gokilvss]

@karelz , We also have similar exception using .Net Core httpclient Async methods only in Linux environment. We wanted to know if the way async methods work in Linux differs from the one that runs in Windows environment. I see couple of other threads having similar exceptions either taskcanceled or operationcanceled. Do you see any possible fixes happening around this area in the coming releases/hotfixes ?

Operation Cancelled True
The operation was canceled.
System.OperationCanceledException: The operation was canceled.
   at System.Net.Http.HttpClient.HandleFinishSendAsyncError(Exception e, CancellationTokenSource cts)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at System.Net.Http.HttpClient.GetStringAsyncCore(Task`1 getTask)

[karelz]

Triage:
Some comments on this issue are about "Why are we throwing OperationCanceledException instead of Timeout?" -- that is tracked separately in dotnet/runtime#21965

The original issue here is either race condition of cancellation + error happening in parallel ... in such case, it is just "bad luck" that the race was lost and cancellation won. We don't expect the window to be large.
The other alternative is that cancellation itself caused new exceptions during cleanup, which are not exceptions that should be interesting to the caller and we intentionally not provide them back.

[ravi6456]

@karelz Hi , I see this errors is closed and tracked for 5.0 milestone.
I am experiencing similar issue but with Azure Fluent SDK
System.OperationCanceledException: The operation was canceled.
at System.Net.Http.HttpClient.HandleFinishSendAsyncError(Exception e, CancellationTokenSource cts)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at Microsoft.Azure.Management.ResourceManager.Fluent.ResourceGroupsOperations.CreateOrUpdateWithHttpMessagesAsync(String resourceGroupName, ResourceGroupInner parameters, Dictionary2 customHeaders, CancellationToken cancellationToken)
at Microsoft.Azure.Management.ResourceManager.Fluent.ResourceGroupImpl.CreateResourceAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Management.ResourceManager.Fluent.Core.ResourceActions.Creatable4.Microsoft.Azure.Management.ResourceManager.Fluent.Core.ResourceActions.IResourceCreator<IResourceT>.CreateResourceAsync(CancellationToken cancellationToken) at Microsoft.Azure.Management.ResourceManager.Fluent.Core.DAG.CreatorTaskItem1.ExecuteAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Management.ResourceManager.Fluent.Core.DAG.TaskGroupBase1.ExecuteNodeTaskAsync(DAGNode1 node, CancellationToken cancellationToken)

As long as the .net core version is 2.1 , it is the same issue discussed above right ?