Runtime: Problem with HttpClient after updating my project from net461 to netcoreapp2.0

[xrkolovos]

I have a piece of code that makes a httpRequest in a page in a domain www.example.gr
i am redirected in a page (auth.example.gr) to login.
I post some data to login and get the right cookies, so i can use the admin page.

This code works fine with .net framework 461.

When i test it with netcore 2.0, when i post the data to Login i get this error.

The operation has been canceled

The read operation failed, see inner exception.
at System.Net.Http.WinHttpResponseStream.d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Net.Http.HttpContent.d__48.MoveNext()

Error while copying content to a stream.
at System.Net.Http.WinHttpResponseStream.d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Net.Http.HttpContent.d__48.MoveNext()

I don't know what else to provide to help you more. Please inform me. Thank you.

[davidsh]

We have seen some reports of this issue but have not yet repro'd the problem.

Please attach a complete Visual Studio solution of the repro. That will allow us to diagnose. Thx.

[xrkolovos]

I can't do this because i order to test it i must provide you with the credentials that don't belong to us.
Any other solution? I can provide you remote desktop.

[davidsh]

We don't need the credentials. But we do need the complete app or a smaller test app that repros the problem. It's important to see the way the code is structured, the Http APIs calls that are being used, and the versions of the NuGet packages that are being used. We also need to either use the same server or instructions for how to set up a test server that works with your app. Is TLS/SSL involved? Is authentication or redirection being used, etc. Understanding all these things helps to figure out what is going on.

[xrkolovos]

Is TLS/SSL involved?

Yes

Is authentication or redirection being used, etc.

Yes, redirection

My code

Makes a httpRequest in a page in a domain www.example.gr
i am redirected in a page (auth.example.gr) to login.
I post some data to login and get the right cookies, so i can use the www.example.gr App

So, in order to test it you must login to get the redirect back to www.example.gr. So, i can't share the credentials. This is not mine enviroment and not mine credentials. I can't have extra credentials.
The auth.example.gr has https (TLS/SSL enabled). The app www.example.gr is only http.

[davidsh]

cc: @karelz

It is not possible for us to diagnose further without a repro. I understand you can't share the credentials. But please share the Visual Studio solution with the source code of the repro so that we can understand more about the types of data being POST'd and the Http APIs calls that are being made. Thx.

[karelz]

@xrkolovos it would be great if you can try to create minimal repro (even though it requires special site credentials to run) ... it will help us understand which code is in place as first step. Thanks!

[xrkolovos]

How would you like to share the code? In a private way.

[xrkolovos]

I send a pm in @karelz twitter

[xrkolovos]

I email you. Please inform here if you found this problem. Thank you

[davidsh]

Thank you for submitting the repro. I was able to repro the problem. The .NET Core test fails. The .NET Framwork test passes. Both tests use a shared library that makes the HttpClient API calls. The shared library is built targetting NetStandard.Library.

The core problem is that there is an HTTPS -> HTTP redirection happening in the sequence of calls. On .NET Framework HTTPS -> HTTP automatic redirection is allowed. But on .NET Core, this is not allowed. This is considered an insecure pattern.

To workaround this, you can disable automatic redirection in the HttpClientHandler. And then parse the "Location: " header from the 3xx responses yourself and then submit new Http Requests. Here is the code you would add to disable automatic redirections in your repro.

```c#
_handler.AllowAutoRedirect = false;

```

I will say, though, that the error message you are getting is confusing. Because this is a POST request that is failing during redirection, it fails with the I/O exception error since it is trying to write the request body. We will try to make a fix to HttpClientHandler so that the error message will be clearer that the request is aborting due to HTTPS -> HTTP automatic redirection not being allowed.

cc: @stephentoub

[davidsh]

I will say, though, that the error message you are getting is confusing. Because this is a POST request that is failing during redirection, it fails with the I/O exception error since it is trying to write the request body. We will try to make a fix to HttpClientHandler so that the error message will be clearer that the request is aborting due to HTTPS -> HTTP automatic redirection not being allowed.

So, it turns out that returning a better error message about disallowing HTTPS -> HTTP is more complicated.

In .NET Core, we disallow automatic redirection from HTTPS to HTTP. When we detect this, our underlying HTTP stack (WinHttp) prevents the auto redirection. It just ends the request and returns the 3xx response. However, it also does something we didn't expect. It closes the request connection to the server. So, after getting the 3xx response, if the HttpClient then tries to read the response body of the 3xx response, it gets an error in the WinHttpResponseStream methods. WinHttpQueryDataAvailable returns 12017 (ERROR_WINHTTP_OPERATION_CANCELLED). Buffering in the response body is a default action of .PostAsync(). You can workaround around that by using HttpCompletionOption.ResponseHeadersRead:

c# var request = new HttpRequestMessage(HttpMethod.Post, secondurl); request.Content = content; using (var response = await _httpClient.SendAsync(request, HttpCompletionOption.ResponseHeadersRead))

So, this is why the error call stack looks like this:

System.Net.Http.HttpRequestException : Error while copying content to a stream.
---- System.IO.IOException : The read operation failed, see inner exception.
-------- System.Net.Http.WinHttpException : The operation has been canceled

Given this information, we will investigate if we should return an error (exception) when doing HTTPS -> HTTP redirection in the first place. Doing such a change though would be a change in behavior in .NET Core from the current 2.0 version.

In summary, though, the root problem is that your server is doing HTTPS -> HTTP redirection. And if that behavior is corrected (or you process the redirects manually), you will avoid this problem.

[xrkolovos]

Thank you for your advices. I did understand the problem was in that area, but the error is very confusing, as you pointed out. That is the issue.

[davidsh]

Closing this issue now since the behavior is understood and dotnet/corefx#24577 will track changing this to an exception.

[SpaceMonkeyForever]

For anyone else reading this after being frustrated for hours, if you want to force allow redirection:

        var request = new HttpRequestMessage(HttpMethod.Get, url);
        var httpResponseMessage = await _client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);

        // IMPORTANT
        // .NET Core does not allow redirection from HTTPs -> HTTP and won't give a proper exception message
        // We have to account for that ourselves by not following redirections and instead creating a new request
        // Note that this behaviour doesn't exist in .NET Framework
        if (httpResponseMessage.Headers.Location != null)
        {
            request = new HttpRequestMessage(HttpMethod.Get, httpResponseMessage.Headers.Location);
            httpResponseMessage = await _client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
        }