Runtime: WinHttp buffer supplied to the function was too small error should not be exposed

[stephentoub]

http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/master/job/windows_nt_release_prtest/80/consoleFull

07:27:26      System.Net.Http.Functional.Tests.HttpClientHandler_ServerCertificates_Test.UseCallback_BadCertificate_ExpectedPolicyErrors(url: "https://wrong.host.badssl.com/", expectedErrors: RemoteCertificateNameMismatch) [FAIL]
07:27:26         System.Net.Http.HttpRequestException : An error occurred while sending the request.
07:27:26         ---- System.Net.Http.WinHttpException : A security error occurred
07:27:26         Stack Trace:
07:27:26               at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
07:27:26               at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
07:27:26            D:\j\workspace\windows_nt_re---37265eab\src\System.Net.Http\src\System\Net\Http\HttpClient.cs(392,0): at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext()
07:27:26            --- End of stack trace from previous location where exception was thrown ---
07:27:26               at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
07:27:26               at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
07:27:26            D:\j\workspace\windows_nt_re---37265eab\src\System.Net.Http\tests\FunctionalTests\HttpClientHandlerTest.ServerCertificates.cs(174,0): at System.Net.Http.Functional.Tests.HttpClientHandler_ServerCertificates_Test.<UseCallback_BadCertificate_ExpectedPolicyErrors>d__9.MoveNext()
07:27:26            --- End of stack trace from previous location where exception was thrown ---
07:27:26               at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
07:27:26               at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
07:27:26            --- End of stack trace from previous location where exception was thrown ---
07:27:26               at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
07:27:26               at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
07:27:26            --- End of stack trace from previous location where exception was thrown ---
07:27:26               at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
07:27:26               at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
07:27:26            ----- Inner Stack Trace -----
07:27:26               at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
07:27:26               at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
07:27:26            D:\j\workspace\windows_nt_re---37265eab\src\System.Net.Http.WinHttpHandler\src\System\Net\Http\WinHttpHandler.cs(843,0): at System.Net.Http.WinHttpHandler.<StartRequest>d__102.MoveNext()

[stephentoub]

Also here:
http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/master/job/windows_nt_release_prtest/74/console

[stephentoub]

Failed again, but this time with a different exception message:
"The buffers supplied to a function was too small"
http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/master/job/windows_nt_release_prtest/92/consoleFull

04:25:45      System.Net.Http.Functional.Tests.HttpClientHandler_ServerCertificates_Test.UseCallback_BadCertificate_ExpectedPolicyErrors(url: "https://wrong.host.badssl.com/", expectedErrors: RemoteCertificateNameMismatch) [FAIL]
04:25:45         System.Net.Http.HttpRequestException : An error occurred while sending the request.
04:25:45         ---- System.Net.Http.WinHttpException : The buffers supplied to a function was too small
04:25:45         Stack Trace:
04:25:45               at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
04:25:45               at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
04:25:45            D:\j\workspace\windows_nt_re---37265eab\src\System.Net.Http\src\System\Net\Http\HttpClient.cs(392,0): at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext()
04:25:45            --- End of stack trace from previous location where exception was thrown ---
04:25:45               at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
04:25:45               at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
04:25:45            D:\j\workspace\windows_nt_re---37265eab\src\System.Net.Http\tests\FunctionalTests\HttpClientHandlerTest.ServerCertificates.cs(174,0): at System.Net.Http.Functional.Tests.HttpClientHandler_ServerCertificates_Test.<UseCallback_BadCertificate_ExpectedPolicyErrors>d__9.MoveNext()
04:25:45            --- End of stack trace from previous location where exception was thrown ---
04:25:45               at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
04:25:45               at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
04:25:45            --- End of stack trace from previous location where exception was thrown ---
04:25:45               at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
04:25:45               at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
04:25:45            --- End of stack trace from previous location where exception was thrown ---
04:25:45               at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
04:25:45               at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
04:25:45            ----- Inner Stack Trace -----
04:25:45               at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
04:25:45               at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
04:25:45            D:\j\workspace\windows_nt_re---37265eab\src\System.Net.Http.WinHttpHandler\src\System\Net\Http\WinHttpHandler.cs(843,0): at System.Net.Http.WinHttpHandler.<StartRequest>d__102.MoveNext()

[stephentoub]

From https://github.com/dotnet/corefx/issues/8038:

The exception of "The buffers supplied to a function was too small" doesn't seem like something due to just network vagaries.

http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/master/job/windows_nt_debug_prtest/408/

[stephentoub]

Again:
http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/master/job/windows_nt_debug_prtest/517/consoleText

"The buffers supplied to a function was too small"

[davidsh]

---- System.Net.Http.WinHttpException : The buffers supplied to a function was too small

That particular WinHTTP error is never supposed to bubble out. It means that we called WinHTTP and didn't give it a buffer big enough to return information. That would mean there is a bug in our code.

We didn't start seeing these errors until we added HttpClientHandler properties to expose server certificate validation callbacks. While this error doesn't repro very often, that is the area of code I'll start looking at.

[stephentoub]

Again here:
http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/master/job/windows_nt_debug_prtest/1026/consoleText

[stephentoub]

Again here:
http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/master/job/windows_nt_release_prtest/1025/consoleText

[stephentoub]

Again here:
http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/master/job/windows_nt_release_prtest/998/

[stephentoub]

Again here:
http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/master/job/windows_nt_release_prtest/1021/consoleText

[tmds]

I also occasionally get this error using System.Net.Http 4.1.0 on Windows with a custom ServerCertificateCustomValidationCallback.

[jacksontbryan]

This error is happening to me in a production environment. Packet capture shows the error happens during key exchange. The certificate is valid. I am doing no custom validation. When it happens it is intermittent with ~30% of requests failing for approx. hour period of time.

HttpClient client = new HttpClient();
client.BaseAddress = new Uri(baseUri);
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
var response = await client.PostAsJsonAsync("LabelService.svc/generateLabel", labelRequest);
--exception is blown here while sending the http request
An error occurred while sending the request -- The buffers supplied to a function was too small

[davidsh]

Packet capture shows the error happens during key exchange. The certificate is valid. I am doing no custom validation. When it happens it is intermittent with ~30% of requests failing for approx. hour period of time.

Thanks for the updated info. It sounds like this is a problem with the low-level windows WinHTTP component.

Do you have a stack trace for where the error "The buffers supplied to a function was too small" is coming from? It would be useful to catch this bug in a debugger since that is the only way to get actionable information to diagnose.

[jacksontbryan]

How would you advise setting up the debugger? This happens once every 2-3 days at seemly random times of day.

Stack:
`` at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()
at System.Net.Http.HttpClient.d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Newgistics.Shipping.Service.Repositories.IntellicoreRepository.d__2.MoveNext() in C:\Users\Bjackson\source\newgistics.shipping.service\src\Newgistics.Shipping.Service\Repositories\IntellicoreRepository.cs:line 142
inner exception >>
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at System.Net.Http.WinHttpHandler.d__101.MoveNext()",
"error":"An error occurred while sending the request.
inner exception >> The buffers supplied to a function was too small
````

[EDIT] Format call stack by @karelz

[davidsh]

If the exception is unhandled,then normally your process was crash. If so, you can use "windbg /I" to set up post-mortem debugging. When the process crashes, Windbg will launch automatically and you'll be able to debug and see threads, stack, etc.

[davidsh]

Could be a product bug....need more investigation.

[clement911]

I'm encountering these same errors in an azure production environment. I run asp.net core on top of the full .net 4.6.1 so I suspect it's not specific to .net core.
It happens as the code is trying to send an email using the SendGrid nuget package, which internally seems to use HttpClient.
Usually the error is 'A security error occurred' (it happened 5 times so far), but there is also 1 occurrence of 'The buffers supplied to a function was too small'.
Here is the top of the stack:

System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: A security error occurred at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.Http.WinHttpHandler.<StartRequest>d__103.MoveNext() --- End of inner exception stack trace --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at SendGrid.Web.d__7.MoveNext()`

[karelz]

@davidsh any reason for moving it to Future?

[davidsh]

I believe this is a real product bug. Getting this exception, for example, surfaced "the buffers supplied to a function was too small" should never occur. That particular exception is usually caught internally and we have logic to do the right thing with WinHTTP (which is to resize our buffer and call WinHTTP again).

We haven't seen this a lot, though. So, in terms of prioritizing all the other important 2.0 bugs, we could probably defer it to Future.

[karelz]

Thanks! That makes sense, moving back to Future.

[karelz]

After re-reading the thread, it actually looks that 2 people hit it in the wild. That's concerning. I think we should look into it in 2.0.

[wapxmas]

Also this issue happens to me randomly, along with "WinHttpException: A security error" sometimes, and only if I using https protocol.

[germanmarky]

I am as well seeing this error in production. My code is posting a SOAP request to an HTTPS endpoint. The request does have an embedded encoded document, but the specific document that last caused the error was only 1.1MB. The additional data is less than 1K. As with others, another error received is "An error occurred in the secure channel support" on the same code block. The server is using TLS 1.2, AES with 256 bit encryption; DH with 2048 bit exchange.

    Public Function HTTPRequest(PostURL)

        Set objHTTP = CreateObject("MSXML2.ServerXMLHttp.6.0")

        objHTTP.open "POST", strURLProduction, False

        objHTTP.setRequestHeader "Content-Type", "text/xml;charset-UTF-8"
        objHTTP.setRequestHeader "SOAPAction", """"""

        objHTTP.send objXMLRequest.xml

        HTTPRequest = objHTTP.responseText

        Set objHTTP = Nothing
    End Function

[germanmarky]

I have implemented the workaround provided as the first solution on StackOverflow to use a MSXML2.XMLHttp.6.0 object instead of MSXML2.ServerXMLHttp.6.0. As this was just implemented, I do not know if this will resolve the error but will post back accordingly.

[karelz]

@steveharter can you please try to repro this issue? It will help us triage it. If we don't have repro, it is likely not actionable in 2.0.

[germanmarky]

The MSXML2.XMLHttp.6.0 object did not resolve the issue. I have since created and implemented a different module using the endpoint WSDL and a COM-Visible DLL in the customers production environment, so unable to revert or reproduce.

[karelz]

@steveharter ping?

[steveharter]

I will kick off a test soon and report back tomorrow. A previous mass-run of Http.Functional tests with all ActiveIssue tests disabled was not successful (or trustworthy) due to certain re-enabled tests to cause hangs, thus requiring selective re-enablement of tests going forward.

[steveharter]

Quick update: this did repro on offline Win2008 CI machine at the 36th run so it is likely common. I'll report on overall frequency at 5,000 runs.

  System.Net.Http.Functional.Tests.HttpClientHandler_ServerCertificates_Test.UseCallback_BadCertificate_ExpectedPolicyErrors(url: \"https://self-signed.badssl.com/\", expectedErrors: RemoteCertificateChainErrors) [FAIL]
      System.Net.Http.HttpRequestException : An error occurred while sending the request.
      ---- System.Net.Http.WinHttpException : The buffers supplied to a function was too small
      Stack Trace:
            at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
            at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
            at System.Net.Http.HttpClient.<FinishSendAsyncBuffered>d__58.MoveNext()
         --- End of stack trace from previous location where exception was thrown ---
            at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
            at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
            at System.Net.Http.Functional.Tests.HttpClientHandler_ServerCertificates_Test.<UseCallback_BadCertificate_ExpectedPolicyErrors>d__12.MoveNext()
         --- End of stack trace from previous location where exception was thrown ---
            at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
         --- End of stack trace from previous location where exception was thrown ---
            at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
         --- End of stack trace from previous location where exception was thrown ---
            at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
         ----- Inner Stack Trace -----
            at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
            at System.Threading.Tasks.RendezvousAwaitable`1.GetResult()
            at System.Net.Http.WinHttpHandler.<StartRequest>d__105.MoveNext()

[davidsh]

This exception:

System.Net.Http.WinHttpException : The buffers supplied to a function was too small

represents a real bug. This exception should not be leaking thru. We can get this error from WinHTTP, but it is by-design when we do...and we then handle it internally (adusting buffer sizes and retry).

So, it's important we find out exactly where in the code this error is coming from.

[steveharter]

Update on frequency of error: the CI machine hung on run # 316 but before that there were 4 failures for UseCallback_BadCertificate_ExpectedPolicyErrors, so ~1% failure rate. They all had the same call stack, however two different exception strings.

Two exceptions reported:
System.Net.Http.WinHttpException : The buffers supplied to a function was too small
and the other two:
System.Net.Http.WinHttpException : A security error occurred

[jacksontbryan]

Hello thread. I worked around this issue when we hit it in a real production Windows environment by switching to HTTP from HTTPS. It appeared to happen during the handshake for TLS.

The frequency was very interesting. It would start happening, then consistently happen for several hours for thousands of requests, then stop happening for days.

[AlekseyMartynov]

Seen this exception today in AppVeyor build logs: https://ci.appveyor.com/project/AlekseyMartynov/cc/build/8#L66

[jacksontbryan]

@karelz I don't think this error is being prioritized sufficiently. I just had a developer hit this error with a local development environment today. When hit, it's an all-stop issue for a production environment.

[karelz]

@jacksontbryan thanks for bringing it up - we track it in 2.0 as a must-have, because it hit 4 customers in the wild.
We just didn't get to it yet.

Are you concerned that it might be punted to Future and not addressed in 2.0 at all? (I can 99%-guarantee that it will be addressed)
Or are you concerned that we didn't get to it yet and you need fix in daily builds / 1.x ASAP?

[jacksontbryan]

I'm concerned it will get punted because it didn't look like a development resource had been assigned.

[karelz]

It hasn't been assigned yet (we have it on backlog and it will be eventually grabbed by someone).
As I said above, I can 99% guarantee it won't be punted. Mainly because of the impact on real-world scenarios (4 customer reports).

Unless it requires 6 months of work and rewriting of half of the Networking stack, we should address it in 2.0. If it is so costly that it doesn't fit into 2.0 final release, we would definitely consider addressing it in servicing release. Again, just based on customer impact. Someone should pick it up in next couple of weeks.
I hope it helps.

[KristinXie1]

This issue is repro on Core Tests in build 20170515.01, detail: https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20170515.01/workItem/System.Net.Http.Functional.Tests/analysis/xunit/System.Net.Http.Functional.Tests.HttpClientHandler_ServerCertificates_Test~2FUseCallback_BadCertificate_ExpectedPolicyErrors(url:%20%5C%22https:~2F~2Fwrong.host.badssl.com~2F%5C%22,%20expectedErrors:%20RemoteCertif...

[KristinXie1]

This issue is repro on Portable Core tests, detail: https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fportable~2Fcli~2F/build/20170515.01/workItem/System.Net.Http.Functional.Tests/analysis/xunit/System.Net.Http.Functional.Tests.HttpClientHandler_ServerCertificates_Test~2FUseCallback_BadCertificate_ExpectedPolicyErrors(url:%20%5C%22https:~2F~2Fself-signed.badssl.com~2F%5C%22,%20expectedErrors:%20RemoteCerti...

[stephentoub]

Failed again here:
https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_netcoreapp_windows_nt_release/46/consoleText

[v-haren]

failed again here: https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_netcoreapp_windows_nt_release/46/testReport/System.Net.Http.Functional.Tests/HttpClientHandler_ServerCertificates_Test/UseCallback_BadCertificate_ExpectedPolicyErrors_url____https___wrong_host_badssl_com_____expectedErrors__RemoteCertificateNameMismatch_/

[karelz]

Plan: Enable it in master, get more info from assert, then make a plan for 2.0 fix.

[stephentoub]

Again here:
https://github.com/dotnet/corefx/issues/20078

[karelz]

@schen381 is the info from above failure sufficient for next step?

[jessewhite007]

Has anyone come up with an effective / temporary work around till this gets resolved. It's proving to be very problematic for me.

[karelz]

@jessewhite007 we do not understand the root cause yet, so not sure if we have any idea about potential workarounds ...

[caesar-chen]

@karelz No, I need a dump to analyze the issue, and my PR is not merged yet.

[karelz]

@schen381 PR for what? A fix? Getting more info?
We have last week before Escrow and this is blocking real world scenarios. @DavidGoll are we on track to get this resolved? Is there some way we can help at this moment?

[caesar-chen]

@karelz PR for getting more info by using Debug.Assert()

[DavidGoll]

@karelz the PR is to add a fail-fast so we can get dumps at the point of failure. This will go into master, not the 2.0.0 fork. If we can get usable dumps, then we have a chance at resolving this before Escrow.

[karelz]

Thanks for info!
Do we have ETA for the fail-fast to be checked in? Any reason why we didn't implement it earlier? (looks like it was the plan already last week)

[caesar-chen]

@karelz There are many discussions for this PR: https://github.com/dotnet/corefx/pull/20011, we will have a ETA after offline discussion today.

[karelz]

Where are we with this? I would like to see daily updates.
This is the scariest bug in Networking at this moment.

[DavidGoll]

No test failures since PR of instrumented code was merged.

[karelz]

Can we ask for more runs in lab? @danmosemsft This is impacting real world scenarios.

I guess it would get in even in Escrow (based on the impact, assuming fix is low-risk), but it would be much better to solve it during Ask mode. cc @Petermarcu as it is risk to get into Escrow.

[jessewhite007]

For what it's worth, I was able to go from it showing up very regularly to not showing up at all in my application simply by going to a single instance of the client for the app vs newing one up for each request coming in. Not sure if that helps or not but you might try multi-threaded experiments to get a repro if you haven't already.

[danmosemsft]

@karelz this hit again

https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20170601.02/workItem/System.Net.Http.Functional.Tests/analysis/xunit/System.Net.Http.Functional.Tests.HttpClientHandler_ServerCertificates_Test~2FUseCallback_BadCertificate_ExpectedPolicyErrors(url:%20%5C%22https:~2F~2Fself-signed.badssl.com~2F%5C%22,%20expectedErrors:%20RemoteCerti...

are we expecting to see more diagnostics?

2017-06-01 00:35:47,644: INFO: proc(54): run_and_log_output: Output:    System.Net.Http.Functional.Tests.HttpClientHandler_ServerCertificates_Test.UseCallback_BadCertificate_ExpectedPolicyErrors(url: \"https://self-signed.badssl.com/\", expectedErrors: RemoteCertificateChainErrors) [FAIL]
2017-06-01 00:35:47,644: INFO: proc(54): run_and_log_output: Output:       System.Net.Http.HttpRequestException : An error occurred while sending the request.
2017-06-01 00:35:47,644: INFO: proc(54): run_and_log_output: Output:       ---- System.Net.Http.WinHttpException : The buffers supplied to a function was too small
2017-06-01 00:35:47,660: INFO: proc(54): run_and_log_output: Output:       Stack Trace:
2017-06-01 00:35:47,801: INFO: proc(54): run_and_log_output: Output:             at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
2017-06-01 00:35:47,801: INFO: proc(54): run_and_log_output: Output:             at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
2017-06-01 00:35:47,801: INFO: proc(54): run_and_log_output: Output:             at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
2017-06-01 00:35:47,801: INFO: proc(54): run_and_log_output: Output:             at System.Net.Http.HttpClient.<FinishSendAsyncBuffered>d__58.MoveNext()
2017-06-01 00:35:47,801: INFO: proc(54): run_and_log_output: Output:          --- End of stack trace from previous location where exception was thrown ---
2017-06-01 00:35:47,801: INFO: proc(54): run_and_log_output: Output:             at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
2017-06-01 00:35:47,801: INFO: proc(54): run_and_log_output: Output:             at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
2017-06-01 00:35:47,801: INFO: proc(54): run_and_log_output: Output:             at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
2017-06-01 00:35:47,801: INFO: proc(54): run_and_log_output: Output:             at System.Net.Http.Functional.Tests.HttpClientHandler_ServerCertificates_Test.<UseCallback_BadCertificate_ExpectedPolicyErrors>d__12.MoveNext()

[stephentoub]

are we expecting to see more diagnostics?

I believe all of the added diagnostics were for debug runs; this appears to be release, unfortunately.
cc: @schen381

[danmosemsft]

Yes, it's release. @karelz has someone looped this test in debug, locally?

[KristinXie1]

This issue is repro on build 20170601.01, detail: https://mc.dot.net/#/product/netcore/200/source/official~2Fcorefx~2Frelease~2F2.0.0~2F/type/test~2Ffunctional~2Fcli~2F/build/20170601.01/workItem/System.Net.Http.Functional.Tests/analysis/xunit/System.Net.Http.Functional.Tests.HttpClientHandler_DangerousAcceptAllCertificatesValidator_Test~2FInvalidCertificateServers_CertificateValidationDisabled_Succeeds(url:%20%5C%22https:~2F~2Fexpired.badssl.c...

[johanskoldekrans]

Hard to follow all your will do/won't do. I have this problem in a production environment and it anxious to get a fix. I even get it on the login page which has nothing fancy whatsoever.

[davidsh]

I believe all of the added diagnostics were for debug runs; this appears to be release, unfortunately.
cc: @schen381

We need to turn on this "Debug.Fail" stuff for release builds as well. We need to be able to generate a crash dump. Is there a "Debug.Fail" for release builds? Unfortunately, we haven't seen this repro yet in DEBUG builds since adding the Debug.Asserts, so we should add fast-fail to release builds also.

[danmosemsft]

The release equivalent of Debug.Fail is Trace.Fail. However it doesn't do FailFast by default. I recommend using Environment.FailFast(ex.Message, ex);. It will dump to console and should pop the debugger if registered. I would hope it would create a Watson dump otherwise which dumpling would find.

[steveharter]

I have an offline CI machine with F5 repro if anyone wants it; PM me.

[davidsh]

I was able to get a repro with either ERROR_WINHTTP_SECURE_FAILURE (12175) or SEC_E_BUFFER_TOO_SMALL (2148074273u) errors. This was on a WIndows Server 2008 machine.

We have collected "netsh trace InternetClient" traces and are sending them to the WinHttp team for analysis.

[davidsh]

We have completed the root cause analysis of this problem. The problem is being caused by a Windows OS bug in SCHANNEL which is the low-level component that handles SSL/TLS handshakes.. This bug has been fixed in the very latest Windows 10 and Windows Server 2016 releases.

This bug can manifest in not just .NET Core HttpClient/WinHttpHandler but also with any code that uses SSL/TLS on Windows such as SslStream, .NET Framework, Internet Explorer.

There is a bug in how TLSv1.2 handshakes are done when the client and server negotiate a DHE based cipher suite. And more specifically, only when a Public Key Length of 127 is negotiated. Key lengths can vary and lengths such as 128 are ok. This is why this problem appears "random" and hard to repro.

The bug is hard to diagnose because the errors from SCHANNEL in Windows get obscured and instead other errors are returned by components such as WinHttp.

See this article for an explanation of the root cause when this bug manifests as a connection error with Internet Explorer:
https://connect.microsoft.com/IE/feedback/details/1253526/tls-serverkeyexchange-with-1024-dhe-may-encode-dh-y-as-127-bytes-breaking-internet-explorer-11

More detailed background:
https://security.stackexchange.com/questions/104845/dhe-rsa-pubkey-length-in-tls-1-2

There are no simple workarounds for this except for the following:

• Upgrade client machines to latest Windows 10 / Windows Server 2016
• Disable DHE* cipher suites on the Windows client or server
  Supported methods include GroupPolicy and powershell cmdlets (the latter is available on Win10).
  Via GroupPolicy: https://msdn.microsoft.com/en-us/library/windows/desktop/bb870930(v=vs.85).aspx
  Via cmdlets: https://technet.microsoft.com/en-us/library/dn296632.aspx
  Via direct registry changes: https://support.microsoft.com/en-us/help/245030/how-to-restrict-the-use-of-certain-cryptographic-algorithms-and-protocols-in-schannel.dll

We will be looking at ways to mitigate the random test failure noise caused by this bug in the CI system by adjusting our tests and test machine configurations.

[KristinXie1]

Failed again on Core tests, detail: https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20170605.02/workItem/System.Net.Http.Functional.Tests/analysis/xunit/System.Net.Http.Functional.Tests.HttpClientHandler_ServerCertificates_Test~2FUseCallback_BadCertificate_ExpectedPolicyErrors(url:%20%5C%22https:~2F~2Fself-signed.badssl.com~2F%5C%22,%20expectedErrors:%20RemoteCerti...

[justdmitry]

My five cents.

I have ASP.NET Core web app (Microsoft.AspNetCore.Mvc 1.1.3) deployed to Azure (as web app). I had about 10-20 errors per day. On May 29 I added project reference to System.Net.Http v4.3.2 (it was v4.3.0 as indirect reference by one of my packages). No more errors for about 10 days.

Today 2hr ago I removed direct reference (and package was downgraded back to 4.3.0) - and I already got one error.

I don't know what Windows versions are running in Azure Web App, but it's low possible that Windows had been downgraded same time I pushed new website version (I use automatic deployment from BitBbucket). So, it looks that System.Net.Http version has some influence on this error appearance.

And one more thing. My app is not targeting netcoreapp - it's targeting net461.

Update after 19 hours: got 9 exceptions in less then 24h. Upgrading back to System.Net.Http 4.3.2 - this obviously helps for my app.

[v-haren]

failed again here: https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_netcoreapp_windows_nt_debug/75/testReport/System.Net.Http.Functional.Tests/HttpClientHandler_DangerousAcceptAllCertificatesValidator_Test/InvalidCertificateServers_CertificateValidationDisabled_Succeeds_url____https___expired_badssl_com____/

[filipw]

We have completed the root cause analysis of this problem. The problem is being caused by a Windows OS bug in SCHANNEL which is the low-level component that handles SSL/TLS handshakes.. This bug has been fixed in the very latest Windows 10 and Windows Server 2016 releases.

does anyone know when this will be applied to Azure Web Apps?
@davidsh do you know if there is a specific Windows 10 / Windows Server 2016 build number that we could relate the presence of the fix to?

I have the issue popping up in production on a .NET Core 1.0.4 app running on Azure Web App even today.

[davidsh]

does anyone know when this will be applied to Azure Web Apps?
@davidsh do you know if there is a specific Windows 10 / Windows Server 2016 build number that we could relate the presence of the fix to?

In terms of which Windows 10 / Windows Server 2016 build numbers have the fix:

This was fixed in 10.0.14393. That means it is fixed in Windows Server 2016 since the RTM release of that server SKU was build 14393. For Windows 10, it means "Windows 10 Version 1607 (Anniversary Update)" which is build 14393. See: https://en.wikipedia.org/wiki/Windows_10_version_history

Azure Web Apps (formally known as Azure WebSites), currently runs on Windows Server 2012. I have heard that they are actively working on upgrading their systems to Windows Server 2016. Many developers are asking for this. And upgrading to Windows Server 2016 also means that Azure Web Apps hosted on that will support HTTP/2.0 as well (since Windows Server 2016 is the first server SKU to support that). Please contact Azure support for more information on their timelines.

[filipw]

@davidsh thanks David, much appreciated! 👍

[peter-bannerflow]

I also have this issue in production when I make calls over HTTPS (using TLS1.2) to an external API. My application using .NET Core 1.1 and is deployed as a web app in Azure. At most times, I see the A security error occurred-error, but I have also seen the The buffers supplied to a function was too small-error.

Update:
I tried updating to System.Net.Http v4.3.2 as @justdmitry suggested but the error still occur.

[justdmitry]

@peter-bannerflow if possible - try to switch framework from netcoreapp1.1 to net46/net461 and check if error disappear. I can't switch my app to netcore due to dependencies, but reverse switch should be ok. Let's check does this bug depend on framework :)

[davidsh]

@peter-bannerflow if possible - try to switch framework from netcoreapp1.1 to net46/net461 and check if error disappear. I can't switch my app to netcore due to dependencies, but reverse switch should be ok. Let's check does this bug depend on framework :)

As I described above, this issue is not related to .NET Core nor .NET Framework. It is a Windows OS bug in low level SCHANNEL and deals with specific cipher suites used in TLS/SSL handshakes.

The only fix available is to upgrade the Windows OS to Windows Server 2016 or the latest Windows 10 OS.

[peter-bannerflow]

@davidsh

There is a bug in how TLSv1.2 handshakes are done when the client and server negotiate a DHE based cipher suite. And more specifically, only when a Public Key Length of 127 is negotiated

I captured traffic from my application during the time of the error and I cannot find any negotiation where a 127 bit public key is used. Could the error occur for other key lengths as well?

[davidsh]

Please refer to this:
https://security.stackexchange.com/questions/104845/dhe-rsa-pubkey-length-in-tls-1-2

There were some additional bugs in the TLS/SSL handshake on Windows dealing with DHE cipher suites that did not deal with 127 bit length. It's possible you are seeing a problem due to that.

If you see this problem on Windows Server 2016 or Windows 10 (build 14393 or later), then we would like to hear about that since the problem regarding DHE handshakes was fixed in those builds.

[NikolaosWakem]

"The buffers supplied to a function was too small" - I'm getting this in 1.3% of calls on Azure Production WebAPI .NETCoreApp 1.1 - After I year - I think we need a fix :-D

[davidsh]

"The buffers supplied to a function was too small" - I'm getting this many times / day on Azure Production WebAPI .NETCoreApp 1.1 - After I year - I think we need a fix :-D

As discussed above, this is not a bug in .NET layer. It is a bug in low-level Windows OS.

The fix is to upgrade client machines for Windows 10 Version 1607 or Windows Server 2016. Also, server machines can be upgraded as well.

[NikolaosWakem]

Can I specify Windows Server 2016 for Azure Web API's somehow?

[davidsh]

Can I specify Windows Server 2016 for Azure Web API's somehow?

I don't know. You might want to follow up with Azure support.

I do know that many customers have been asking for Azure Web Apps to use Windows Server 2016 as the backend. It currently uses Windows Server 2012. Customers would like Server 2016 for Web Apps since that version of Windows OS has HTTP/2 support for example.

[davidsh]

See: https://social.msdn.microsoft.com/Forums/azure/en-US/5b584d52-7907-4058-9324-932a2215ce0d/server-2016-on-app-service?forum=windowsazurewebsitespreview

[NikolaosWakem]

thanks.. have added comments to that and https://feedback.azure.com/forums/169385-web-apps/suggestions/9552936-enable-http-2-on-azure-web-apps .. fingers crossed :-D

[ChristophAlbert]

Is there a way to get more information about the circumstances under which these SChannel bugs occur and in which version they were fixed? Something like a change log?
I see the described errors on Windows 7 and 2012, but so far not on Windows 10 - even with build numbers smaller than 1607. I can also rule out that the observed errors are caused by public key length 127.

[davidsh]

Is there a way to get more information about the circumstances under which these SChannel bugs occur and in which version they were fixed?

That information is already in this issue discussion by my comments above:

In terms of which Windows 10 / Windows Server 2016 build numbers have the fix:
This was fixed in 10.0.14393. That means it is fixed in Windows Server 2016 since the RTM release of that server SKU was build 14393. For Windows 10, it means "Windows 10 Version 1607 (Anniversary Update)" which is build 14393. See: https://en.wikipedia.org/wiki/Windows_10_version_history

You can also workaround the problem by disabling DHE cipher suites on the client or server machines. See discussion above.

[ChristophAlbert]

Well, I was actually hoping for something a little more detailed than "There were some additional bugs in the TLS/SSL handshake on Windows dealing with DHE cipher suites that did not deal with 127 bit length." - it's kind of difficult to judge whether an "additional bug" is afflicting my application ;) - or if there is another problem in the infrastructure or client implementation.
Also, according to the discussion above (and my experience), all that's clear is that there are SChannel bugs in Windows 7, and no (known) SChannel bugs in Win10 1607, but it seems unlikely that there were absolutely no fixes in between - especially since I already can't reproduce the error with Win10 1511.
Unfortunately, I need to talk to servers which accept nothing but DH, and I don't have control over them.

So, there's no detailed change log?

[davidsh]

So, there's no detailed change log?

There was no detailed change log for this. If you can't control which servers you talk with, and you can't disable the DHE ciphers on the server, then you'll need to update the Windows OS on the client.

[KristinXie1]

release/2.0.0 failure

failed again on Core Tests (2.0.0), detail: https://mc.dot.net/#/product/netcore/200/source/official~2Fcorefx~2Frelease~2F2.0.0~2F/type/test~2Ffunctional~2Fcli~2F/build/20170816.01/workItem/System.Net.Http.Functional.Tests/analysis/xunit/System.Net.Http.Functional.Tests.HttpClientHandler_DangerousAcceptAllCertificatesValidator_Test~2FInvalidCertificateServers_CertificateValidationDisabled_Succeeds(url:%20%5C%22https:~2F~2Fwrong.host.badss...

[KristinXie1]

release/2.0.0 failure

https://mc.dot.net/#/product/netcore/200/source/official~2Fcorefx~2Frelease~2F2.0.0~2F/type/test~2Ffunctional~2Fcli~2F/build/20170927.01/workItem/System.Net.Http.Functional.Tests/analysis/xunit/System.Net.Http.Functional.Tests.HttpClientHandler_DangerousAcceptAllCertificatesValidator_Test~2FInvalidCertificateServers_CertificateValidationDisabled_Succeeds(url:%20%5C%22https:~2F~2Fwrong.host.badss...

The test System.Net.Http.Functional.Tests.HttpClientHandler_DangerousAcceptAllCertificatesValidator_Test/InvalidCertificateServers_CertificateValidationDisabled_Succeeds(url: \"https://wrong.host.badss... has failed.

System.Net.Http.HttpRequestException : An error occurred while sending the request.\r
    ---- System.Net.Http.WinHttpException : The buffers supplied to a function was too small

        Stack Trace:

           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at System.Net.Http.HttpClient.<FinishSendAsyncBuffered>d__58.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at System.Net.Http.Functional.Tests.HttpClientHandler_DangerousAcceptAllCertificatesValidator_Test.<InvalidCertificateServers_CertificateValidationDisabled_Succeeds>d__3.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    ----- Inner Stack Trace -----
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Threading.Tasks.RendezvousAwaitable`1.GetResult()
       at System.Net.Http.WinHttpHandler.<StartRequest>d__105.MoveNext()

Build : 2.0.0 - 20170927.01 (Core Tests)
Failing configurations:

• Windows.81.Amd64-x86
  
  
  
  • Release
  
  

[karelz]

Triage: We need to disable the test only on older OS's (Win7, Win8.1, Win10 before build - see above https://github.com/dotnet/corefx/issues/7812#issuecomment-313154522).
@pjanotti can you please help with this one?

[karelz]

@pjanotti there is still one test disabled against this bug:
https://github.com/dotnet/corefx/blob/0fd3d2c5bcb1a83e7ebd8025e4810e11e231ce03/src/System.Net.Security/tests/FunctionalTests/SslStreamSystemDefaultsTest.cs#L34

Is that on purpose? Can it be re-enabled? Or do we need to track more work? (Maybe there was mix up of multiple problem here?)

[pjanotti]

Somehow I missed that one... re-opening...

[pjanotti]

/cc @Priya91 @wfurt @davidsh (area owners)
When trying to re-enable https://github.com/dotnet/corefx/blob/0fd3d2c5bcb1a83e7ebd8025e4810e11e231ce03/src/System.Net.Security/tests/FunctionalTests/SslStreamSystemDefaultsTest.cs#L46

The test fails when
```c#
[InlineData(null, SslProtocols.Tls11)]
[InlineData(SslProtocols.Tls11, null)]

Since `_clientStream.HashAlgorithm` becomes Sha1 which is not expected by:

```c#
            if (PlatformDetection.IsWindows && PlatformDetection.WindowsVersion >= 10)
            {
                Assert.True(_clientStream.HashAlgorithm == HashAlgorithmType.Sha256 ||
                            _clientStream.HashAlgorithm == HashAlgorithmType.Sha384 ||
                            _clientStream.HashAlgorithm == HashAlgorithmType.Sha512);
            }

Is this a product bug? The test implies that we shouldn't be going back to Sha1.

[Priya91]

@bartonjs Do you know if the above test is valid, that on Windows 10, the hashalgorithm cannot be Sha1?

[bartonjs]

That claim probably holds when the protocol is 1.2; but 1.1 didn't have any SHA256/SHA384/SHA512 options.

If you look at all of the entries marked as "TLS 1.1" on https://msdn.microsoft.com/en-us/library/windows/desktop/mt808163(v=vs.85).aspx you'll see that they just say "SHA" (except for the couple that say "MD5").

[pvasek]

@davidsh Is there any KB for that issue? I am curious about updates for older systems.

[davidsh]

This issue was fixed in Windows 10 releases:

In terms of which Windows 10 / Windows Server 2016 build numbers have the fix:
This was fixed in 10.0.14393. That means it is fixed in Windows Server 2016 since the RTM release of that server SKU was build 14393. For Windows 10, it means "Windows 10 Version 1607 (Anniversary Update)" which is build 14393. See: https://en.wikipedia.org/wiki/Windows_10_version_history

I am not aware if there are KB articles directly addressing this.

[nickjags]

We had issues related to our https .NET application (F5 BIGIP Load balanced) where it used to break intermittently for 1 hour window. and then started working again by itself after an hour. The server event logs showed "Schannel Error - The TLS protocol defined fatal alert code is 40, while the application logs showed the error "HttpRequestException: The buffers supplied to a function was too small ". The issue was due to what is mentioned in the BIGIP F5 article https://support.f5.com/csp/article/K40424522 and was resolved when we disabled the DHE/EDH cypher on the impacted application VIP.

[MgSam]

We're seeing this today on a production server Windows Server 2012 R2, .NET Core 3.1.

My understanding based on reading this thread was that it was supposed to be have been fixed in .NET Core 2.1?

Is the SSL workaround still the best way to handle this?