Runtime: HttpStress app generating query strings explicitly rejected by Kestrel default limits

Created on 16 Jul 2019 · 19Comments · Source: dotnet/runtime

```C#
C:\git\corefx\src\System.Net.Httptests\StressTests\HttpStress>dotnet run -n 10 -maxContentLength 4000 -http 2.0 -ops 3 -numParameters 4
.NET Core: 3.0.0-preview7-27902-19
ASP.NET Core: 3.0.0-preview7.19353.9
Tracing: False
ASP.NET Log: False
Concurrency: 10
Content Length: 4000
HTTP Versions: 2.0
Lifetime: (infinite)
Operations: GET Parameters
Random Seed: 1327321578
Query Parameters: 4

All the inputs are valid uris as otherwise the system throws uri is too long exception.

Starting 10 client workers.
Error from iteration 21 (GET Parameters) in task 4 with 150 successes / 8 fails:
System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.IO.IOException: The request was aborted.
---> System.Net.Http.Http2ProtocolException: The HTTP/2 request failed with protocol error 'COMPRESSION_ERROR' (0x9).
--- End of inner exception stack trace ---
at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders()
at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken)
at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Program.<>c__DisplayClass1_0.<b__5>d.MoveNext() in C:\git\corefx\src\System.Net.Httptests\StressTests\HttpStress\Program.cs:line 175
--- End of stack trace from previous location where exception was thrown ---
at Program.<>c__DisplayClass1_4.<b__43>d.MoveNext() in C:\git\corefx\src\System.Net.Httptests\StressTests\HttpStress\Program.cs:line 634

Another exception thrown was

Error from iteration 19 (GET Parameters) in task 3 with 363 successes / 16 fails:
System.ArgumentException: Destination is too short. (Parameter 'destination')
at System.ReadOnlyMemory1.CopyTo(Memory1 destination)
at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush)
at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Program.<>c__DisplayClass1_0.<b__5>d.MoveNext() in C:\git\corefx\src\System.Net.Httptests\StressTests\HttpStress\Program.cs:line 175
--- End of stack trace from previous location where exception was thrown ---
at Program.<>c__DisplayClass1_4.<b__43>d.MoveNext() in C:\git\corefx\src\System.Net.Httptests\StressTests\HttpStress\Program.cs:line 634

On http1.1 sometimes the assertion fails  but it never throws a httpException. The returned value in these cases is an empty string

at Program.g__ValidateContent|1_1(String expectedContent, String actualContent) in C:\git\corefx\src\System.Net.Httptests\StressTests\HttpStress\Program.cs:line 122
at Program.<>c__DisplayClass1_0.<b__5>d.MoveNext() in C:\git\corefx\src\System.Net.Httptests\StressTests\HttpStress\Program.cs:line 178
--- End of stack trace from previous location where exception was thrown ---
at Program.<>c__DisplayClass1_4.<b__43>d.MoveNext() in C:\git\corefx\src\System.Net.Httptests\StressTests\HttpStress\Program.cs:line 634

[15-07-2019 15:17:17] Total: 30,817
GET Parameters Success: 30,395 Fail: 427

Error from iteration 2894 (GET Parameters) in task 6 with 30398 successes / 427 fails:
System.Exception: Expected response content "123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789090123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234", got ""
at Program.g__ValidateContent|1_1(String expectedContent, String actualContent) in C:\git\corefx\src\System.Net.Httptests\StressTests\HttpStress\Program.cs:line 122
at Program.<>c__DisplayClass1_0.<b__5>d.MoveNext() in C:\git\corefx\src\System.Net.Httptests\StressTests\HttpStress\Program.cs:line 178
--- End of stack trace from previous location where exception was thrown ---
at Program.<>c__DisplayClass1_4.<b__43>d.MoveNext() in C:\git\corefx\src\System.Net.Httptests\StressTests\HttpStress\Program.cs:line 634
```

cc @stephentoub

Source

Anipik

Most helpful comment

Just to confirm, changing the app to set both ko.Limits.Http2.MaxRequestHeaderFieldSize and ko.Limits.MaxRequestLineSize to a much larger value makes the errors go away. So this is a bug in the stress app rather than in HttpClient.

stephentoub on 16 Jul 2019

👍2 😄1

All 19 comments

Input command dotnet run -n 10 -maxContentLength 7000 -http 2.0 -ops 3 -numParameters 1

Another Different exception being thrown

Error from iteration 2556 (GET Parameters) in task 9 with 25486 successes / 10 fails:
System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: The request was aborted.
 ---> System.Net.Http.Http2ProtocolException: The HTTP/2 request failed with protocol error 'NO_ERROR' (0x0).
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders()
   at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Program.<>c__DisplayClass1_0.<<Run>b__5>d.MoveNext() in C:\git\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 175
--- End of stack trace from previous location where exception was thrown ---
   at Program.<>c__DisplayClass1_4.<<Run>b__43>d.MoveNext() in C:\git\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 634

Anipik on 16 Jul 2019

I am not sure if there is any length constraint on the size of the get query parameter but the test starts failing on 2.0 after 8100 characters and starts returning an empty string on 1.1

Anipik on 16 Jul 2019

Another case of Destination is too short

```c#
C:\git\corefx\src\System.Net.Httptests\StressTests\HttpStress>dotnet run -n 70 -maxContentLength 400 -http 2.0 -ops 3 -numParameters 35
.NET Core: 3.0.0-preview7-27902-19
ASP.NET Core: 3.0.0-preview7.19353.9
Tracing: False
ASP.NET Log: False
Concurrency: 70
Content Length: 400
HTTP Versions: 2.0
Lifetime: (infinite)
Operations: GET Parameters
Random Seed: 1403076020
Query Parameters: 35
Client Max Buffer Size: 10
Server Buffer Size: 5

Starting server.
Starting 70 client workers.
Error from iteration 6 (GET Parameters) in task 5 with 71 successes / 1 fails:
System.ArgumentException: Destination is too short. (Parameter 'destination')
at System.ReadOnlyMemory1.CopyTo(Memory1 destination)
at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush)
at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Program.<>c__DisplayClass1_0.<b__5>d.MoveNext() in C:\git\corefx\src\System.Net.Httptests\StressTests\HttpStress\Program.cs:line 175
--- End of stack trace from previous location where exception was thrown ---
at Program.<>c__DisplayClass1_4.<b__43>d.MoveNext() in C:\git\corefx\src\System.Net.Httptests\StressTests\HttpStress\Program.cs:line 634
```

Anipik on 16 Jul 2019

Another different exception dotnet run -n 10 -maxContentLength 400 -http 2.0 -ops 3 -numParameters 100

``Error from iteration 16 (GET Parameters) in task 9 with 15 successes / 12 fails: System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.. ---> System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine. --- End of inner exception stack trace --- at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask1 task, Int32 min, Int32 initial)
at System.Net.Security.SslStream.ReadAsyncInternalTReadAdapter
at System.Net.Http.Http2Connection.EnsureIncomingBytesAsync(Int32 minReadBytes)
at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame)
at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()
--- End of inner exception stack trace ---
at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Program.<>c__DisplayClass1_0.<b__5>d.MoveNext() in C:\git\corefx\src\System.Net.Httptests\StressTests\HttpStress\Program.cs:line 175
--- End of stack trace from previous location where exception was thrown ---
at Program.<>c__DisplayClass1_4.<b__43>d.MoveNext() in C:\git\corefx\src\System.Net.Httptests\StressTests\HttpStress\Program.cs:line 634

General excpetion thrown is this

Error from iteration 11 (GET Parameters) in task 8 with 7 successes / 10 fails:
System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.IO.IOException: The request was aborted.
---> System.Net.Http.Http2ProtocolException: The HTTP/2 request failed with protocol error 'COMPRESSION_ERROR' (0x9).
--- End of inner exception stack trace ---
at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders()
at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken)
at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Program.<>c__DisplayClass1_0.<b__5>d.MoveNext() in C:\git\corefx\src\System.Net.Httptests\StressTests\HttpStress\Program.cs:line 175
--- End of stack trace from previous location where exception was thrown ---
at Program.<>c__DisplayClass1_4.<b__43>d.MoveNext() in C:\git\corefx\src\System.Net.Httptests\StressTests\HttpStress\Program.cs:line 634

[15-07-2019 15:42:20] Total: 21
GET Parameters Success: 11 Fail: 10
```

Anipik on 16 Jul 2019

@anipik, the HttpStress app runs with whatever SDK you have installed, not what's in the repo, and the output shows you're running with 3.0.0-preview7-27902-19. That's not particularly recent. Can you please uninstall your SDK and install a new one from https://dotnetcli.blob.core.windows.net/dotnet/Sdk/master/dotnet-sdk-latest-win-x64.exe. At least some of the call stacks you're sharing look like things already fixed.

stephentoub on 16 Jul 2019

👍1

I updated the sdk and the call stack changed to

Error from iteration 4414 (GET Parameters) in task 2 with 41820 successes / 988 fails:
System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: The response ended prematurely, with at least 9 additional bytes expected.
   at System.Net.Http.Http2Connection.ReadAtLeastAsync(Stream stream, Memory`1 buffer, Int32 minReadBytes)
   at System.Net.Http.Http2Connection.EnsureIncomingBytesAsync(Int32 minReadBytes)
   at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame)
   at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Program.<>c__DisplayClass1_0.<<Run>b__5>d.MoveNext() in C:\git\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 175
--- End of stack trace from previous location where exception was thrown ---
   at Program.<>c__DisplayClass1_4.<<Run>b__43>d.MoveNext() in C:\git\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 634

Error from iteration 3928 (GET Parameters) in task 7 with 41779 successes / 988 fails:
System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
   at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
   at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
   at System.Net.Http.Http2Connection.ReadAtLeastAsync(Stream stream, Memory`1 buffer, Int32 minReadBytes)
   at System.Net.Http.Http2Connection.EnsureIncomingBytesAsync(Int32 minReadBytes)
   at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame)
   at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Program.<>c__DisplayClass1_0.<<Run>b__5>d.MoveNext() in C:\git\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 175
--- End of stack trace from previous location where exception was thrown ---
   at Program.<>c__DisplayClass1_4.<<Run>b__43>d.MoveNext() in C:\git\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 634

Anipik on 16 Jul 2019

Have you tried adding -aspnetlog to see if these failures end up being correlated with failures on the server side? If the server is abruptly closing a connection due to some exception it hits due to a bug, that's not something we can do much about on the client, and we'd expect an error like this. If we're sending it bad data and that's why it closes the connection, that's more interesting.

stephentoub on 16 Jul 2019

I'm not clear on what this test is actually doing. You say "passing long get parameters"; what does this mean? How long?

If the resulting Uri is really big -- like around 16K or more -- we may be getting into the path where we need to send a CONTINUATION frame, and possibly we have an issue here since this is rare. Or possibly ASP.NET has an issue here.

geoffkizer on 16 Jul 2019

Would be interesting to run this against IIS (or something else) to see if we can repro without ASP.NET. I'm guessing this is easy to do, assuming it's a GET with a fixed path.

geoffkizer on 16 Jul 2019

Example url:

https://localhost:5001/variables?Var0=sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss

This is Var0= followed by 8155 characters. 8154 works, 8155 results in failure.

stephentoub on 16 Jul 2019

👍1

You say "passing long get parameters"; what does this mean? How long?

So the error gets reproduced when the uri size is greater than 8100 chars. It could be done with just one parameter or with multiple params

Anipik on 16 Jul 2019

My guess is we're hitting this:
https://github.com/aspnet/AspNetCore/blob/7b56439ca1cbc384f1e9648f4ae983d3b39119d8/src/Servers/Kestrel/Core/src/KestrelServerLimits.cs#L21
and it's expected, and the PROTOCOL_ERROR that comes back is expected. Then those other errors are just the result of requests getting killed and not being retried (yet).

stephentoub on 16 Jul 2019

just a note http 1.1 just returns an empty string

Anipik on 16 Jul 2019

Based on what I wrote above (that url length maps exactly to the kestrel limit I linked to), I don't think there's a client bug here. I think it's dutifully sending what was requested, and kestrel has explicit limits set that block it.

I just spoke with @jkotalik, and he found https://github.com/aspnet/AspNetCore/blob/81b757afcc15c02fc89f35f47cc96a148b09648f/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Stream.cs#L208.

So I suggest the "fix" here is to fix the stress app to work within the confines of what the server allows, or to eat this exception when it occurs.

stephentoub on 16 Jul 2019

👍1

stephentoub on 16 Jul 2019

👍2 😄1

(I'm marking this as 3.0 as it needs to be fixed asap in order to not hold up using the stress app in general to find other issues.)

stephentoub on 16 Jul 2019

i will try to put up a pr for this tomorrow :)

Anipik on 16 Jul 2019

👍1

Going to fix this now to unblock the app.