Runtime: Stress issues with HttpClient, Kestrel, and HTTP/2

[stephentoub]

(I'm opening one issue right now because I don't have individual repros for each of the following issues, and it’s possible one or more are related. We can decide if/when to break out individual issues.)

I wrote a simple stress program to exercise many concurrent requests from HttpClient against Kestrel, specifically for HTTP/2. In doing so, I encountered a variety of failures in a very short period of time (seconds). I've not investigated any of these. Some are obviously issues with SocketsHttpHandler, but some could be on either end of the connection (and a few are likely to be Kestrel). The below is based on the most recent daily build as of now:

   .NET Core: 3.0.0-preview7-27825-06
ASP.NET Core: 3.0.0-preview7.19325.3

These were encountered by varying some of the constants (e.g. ConcurrentRequests, AppxContentLength) as well as by commenting and uncommenting out which client operations were available to be performed in a given run.

Error 1:

System.IO.IOException: The request was aborted.
 ---> System.Net.Http.Http2ProtocolException: The HTTP/2 request failed with protocol error 'FRAME_SIZE_ERROR' (0x6).
   at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame)
   at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span`1 buffer)
   at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, CancellationToken cancellationToken)
   at System.IO.Stream.CopyToAsyncInternal(Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.GetStringAsyncCore(Task`1 getTask)
   at Program.<>c__DisplayClass0_2.<<Main>b__17>d.MoveNext() in C:\Users\stoub\source\repos\HTTP2Stress\HTTP2Stress\Program.cs:line 173```

Error 2:

System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: The request was aborted.
 ---> System.Net.Http.Http2ProtocolException: The HTTP/2 request failed with protocol error 'NO_ERROR' (0x0).
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders()
   at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync()
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at System.Net.Http.HttpClient.GetStringAsyncCore(Task`1 getTask)
   at Program.<>c__DisplayClass0_2.<<Main>b__13>d.MoveNext() in C:\Users\stoub\source\repos\HTTP2Stress\HTTP2Stress\Program.cs:line 177

Error 3:

System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Net.Http.RedirectHandler.GetUriForRedirect(Uri requestUri, HttpResponseMessage response)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Program.<>c__DisplayClass0_0.<<Main>b__1>d.MoveNext() in C:\Users\stoub\source\repos\HTTP2Stress\HTTP2Stress\Program.cs:line 100
--- End of stack trace from previous location where exception was thrown ---
   at Program.<>c__DisplayClass0_2.<<Main>b__13>d.MoveNext() in C:\Users\stoub\source\repos\HTTP2Stress\HTTP2Stress\Program.cs:line 173

Error 4:

System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Http.Http2ProtocolException: The HTTP/2 request failed with protocol error 'PROTOCOL_ERROR' (The HTTP/2 request encountered a protocol error, receiving 'data' while in the state 'ExpectingStatus'.).
   at System.Net.Http.Http2Connection.Http2Stream.OnResponseData(ReadOnlySpan`1 buffer, Boolean endStream)
   at System.Net.Http.Http2Connection.ProcessDataFrame(FrameHeader frameHeader)
   at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Program.<>c__DisplayClass0_0.<<Main>b__1>d.MoveNext() in C:\Users\stoub\source\repos\HTTP2Stress\HTTP2Stress\Program.cs:line 65
--- End of stack trace from previous location where exception was thrown ---
   at Program.<>c__DisplayClass0_2.<<Main>b__13>d.MoveNext() in C:\Users\stoub\source\repos\HTTP2Stress\HTTP2Stress\Program.cs:line 169

Error 5:

System.InvalidOperationException: The stream was already consumed. It cannot be read again.
   at System.Net.Http.HttpConnectionResponseContent.ConsumeStream()
   at System.Net.Http.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Program.<>c__DisplayClass0_0.<<Main>b__1>d.MoveNext() in C:\Users\stoub\source\repos\HTTP2Stress\HTTP2Stress\Program.cs:line 50
--- End of stack trace from previous location where exception was thrown ---
   at Program.<>c__DisplayClass0_2.<<Main>b__13>d.MoveNext() in C:\Users\stoub\source\repos\HTTP2Stress\HTTP2Stress\Program.cs:line 154

Error 6:

System.Exception: Unexpected response.  Got: 1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456
   at Program.<>c__DisplayClass0_2.<<Main>b__13>d.MoveNext() in C:\Users\stoub\source\repos\HTTP2Stress\HTTP2Stress\Program.cs:line 169

Error 7:

System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'CreditManager'.
   at System.Net.Http.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush)
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Program.<>c__DisplayClass0_0.<<Main>b__5>d.MoveNext() in C:\Users\stoub\source\repos\HTTP2Stress\HTTP2Stress\Program.cs:line 100
--- End of stack trace from previous location where exception was thrown ---
   at Program.<>c__DisplayClass0_2.<<Main>b__17>d.MoveNext() in C:\Users\stoub\source\repos\HTTP2Stress\HTTP2Stress\Program.cs:line 173

Error 8:

System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: The request was aborted.
 ---> System.Net.Http.Http2ProtocolException: The HTTP/2 request failed with protocol error 'STREAM_CLOSED' (0x5).
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders()
   at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync()
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Program.<>c__DisplayClass0_0.<<Main>b__1>d.MoveNext() in C:\Users\stoub\source\repos\HTTP2Stress\HTTP2Stress\Program.cs:line 105
--- End of stack trace from previous location where exception was thrown ---
   at Program.<>c__DisplayClass0_2.<<Main>b__13>d.MoveNext() in C:\Users\stoub\source\repos\HTTP2Stress\HTTP2Stress\Program.cs:line 177

Error 9:

System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: The response ended prematurely, with at least 213 additional bytes expected.
   at System.Net.Http.Http2Connection.ReadAtLeastAsync(Stream stream, Memory`1 buffer, Int32 minReadBytes)
   at System.Net.Http.Http2Connection.EnsureIncomingBytesAsync(Int32 minReadBytes)
   at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame)
   at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Program.<>c__DisplayClass0_0.<<Main>b__1>d.MoveNext() in C:\Users\stoub\source\repos\HTTP2Stress\HTTP2Stress\Program.cs:line 105
--- End of stack trace from previous location where exception was thrown ---
   at Program.<>c__DisplayClass0_2.<<Main>b__13>d.MoveNext() in C:\Users\stoub\source\repos\HTTP2Stress\HTTP2Stress\Program.cs:line 177

Error 10:

System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Http.Http2ProtocolException: The HTTP/2 request failed with protocol error 'PROTOCOL_ERROR' (Received an HTTP/2 pseudo-header as a trailing header.).
   at System.Net.Http.Http2Connection.Http2Stream.OnResponseHeader(ReadOnlySpan`1 name, ReadOnlySpan`1 value)
   at System.Net.Http.Http2Connection.<>c.<.cctor>b__86_0(Object state, ReadOnlySpan`1 name, ReadOnlySpan`1 value)
   at System.Net.Http.HPack.HPackDecoder.OnIndexedHeaderField(Int32 index, HeaderCallback onHeader, Object onHeaderState)
   at System.Net.Http.HPack.HPackDecoder.Decode(ReadOnlySpan`1 data, Boolean endHeaders, HeaderCallback onHeader, Object onHeaderState)
   at System.Net.Http.Http2Connection.ProcessHeadersFrame(FrameHeader frameHeader)
   at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Program.<>c__DisplayClass0_0.<<Main>b__1>d.MoveNext() in C:\Users\stoub\source\repos\HTTP2Stress\HTTP2Stress\Program.cs:line 100
--- End of stack trace from previous location where exception was thrown ---
   at Program.<>c__DisplayClass0_2.<<Main>b__13>d.MoveNext() in C:\Users\stoub\source\repos\HTTP2Stress\HTTP2Stress\Program.cs:line 173

Code:
```C#
using Microsoft.AspNetCore;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using System;
using System.IO;
using System.Linq;
using System.Net.Http;
using System.Net.Security;
using System.Runtime.InteropServices;
using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;
using System.Threading;
using System.Threading.Tasks;

public class Program
{
public static void Main(string[] args)
{
Console.WriteLine(" .NET Core: " + Path.GetFileName(Path.GetDirectoryName(typeof(object).Assembly.Location)));
Console.WriteLine("ASP.NET Core: " + Path.GetFileName(Path.GetDirectoryName(typeof(WebHost).Assembly.Location)));

    const int ConcurrentRequests = 10; // 90;
    const int HttpsPort = 5001;
    const int AppxContentLength = 10_000; // 300_000
    const int PrintEveryNRequests = 1000;

    var httpVersion = new Version(2, 0);
    var serverUri = new Uri("https://localhost:" + HttpsPort);
    string content = string.Concat(Enumerable.Repeat("1234567890", AppxContentLength / 10));

    void ValidateResponseVersion(HttpResponseMessage m)
    {
        if (m.Version != httpVersion)
        {
            throw new Exception("Unexpected response version: " + m.Version);
        }
    }

    var clientOperations = new (string, Func<HttpClient, Task<string>>)[]
    {
        ("GET", client => client.GetStringAsync(serverUri)),

        ("POST", async client =>
        {
            using (HttpResponseMessage m = await client.PostAsync(serverUri, new StringContent(content)))
            {
                ValidateResponseVersion(m);
                return await m.Content.ReadAsStringAsync();
            }
        }),

        ("POST w/ Expect: 100-continue", async client =>
        {
            using (var req = new HttpRequestMessage(HttpMethod.Post, serverUri) { Version = httpVersion })
            {
                req.Content = new StringContent(content);
                req.Headers.ExpectContinue = true;
                using (HttpResponseMessage m = await client.SendAsync(req, HttpCompletionOption.ResponseHeadersRead))
                {
                    ValidateResponseVersion(m);
                    return await m.Content.ReadAsStringAsync();
                }
            }
        }),

        ("GET w/ Cancellation", async client =>
        {
            using (var req = new HttpRequestMessage(HttpMethod.Get, serverUri) { Version = httpVersion })
            {
                try
                {
                    var cts = new CancellationTokenSource();
                    Task<HttpResponseMessage> t = client.SendAsync(req, HttpCompletionOption.ResponseHeadersRead, cts.Token);
                    await Task.Delay(1);
                    cts.Cancel();
                    using (HttpResponseMessage m = await t)
                    {
                        ValidateResponseVersion(m);
                        return await m.Content.ReadAsStringAsync();
                    }
                }
                catch (OperationCanceledException) { return null; }
            }
        }),

        ("POST w/ cancellation", async client =>
        {
            using (var req = new HttpRequestMessage(HttpMethod.Post, serverUri) { Version = httpVersion })
            {
                try
                {
                    req.Content = new StringContent(content);
                    var cts = new CancellationTokenSource();
                    Task<HttpResponseMessage> t = client.SendAsync(req, HttpCompletionOption.ResponseHeadersRead, cts.Token);
                    await Task.Delay(1);
                    cts.Cancel();
                    using (HttpResponseMessage m = await t)
                    {
                        ValidateResponseVersion(m);
                        return await m.Content.ReadAsStringAsync();
                    }
                }
                catch (OperationCanceledException) { return null; }
            }
        }),
    };

    Console.WriteLine("Starting kestrel...");
    WebHost
        .CreateDefaultBuilder(args)
        .ConfigureLogging(log => log.AddFilter("Microsoft.AspNetCore", level => false))
        .UseKestrel(ko =>
        {
            ko.ListenLocalhost(HttpsPort, listenOptions =>
            {
                using (RSA rsa = RSA.Create())
                {
                    var certReq = new CertificateRequest("CN=localhost", rsa, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1);
                    certReq.CertificateExtensions.Add(new X509BasicConstraintsExtension(false, false, 0, false));
                    certReq.CertificateExtensions.Add(new X509EnhancedKeyUsageExtension(new OidCollection { new Oid("1.3.6.1.5.5.7.3.1") }, false));
                    certReq.CertificateExtensions.Add(new X509KeyUsageExtension(X509KeyUsageFlags.DigitalSignature, false));
                    X509Certificate2 cert = certReq.CreateSelfSigned(DateTimeOffset.UtcNow.AddMonths(-1), DateTimeOffset.UtcNow.AddMonths(1));
                    if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
                    {
                        cert = new X509Certificate2(cert.Export(X509ContentType.Pfx));
                    }
                    listenOptions.UseHttps(cert);
                }
            });
        })
        .Configure(app =>
        {
            app.UseRouting();
            app.UseEndpoints(endpoints =>
            {
                endpoints.MapGet("/", async context =>
                {
                    await context.Response.WriteAsync(content);
                });
                endpoints.MapPost("/", async context =>
                {
                    await context.Request.Body.CopyToAsync(context.Response.Body);
                });
            });
        })
        .Build()
        .Start();

    Console.WriteLine("Starting clients...");
    using (var handler = new SocketsHttpHandler() { SslOptions = new SslClientAuthenticationOptions { RemoteCertificateValidationCallback = delegate { return true; } } })
    using (var client = new HttpClient(handler) { DefaultRequestVersion = httpVersion })
    {
        long success = 0, fail = 0;
        void Increment(ref long counter)
        {
            if (Interlocked.Increment(ref counter) % PrintEveryNRequests == 0)
            {
                Console.WriteLine($"Success: {Volatile.Read(ref success)}, Fail: {Volatile.Read(ref fail)}");
            }
        }

        Task.WaitAll(Enumerable.Range(0, ConcurrentRequests).Select(taskNum => Task.Run(async () =>
        {
            Console.WriteLine("Starting client " + taskNum);
            for (long i = 0; ; i++)
            {
                (string operation, Func<HttpClient, Task<string>> func) = clientOperations[i % clientOperations.Length];
                try
                {
                    string result = await func(client);

                    if (result != null && result != content)
                    {
                        throw new Exception("Unexpected response.  Got: " + result);
                    }

                    Increment(ref success);
                }
                catch (Exception e)
                {
                    Increment(ref fail);
                    lock (Console.Out)
                    {
                        Console.ForegroundColor = ConsoleColor.Yellow;
                        Console.WriteLine($"Error from iteration {i} ({operation}) in task {taskNum} with {success} successes / {fail} fails:");
                        Console.ResetColor();
                        Console.WriteLine(e);
                        Console.WriteLine();
                    }
                }
            }
        })).ToArray());
    }
}

}
```

cc: @geoffkizer, @wfurt, @scalablecory, @davidsh, @halter73

[geoffkizer]

Error 1 is strange; it could be a bogus frame from Kestrel, but it could also be we've got some problem reading frames where we're trying to read from the wrong offset or something like that.

Seems like adding some logging here would help. For example, when we read what appears to be a bogus frame length, log what we think is the frame type/length/flags/stream ID. That would help clarify if the problem is on our end or not.

[davidfowl]

This might be related to https://github.com/aspnet/AspNetCore/issues/11560 where Kestrel is currently cutting off HTTP/2 responses because of implementation differences between the StreamPipeWriter and the DefaultPipeWriter.

[halter73]

I agree with @davidfowl. This is the first issue I linked you to in our teams chat @stephentoub. At the time, I wasn't sure if you were running against our preview7 bits, but since you are, this seems like a very likely culprit.

Like @davidfowl mentions, it looks like a post-preview6 regression in Kestrel that was caused by Kestrel recently switching from using DefaultPipeWriter to using StreamPipeWriter for some scenarios. It turns out, that if you call StreamPipeWriter.Complete() before the ValueTask returned from StreamPipeWriter.FlushAsync() completes, StreamPipeWriter will double-return MemoryPool blocks. This was completely safe before with DefaultPipeWriter.

Since double-return/use-after-free bugs are really bad, I think we should look to fix the StreamPipeWriter ASAP in addition to looking at improving Kestrel's usage of the PipeWriter.

Edit: We're probably not actually double-freeing. https://github.com/aspnet/AspNetCore/issues/11560#issuecomment-505762781

[wfurt]

should we re-run this with different version of asp.net or try it with IIS?

[davidfowl]

Downgrade to preview6 😄 or wait until we fix it.

[geoffkizer]

For simple cases (e.g. GET only) we can also use IIS.

I ran a version of this against local IIS (GET only, no cancellation, response = "Hello world") and ran for ~2 minutes with no errors. So that's good.

[geoffkizer]

We have an issue with receiving frames after cancellation that shows up if you enable the "GET w/cancellation" variation here. Filed issue dotnet/corefx#38913.

[stephentoub]

Most of these failures do appear to be due to Kestrel (presumably the known regression(s) mentioned by @davidfowl and @halter73). I just tried running with a hacked together Preview7 System.Net.Http but a Preview6 Kestrel. Running with 200 concurrent requests on the same client, and removing the cancellation client operations from the mix (#38913), I successfully ran for several minutes without incurring any failures.

[stephentoub]

I spoke a bit too soon. Again using the latest HttpClient bits (built from master this morning) but Preview6 kestrel, setting ConcurrentRequests to 10 and setting AppxContentLength to 50_000, and commenting out the cancellation actions, I started hitting the following error eventually (note I’m also now on my less powerful laptop, as my office’s power went out, so it might require larger numbers to repro this elsewhere):

Error 11:

Error from iteration 12683 (POST w/ Expect: 100-continue) in task 6 with 133629 successes / 1 fails:
System.ArgumentException: Destination is too short.
Parameter name: destination
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush)
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in C:\Users\stoub\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1466
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) in C:\Users\stoub\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs:line 539
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in C:\Users\stoub\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\RedirectHandler.cs:line 33
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) in C:\Users\stoub\repos\corefx\src\System.Net.Http\src\System\Net\Http\HttpClient.cs:line 546
   at Program.<>c__DisplayClass0_0.<<Main>b__3>d.MoveNext() in c:\Users\stoub\Desktop\HTTPStress\Program.cs:line 68
--- End of stack trace from previous location where exception was thrown ---
   at Program.<>c__DisplayClass0_3.<<Main>b__15>d.MoveNext() in c:\Users\stoub\Desktop\HTTPStress\Program.cs:line 175

Once that happens, everything falls apart, and we get lots of cancellation exceptions from other requests, until everything halts. That’s obviously a SocketsHttpHandler issue; there’s nothing the remote server should be able to do to trigger that (even if the server sent an invalid frame, that still shouldn’t be the failure mode).

[stephentoub]

Also hit Error 7 again:

Error from iteration 1 (POST w/ Expect: 100-continue) in task 2 with 0 successes / 6 fails:
System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'CreditManager'.
   at System.Net.Http.CreditManager.RequestCreditAsync(Int32 amount, CancellationToken cancellationToken) in C:\Users\stoub\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\CreditManager.cs:line 50
   at System.Net.Http.Http2Connection.SendStreamDataAsync(Int32 streamId, ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in C:\Users\stoub\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1069
   at System.Net.Http.Http2Connection.Http2Stream.SendDataAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in C:\Users\stoub\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Stream.cs:line 629
   at System.Net.Http.HttpContent.CopyToAsyncCore(ValueTask copyTask) in C:\Users\stoub\repos\corefx\src\System.Net.Http\src\System\Net\Http\HttpContent.cs:line 361
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.CopyToAsyncCore(ValueTask copyTask) in C:\Users\stoub\repos\corefx\src\System.Net.Http\src\System\Net\Http\HttpContent.cs:line 365
   at System.Net.Http.Http2Connection.Http2Stream.SendRequestBodyAsync(CancellationToken cancellationToken) in C:\Users\stoub\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Stream.cs:line 123
   at System.Net.Http.Http2Connection.Http2Stream.SendRequestBodyWithExpect100ContinueAsync(CancellationToken cancellationToken) in C:\Users\stoub\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Stream.cs:line 176
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in C:\Users\stoub\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1383
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) in C:\Users\stoub\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs:line 539
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in C:\Users\stoub\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\RedirectHandler.cs:line 33
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) in C:\Users\stoub\repos\corefx\src\System.Net.Http\src\System\Net\Http\HttpClient.cs:line 528
   at Program.<>c__DisplayClass0_0.<<Main>b__1>d.MoveNext() in c:\Users\stoub\Desktop\HTTPStress\Program.cs:line 62
--- End of stack trace from previous location where exception was thrown ---
   at Program.<>c__DisplayClass0_3.<<Main>b__13>d.MoveNext() in c:\Users\stoub\Desktop\HTTPStress\Program.cs:line 175

[geoffkizer]

Re error 11, where is this exception coming from?

[geoffkizer]

Re error 11, this line looks suspect: https://github.com/dotnet/corefx/blob/f0ec881bbb7945e35e3cf6862ea309b1143659f7/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs#L1008

We're probably usually just getting lucky that there's enough space.

Seems like we could catch this sort of thing a bit better by returning a Memory from StartWriteAsync and writing into it, rather than writing directly into _outgoingBuffer.

[stephentoub]

Re error 11, where is this exception coming from?

I switched over to a debug build of System.Net.Http.dll and got this:

Error from iteration 43 (POST w/ Expect: 100-continue) in task 44 with 2184 successes / 1 fails:
System.ArgumentException: Destination is too short.
Parameter name: destination
   at System.ReadOnlyMemory`1.CopyTo(Memory`1 destination)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1023
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1378
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs:line 539
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\RedirectHandler.cs:line 33
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\HttpClient.cs:line 528
   at Program.<>c__DisplayClass0_0.<<Main>b__1>d.MoveNext() in C:\Users\stoub\source\repos\HTTP2Stress\HTTP2Stress\Program.cs:line 60
--- End of stack trace from previous location where exception was thrown ---
   at Program.<>c__DisplayClass0_2.<<Main>b__13>d.MoveNext() in C:\Users\stoub\source\repos\HTTP2Stress\HTTP2Stress\Program.cs:line 180

which suggests it's coming from here:
https://github.com/dotnet/corefx/blob/797afb9194c3d69a12e838b331179043a61c9542/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs#L1023

[davidfowl]

that _outgoingBuffer looks alot like a PipeWriter (I couldn't help it)

[geoffkizer]

It does indeed! And it's only 130 lines of code :)

[davidfowl]

Great minds...

[geoffkizer]

dotnet/corefx#38953 should fix error 11.

dotnet/corefx#38918 fixes a lot of the issues with cancellation, but I'm still seeing some weird behavior. Running with just GET and GET with cancellation against local IIS with a response of around 300K, things run fine for a while (a couple minutes, no failures) then all the requests time out at the same time, and then things start running again, rinse and repeat. Not sure what's up here. @stephentoub do you see this in your Kestrel runs?

[stephentoub]

@stephentoub do you see this in your Kestrel runs?

With https://github.com/dotnet/corefx/pull/38995 on top of master, I've been running with ConcurrentRequests=40, AppxContentLength=300_000, and GET/POST/POST100Continue/GETCancellation for the last 50 minutes or so and have 0 failures.

However, when I set the concurrency level higher, like ConcurrentRequests=200, when the program first starts up I get a bunch of REFUSED_STREAM errors like:

Error from iteration 0 (GET) in task 151 with 109 successes / 6 fails:
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: The request was aborted. ---> System.Net.Http.Http2ProtocolException: The HTTP/2 request failed with protocol error 'REFUSED_STREAM' (0x7).
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders() in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Stream.cs:line 492
   at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Stream.cs:line 523
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1505
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1541
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs:line 541
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\RedirectHandler.cs:line 33
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\HttpClient.cs:line 528
   at System.Net.Http.HttpClient.GetStringAsyncCore(Task`1 getTask) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\HttpClient.cs:line 156
   at Program.<>c__DisplayClass0_3.<<Main>b__16>d.MoveNext() in c:\Users\stoub\source\repos\HTTP2Stress\HTTP2Stress\Program.cs:line 189

and STREAM_CLOSED errors like:

Error from iteration 1 (POST) in task 94 with 196 successes / 18 fails:
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: The request was aborted. ---> System.Net.Http.Http2ProtocolException: The HTTP/2 request failed with protocol error 'STREAM_CLOSED' (0x5).
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders() in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Stream.cs:line 492
   at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Stream.cs:line 523
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1505
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1541
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs:line 541
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\RedirectHandler.cs:line 33
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\HttpClient.cs:line 496
   at Program.<>c__DisplayClass0_0.<<Main>b__2>d.MoveNext() in c:\Users\stoub\source\repos\HTTP2Stress\HTTP2Stress\Program.cs:line 64
--- End of stack trace from previous location where exception was thrown ---
   at Program.<>c__DisplayClass0_3.<<Main>b__16>d.MoveNext() in c:\Users\stoub\source\repos\HTTP2Stress\HTTP2Stress\Program.cs:line 189

In my most recent run, I got 36 in total, all within the first few hundred requests, and then after then, everything starts running fine and continues to process fairly constantly.

[stephentoub]

Spoke too soon. After those 36 failures, it ran for around a minute and a half, and then all of a sudden hit a slew of ~200 failures like:

Error from iteration 1 (POST) in task 137 with 95755 successes / 203 fails:
System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Net.Http.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\TaskCompletionSourceWithCancellation.cs:line 28
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1029
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1462
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs:line 541
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\RedirectHandler.cs:line 33
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\HttpClient.cs:line 496
   at Program.<>c__DisplayClass0_0.<<Main>b__2>d.MoveNext() in c:\Users\stoub\source\repos\HTTP2Stress\HTTP2Stress\Program.cs:line 69
--- End of stack trace from previous location where exception was thrown ---
   at Program.<>c__DisplayClass0_3.<<Main>b__16>d.MoveNext() in c:\Users\stoub\source\repos\HTTP2Stress\HTTP2Stress\Program.cs:line 189

Then went back to processing normally. It's been running for several more minutes since and I've not seen that again.

[geoffkizer]

Re REFUSED_STREAM: This is presumably coming from Kestrel. This is just a guess, but perhaps Kestrel has a default for SETTINGS_MAX_CONCURRENT_STREAMS that we're exceeding because we haven't actually received the SETTINGS frame from Kestrel yet? (That would explain why we only see this at startup; once we process the SETTINGS, everything is fine.)

Per https://tools.ietf.org/html/rfc7540#section-8.1.4, we can retry the request when we get REFUSED_STREAM; however it's not obvious what the best way to handle this is... if we just immediately retry we'll likely get REFUSED_STREAM again.

[geoffkizer]

Re STREAM_CLOSED, I have no idea what's up here. Possibly it's related to the REFUSED_STREAM thing (since both only occur at startup), but I can't see how it would be...

[geoffkizer]

it ran for around a minute and a half, and then all of a sudden hit a slew of ~200 failures like:

This sounds similar to what I was seeing with IIS. In my app, I had it print to console on every iteration so I could monitor progress closely. What I'd see is that it would run fine for a minute or two, then progress would stop completely, and then sometime later, all requests would timeout at effectively the same time. Does that sound similar to what you're seeing?

I was running with just GET and GET w/ cancellation, with a content size of around 300K. Might be good to determine if this is cancellation related, or happens without it as well.

[geoffkizer]

Re STREAM_CLOSED, here's a theory. For POST, we are sending a HEADERS frame followed by one or more DATA frames, and we send these before we get the REFUSED_STREAM error. So when Kestrel gets the HEADERS frame, it sends REFUSED_STREAM. Then when Kestrel gets the DATA frame(s), it sends STREAM_CLOSED because it previously refused the stream. We process both errors, and the second overwrites the first.

This is a good reason to NOT overwrite errors like this, btw.

We could get some evidence for this by doing a run with only GET and see if we still get STREAM_CLOSED, or just REFUSED_STREAM.

[geoffkizer]

Re STREAM_CLOSED: All that said, it's not at all clear to me where or how the exception would be getting overwritten. Unfortunately, there are a lot of places where we set _abortException and it's not clear where this is getting set to STREAM_CLOSED. Would be nice to clean this all up.

[geoffkizer]

I added an issue specifically for retrying REFUSED_STREAM. I actually think we can just always do this, so it should be relatively straightforward to implement.

However, it would be good to understand the STREAM_CLOSED error here better first...

[davidfowl]

FYI the latest build of ASP.NET Core has the kestrel fix.

[stephentoub]

@davidfowl, thanks. I just installed the latest from https://github.com/dotnet/core-sdk/blob/master/README.md#installers-and-binaries, which includes:

   .NET Core: 3.0.0-preview7-27826-20
ASP.NET Core: 3.0.0-preview7.19326.10

That contains the fixes? I'm not seeing the same errors I was seeing, but I do see failure events being logged by ASP.NET, e.g. a bunch of failures that look like this:

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HLNRDQ04UI61", Request id "0HLNRDQ04UI61:00000E5F": An unhandled exception was thrown by the application.
System.InvalidOperationException: Writing is not allowed after writer was completed.
   at System.IO.Pipelines.ThrowHelper.ThrowInvalidOperationException_NoWritingAllowed()
   at System.IO.Pipelines.Pipe.GetSpan(Int32 sizeHint)
   at System.IO.Pipelines.Pipe.DefaultPipeWriter.GetSpan(Int32 sizeHint)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2OutputProducer.GetSpan(Int32 sizeHint)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.GetSpan(Int32 sizeHint)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponsePipeWriter.GetSpan(Int32 sizeHint)
   at Microsoft.AspNetCore.Http.HttpResponseWritingExtensions.Write(HttpResponse response, String text, Encoding encoding)
   at Microsoft.AspNetCore.Http.HttpResponseWritingExtensions.WriteAsync(HttpResponse response, String text, Encoding encoding, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Http.HttpResponseWritingExtensions.WriteAsync(HttpResponse response, String text, CancellationToken cancellationToken)
   at Program.<>c__DisplayClass0_0.<<Main>b__12>d.MoveNext() in d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 196
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

Is that expected, and the errors are just being eaten?

[davidfowl]

Not expected, thanks, taking a look

[davidfowl]

Are you using that stress test you have above?

[halter73]

I'm looking at this too.

@davidfowl you can repro with https://github.com/stephentoub/corefx/tree/httpstress/src/System.Net.Http/tests/StressTests/HttpStress

[davidfowl]

I’m thinking maybe we check in something basic as a functional test until we have better coverage

[halter73]

https://github.com/aspnet/AspNetCore/pull/11675

OK. So I think this is was introduced without being caught mostly because the vast majority of Kestrel tests use the response Stream instead of the response PipeWriter, and HttpResponseWritingExtensions.WriteAsync uses the PipeWriter directly.

We should look at getting better coverage here, because HttpResponse.Body.WriteAsync() already correctly no-ops in this case because that calls Http2OutputProducer.WriteDataAsync() which currently checks the _completed flag unlike Http2OutputProducer.GetSpan/Advance().

I'm not sure we want to run every test twice, one with the request/response Stream and one with the request/response Pipes, but that might be a good temporary solution. Long term, I'd only want to run targeted tests against both APIs, tests that could actually be reasonably individually impacted by pipe bugs.

[halter73]

The InvalidOperationException from the PipeWriter after an RST_STREAM frame will be fixed in preview7. I just merged the PR, so there are no builds with the fix yet.

[davidfowl]

Builds are available now.

[stephentoub]

Using the latest SDK (with ASP.NET Core 3.0.0-preview7.19330.2) and a local build of System.Net.Http with the bits from master as of right now, I get a fast and steady stream of these exceptions, even with a concurrency level of 1:

Error from iteration 1904 (GET) in task 0 with 1852 successes / 53 fails:
System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: The request was aborted.
 ---> System.Net.Http.Http2ProtocolException: The HTTP/2 request failed with protocol error 'STREAM_CLOSED' (0x5).
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.Http2Stream.CheckIfDisposedOrAborted() in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Stream.cs:line 517
   at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders() in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Stream.cs:line 545
   at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Stream.cs:line 573
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1557
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1557
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs:line 541
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\RedirectHandler.cs:line 33
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\HttpClient.cs:line 521
   at Program.<>c__DisplayClass1_0.<<Run>b__1>d.MoveNext() in d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 99
--- End of stack trace from previous location where exception was thrown ---
   at Program.<>c__DisplayClass1_2.<<Run>b__28>d.MoveNext() in d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 373

The ASP.NET log shows errors like:

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HLNTNM18H99D", Request id "0HLNTNM18H99D:0000000F": An unhandled exception was thrown by the application.
System.IO.IOException: The client reset the request stream.
   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
   at System.IO.Pipelines.Pipe.GetReadAsyncResult()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2MessageBody.ReadAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.CopyToAsyncInternal(Stream destination, CancellationToken cancellationToken)
   at Program.<>c.<<Run>b__1_20>d.MoveNext() in d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 299
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

[stephentoub]

The above seem to be triggered by the "POST Cancellation" client operation, e.g. I haven't yet seen them if I specify "-ops 0 1 2 3 4 5 6" on the command-line to the stress program, leaving off 7.

However, with a concurrency level higher than 1, e.g.

dotnet run -c Release -n 10 -ops 0 1 2 3 4 5 6

I do see a fair number of errors like this:

Error from iteration 738 (GET Aborted) in task 3 with 7316 successes / 1 fails:
System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: The request was aborted.
 ---> System.Net.Http.Http2ProtocolException: The HTTP/2 request failed with protocol error 'NO_ERROR' (0x0).
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.Http2Stream.CheckIfDisposedOrAborted() in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Stream.cs:line 517
   at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders() in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Stream.cs:line 528
   at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Stream.cs:line 559
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1557
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1557
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs:line 541
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\RedirectHandler.cs:line 33
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\HttpClient.cs:line 528
   at System.Net.Http.HttpClient.GetStringAsyncCore(Task`1 getTask) in d:\repos\corefx\src\System.Net.Http\src\System\Net\Http\HttpClient.cs:line 156
   at Program.<>c__DisplayClass1_0.<<Run>b__4>d.MoveNext() in d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 138
--- End of stack trace from previous location where exception was thrown ---
   at Program.<>c__DisplayClass1_2.<<Run>b__28>d.MoveNext() in d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 373

[geoffkizer]

I'm guessing the STREAM_CLOSED stuff is dotnet/corefx#39016.

The NO_ERROR stuff is strange. Possibly could be dotnet/corefx#39013, but then that raises the question of why we are seeing GOAWAY here. Looking at the trace would be useful.

[stephentoub]

I just installed the latest SDK (w/ ASP.NET Core 3.0.0-preview7.19351.4 and .NET Core 3.0.0-preview7-27901-06), and then built System.Net.Http from source based on the latest in master plus cherry-picking dotnet/corefx#39107 and dotnet/corefx#39117. Definitely improved, but still some issues.

In some runs, everything works well. In others, I see errors like these (they usually come in clusters):

Error from iteration 715 (GET Aborted) in task 4 with 8717 successes / 12 fails:
System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: The request was aborted.
 ---> System.Net.Http.Http2ConnectionException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'NO_ERROR' (0x0).
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.Http2Stream.CheckIfDisposedOrAborted() in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Stream.cs:line 517
   at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders() in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Stream.cs:line 528
   at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Stream.cs:line 559
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1580
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1580
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs:line 541
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\RedirectHandler.cs:line 33
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\HttpClient.cs:line 528
   at System.Net.Http.HttpClient.GetStringAsyncCore(Task`1 getTask) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\HttpClient.cs:line 156
   at Program.<>c__DisplayClass1_0.<<Run>b__4>d.MoveNext() in d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 138
--- End of stack trace from previous location where exception was thrown ---
   at Program.<>c__DisplayClass1_2.<<Run>b__28>d.MoveNext() in d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 375

Also, if I increase the concurrency level, e.g.

d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress>dotnet run -c Release -n 100

I frequently won't see any errors, but relatively early into the run, no progress is made (note how after a few seconds, the numbers all stay constant):

d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress>dotnet run -c Release -n 100
     .NET Core: 3.0.0-preview7-27901-06
  ASP.NET Core: 3.0.0-preview7.19351.4
       Tracing: False
   ASP.NET Log: False
   Concurrency: 100
Content Length: 1000
  HTTP Version: 2.0
    Operations: GET, GET Headers, GET Cancellation, GET Aborted, POST, POST Duplex, POST ExpectContinue, POST Cancellation

Starting server.
Starting 100 client workers.
[7/2/2019 10:53:19 AM] Total: 6,600
        GET                           Success: 826      Fail: 0
        GET Headers                   Success: 831      Fail: 0
        GET Cancellation              Success: 839      Fail: 0
        GET Aborted                   Success: 825      Fail: 0
        POST                          Success: 832      Fail: 0
        POST Duplex                   Success: 835      Fail: 0
        POST ExpectContinue           Success: 810      Fail: 0
        POST Cancellation             Success: 821      Fail: 0

[7/2/2019 10:53:20 AM] Total: 14,000
        GET                           Success: 1,750    Fail: 0
        GET Headers                   Success: 1,760    Fail: 0
        GET Cancellation              Success: 1,756    Fail: 0
        GET Aborted                   Success: 1,754    Fail: 0
        POST                          Success: 1,762    Fail: 0
        POST Duplex                   Success: 1,748    Fail: 0
        POST ExpectContinue           Success: 1,747    Fail: 0
        POST Cancellation             Success: 1,753    Fail: 0

[7/2/2019 10:53:21 AM] Total: 18,876
        GET                           Success: 2,357    Fail: 0
        GET Headers                   Success: 2,368    Fail: 0
        GET Cancellation              Success: 2,381    Fail: 0
        GET Aborted                   Success: 2,344    Fail: 0
        POST                          Success: 2,356    Fail: 0
        POST Duplex                   Success: 2,364    Fail: 0
        POST ExpectContinue           Success: 2,347    Fail: 0
        POST Cancellation             Success: 2,359    Fail: 0

[7/2/2019 10:53:22 AM] Total: 18,876
        GET                           Success: 2,357    Fail: 0
        GET Headers                   Success: 2,368    Fail: 0
        GET Cancellation              Success: 2,381    Fail: 0
        GET Aborted                   Success: 2,344    Fail: 0
        POST                          Success: 2,356    Fail: 0
        POST Duplex                   Success: 2,364    Fail: 0
        POST ExpectContinue           Success: 2,347    Fail: 0
        POST Cancellation             Success: 2,359    Fail: 0

[7/2/2019 10:53:23 AM] Total: 18,876
        GET                           Success: 2,357    Fail: 0
        GET Headers                   Success: 2,368    Fail: 0
        GET Cancellation              Success: 2,381    Fail: 0
        GET Aborted                   Success: 2,344    Fail: 0
        POST                          Success: 2,356    Fail: 0
        POST Duplex                   Success: 2,364    Fail: 0
        POST ExpectContinue           Success: 2,347    Fail: 0
        POST Cancellation             Success: 2,359    Fail: 0

[7/2/2019 10:53:24 AM] Total: 18,876
        GET                           Success: 2,357    Fail: 0
        GET Headers                   Success: 2,368    Fail: 0
        GET Cancellation              Success: 2,381    Fail: 0
        GET Aborted                   Success: 2,344    Fail: 0
        POST                          Success: 2,356    Fail: 0
        POST Duplex                   Success: 2,364    Fail: 0
        POST ExpectContinue           Success: 2,347    Fail: 0
        POST Cancellation             Success: 2,359    Fail: 0

[7/2/2019 10:53:25 AM] Total: 18,876
        GET                           Success: 2,357    Fail: 0
        GET Headers                   Success: 2,368    Fail: 0
        GET Cancellation              Success: 2,381    Fail: 0
        GET Aborted                   Success: 2,344    Fail: 0
        POST                          Success: 2,356    Fail: 0
        POST Duplex                   Success: 2,364    Fail: 0
        POST ExpectContinue           Success: 2,347    Fail: 0
        POST Cancellation             Success: 2,359    Fail: 0

[7/2/2019 10:53:26 AM] Total: 18,876
        GET                           Success: 2,357    Fail: 0
        GET Headers                   Success: 2,368    Fail: 0
        GET Cancellation              Success: 2,381    Fail: 0
        GET Aborted                   Success: 2,344    Fail: 0
        POST                          Success: 2,356    Fail: 0
        POST Duplex                   Success: 2,364    Fail: 0
        POST ExpectContinue           Success: 2,347    Fail: 0
        POST Cancellation             Success: 2,359    Fail: 0

[7/2/2019 10:53:27 AM] Total: 18,876
        GET                           Success: 2,357    Fail: 0
        GET Headers                   Success: 2,368    Fail: 0
        GET Cancellation              Success: 2,381    Fail: 0
        GET Aborted                   Success: 2,344    Fail: 0
        POST                          Success: 2,356    Fail: 0
        POST Duplex                   Success: 2,364    Fail: 0
        POST ExpectContinue           Success: 2,347    Fail: 0
        POST Cancellation             Success: 2,359    Fail: 0

[7/2/2019 10:53:28 AM] Total: 18,876
        GET                           Success: 2,357    Fail: 0
        GET Headers                   Success: 2,368    Fail: 0
        GET Cancellation              Success: 2,381    Fail: 0
        GET Aborted                   Success: 2,344    Fail: 0
        POST                          Success: 2,356    Fail: 0
        POST Duplex                   Success: 2,364    Fail: 0
        POST ExpectContinue           Success: 2,347    Fail: 0
        POST Cancellation             Success: 2,359    Fail: 0

[7/2/2019 10:53:29 AM] Total: 18,876
        GET                           Success: 2,357    Fail: 0
        GET Headers                   Success: 2,368    Fail: 0
        GET Cancellation              Success: 2,381    Fail: 0
        GET Aborted                   Success: 2,344    Fail: 0
        POST                          Success: 2,356    Fail: 0
        POST Duplex                   Success: 2,364    Fail: 0
        POST ExpectContinue           Success: 2,347    Fail: 0
        POST Cancellation             Success: 2,359    Fail: 0

@davidfowl, @halter73, something also appears to be going wrong on the server side (which may or may not be related to the above). When I enable logging ASP.NET warnings and errors:

d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress>dotnet run -c Release -n 100 -aspnetlog

I see a ton of "fails" like this:

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HLNUVKEU3F5M", Request id "0HLNUVKEU3F5M:00000155": An unhandled exception was thrown by the application.
System.IO.IOException: The client reset the request stream.
   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
   at System.IO.Pipelines.Pipe.GetReadAsyncResult()
   at System.IO.Pipelines.Pipe.DefaultPipeReader.GetResult(Int16 token)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2MessageBody.ReadAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.CopyToAsyncInternal(Stream destination, CancellationToken cancellationToken)
   at Program.<>c.<<Run>b__1_20>d.MoveNext() in d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 301
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

and maybe that's expected? But then I start to see tons of exceptions from the client like this:

Error from iteration 41 (GET Headers) in task 33 with 680 successes / 14 fails:
System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Http.HttpRequestException: Received an invalid header name: '
 '.
   at System.Net.Http.Http2Connection.Http2Stream.OnResponseHeader(ReadOnlySpan`1 name, ReadOnlySpan`1 value) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Stream.cs:line 308
   at System.Net.Http.Http2Connection.<>c.<.cctor>b__88_0(Object state, ReadOnlySpan`1 name, ReadOnlySpan`1 value) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 310
   at System.Net.Http.HPack.HPackDecoder.OnHeaderComplete(HeaderCallback onHeader, Object onHeaderState, ReadOnlySpan`1 headerName, ReadOnlySpan`1 headerValue) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HPack\HPackDecoder.cs:line 438
   at System.Net.Http.HPack.HPackDecoder.Decode(ReadOnlySpan`1 data, Boolean endHeaders, HeaderCallback onHeader, Object onHeaderState) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HPack\HPackDecoder.cs:line 288
   at System.Net.Http.Http2Connection.ProcessHeadersFrame(FrameHeader frameHeader) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 333
   at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync() in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 235
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1576
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs:line 541
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\RedirectHandler.cs:line 33
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\HttpClient.cs:line 496
   at Program.<>c__DisplayClass1_0.<<Run>b__2>d.MoveNext() in d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 104
--- End of stack trace from previous location where exception was thrown ---
   at Program.<>c__DisplayClass1_2.<<Run>b__28>d.MoveNext() in d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 375

or like:

Error from iteration 93 (POST Duplex) in task 84 with 1882 successes / 82 fails:
System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Http.Http2ConnectionException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'FRAME_SIZE_ERROR' (0x6).
   at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 204
   at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync() in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 229
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1055
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1497
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1576
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs:line 541
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\RedirectHandler.cs:line 33
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\HttpClient.cs:line 496
   at Program.<>c__DisplayClass1_0.<<Run>b__6>d.MoveNext() in d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 171
--- End of stack trace from previous location where exception was thrown ---
   at Program.<>c__DisplayClass1_2.<<Run>b__28>d.MoveNext() in d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 375

which suggests the server is sending erroneous/corrupted data.

(That latter stack trace is also very strange from a client perspective, with all of those repeated ExceptionDispatchInfo.Throw+SendAsync+SendHeadersAsync frames... that suggests that we've got a loop or recursive calls somewhere that are unnecessarily throwing the same exception over and over, or that there's a bug in exception propagation / stack trace data, or maybe that we're repeatedly rethrowing the same exception over and over and over.)

[stephentoub]

(For the client stack trace oddity, I think what we're seeing here is many concurrent requests all failing on the same connection, and all concurrently using ExceptionDispathInfo.Throw to throw the same _abortException instance; all of them are thus mutating the same Exception object, and they all end up seeing each other's stack frames. I'll prove that out and then submit a PR to fix it if it's the case.)

[stephentoub]

@halter73, @davidfowl, I've also been seeing this showing up from the ASP.NET log:

warn: Microsoft.AspNetCore.Server.Kestrel[0]
      Connection processing ended abnormally.
System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
   at System.IO.Pipelines.BufferSegment.set_End(Int32 value)
   at System.IO.Pipelines.Pipe.AllocateWriteHeadSynchronized(Int32 sizeHint)
   at System.IO.Pipelines.Pipe.GetSpan(Int32 sizeHint)
   at System.IO.Pipelines.Pipe.DefaultPipeWriter.GetSpan(Int32 sizeHint)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2FrameWriter.WriteHeader(Http2Frame frame, PipeWriter output)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2FrameWriter.WriteHeaderUnsynchronized()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2FrameWriter.WriteRstStreamAsync(Int32 streamId, Http2ErrorCode errorCode)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2OutputProducer.WriteRstStreamAsync(Http2ErrorCode error)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Stream.ResetAndAbort(ConnectionAbortedException abortReason, Http2ErrorCode error)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Stream.OnErrorAfterResponseStarted()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProduceEnd()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequestsAsync[TContext](IHttpApplication`1 application)

It also sometimes shows up like this:

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HLNV04RDIO5I", Request id "0HLNV04RDIO5I:00000703": An unhandled exception was thrown by the application.
System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
   at System.IO.Pipelines.BufferSegment.set_End(Int32 value)
   at System.IO.Pipelines.Pipe.AllocateWriteHeadSynchronized(Int32 sizeHint)
   at System.IO.Pipelines.Pipe.GetSpan(Int32 sizeHint)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2FrameWriter.WriteHeader(Http2Frame frame, PipeWriter output)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2FrameWriter.WriteHeaderUnsynchronized()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2FrameWriter.FinishWritingHeaders(Int32 streamId, Int32 payloadLength, Boolean done)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2FrameWriter.WriteResponseHeaders(Int32 streamId, Int32 statusCode, Http2HeadersFrameFlags headerFrameFlags, IHeaderDictionary headers)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2OutputProducer.WriteResponseHeaders(Int32 statusCode, String ReasonPhrase, HttpResponseHeaders responseHeaders, Boolean autoChunk, Boolean appCompleted)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProduceStart(Boolean appCompleted)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.InitializeResponseAsync(Int32 firstWriteByteCount)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.Microsoft.AspNetCore.Http.Features.IHttpResponseStartFeature.StartAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Http.DefaultHttpResponse.StartAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Http.HttpResponseWritingExtensions.WriteAsync(HttpResponse response, String text, Encoding encoding, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Http.HttpResponseWritingExtensions.WriteAsync(HttpResponse response, String text, CancellationToken cancellationToken)
   at Program.<>c__DisplayClass1_0.<<Run>b__19>d.MoveNext() in d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 295
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

and sometimes like this:

crit: Microsoft.AspNetCore.Server.Kestrel[0]
      Http2OutpuProducer.ProcessDataWrites() observed an unexpected exception.
System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
   at System.IO.Pipelines.BufferSegment.set_End(Int32 value)
   at System.IO.Pipelines.Pipe.CommitUnsynchronized()
   at System.IO.Pipelines.Pipe.PrepareFlush(CompletionData& completionData, ValueTask`1& result, CancellationToken cancellationToken)
   at System.IO.Pipelines.Pipe.FlushAsync(CancellationToken cancellationToken)
   at System.IO.Pipelines.Pipe.DefaultPipeWriter.FlushAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.TimeFlushAsync(MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2OutputProducer.ProcessDataWrites()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2OutputProducer.ProcessDataWrites()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2OutputProducer.ProcessDataWrites()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2OutputProducer.ProcessDataWrites()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2OutputProducer.ProcessDataWrites()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2OutputProducer.ProcessDataWrites()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2OutputProducer.ProcessDataWrites()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2OutputProducer.ProcessDataWrites()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2OutputProducer.ProcessDataWrites()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2OutputProducer.ProcessDataWrites()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2OutputProducer.ProcessDataWrites()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2OutputProducer.ProcessDataWrites()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2OutputProducer.ProcessDataWrites()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.AwaitLastFlushAndTimeFlushAsync(ValueTask`1 lastFlushTask, MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2OutputProducer.ProcessDataWrites()

[stephentoub]

@davidfowl, @benaadams, given the 😕 faces, I assume this isn't a known issue :wink:, so I've opened https://github.com/aspnet/AspNetCore/issues/11804 for it. Presumably the error is due to using Pipe in an unexpected way; if it's using Pipe correctly, then the issue could be moved to corefx.

[benaadams]

, given the 😕 faces,

The and sometimes like this: stack trace hurts me. Not sure if its expected (might be); but feels like it could go to an infinite depth chain; just wondering at what point it awaits to unwind...

[benaadams]

System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
   at System.IO.Pipelines.BufferSegment.set_End(Int32 value)

Is from Memory = AvailableMemory.Slice(0, value) which would suggest value is out of range; either concurrent Writing; or writing to a Reset BufferSegment where AvailableMemory is default; so Read disposing of a Writer's block.

But that's just a guess :)

[benaadams]

Guessing is from:

endpoints.MapPost("/", async context =>
{
    await context.Request.Body.CopyToAsync(context.Response.Body);
});

Stream Dispose completes the reader and writer https://github.com/dotnet/corefx/pull/36160; could this happen while the copy operation is still happening? (I assume not?) Just the length of those FlushAsync chains make me nervous

[stephentoub]

Guessing is from

Am I using the ASP.NET API incorrectly there? I realize this particular case is relying on the duplex behavior, but that should be supported, right? That said, you're looking at a slightly out-of-date version of the test. That MapPost is still there, but as:
https://github.com/dotnet/corefx/blob/b132f92c28e4c4f696610f48cd9b3afb78b5eff3/src/System.Net.Http/tests/StressTests/HttpStress/Program.cs#L302-L306
and I still see these failures when I disable that duplex client operation, skipping '5' which is:
https://github.com/dotnet/corefx/blob/b132f92c28e4c4f696610f48cd9b3afb78b5eff3/src/System.Net.Http/tests/StressTests/HttpStress/Program.cs#L166-L174
e.g. when I run:

dotnet run -c Release -ops 0 1 2 3 4 6 7 -n 100 -aspnetlog

I end up seeing a lot of errors like:

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HLNV43T7AG5S", Request id "0HLNV43T7AG5S:0000019B": An unhandled exception was thrown by the application.
System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
   at System.IO.Pipelines.BufferSegment.set_End(Int32 value)
   at System.IO.Pipelines.Pipe.AllocateWriteHeadSynchronized(Int32 sizeHint)
   at System.IO.Pipelines.Pipe.GetSpan(Int32 sizeHint)
   at System.IO.Pipelines.Pipe.DefaultPipeWriter.GetSpan(Int32 sizeHint)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2FrameWriter.WriteHeader(Http2Frame frame, PipeWriter output)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2FrameWriter.WriteHeaderUnsynchronized()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2FrameWriter.FinishWritingHeaders(Int32 streamId, Int32 payloadLength, Boolean done)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2FrameWriter.WriteResponseHeaders(Int32 streamId, Int32 statusCode, Http2HeadersFrameFlags headerFrameFlags, IHeaderDictionary headers)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2OutputProducer.WriteResponseHeaders(Int32 statusCode, String ReasonPhrase, HttpResponseHeaders responseHeaders, Boolean autoChunk, Boolean appCompleted)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProduceStart(Boolean appCompleted)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.InitializeResponseAsync(Int32 firstWriteByteCount)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.Microsoft.AspNetCore.Http.Features.IHttpResponseStartFeature.StartAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Http.DefaultHttpResponse.StartAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Http.HttpResponseWritingExtensions.WriteAsync(HttpResponse response, String text, Encoding encoding, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Http.HttpResponseWritingExtensions.WriteAsync(HttpResponse response, String text, CancellationToken cancellationToken)
   at Program.<>c__DisplayClass1_0.<<Run>b__19>d.MoveNext() in d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 293
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

[benaadams]

Am I using the ASP.NET API incorrectly there?

Doesn't look unusual.

Is it all from the Http2 side or do you see these issues if you use Http1.1 instead? (I'm not too familiar with how the Http2 works in Kestrel)

[stephentoub]

Is it all from the Http2 side

Yes. Though I see a different set of issues with HTTP/1.1, e.g. running
```C#
dotnet run -c Release -n 100 -http 1.1 -aspnetlog

I sporadically see failures like this from the client:

Error from iteration 1419 (GET Aborted) in task 27 with 138667 successes / 138 fails:
System.Net.Http.HttpRequestException: Only one usage of each socket address (protocol/network address/port) is normally permitted.
---> System.Net.Sockets.SocketException (10048): Only one usage of each socket address (protocol/network address/port) is normally permitted.
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken) in D:reposcorefxsrcSystem.Net.HttpsrcSystemNetHttpSocketsHttpHandlerConnectHelper.cs:line 97
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken) in D:reposcorefxsrcSystem.Net.HttpsrcSystemNetHttpSocketsHttpHandlerConnectHelper.cs:line 97
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken) in D:reposcorefxsrcSystem.Net.HttpsrcSystemNetHttpSocketsHttpHandlerHttpConnectionPool.cs:line 662
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:reposcorefxsrcSystem.Net.HttpsrcSystemNetHttpSocketsHttpHandlerHttpConnectionPool.cs:line 675
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:reposcorefxsrcSystem.Net.HttpsrcSystemNetHttpSocketsHttpHandlerHttpConnectionPool.cs:line 345
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) in D:reposcorefxsrcSystem.Net.HttpsrcSystemNetHttpSocketsHttpHandlerHttpConnectionPool.cs:line 524
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:reposcorefxsrcSystem.Net.HttpsrcSystemNetHttpSocketsHttpHandlerRedirectHandler.cs:line 33
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\HttpClient.cs:line 546 at System.Net.Http.HttpClient.GetStringAsyncCore(Task1 getTask) in D:reposcorefxsrcSystem.Net.HttpsrcSystemNetHttpHttpClient.cs:line 188
at Program.<>c__DisplayClass1_0.<b__4>d.MoveNext() in d:reposcorefxsrcSystem.Net.HttptestsStressTestsHttpStressProgram.cs:line 165
--- End of stack trace from previous location where exception was thrown ---
at Program.<>c__DisplayClass1_2.<b__28>d.MoveNext() in d:reposcorefxsrcSystem.Net.HttptestsStressTestsHttpStressProgram.cs:line 384

and errors like this from the server:

crit: Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware[0]
SslDuplexPipe.WriteOutputAsync
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'SslStream'.
at System.Net.Security.SslStream.ThrowIfExceptional()
at System.Net.Security.SslStream.CheckThrow(Boolean authSuccessCheck, Boolean shutdownCheck)
at System.Net.Security.SslStream.WriteAsync(ReadOnlyMemory1 buffer, CancellationToken cancellationToken) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.DuplexPipeStreamAdapter1.WriteOutputAsync()
```

[jkotalik]

@davidfowl that last error with SslStream seems potentially new.

@stephentoub do you know if the errors the clients and server sees are 1 to 1? Ex: every time the client sees that error, the server also hits that error?

[stephentoub]

do you know if the errors the clients and server sees are 1 to 1?

@jkotalik, I don't think they are.

[benaadams]

Port exhaustion might be coming from not shutting down Kestrel in a controlled way and the client sockets being stuck in TIME_WAIT?

Rather than

WebHost.CreateDefaultBuilder()
// ...
 .Build()
 .Start();

You could do something more like

var host = WebHost.CreateDefaultBuilder()
// ...
 .Build();

Task kestrelTask = host.StartAsync();

// ... later

Task shutdown = host.StopAsync();
await Task.WhenAll(shutdown, kestrelTask); // only really need to wait shutdown

[stephentoub]

Port exhaustion might be coming from not shutting down Kestrel in a controlled way and the client sockets being stuck in TIME_WAIT?

It's definitely due to lots of TIME_WAITs, but not from not shutting down Kestrel correctly; it happens in a single run when my machine starts with practically nothing in TIME_WAIT.

[krwq]

I've seen one similar failure on regular test run on my Win10 box:

      System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_Cookies_Http2.GetAsync_SetCookieContainer_CookieSent(cookieName: "foo", cookieValue: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"..., useCookies: False) [FAIL]
        System.Net.Http.HttpRequestException : Only one usage of each socket address (protocol/network address/port) is normally permitted.
        ---- System.Net.Sockets.SocketException : Only one usage of each socket address (protocol/network address/port) is normally permitted.
        Stack Trace:
          F:\src\corefx4\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\ConnectHelper.cs(84,0): at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
          F:\src\corefx4\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(626,0): at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cance
  llationToken)
          F:\src\corefx4\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(416,0): at System.Net.Http.HttpConnectionPool.GetHttp2ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationTo
  ken)
          F:\src\corefx4\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(524,0): at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationTo
  ken cancellationToken)
          F:\src\corefx4\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\RedirectHandler.cs(33,0): at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
          F:\src\corefx4\src\System.Net.Http\src\System\Net\Http\HttpClient.cs(496,0): at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
          F:\src\corefx4\src\System.Net.Http\tests\FunctionalTests\HttpClientHandlerTest.Cookies.cs(74,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Cookies.<>c__DisplayClass10_0.<<GetAsync_SetCookieContainer_CookieSent>
  b__0>d.MoveNext()
          --- End of stack trace from previous location where exception was thrown ---
          F:\src\corefx4\src\Common\tests\System\Threading\Tasks\TaskTimeoutExtensions.cs(83,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
          F:\src\corefx4\src\Common\tests\System\Threading\Tasks\TaskTimeoutExtensions.cs(111,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
          F:\src\corefx4\src\Common\tests\System\Net\Http\GenericLoopbackServer.cs(31,0): at System.Net.Test.Common.LoopbackServerFactory.<>c__DisplayClass5_0.<<CreateClientAndServerAsync>b__0>d.MoveNext()
          --- End of stack trace from previous location where exception was thrown ---
          F:\src\corefx4\src\Common\tests\System\Threading\Tasks\TaskTimeoutExtensions.cs(38,0): at System.Threading.Tasks.TaskTimeoutExtensions.TimeoutAfter(Task task, TimeSpan timeout)
          F:\src\corefx4\src\Common\tests\System\Net\Http\Http2LoopbackServer.cs(215,0): at System.Net.Test.Common.Http2LoopbackServerFactory.CreateServerAsync(Func`3 funcAsync, Int32 millisecondsTimeout)
          F:\src\corefx4\src\Common\tests\System\Threading\Tasks\TaskTimeoutExtensions.cs(38,0): at System.Threading.Tasks.TaskTimeoutExtensions.TimeoutAfter(Task task, TimeSpan timeout)
          F:\src\corefx4\src\System.Net.Http\tests\FunctionalTests\HttpClientHandlerTest.Cookies.cs(65,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Cookies.GetAsync_SetCookieContainer_CookieSent(String cookieName, Strin
  g cookieValue, Boolean useCookies)
          --- End of stack trace from previous location where exception was thrown ---
          ----- Inner Stack Trace -----
          F:\src\corefx4\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\ConnectHelper.cs(65,0): at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)

[geoffkizer]

In others, I see errors like these (they usually come in clusters):

This is almost certainly dotnet/corefx#39013. I believe what's happening is that the server is sending a GOAWAY with error = NO_ERROR, and we are incorrectly aborting (jnstead of retrying) the requests that haven't been processed by the server.

It's weird that Kestrel is sending GOAWAY with error = NO_ERROR here, and may indicate a Kestrel issue. If it was a client-caused problem, I would expect an error other than NO_ERROR here. Regardless, the client needs to handle this properly by retrying.

[geoffkizer]

(For the client stack trace oddity, I think what we're seeing here is many concurrent requests all failing on the same connection, and all concurrently using ExceptionDispathInfo.Throw to throw the same _abortException instance

Yeah, that's almost certainly what's happening. We're throwing this error while processing incoming frames, it gets caught and passed to Http2Stream.OnAbort for all the (incorrectly) aborted streams, and then we appear to rethrow it here:

https://github.com/dotnet/corefx/blob/cc6435bb9ec069c57a3d155944b5729b3f06439f/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs#L909

This needs to get fixed to wrap it in an IOException, like we do other places.

[geoffkizer]

It's definitely due to lots of TIME_WAITs, but not from not shutting down Kestrel correctly; it happens in a single run when my machine starts with practically nothing in TIME_WAIT.

I suspect this is just that you're cancelling a lot of requests, causing lots of connections to be closed and thus exhausting ports because they are all in TIME_WAIT. I.e., expected behavior.

[stephentoub]

I suspect this is just that you're cancelling a lot of requests

That's probably true.

[geoffkizer]

Re ExceptionDispatchInfo, it could also be here:

https://github.com/dotnet/corefx/blob/cc6435bb9ec069c57a3d155944b5729b3f06439f/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs#L1052

I don't understand why we are checking _abortException here. Seems like we should always throw the retryable exception here. @wfurt, thoughts?

[benaadams]

I think auto reuse port is on by default in Win10/Server2016; so you could try using a range of localhost addresses to increase the port range?

e.g. rather than localhost use 127.0.0.1 then 127.0.0.2 then 127.0.0.3, etc

May or may not help... YMMV etc 😉

[geoffkizer]

The ExceptionDispatchInfo issues should be fixed in the new commit I pushed on dotnet/corefx#39107

[benaadams]

Might also need to set

socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, optionValue: true);

on the client socket before connecting first though?

[stephentoub]

https://github.com/dotnet/corefx/issues/38911#issuecomment-507891699

I already put up a PR for that as well.
https://github.com/dotnet/corefx/pull/39135

[benaadams]

Might be something similar(ish) in Kestrel with the odd stack trace; same exception is used in two different paths; both of which I think then use EDI on it

[benaadams]

For the long chain exception https://github.com/aspnet/AspNetCore/pull/11827

[benaadams]

Fix for Pipelines exceptions https://github.com/dotnet/corefx/pull/39158

[stephentoub]

In positive news, with the latest SDK, bits from master for System.Net.Http + cherry-picking dotnet/corefx#39127, and restricting to running the stress tests only with the "successful" operations (e.g. no cancellation, no abort):

d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress>dotnet run -c Release -ops 0 1 4 5 6 -aspnetlog

things run fairly smoothly, e.g. after a few minutes, :

[7/3/2019 5:03:37 PM] Total: 12,412,742
        GET                           Success: 2,482,549        Fail: 0
        GET Headers                   Success: 2,482,552        Fail: 0
        POST                          Success: 2,482,551        Fail: 3
        POST Duplex                   Success: 2,482,549        Fail: 6
        POST ExpectContinue           Success: 2,482,497        Fail: 56

which is a 99.999% success rate. The failures are all of a form where ASP.NET incurs a failure like:

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HLNVV1RJH9DH", Request id "0HLNVV1RJH9DH:00091A39": An unhandled exception was thrown by the application.
System.IO.IOException: The request stream was aborted.
 ---> Microsoft.AspNetCore.Connections.ConnectionAbortedException: The HTTP/2 connection faulted.
 ---> System.IO.IOException: The read operation failed, see inner exception.
 ---> System.InvalidOperationException: Reading is already in progress.
   at System.IO.Pipelines.ThrowHelper.ThrowInvalidOperationException_AlreadyReading()
   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
   at System.IO.Pipelines.Pipe.ReadAsync(CancellationToken token)
   at System.IO.Pipelines.Pipe.DefaultPipeReader.ReadAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.DuplexPipeStream.ReadAsyncInternal(Memory`1 destination, CancellationToken cancellationToken)
   at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
   at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
   at System.IO.Pipelines.StreamPipeReader.ReadAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessRequestsAsync[TContext](IHttpApplication`1 application)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
   at System.IO.Pipelines.Pipe.GetReadAsyncResult()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2MessageBody.ReadAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.CopyToAsyncInternal(Stream destination, CancellationToken cancellationToken)
   at Program.<>c.<<Run>b__1_20>d.MoveNext() in d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 318
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

and then the client operation immediately fails with:

Error from iteration 976309 (POST ExpectContinue) in task 10 with 11786021 successes / 65 fails:
System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: The response ended prematurely, with at least 9 additional bytes expected.
   at System.Net.Http.Http2Connection.ReadAtLeastAsync(Stream stream, Memory`1 buffer, Int32 minReadBytes) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1687
   at System.Net.Http.Http2Connection.EnsureIncomingBytesAsync(Int32 minReadBytes) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 163
   at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 210
   at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync() in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 229
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1614
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs:line 541
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\RedirectHandler.cs:line 33
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\HttpClient.cs:line 571
   at Program.<>c__DisplayClass1_0.<<Run>b__7>d.MoveNext() in d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 206
--- End of stack trace from previous location where exception was thrown ---
   at Program.<>c__DisplayClass1_2.<<Run>b__28>d.MoveNext() in d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 400

The client failure here seems correct (the connection gets closed erroneously and the client is unable to read the frame header), and that the likely cause is some kind of concurrency problem on the server: "Reading is already in progress.". That's tracked by https://github.com/aspnet/AspNetCore/issues/11804.

[karelz]

@stephentoub is there anything left in this bug for 3.0?
Or do we want to spin off separate issues for tracking bug fixes?

[stephentoub]

There are still failures here that we believe are due to known issues in ASP.NET actively being worked on, but we don't know if those fixes will fully address all such failures or if there's something else lurking. We should keep this open until those fixes are in place, and then re-evaluate.

[karelz]

Thanks @stephentoub. Who will rerun the test(s) once we have the fixes? You or @Anipik / @eiriktsarpalis?
(Just trying to figure out who should track this issue and the dependent ASP.NET fixes)

[eiriktsarpalis]

Not sure if anybody has noticed, but I'm experiencing a stalling behaviour when I run the stress suite on my machine. After ~1200 requests, all of the request counters stop showing any progress. This lasts for 1 minute and 40 seconds, after which time _all_ client threads will throw one TaskCanceledException:

Error from iteration 253 (POST) in task 4 with 2589 successes / 24 fails:
System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Net.Http.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush)
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Program.<>c__DisplayClass1_0.<<Run>b__8>d.MoveNext() in C:\Users\eitsarpa\devel\dotnet\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 251
--- End of stack trace from previous location where exception was thrown ---
   at Program.<>c__DisplayClass1_3.<<Run>b__40>d.MoveNext() in C:\Users\eitsarpa\devel\dotnet\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 590

This might result in a bit of progress, but the test suite will quickly reach the same stalled state as before.

It reproduces pretty consistently on my machine, and happens irrespective of any previous failures.

The issue does not reproduce using http v1.1. I'm using dotnet sdk v3.0.100-preview8-013214.

[stephentoub]

@eiriktsarpalis, most likely this is https://github.com/dotnet/corefx/issues/39461#issuecomment-511171311. There are 12 operations, so after ~1200 requests we'll have done the GET Partial test 100 times, which is the magic number to trigger it. Try removing that op from your run and seeing if it goes away.

[eiriktsarpalis]

Thanks, verified as well that GET Partial was causing the issue.

[stephentoub]

With an SDK from last night, and omitting the cancellation and partial operations, things run much more smoothly for me. I do see sporadic failures, in particular in the POST Duplex Slow operation, e.g.

Error from iteration 84353 (POST Duplex Slow) in task 3 with 1019174 successes / 91 fails:
System.Net.Http.HttpRequestException: Error while copying content to a stream.
 ---> System.IO.IOException: The request was aborted.
 ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteAsync|210_1[TWriteAdapter](ValueTask writeTask, Byte[] bufferToReturn)
   at System.Net.Security.SslStream.WriteAsyncInternal[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
   at System.Net.Http.Http2Connection.FlushOutgoingBytesAsync()
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.Http2Stream.CheckIfDisposedOrAborted()
   at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span`1 buffer)
   at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, CancellationToken cancellationToken)
   at System.IO.Stream.CopyToAsyncInternal(Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
   at System.Net.Http.HttpContent.WaitAndReturnAsync[TState,TResult](Task waitTask, TState state, Func`2 returnFunc)
   at Program.<>c__DisplayClass1_0.<<Run>b__10>d.MoveNext() in d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 284
--- End of stack trace from previous location where exception was thrown ---
   at Program.<>c__DisplayClass1_3.<<Run>b__40>d.MoveNext() in d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 590

[stephentoub]

I'm going to close this uber issue, and we can open individual issues for additional problems. Thanks, all.

[davidfowl]

Any sever side issues pop up?

[stephentoub]

Any sever side issues pop up?

Hard to say: the "read already in progress" failure happens frequently and could be hiding other issues.

[eiriktsarpalis]

I just came across this error:

Error from iteration 2237 (POST ExpectContinue) in task 6 with 17227 successes / 8 fails:
System.Net.Http.HttpRequestException: Error while copying content to a stream.
 ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'CreditManager:Http2Connection:_connectionWindow'.
   at System.Net.Http.CreditManager.RequestCreditAsync(Int32 amount, CancellationToken cancellationToken)
   at System.Net.Http.Http2Connection.SendStreamDataAsync(Int32 streamId, ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
   at System.Net.Http.Http2Connection.Http2Stream.SendDataAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
   at System.Net.Http.HttpContent.CopyToAsyncCore(ValueTask copyTask)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.CopyToAsyncCore(ValueTask copyTask)
   at System.Net.Http.Http2Connection.Http2Stream.SendRequestBodyAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http2Connection.Http2Stream.SendRequestBodyWithExpect100ContinueAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Program.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in C:\Users\eitsarpa\devel\dotnet\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 666
   at Program.<>c__DisplayClass1_0.<<Run>b__10>d.MoveNext() in C:\Users\eitsarpa\devel\dotnet\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 281
--- End of stack trace from previous location where exception was thrown ---
   at Program.<>c__DisplayClass1_3.<<Run>b__38>d.MoveNext() in C:\Users\eitsarpa\devel\dotnet\corefx\src\System.Net.Http\tests\StressTests\HttpStress\Program.cs:line 565

Also lots of manifestations of error 9.

[jkotalik]

error 9 should be address by either https://github.com/aspnet/AspNetCore/pull/12204 or https://github.com/dotnet/corefx/pull/39228 (whatever is merged first).

[eiriktsarpalis]

I thought I'd update this thread with some issues I came across when running the suite in Ubuntu:

Repro steps:

• OS: Ubuntu 18.04 LTS
• Dotnet nightly build 5.0.100-alpha1-013628
• Using the stress suite from this PR, run without arguments.

HttpStress Run Final Report

[07/31/2019 20:34:25] Total: 503,081 Runtime: 00:15:10
        GET                           Success: 36,598   Canceled: 2,099 Fail: 1
        GET Partial                   Success: 36,552   Canceled: 2,145 Fail: 2
        GET Headers                   Success: 36,250   Canceled: 2,449 Fail: 0
        GET Parameters                Success: 36,494   Canceled: 2,205 Fail: 0
        GET Aborted                   Success: 36,515   Canceled: 2,184 Fail: 0
        POST                          Success: 36,622   Canceled: 2,077 Fail: 0
        POST Multipart Data           Success: 36,561   Canceled: 2,136 Fail: 2
        POST Duplex                   Success: 36,600   Canceled: 2,098 Fail: 1
        POST Duplex Slow              Success: 37,101   Canceled: 1,560 Fail: 38
        POST Duplex Dispose           Success: 36,557   Canceled: 2,142 Fail: 0
        POST ExpectContinue           Success: 36,469   Canceled: 2,222 Fail: 7
        HEAD                          Success: 36,570   Canceled: 2,127 Fail: 0
        PUT                           Success: 36,674   Canceled: 2,023 Fail: 0

There were a total of 51 failures classified into 4 different types:

Failure Type 1/4:
System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Net.Http.Http2Connection.AcquireWriteLockAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http2Connection.StartWriteAsync(Int32 writeBytes, CancellationToken cancellationToken)
   at System.Net.Http.Http2Connection.SendStreamDataAsync(Int32 streamId, ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
   at System.Net.Http.Http2Connection.Http2Stream.SendDataAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
   at System.Net.Http.HttpContent.CopyToAsyncCore(ValueTask copyTask)
   at System.Net.Http.Http2Connection.Http2Stream.SendRequestBodyAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in /home/eitsarpa/devel/dotnet/corefx/src/System.Net.Http/tests/StressTests/HttpStress/ClientOperations.cs:line 82
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_10>d.MoveNext() in /home/eitsarpa/devel/dotnet/corefx/src/System.Net.Http/tests/StressTests/HttpStress/ClientOperations.cs:line 350
--- End of stack trace from previous location where exception was thrown ---
   at HttpStress.StressClient.<>c__DisplayClass11_0.<<StartCore>g__RunWorker|0>d.MoveNext() in /home/eitsarpa/devel/dotnet/corefx/src/System.Net.Http/tests/StressTests/HttpStress/StressClient.cs:line 106

        POST Duplex Slow              : 1
        POST ExpectContinue           : 5
        TOTAL                         : 6

Failure Type 2/4:
System.Net.Http.HttpRequestException: Error while copying content to a stream.
 ---> System.IO.IOException: The request was aborted.
 ---> System.IO.IOException: Unable to read data from the transport connection: Broken pipe.
 ---> System.Net.Sockets.SocketException (32): Broken pipe
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteAsync|210_1[TWriteAdapter](ValueTask writeTask, Byte[] bufferToReturn)
   at System.Net.Security.SslStream.WriteAsyncInternal[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
   at System.Net.Http.Http2Connection.FlushOutgoingBytesAsync()
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
   at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span`1 buffer)
   at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)
   at System.IO.Stream.CopyToAsyncInternal(Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in /home/eitsarpa/devel/dotnet/corefx/src/System.Net.Http/tests/StressTests/HttpStress/ClientOperations.cs:line 82
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_8>d.MoveNext() in /home/eitsarpa/devel/dotnet/corefx/src/System.Net.Http/tests/StressTests/HttpStress/ClientOperations.cs:line 318
--- End of stack trace from previous location where exception was thrown ---
   at HttpStress.StressClient.<>c__DisplayClass11_0.<<StartCore>g__RunWorker|0>d.MoveNext() in /home/eitsarpa/devel/dotnet/corefx/src/System.Net.Http/tests/StressTests/HttpStress/StressClient.cs:line 106

        GET                           : 1
        POST Duplex                   : 1
        POST Duplex Slow              : 11
        POST ExpectContinue           : 2
        TOTAL                         : 15

Failure Type 3/4:
System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at System.Threading.SemaphoreSlim.WaitUntilCountOrTimeoutAsync(TaskNode asyncWaiter, Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Net.Http.Http2Connection.AcquireWriteLockAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http2Connection.StartWriteAsync(Int32 writeBytes, CancellationToken cancellationToken)
   at System.Net.Http.Http2Connection.SendStreamDataAsync(Int32 streamId, ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
   at System.Net.Http.Http2Connection.Http2Stream.SendDataAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
   at System.Net.Http.MultipartContent.SerializeToStreamAsyncCore(Stream stream, TransportContext context, CancellationToken cancellationToken)
   at System.Net.Http.HttpContent.CopyToAsyncCore(ValueTask copyTask)
   at System.Net.Http.Http2Connection.Http2Stream.SendRequestBodyAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in /home/eitsarpa/devel/dotnet/corefx/src/System.Net.Http/tests/StressTests/HttpStress/ClientOperations.cs:line 82
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_6>d.MoveNext() in /home/eitsarpa/devel/dotnet/corefx/src/System.Net.Http/tests/StressTests/HttpStress/ClientOperations.cs:line 290
--- End of stack trace from previous location where exception was thrown ---
   at HttpStress.StressClient.<>c__DisplayClass11_0.<<StartCore>g__RunWorker|0>d.MoveNext() in /home/eitsarpa/devel/dotnet/corefx/src/System.Net.Http/tests/StressTests/HttpStress/StressClient.cs:line 106

        POST Multipart Data           : 2
        TOTAL                         : 2

Failure Type 4/4:
System.Net.Http.HttpRequestException: Error while copying content to a stream.
 ---> System.IO.IOException: The request was aborted.
 ---> System.IO.IOException: The response ended prematurely, with at least 9 additional bytes expected.
   at System.Net.Http.Http2Connection.ReadAtLeastAsync(Stream stream, Memory`1 buffer, Int32 minReadBytes)
   at System.Net.Http.Http2Connection.EnsureIncomingBytesAsync(Int32 minReadBytes)
   at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame)
   at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
   at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span`1 buffer)
   at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)
   at System.IO.Stream.CopyToAsyncInternal(Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in /home/eitsarpa/devel/dotnet/corefx/src/System.Net.Http/tests/StressTests/HttpStress/ClientOperations.cs:line 82
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_8>d.MoveNext() in /home/eitsarpa/devel/dotnet/corefx/src/System.Net.Http/tests/StressTests/HttpStress/ClientOperations.cs:line 318
--- End of stack trace from previous location where exception was thrown ---
   at HttpStress.StressClient.<>c__DisplayClass11_0.<<StartCore>g__RunWorker|0>d.MoveNext() in /home/eitsarpa/devel/dotnet/corefx/src/System.Net.Http/tests/StressTests/HttpStress/StressClient.cs:line 106

        GET Partial                   : 2
        POST Duplex Slow              : 26
        TOTAL                         : 28

Note that the client is hitting an in-proc kestrel server, so not sure if this is hitting kestrel issues on linux.

[jkotalik]

The first two seem client related, the next two I'm not certain about. Can you file another issue for discoverability and potentially get server logs s.t. we can figure out what to do with these errors?

Also, did you hit these errors on Windows?

[eiriktsarpalis]

Also, did you hit these errors on Windows?

The issues were Kestrel related. Was able to make errors go away by stress testing the client on linux against a http.sys server. Reproduced the issues with linux client against windows kestrel too. Found similar issues in OSX.

I will follow-up with more concrete issues in the aspnet repo once I have isolated the problems.

[anurse]

I will follow-up with more concrete issues in the aspnet repo once I have isolated the problems.

Excellent. It will be useful to have information about how a customer could encounter these issues since that justification will be necessary to determine if fixes land in 3.0 or 5.0.