Runtime: Low-level HTTP APIs

[scalablecory]

We've got desires for a) code sharing between HttpClient and Kestrel, and b) very precise control for partners with advanced needs (e.g. reverse proxies).

What do we think of an XmlReader-like API for low level HTTP? Minimal allocation, no connection pooling, no proxy support, etc. -- something like:

```c#
HttpStreamReader reader = ...;
HttpStreamWriter writer = ...;
ReadOnlySpan uri = ...;

await writer.WriteRequestAsync(HttpMethod.Get, uri);
await writer.WriteHeaderAsync(..., ...);
await writer.FlushMessage();

HttpTokenType token = await reader.ReadNextAsync();
Debug.Assert(token == HttpTokenType.Response);

if(reader.MessageStatusCode != 200)
{
throw ...;
}

while((token = await reader.ReadNextAsync()) == HttpTokenType.Header)
{
Span name = reader.HeaderName;
Span value = reader.HeaderValue;
UseHeader(name, value);
}

while(token == HttpTokenType.Content)
{
UseContent(reader.Content);
token = await reader.ReadNextAsync();
}

while(token == HttpTokenType.TrailingHeader)
{
UseHeader(reader.HeaderName, reader.HeaderValue);
token = await reader.ReadNextAsync();
}

Debug.Assert(token == HttpTokenType.MessageComplete);
```

@geoffkizer @stephentoub @davidfowl @davidsh @anurse

[Joe4evr]

Isn't this the entire point of ASP.NET's Project Bedrock?

[scalablecory]

@Joe4evr Bedrock is a connection establishment abstraction, not a HTTP API.

[Joe4evr]

Yeah, but the HTTP APIs would built on top of Bedrock.

[davidfowl]

What do we think of an XmlReader-like API for low level HTTP? Minimal allocation, no connection pooling, no proxy support, etc. -- something like:

You'd need a reader and writer API for HTTP/1 and 2 (maybe 3 as well?) for server and client. I worry about making the writing and reader of each header asynchronous. I also worry about using this for reading the body. We have lots of optimizations to avoid copying and this would likely negate that (though we could just skip the body altogether) or have different types for those readers.

A callback API, though much less convenient, would likely be more efficient here.

[Rodrigo-Andrade]

OMG plz do! Several bottlenecks in many projects i work are on HttpClient.

[scalablecory]

@Rodrigo-Andrade do you have more info? We'd be interested in hearing...

[Rodrigo-Andrade]

@scalablecory sory, for some reason i didn't see this sooner.

We have a bunch of microservices and an API Gateway that runs on dotnet core. I am kind of the "performance guy" and every time there are issues of performance i get to profile a different system on a different team. After most obvious optimizations, most performance to win are from HttpClient.

Things got really better, but making http requests from the HttpClient model is still too costly.

I cobbled up a simple http1.1 client that departs from HttpRequestMessage and the win was obvious. I could buffer most stuff, bypass most parsing and checking (since most requests can be proven well formed), allocating way less. Many workloads have more headers and long uris then big bodies, so serialization is a lesser issue (specially with serializers like MessagePack)

The cobbled up client is already a lot of code (even when its only http1.1, no ssl suport etc), so it would be nice to have lower level apis for this stuff, so we could benefit from all the work on http2 and 3 you guys are doing.

Tell me if you need more details (ill be sure to be checking this)