Runtime: System.Net.Quic initial API review

[geoffkizer]

This is the first cut at APIs for using the QUIC protocol.

A brief overview of QUIC is available here: https://blog.cloudflare.com/the-road-to-quic/

namespace System.Net.Quic.Experimental
{
    public sealed class QuicConnection : IDisposable
    {
        public QuicConnection(IPEndPoint remoteEndPoint, System.Net.Security.SslClientAuthenticationOptions sslClientAuthenticationOptions, IPEndPoint localEndPoint = null, bool mock = false);
        public System.Threading.Tasks.ValueTask ConnectAsync(System.Threading.CancellationToken cancellationToken = default);
        public bool Connected { get; }
        public IPEndPoint LocalEndPoint { get; }
        public IPEndPoint RemoteEndPoint { get; }
        public SslApplicationProtocol NegotiatedApplicationProtocol [ get; }
        public QuicStream CreateUnidirectionalStream();
        public QuicStream CreateBidirectionalStream();
        public System.Threading.Tasks.ValueTask<QuicStream> AcceptStreamAsync(System.Threading.CancellationToken cancellationToken = default);
        public void Close();
    }
    public sealed class QuicListener : IDisposable
    {
        public QuicListener(IPEndPoint listenEndPoint, System.Net.Security.SslServerAuthenticationOptions sslServerAuthenticationOptions, bool mock = false);
        public IPEndPoint ListenEndPoint { get; }
        public System.Threading.Tasks.ValueTask<QuicConnection> AcceptConnectionAsync(System.Threading.CancellationToken cancellationToken = default);
        public void Close();
    }
    public sealed class QuicStream : System.IO.Stream
    {
        internal QuicStream() { }
        public long StreamId { get; }
        public void ShutdownRead();
        public void ShutdownWrite();
    }
}

Notes:

Generally, this object model is inspired by the existing System.Net.Sockets API -- Connect, Accept, LocalEndPoint, RemoteEndPoint, etc. Some differences worth noting:

(1) QuicListener and QuicConnection take SSL options classes, since QUIC always uses TLS
(2) These classes use IPEndPoint, since they are IP-specific (vs Socket which uses EndPoint and AddressFamily etc)
(3) Since QUIC supports multiplexing, there is a distinction here between QuicConnection and QuicStream that doesn't exist in Sockets

The "mock" arguments to the QuicListener and QuicConnection constructors cause the "mock QUIC" implementation to be used. This is meant as a temporary measure while real QUIC protocol support is implemented and matures. This allows us to make progress on HTTP3 in parallel. These arguments will be removed before shipping.

There are currently no synchronous equivalents defined for async APIs like ConnectAsync. This matches HttpClient API definition.

I've added "Experimental" to the namespace here to position this work as experimental and not yet committed.

[geoffkizer]

cc @stephentoub @dotnet/ncl @jkotalik @anurse @davidfowl

[scalablecory]

Presumably the fields will actually be properties.

[scalablecory]

How will QuicListener.Close be different from QuicListener.Dispose? Same for QuicConnection.Close()

[scalablecory]

Does it make sense to implement IAsyncDisposable?

[scalablecory]

Is it conceivable that QUIC might ever be asked for in non-IP networking? Should those IPEndPoint be made EndPoint to allow for this?

[scalablecory]

(work for a separate API proposal, perhaps: it would be super super nice if NetworkStream and QuicStream both implemented some base class or interface that provides ShutdownRead and ShutdownWrite.)

[geoffkizer]

Presumably the fields will actually be properties.

Yep, fixed, thanks

[geoffkizer]

How will QuicListener.Close be different from QuicListener.Dispose? Same for QuicConnection.Close()

They are the same. That's how Socket works too. I suppose we could just get rid of Close; I don't have a strong opinion here

[geoffkizer]

Does it make sense to implement IAsyncDisposable?

At this point I don't see any reason to, but we can revisit in the future if necessary.

[geoffkizer]

Is it conceivable that QUIC might ever be asked for in non-IP networking? Should those IPEndPoint be made EndPoint to allow for this?

I think it's pretty doutbful. QUIC is explicitly defined in terms of UDP.

[scalablecory]

How will QuicListener.Close be different from QuicListener.Dispose? Same for QuicConnection.Close()

They are the same. That's how Socket works too. I suppose we could just get rid of Close; I don't have a strong opinion here

I feel like I've seen guidance to not do this anymore, but can't find it. @bartonjs anything in our guidelines about this?

[omariom]

Any plans to have PipeReader and PipeWriter in the API?

[bartonjs]

I feel like I've seen guidance to not do this anymore, but can't find it. @bartonjs anything in our guidelines about this?

We haven't made it to the chapter with the Dispose Pattern in it yet with our systemic analysis of the guidelines, but the current guideline is:

• CONSIDER providing method Close(), in addition to the Dispose(), if close is standard terminology in the area.

It's a CONSIDER, so if it feels wrong, don't do it (things can be added later easier than they can be removed). If you run across where we said "don't do that", it would be useful for justifying any changes to that guideline.

If you do add Close(), Close() and Dispose() are supposed to be identical.

[omariom]

Any plans to have PipeReader and PipeWriter in the API?

Something like this?

```C#
public sealed class QuicConnection
{
public QuicPipeWriter CreateUnidirectionalStream();
public QuicDuplexPipe CreateBidirectionalStream();

     // Not sure what should be returned here 
     // as the result can either be a unidirectional or a bidirectional stream.
    public ValueTask<QuicPipeReader|QuicDuplexPipe> AcceptStreamAsync(CancellationToken);
}

```

[jkotalik]

@omariom I don't think we are against having PipeReader/PipeWriter apis here, but we would first need to discuss adding Pipelines to Microsoft.NetCore.App first (right now it's a standalone package). We may have something that can be used in AspNetCore as well.

[jkotalik]

@geoffkizer I don't see how the bool Connected would be used reliably; an event or callback may be better (like OnConnected).

[jkotalik]

Does SslClientAuthenticationOptions apply to both client and server scenarios?

[jkotalik]

Why are QuicConnection, QuicStream, and QuicListener sealed? Can they be virtual?

[jkotalik]

As @scalablecory mentioned, DisposeAsync and CloseAsync instead of sync.

[omariom]

Connect instead of Create?
C# public QuicStream ConnectUnidirectionalStream(); public QuicStream ConnectBidirectionalStream();

[omariom]

In the API streams are created synchronously. What transitions them to Connected state?

[omariom]

Instead of having public ctor and ConnectAsync method on QuicConnection
there could be QuicConnector class with its ConnectAsync returning a QuicConnection
as QuicListener's AcceptConnectionAsync does.

[omariom]

@jkotalik Another argument for having pipes natively in .NET QUIC is that
if in the future MSQUIC team decides to use Registered IO or io_uring and expose zero-copy abilities,
only pipes API will allow to get advantage of that.

[geoffkizer]

QuicStream.Connected shouldn't exist; I've removed it above.

A QUIC stream doesn't really have a separate Connect operation. You just start sending on it.

[anurse]

public QuicConnection(IPEndPoint remoteEndPoint, System.Net.Security.SslClientAuthenticationOptions sslClientAuthenticationOptions, IPEndPoint localEndPoint = null, bool mock = false);

I think there may be value in other EndPoint types at some point, but we can add ctors later. One thing that comes to mind is that we could use an EndPoint to coordinate the mock in-memory version. If we add a ctor like this:

public QuicConnection(MemoryEndPoint memoryEndPoint, System.Net.Security.SslClientAuthenticationOptions sslClientAuthenticationOptions)

Where MemoryEndPoint is mostly an "opaque" type from a user perspective that helps coordinate things so that two QuicConnection that are given the same MemoryEndPoint can talk to each other. Kestrel and HttpClient can be tested in memory then by using the same instance.

Since that's all just temporary during feature development, it wouldn't need a huge amount of rigorous design, as long as we were very clear on when we were getting rid of it. It could even be marked Obsolete in advance or something 🐱.

        public QuicStream CreateUnidirectionalStream();
        public QuicStream CreateBidirectionalStream();

I like the name Open, but that's just a preference.

Any thoughts on an API that lets you set the stream ID? I'm thinking of possible protocols that assign semantics to specific stream IDs. HTTP/3 doesn't do that (instead you have to send a "stream type" message as the first message on a stream) but some protocol might. We would probably want to validate that the stream ID is valid for the purpose (which may mean the connection has to know if it's the client or the server...). Just a thought.

public int StreamId { get; }

Should probably be a long? Or even a ulong (though we generally don't use those)? The largest stream ID is 2^62-1 (Source).

        public void ShutdownRead();
        public void ShutdownWrite();

Socket has a single Shutdown API with an enum to select how to shutdown (Read, Write or Both). Should we do the same? We could even borrow the same enum.

public sealed class QuicStream : System.IO.Stream

Not sure I like inheriting here. I'd prefer a model where the stream is exposed as a property. That could also give us the option to expose pipes later and change which is "authoritative" (i.e. change to a Pipe and make the Stream a wrapper around the Pipe, or vice versa).

[geoffkizer]

One thing that comes to mind is that we could use an EndPoint to coordinate the mock in-memory version.

I agree this is interesting. IPEndPoint is not sealed, so I think we could just derive a "mock" endpoint type from it.

[geoffkizer]

I like the name Open, but that's just a preference.

Open is good; I think I may prefer it to Create too. But I don't have a strong opinion.

[geoffkizer]

Any thoughts on an API that lets you set the stream ID?

My (perhaps naive) assumption is that higher-level protocols won't rely on this. I think at one point HTTP3 did, but changed it so they don't anymore. It's problematic in some ways and using the HTTP3 approach seems better in general.

That said, it seems straightforward to add in the future if necessary.

[geoffkizer]

[StreamId] Should probably be a long

Yep -- updated above.

[bartonjs]

As @scalablecory mentioned, DisposeAsync and CloseAsync instead of sync.

Not "instead of". Our (unpublished) async guidance is to add a sync version of every async thing (for high level types).

Sync-only: OK. Async-only: not OK (except speciality low-level things)

[geoffkizer]

Does SslClientAuthenticationOptions apply to both client and server scenarios?

SslClientAuthenticationOptions applies to client scenarios. SslServerAuthenticationOptions applies to server scenarios.

[geoffkizer]

Socket has a single Shutdown API with an enum to select how to shutdown (Read, Write or Both).

I prefer just having two different APIs, it seems simpler that way. Both is equivalent to Close().

[stephentoub]

CreateUnidirectionalStream()

I understand the meaning of "bidirectional" stream. But with "unidirectional"... which direction is it? System.IO.Pipes uses a PipeDirection enum for In, Out, and InOut... maybe something like that is relevant here? (Or maybe I'm totally misunderstanding the concept.)

I suppose we could just get rid of Close

I think we should get rid of Close. We can always add it later if it proves meaningful, but I doubt it will.

[geoffkizer]

Instead of having public ctor and ConnectAsync method on QuicConnection
there could be QuicConnector class with its ConnectAsync returning a QuicConnection
as QuicListener's AcceptConnectionAsync does.

Yeah, that's definitely a possibility. I'm not sure it adds enough value to be worth creating another class, though.

[jkotalik]

SslClientAuthenticationOptions applies to client scenarios. SslServerAuthenticationOptions applies to server scenarios.

Makes sense. Will AcceptConnectionAsync use an internal constructor to new up the QuicConnection?

Sync-only: OK. Async-only: not OK (except speciality low-level things)

Shutdown and Close in MsQuic I believe are always async, so are we okay with having both if the synchronous ones will always be sync over async?

[anurse]

My (perhaps naive) assumption is that higher-level protocols won't rely on this. I think at one point HTTP3 did, but changed it so they don't anymore.

Fair point, and that's why I'm kinda waffling on it. We could add the API later as an overload.

I understand the meaning of "bidirectional" stream. But with "unidirectional"... which direction is it?

It's known what the direction is. Since you are initiating it, it means you are writing to it. Bidirectional just allows the remote party to also send on that stream. ("Unidirectional streams carry data in one direction: from the initiator of the stream to its peer." Source)

[geoffkizer]

But with "unidirectional"... which direction is it?

It's always outbound. That is, the peer that creates a unidirectional stream can write to it but not read; the peer that accepts the unidirectional stream can read but not write.

There is no way for a peer to create an inbound-only stream in QUIC.

"Unidirectional" and "bidirectional" are the terms the RFC uses so I'm inclined to stick with them.

[scalablecory]

I prefer just having two different APIs, it seems simpler that way. Both is equivalent to Close().

I also think I prefer two APIs, though symmetry with Socket.Shutdown wouldn't be a terrible thing.

[scalablecory]

I prefer just having two different APIs, it seems simpler that way. Both is equivalent to Close().

My assumption was that Close/Dispose would have behavior similar to Socket.Dispose, but it seems that was incorrect. (I bet users will make the same mistake -- we need better naming in that case)

How do we achieve a reset stream VS a shutdown & cleanly closed stream?

[geoffkizer]

My assumption was that Close/Dispose would have behavior similar to Socket.Dispose

In what sense? Specifically re graceful vs abortive shutdown? Close/Dispose on Socket is not necessarily abortive, and Shutdown is not necessarily graceful.

How do we achieve a reset stream VS a shutdown & cleanly closed stream?

Yeah, I agree we need to clarify how this works. Let me think about it a bit. QUIC is a bit different than TCP here in that you can reset one direction without resetting the other.

[stephentoub]

There is no way for a peer to create an inbound-only stream in QUIC.

Got it.

[omariom]

If the connection has been migrated, does it change its RemoteEndPoint?
Should ConnectionId be exposed as well/instead?

[davidfowl]

Doesn’t shutdown need to be fully asynchronous or at least allow for it? This is where it helps to have multiple implementations, it usually shows other ways APIs can be used.

[anurse]

Should ConnectionId be exposed as well/instead?

The extra complexity there is that a peer in a QUIC connection can have multiple connection IDs.

We should be considering how connection migration may affect these APIs, though I think it's reasonable to call some of that out-of-scope for 5.0. The LocalEndPoint and RemoteEndPoint could both change (though the LocalEndPoint would only change via user action, we probably don't need to worry as much about that). The set of remote ConnectionId values can change at any time (via NEW_CONNECTION_ID and RETIRE_CONNECTION_ID frames). I believe we have more control on the local ones (depending on our implementation).

I think that if we leave ConnectionId off for now, and indicate via docs that the RemoteEndPoint value may change if the connection is migrated, we probably cover enough mileage to get us in a good state for 5.0 in re: connection migration.

[geoffkizer]

I added QuicConnection.NegotiatedApplicationProtocol above, to match SslStream.NegotiatedApplicationProtocol.

As @Tratcher pointed out on the linked issue, there are a bunch of TLS-related properties that SslStream exposes that we'll want to have here as well. These include LocalCertificate, RemoteCertificate, SslVersion, etc. We should review these and add as appropriate.

[am11]

Will QUIC, at some point, use UDP at its core in implementation detail, as it is described in its design?

ps: dotnet UdpClient did not get any span/memory improvements, I was thinking if QUIC implementation depends on it, that might improve existing UDP APIs as well?

[Tratcher]

@am11 yes, we have PRs that are UDP based (https://github.com/dotnet/runtime/pull/427). They do not go through UdpClient though, they use native libraries that implement the QUIC protocol over UDP.

[karelz]

The initial API was checked in. We will do full API later when QUIC is mature and finalized (post 5.0).