Runtime: Performance of multi-buffer socket write APIs (scatter-gather)

Created on 24 May 2020 · 9Comments · Source: dotnet/runtime

As part of looking at multiplexing and perf for Linux, I did a quick benchmark of "gather socket I/O", i.e. writing multiple buffers to a socket in one go. This would open up some interesting possibilities to further push perf (and also considerably simplify things for the design). Here are some quick results and observations:

Socket.SendPacketsAsync seems to be particularly bad on Linux, since... it doesn't actually write the buffers in one go, but rather writes them one-by-one.
Socket.SendPacketsAsync also requires providing a list of SendPacketsElement, which is an immutable reference type. That means that unless the buffers and offsets/counts are completely fixed, heap allocations are necessary for each write.
Socket.SendAsync with SocketAsyncEventArgs.BufferList is much better ("only" 3 times slower than regular single-buffer). However, each time SocketAsyncEventArgs.BufferList is set, the whole array is copied internally to avoid issues with subsequent changes. This again means that it's impossible to change the buffers/counts/offsets without triggering an expensive re-copying. Note that this re-copying is not done in the benchmark below, which reuses the same buffer list.
The parameters (NumBuffers/BufferSize) are currently set to a typical Npgsql muiltiplexing scenario for TE Fortunes, but can obviously be played with.

In other words, there's definitely work needed (and possibly new APIs) if these are to be useful in a high-perf scenario. Hopefully I didn't do miss anything or do any silliness in the benchmark, let me know.

Results:

BenchmarkDotNet=v0.12.0, OS=ubuntu 20.04
Intel Xeon W-2133 CPU 3.60GHz, 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=5.0.100-preview.3.20216.6
  [Host]     : .NET Core 3.1.1 (CoreCLR 4.700.19.60701, CoreFX 4.700.19.60801), X64 RyuJIT
  DefaultJob : .NET Core 3.1.1 (CoreCLR 4.700.19.60701, CoreFX 4.700.19.60801), X64 RyuJIT

| Method | NumBuffers | BufferSize | Mean | Error | StdDev |
|--------------------------------- |----------- |----------- |----------:|----------:|----------:|
| NetworkStreamSingleBuffer | 25 | 36 | 1.198 us | 0.0114 us | 0.0106 us |
| SocketAsyncEventArgsSingleBuffer | 25 | 36 | 1.145 us | 0.0110 us | 0.0092 us |
| SocketAsyncEventArgsBufferList | 25 | 36 | 3.616 us | 0.0158 us | 0.0132 us |
| SocketAsyncEventArgsSendPackets | 25 | 36 | 42.723 us | 0.2419 us | 0.2145 us |

Benchmark code

```c#
public class Program
{
const int Port = 9595;
const string Address = "127.0.0.1";

DummyServer _server;
Socket _socket;
SocketAsyncEventArgs _args;
NetworkStream _networkStream;
ManualResetEventSlim _mre;

[Params(25)]
public int NumBuffers { get; set; }

[Params(36)]
public int BufferSize { get; set; }

Memory<byte> _singleBuffer;

public void SetupServerAndConnect()
{
    _server = new DummyServer();
    _server.Start();
    _socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
    _socket.Connect(new IPEndPoint(IPAddress.Parse(Address), Port));
}

[GlobalCleanup]
public async Task Cleanup()
{
    _args?.Dispose();
    _args = null;
    _socket.Close();
    _socket = null;
    _networkStream = null;
    await _server.FinishTask;
}

[GlobalSetup(Target = nameof(NetworkStreamSingleBuffer))]
public void SetupNetworkStreamSingleBuffer()
{
    SetupServerAndConnect();
    _networkStream = new NetworkStream(_socket, ownsSocket: false);
    _singleBuffer = new byte[NumBuffers * BufferSize];
}

[GlobalSetup(Target = nameof(SocketAsyncEventArgsSingleBuffer))]
public void SetupSocketAsyncEventArgsSingleBuffer()
{
    SetupServerAndConnect();
    _args = new SocketAsyncEventArgs();
    _args.SetBuffer(new byte[NumBuffers * BufferSize]);
    _mre = new ManualResetEventSlim();
    _args.Completed += (s, e) => _mre.Set();
}

[GlobalSetup(Target = nameof(SocketAsyncEventArgsBufferList))]
public void SetupSocketAsyncEventArgsBufferList()
{
    SetupServerAndConnect();
    var bufferList = new List<ArraySegment<byte>>(NumBuffers);
    for (var i = 0; i < NumBuffers; i++)
        bufferList.Add(new byte[BufferSize]);
    _args = new SocketAsyncEventArgs
    {
        BufferList = bufferList
    };
    _mre = new ManualResetEventSlim();
    _args.Completed += (s, e) => _mre.Set();
}

[GlobalSetup(Target = nameof(SocketAsyncEventArgsSendPackets))]
public void SetupSocketAsyncEventArgsSendPackets()
{
    SetupServerAndConnect();
    _args = new SocketAsyncEventArgs
    {
        SendPacketsElements = new SendPacketsElement[NumBuffers]
    };
    for (var i = 0; i < NumBuffers; i++)
        _args.SendPacketsElements[i] = new SendPacketsElement(new byte[BufferSize]);
    _mre = new ManualResetEventSlim();
    _args.Completed += (s, e) => _mre.Set();
}

[Benchmark]
public void NetworkStreamSingleBuffer()
{
    var t = _networkStream.WriteAsync(_singleBuffer);
    if (!t.IsCompletedSuccessfully)
        throw new Exception("Non-synchronous completion");
}

[Benchmark]
public void SocketAsyncEventArgsSingleBuffer()
{
    // Never completes asynchronously, but adding MRE and event handling because SendPacketsAsync needs it
    _mre.Reset();
    if (_socket.SendAsync(_args))
        _mre.Wait();
}

[Benchmark]
public void SocketAsyncEventArgsBufferList()
{
    // Never completes asynchronously, but adding MRE and event handling because SendPacketsAsync needs it
    _mre.Reset();
    if (_socket.SendAsync(_args))
        _mre.Wait();
}

[Benchmark]
public void SocketAsyncEventArgsSendPackets()
{
    // Note: SendPacketsAsync seems to always returns true (non-synchronous completion)
    _mre.Reset();
    if (_socket.SendPacketsAsync(_args))
        _mre.Wait();
}

static void Main() => BenchmarkRunner.Run<Program>();

class DummyServer
{
    TcpListener _listener;

    public Task FinishTask { get; private set; }

    public void Start()
    {
        _listener = new TcpListener(IPAddress.Parse(Address), Port);
        _listener.Start();
        FinishTask = Task.Run(Process);
    }

    void Process()
    {
        var client = _listener.AcceptTcpClient();
        var buf = new byte[8192];
        var stream = client.GetStream();
        while (stream.Read(buf, 0, buf.Length) != 0)
        {
            // Do nothing with the data
        }
    }
}

}
```

/cc @stephentoub @davidfowl @ajcvickers

api-suggestion area-System.Net.Sockets tenet-performance

Source

roji

Most helpful comment

@roji Linux defines UIO_FASTIOV to 8. This means up to 8 iovs (NumBuffers) will be stackalloced in the kernel. It is an interesting nr to try in the benchmarks.

As the Windows version of this does. Not sure why this one wasn't changed similarly.

@stephentoub I will take a look at doing this for Unix too.

tmds on 25 May 2020

❤2 👍1

All 9 comments

Tagging subscribers to this area: @dotnet/ncl
Notify danmosemsft if you want to be subscribed.

msftbot[bot] on 24 May 2020

the whole array is copied internally to avoid issues with subsequent changes. This again means that it's impossible to change the buffers/counts/offsets without triggering an expensive re-copying

How many buffers are you using such that this copy shows up in profiles as being "expensive"? This is copying the buffer references/bounds, not the contents of the buffers.

stephentoub on 24 May 2020

Socket.SendAsync with SocketAsyncEventArgs.BufferList is much better ("only" 3 times slower than regular single-buffer).

From a quick look at the benchmark, you're always copying N bytes, and it's either all in one buffer or split across M buffers? All in one buffer is always going to be faster. The interesting thing to measure/compare is the overhead it takes to get M buffers into one buffer and sending that vs sending the M buffers sequentially vs sending them all at once.

Or am I misunderstanding what's being demonstrated here?

stephentoub on 24 May 2020

Socket.SendPacketsAsync seems to be particularly bad on Linux, since... it doesn't actually write the buffers in one go, but rather writes them one-by-one.

This could certainly be improved. The API that exists is closely tied to the Windows API it was originally wrapping, and supports "packets" that are either buffers or files. We need to maintain the relative ordering, and we use sendfile for the files. But the implementation could be changed to build up a list of adjacent buffers and send those in one shot. We'd need to measure to see at what point the overhead of doing so is worthwhile.

stephentoub on 24 May 2020

I probably should have provided a bit more context here. In my current prototype, multiple producers encode directly into different segments of a single large buffer, which is then written and flushed; the disadvantage is that encoding is serialized, since lengths aren't necessarily known in advance, and management of multiple encoding producers would be quite complicated. The ideal situation is for each producer to encode on its own (pooled) buffer, which then gets written/flushed along with the others via a single gather I/O operation. Hence the baseline here was comparing to a single buffer write, without implying that the gather operation should be faster or equal to a one-buffer write.

I've updated the benchmark as suggested, parameterized different buffer numbers and sizes, and executed on both Linux and Windows (different machines though!).

| Benchmark name | Description |
|-------------------------|-----------------|
| NetworkStreamSingleBuffer | Send a single large buffer
| NetworkStreamMultipleBuffers | Send multiple buffers sequentially
| NetworkStreamCopyMultipleBuffersIntoSingle | Copy multiple buffers into a single, large rented array, and send that
| SocketAsyncEventArgsBufferList | Send multiple buffers in one go with the BufferList API
| SocketAsyncEventArgsSendPackets | Send multiple buffers in one go with the SendPackets API

All tests send the same amount of bytes per method call. Full results are below, here are some observations:

BufferList allocates on Linux, but not on Windows
On Linux, copying multiple buffers is better than BufferList for smaller buffer sizes (100 < N < 1000), regardless of the number of buffers. On Windows, BufferList is (almost) always better than copying. This may suggest possible room for improvement in the BufferList implementation for Linux.
In some cases, NetworkStreamSingleBuffer performs better than SocketAsyncEventArgsBufferList. In these cases, it seems that NetworkStreamSingleBuffer allocates.
On Windows, SendPackets is much better than on Linux (as already mentioned), but still usually performs worse than BufferList (and also allocates), except with the bigger buffer sizes (1000 < N < 10000). The number of buffers doesn't seem to matter.
SendPackets can be tweaked via SendPacketsFlags, though that looks like it's specific to the file sending mode of SendPackets, not to memory buffer (not sure).

Linux results

BenchmarkDotNet=v0.12.0, OS=ubuntu 20.04
Intel Xeon W-2133 CPU 3.60GHz, 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=5.0.100-preview.3.20216.6
  [Host]     : .NET Core 3.1.1 (CoreCLR 4.700.19.60701, CoreFX 4.700.19.60801), X64 RyuJIT
  DefaultJob : .NET Core 3.1.1 (CoreCLR 4.700.19.60701, CoreFX 4.700.19.60801), X64 RyuJIT

Method | NumBuffers | BufferSize | Mean | Error | StdDev | Median | Gen 0 | Gen 1 | Gen 2 | Allocated |
|----------- |----------- |-----------:|----------:|-----------:|-----------:|-------:|------:|------:|----------:|
Buffer | 10 | 10 | 1.681 us | 0.0333 us | 0.0356 us | 1.684 us | - | - | - | - |
10 | 10 | 16.244 us | 0.3210 us | 0.3153 us | 16.119 us | - | - | - | - |
10 | 10 | 1.683 us | 0.1376 us | 0.4058 us | 1.895 us | - | - | - | - |
10 | 10 | 3.185 us | 0.0341 us | 0.0302 us | 3.179 us | 0.0648 | - | - | 288 B |
10 | 10 | 17.036 us | 0.1362 us | 0.1138 us | 17.038 us | 0.2136 | - | - | 1017 B |
Buffer | 10 | 100 | 1.253 us | 0.0029 us | 0.0026 us | 1.254 us | - | - | - | - |
10 | 100 | 15.294 us | 0.8241 us | 2.4170 us | 16.354 us | - | - | - | - |
10 | 100 | 1.430 us | 0.0084 us | 0.0074 us | 1.433 us | - | - | - | - |
10 | 100 | 2.309 us | 0.0131 us | 0.0109 us | 2.311 us | 0.0648 | - | - | 288 B |
10 | 100 | 16.818 us | 0.7963 us | 2.2329 us | 17.794 us | 0.2136 | - | - | 1016 B |
Buffer | 10 | 1000 | 3.098 us | 0.0618 us | 0.0962 us | 3.076 us | - | - | - | - |
10 | 1000 | 12.139 us | 0.0391 us | 0.0346 us | 12.131 us | - | - | - | - |
10 | 1000 | 5.060 us | 0.0998 us | 0.1668 us | 5.131 us | - | - | - | - |
10 | 1000 | 4.140 us | 0.0728 us | 0.0681 us | 4.143 us | 0.0610 | - | - | 288 B |
10 | 1000 | 13.068 us | 0.1886 us | 0.1672 us | 13.064 us | 0.2289 | - | - | 1016 B |
Buffer | 10 | 10000 | 22.499 us | 0.4488 us | 1.1094 us | 22.485 us | - | - | - | 2 B |
10 | 10000 | 31.141 us | 0.5615 us | 0.4689 us | 31.300 us | - | - | - | 1 B |
10 | 10000 | 30.042 us | 0.5942 us | 1.1448 us | 30.232 us | - | - | - | - |
10 | 10000 | 22.884 us | 0.4510 us | 0.9415 us | 22.855 us | 0.0610 | - | - | 290 B |
10 | 10000 | 31.418 us | 0.6147 us | 1.0099 us | 31.234 us | 0.1831 | - | - | 1017 B |
Buffer | 20 | 10 | 1.050 us | 0.0326 us | 0.0931 us | 1.007 us | - | - | - | - |
20 | 10 | 31.815 us | 0.2820 us | 0.2500 us | 31.795 us | - | - | - | - |
20 | 10 | 1.656 us | 0.1561 us | 0.4602 us | 1.438 us | - | - | - | - |
20 | 10 | 5.307 us | 0.1056 us | 0.2488 us | 5.320 us | 0.1221 | - | - | 528 B |
20 | 10 | 34.372 us | 0.3223 us | 0.3015 us | 34.268 us | 0.4272 | - | - | 1896 B |
Buffer | 20 | 100 | 1.472 us | 0.0066 us | 0.0058 us | 1.470 us | - | - | - | - |
20 | 100 | 30.565 us | 1.8321 us | 5.4020 us | 33.213 us | - | - | - | - |
20 | 100 | 1.706 us | 0.0068 us | 0.0063 us | 1.705 us | - | - | - | - |
20 | 100 | 3.654 us | 0.0196 us | 0.0184 us | 3.659 us | 0.1221 | - | - | 528 B |
20 | 100 | 30.806 us | 2.0281 us | 5.9798 us | 34.251 us | 0.4272 | - | - | 1896 B |
Buffer | 20 | 1000 | 5.134 us | 0.1023 us | 0.2090 us | 5.130 us | - | - | - | - |
20 | 1000 | 24.738 us | 0.4434 us | 0.4147 us | 24.732 us | - | - | - | - |
20 | 1000 | 8.631 us | 0.1723 us | 0.2683 us | 8.573 us | - | - | - | - |
20 | 1000 | 6.891 us | 0.1000 us | 0.0886 us | 6.848 us | 0.1221 | - | - | 528 B |
20 | 1000 | 25.447 us | 0.1552 us | 0.1376 us | 25.402 us | 0.4272 | - | - | 1896 B |
Buffer | 20 | 10000 | 43.661 us | 0.8650 us | 1.3210 us | 43.957 us | - | - | - | 4 B |
20 | 10000 | 60.014 us | 1.1742 us | 1.0984 us | 60.079 us | - | - | - | 2 B |
20 | 10000 | 54.904 us | 1.0826 us | 2.3071 us | 54.624 us | - | - | - | 1 B |
20 | 10000 | 41.990 us | 0.8255 us | 1.3093 us | 41.916 us | 0.1221 | - | - | 528 B |
20 | 10000 | 61.259 us | 1.4364 us | 1.4107 us | 61.050 us | 0.3662 | - | - | 1897 B |
Buffer | 30 | 10 | 1.057 us | 0.0144 us | 0.0113 us | 1.054 us | - | - | - | - |
30 | 10 | 49.629 us | 0.9878 us | 1.2131 us | 49.217 us | - | - | - | - |
30 | 10 | 1.409 us | 0.0282 us | 0.0602 us | 1.382 us | - | - | - | - |
30 | 10 | 4.709 us | 0.3219 us | 0.9492 us | 4.342 us | 0.1755 | - | - | 768 B |
30 | 10 | 53.295 us | 1.0575 us | 1.7375 us | 52.644 us | 0.6104 | - | - | 2776 B |
Buffer | 30 | 100 | 1.692 us | 0.0064 us | 0.0057 us | 1.691 us | - | - | - | - |
30 | 100 | 44.571 us | 3.0315 us | 8.9385 us | 49.222 us | - | - | - | - |
30 | 100 | 2.195 us | 0.0305 us | 0.0286 us | 2.198 us | - | - | - | - |
30 | 100 | 4.931 us | 0.0334 us | 0.0296 us | 4.940 us | 0.1755 | - | - | 768 B |
30 | 100 | 48.878 us | 1.9808 us | 5.8406 us | 51.348 us | 0.6104 | - | - | 2776 B |
Buffer | 30 | 1000 | 6.896 us | 0.1521 us | 0.2030 us | 6.852 us | - | - | - | - |
30 | 1000 | 37.226 us | 0.3521 us | 0.3294 us | 37.185 us | - | - | - | - |
30 | 1000 | 9.256 us | 0.1843 us | 0.4007 us | 9.240 us | - | - | - | - |
30 | 1000 | 9.996 us | 0.1969 us | 0.3290 us | 10.015 us | 0.1678 | - | - | 768 B |
30 | 1000 | 37.859 us | 0.1902 us | 0.1485 us | 37.917 us | 0.6104 | - | - | 2776 B |
Buffer | 30 | 10000 | 65.492 us | 1.2892 us | 2.1892 us | 65.761 us | - | - | - | 5 B |
30 | 10000 | 89.947 us | 1.7124 us | 1.6818 us | 90.394 us | - | - | - | 1 B |
30 | 10000 | 106.631 us | 2.4397 us | 7.1554 us | 106.002 us | - | - | - | 4 B |
30 | 10000 | 61.315 us | 1.2136 us | 2.9771 us | 61.580 us | 0.1221 | - | - | 779 B |
30 | 10000 | 93.916 us | 1.8228 us | 1.8719 us | 93.266 us | 0.6104 | - | - | 2777 B |
Buffer | 40 | 10 | 1.112 us | 0.0171 us | 0.0151 us | 1.110 us | - | - | - | - |
40 | 10 | 64.648 us | 1.0430 us | 0.9246 us | 64.515 us | - | - | - | - |
40 | 10 | 1.589 us | 0.0434 us | 0.1266 us | 1.581 us | - | - | - | - |
40 | 10 | 7.375 us | 0.0524 us | 0.0490 us | 7.367 us | 0.2289 | - | - | 1008 B |
40 | 10 | 67.361 us | 0.9693 us | 0.9067 us | 67.433 us | 0.7324 | - | - | 3656 B |
Buffer | 40 | 100 | 1.818 us | 0.0347 us | 0.0341 us | 1.815 us | - | - | - | - |
40 | 100 | 60.379 us | 3.5290 us | 10.4052 us | 65.571 us | - | - | - | - |
40 | 100 | 2.198 us | 0.0106 us | 0.0094 us | 2.201 us | - | - | - | - |
40 | 100 | 6.228 us | 0.0207 us | 0.0184 us | 6.229 us | 0.2289 | - | - | 1008 B |
40 | 100 | 62.662 us | 3.1433 us | 9.2681 us | 67.246 us | 0.7935 | - | - | 3656 B |
Buffer | 40 | 1000 | 9.133 us | 0.1765 us | 0.2295 us | 9.068 us | - | - | - | - |
40 | 1000 | 48.641 us | 0.4517 us | 0.4004 us | 48.591 us | - | - | - | - |
40 | 1000 | 17.113 us | 0.3383 us | 0.5064 us | 17.107 us | - | - | - | - |
40 | 1000 | 12.708 us | 0.2476 us | 0.2316 us | 12.670 us | 0.2289 | - | - | 1009 B |
40 | 1000 | 50.512 us | 0.1785 us | 0.1670 us | 50.471 us | 0.7935 | - | - | 3656 B |
Buffer | 40 | 10000 | 88.075 us | 1.7412 us | 3.0496 us | 88.145 us | - | - | - | 7 B |
40 | 10000 | 117.651 us | 2.2418 us | 2.4917 us | 117.356 us | - | - | - | 1 B |
40 | 10000 | 112.583 us | 2.7073 us | 7.9824 us | 111.997 us | - | - | - | 4 B |
40 | 10000 | 78.671 us | 1.7015 us | 5.0168 us | 78.457 us | 0.1221 | - | - | 1010 B |
40 | 10000 | 121.491 us | 2.3513 us | 2.8877 us | 121.659 us | 0.7324 | - | - | 3658 B |
Buffer | 50 | 10 | 1.143 us | 0.0224 us | 0.0343 us | 1.123 us | - | - | - | - |
50 | 10 | 81.149 us | 0.7877 us | 0.7368 us | 81.007 us | - | - | - | - |
50 | 10 | 1.573 us | 0.0314 us | 0.0875 us | 1.531 us | - | - | - | - |
50 | 10 | 8.475 us | 0.1018 us | 0.0903 us | 8.460 us | 0.2747 | - | - | 1248 B |
50 | 10 | 83.192 us | 0.5231 us | 0.4893 us | 83.153 us | 0.9766 | - | - | 4536 B |
Buffer | 50 | 100 | 1.982 us | 0.0220 us | 0.0195 us | 1.990 us | - | - | - | - |
50 | 100 | 75.753 us | 3.7625 us | 11.0939 us | 80.621 us | - | - | - | - |
50 | 100 | 3.133 us | 0.0623 us | 0.0667 us | 3.098 us | - | - | - | - |
50 | 100 | 7.461 us | 0.0346 us | 0.0307 us | 7.464 us | 0.2823 | - | - | 1248 B |
50 | 100 | 75.057 us | 5.5594 us | 16.3921 us | 84.178 us | 0.9766 | - | - | 4536 B |
Buffer | 50 | 1000 | 10.730 us | 0.2130 us | 0.2279 us | 10.659 us | - | - | - | - |
50 | 1000 | 61.631 us | 0.6201 us | 0.5801 us | 61.501 us | - | - | - | - |
50 | 1000 | 17.215 us | 0.3236 us | 0.4208 us | 17.232 us | - | - | - | - |
50 | 1000 | 15.862 us | 0.3152 us | 0.4908 us | 15.870 us | 0.2747 | - | - | 1249 B |
50 | 1000 | 63.352 us | 0.2720 us | 0.2123 us | 63.364 us | 0.9766 | - | - | 4536 B |
Buffer | 50 | 10000 | 108.719 us | 2.1416 us | 3.2054 us | 108.736 us | - | - | - | 12 B |
50 | 10000 | 148.399 us | 3.2352 us | 4.4284 us | 148.460 us | - | - | - | 2 B |
50 | 10000 | 127.577 us | 2.5284 us | 6.4357 us | 128.348 us | - | - | - | 1 B |
50 | 10000 | 98.270 us | 2.0086 us | 5.8909 us | 97.926 us | 0.2441 | - | - | 1311 B |
50 | 10000 | 153.122 us | 3.0482 us | 4.1725 us | 153.159 us | 0.9766 | - | - | 4539 B |

Windows results

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.18363.836 (1909/November2018Update/19H2)
Intel Core i7-9750H CPU 2.60GHz, 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=3.1.300
  [Host]     : .NET Core 3.1.4 (CoreCLR 4.700.20.20201, CoreFX 4.700.20.22101), X64 RyuJIT
  DefaultJob : .NET Core 3.1.4 (CoreCLR 4.700.20.20201, CoreFX 4.700.20.22101), X64 RyuJIT

| Method | NumBuffers | BufferSize | Mean | Error | StdDev | Median | Ratio | RatioSD | Gen 0 | Gen 1 | Gen 2 | Allocated |
|------------------------------------------- |----------- |----------- |-----------:|-----------:|-----------:|-----------:|------:|--------:|-------:|------:|------:|----------:|
| NetworkStreamSingleBuffer | 10 | 10 | 5.310 μs | 0.1059 μs | 0.1680 μs | 5.341 μs | 1.00 | 0.00 | - | - | - | - |
| NetworkStreamMultipleBuffers | 10 | 10 | 51.103 μs | 0.3414 μs | 0.3026 μs | 51.090 μs | 9.76 | 0.50 | - | - | - | - |
| NetworkStreamCopyMultipleBuffersIntoSingle | 10 | 10 | 5.314 μs | 0.1062 μs | 0.1305 μs | 5.345 μs | 1.01 | 0.05 | - | - | - | - |
| SocketAsyncEventArgsBufferList | 10 | 10 | 5.134 μs | 0.0319 μs | 0.0299 μs | 5.144 μs | 0.98 | 0.05 | - | - | - | - |
| SocketAsyncEventArgsSendPackets | 10 | 10 | 13.156 μs | 0.0773 μs | 0.0685 μs | 13.159 μs | 2.51 | 0.13 | 0.0458 | - | - | 368 B |
| | | | | | | | | | | | | |
| NetworkStreamSingleBuffer | 10 | 100 | 5.251 μs | 0.1036 μs | 0.2162 μs | 5.269 μs | 1.00 | 0.00 | - | - | - | - |
| NetworkStreamMultipleBuffers | 10 | 100 | 50.448 μs | 1.3837 μs | 4.0365 μs | 49.798 μs | 9.34 | 1.01 | - | - | - | - |
| NetworkStreamCopyMultipleBuffersIntoSingle | 10 | 100 | 6.132 μs | 0.1214 μs | 0.1661 μs | 6.097 μs | 1.18 | 0.11 | - | - | - | - |
| SocketAsyncEventArgsBufferList | 10 | 100 | 5.820 μs | 0.1155 μs | 0.2359 μs | 5.852 μs | 1.11 | 0.08 | - | - | - | - |
| SocketAsyncEventArgsSendPackets | 10 | 100 | 15.997 μs | 0.3178 μs | 0.5567 μs | 16.029 μs | 3.06 | 0.21 | 0.0458 | - | - | 368 B |
| | | | | | | | | | | | | |
| NetworkStreamSingleBuffer | 10 | 1000 | 5.611 μs | 0.1757 μs | 0.5180 μs | 5.629 μs | 1.00 | 0.00 | - | - | - | - |
| NetworkStreamMultipleBuffers | 10 | 1000 | 55.607 μs | 2.0389 μs | 5.9152 μs | 56.986 μs | 10.00 | 1.40 | - | - | - | - |
| NetworkStreamCopyMultipleBuffersIntoSingle | 10 | 1000 | 7.039 μs | 0.3720 μs | 1.0852 μs | 6.500 μs | 1.27 | 0.25 | - | - | - | 2 B |
| SocketAsyncEventArgsBufferList | 10 | 1000 | 5.623 μs | 0.2575 μs | 0.7592 μs | 5.927 μs | 1.01 | 0.18 | - | - | - | - |
| SocketAsyncEventArgsSendPackets | 10 | 1000 | 18.711 μs | 0.0710 μs | 0.0664 μs | 18.718 μs | 3.43 | 0.29 | 0.0305 | - | - | 368 B |
| | | | | | | | | | | | | |
| NetworkStreamSingleBuffer | 10 | 10000 | 35.910 μs | 1.2629 μs | 3.6437 μs | 36.673 μs | 1.00 | 0.00 | - | - | - | - |
| NetworkStreamMultipleBuffers | 10 | 10000 | 58.625 μs | 1.8727 μs | 5.5218 μs | 60.418 μs | 1.66 | 0.32 | - | - | - | - |
| NetworkStreamCopyMultipleBuffersIntoSingle | 10 | 10000 | 49.577 μs | 0.7218 μs | 0.6752 μs | 49.608 μs | 1.70 | 0.50 | - | - | - | 19 B |
| SocketAsyncEventArgsBufferList | 10 | 10000 | 37.742 μs | 0.7500 μs | 0.8637 μs | 37.910 μs | 1.22 | 0.33 | - | - | - | - |
| SocketAsyncEventArgsSendPackets | 10 | 10000 | 42.494 μs | 0.2339 μs | 0.2188 μs | 42.421 μs | 1.46 | 0.45 | - | - | - | 368 B |
| | | | | | | | | | | | | |
| NetworkStreamSingleBuffer | 20 | 10 | 5.206 μs | 0.0296 μs | 0.0277 μs | 5.211 μs | 1.00 | 0.00 | - | - | - | - |
| NetworkStreamMultipleBuffers | 20 | 10 | 98.887 μs | 1.3435 μs | 1.2567 μs | 99.441 μs | 18.99 | 0.27 | - | - | - | 1 B |
| NetworkStreamCopyMultipleBuffersIntoSingle | 20 | 10 | 5.603 μs | 0.0231 μs | 0.0216 μs | 5.612 μs | 1.08 | 0.01 | - | - | - | - |
| SocketAsyncEventArgsBufferList | 20 | 10 | 5.382 μs | 0.0234 μs | 0.0219 μs | 5.382 μs | 1.03 | 0.01 | - | - | - | - |
| SocketAsyncEventArgsSendPackets | 20 | 10 | 14.400 μs | 0.0550 μs | 0.0514 μs | 14.413 μs | 2.77 | 0.02 | 0.1068 | - | - | 688 B |
| | | | | | | | | | | | | |
| NetworkStreamSingleBuffer | 20 | 100 | 5.536 μs | 0.0417 μs | 0.0391 μs | 5.543 μs | 1.00 | 0.00 | - | - | - | - |
| NetworkStreamMultipleBuffers | 20 | 100 | 99.341 μs | 0.4334 μs | 0.3842 μs | 99.341 μs | 17.95 | 0.13 | - | - | - | - |
| NetworkStreamCopyMultipleBuffersIntoSingle | 20 | 100 | 5.732 μs | 0.0278 μs | 0.0247 μs | 5.729 μs | 1.04 | 0.01 | - | - | - | - |
| SocketAsyncEventArgsBufferList | 20 | 100 | 4.944 μs | 0.3061 μs | 0.9025 μs | 5.438 μs | 0.88 | 0.20 | - | - | - | - |
| SocketAsyncEventArgsSendPackets | 20 | 100 | 14.547 μs | 0.1657 μs | 0.1550 μs | 14.470 μs | 2.63 | 0.03 | 0.1068 | - | - | 688 B |
| | | | | | | | | | | | | |
| NetworkStreamSingleBuffer | 20 | 1000 | 8.988 μs | 0.1629 μs | 0.1444 μs | 9.036 μs | 1.00 | 0.00 | - | - | - | 3 B |
| NetworkStreamMultipleBuffers | 20 | 1000 | 101.225 μs | 4.5661 μs | 13.3915 μs | 104.686 μs | 10.39 | 3.04 | - | - | - | - |
| NetworkStreamCopyMultipleBuffersIntoSingle | 20 | 1000 | 12.872 μs | 0.2839 μs | 0.8326 μs | 12.392 μs | 1.59 | 0.04 | - | - | - | 5 B |
| SocketAsyncEventArgsBufferList | 20 | 1000 | 7.942 μs | 0.1455 μs | 0.3037 μs | 7.842 μs | 0.92 | 0.03 | - | - | - | - |
| SocketAsyncEventArgsSendPackets | 20 | 1000 | 26.798 μs | 0.0947 μs | 0.0886 μs | 26.796 μs | 2.98 | 0.05 | 0.0916 | - | - | 688 B |
| | | | | | | | | | | | | |
| NetworkStreamSingleBuffer | 20 | 10000 | 66.829 μs | 1.3360 μs | 1.5904 μs | 66.775 μs | 1.00 | 0.00 | - | - | - | 29 B |
| NetworkStreamMultipleBuffers | 20 | 10000 | 89.588 μs | 1.1547 μs | 1.0236 μs | 89.430 μs | 1.33 | 0.04 | - | - | - | 16 B |
| NetworkStreamCopyMultipleBuffersIntoSingle | 20 | 10000 | 100.458 μs | 1.2161 μs | 1.1376 μs | 100.783 μs | 1.50 | 0.04 | - | - | - | 35 B |
| SocketAsyncEventArgsBufferList | 20 | 10000 | 73.214 μs | 1.0808 μs | 0.9026 μs | 72.933 μs | 1.09 | 0.03 | - | - | - | - |
| SocketAsyncEventArgsSendPackets | 20 | 10000 | 67.810 μs | 0.2853 μs | 0.2802 μs | 67.812 μs | 1.01 | 0.02 | - | - | - | 688 B |
| | | | | | | | | | | | | |
| NetworkStreamSingleBuffer | 30 | 10 | 5.241 μs | 0.0344 μs | 0.0287 μs | 5.254 μs | 1.00 | 0.00 | - | - | - | - |
| NetworkStreamMultipleBuffers | 30 | 10 | 154.837 μs | 0.4575 μs | 0.4056 μs | 154.853 μs | 29.54 | 0.19 | - | - | - | - |
| NetworkStreamCopyMultipleBuffersIntoSingle | 30 | 10 | 5.582 μs | 0.1117 μs | 0.3000 μs | 5.631 μs | 1.04 | 0.10 | - | - | - | - |
| SocketAsyncEventArgsBufferList | 30 | 10 | 5.343 μs | 0.0127 μs | 0.0106 μs | 5.340 μs | 1.02 | 0.01 | - | - | - | - |
| SocketAsyncEventArgsSendPackets | 30 | 10 | 15.146 μs | 0.0682 μs | 0.0638 μs | 15.144 μs | 2.89 | 0.02 | 0.1526 | - | - | 1008 B |
| | | | | | | | | | | | | |
| NetworkStreamSingleBuffer | 30 | 100 | 5.690 μs | 0.0501 μs | 0.0444 μs | 5.697 μs | 1.00 | 0.00 | - | - | - | - |
| NetworkStreamMultipleBuffers | 30 | 100 | 144.650 μs | 4.5357 μs | 13.2309 μs | 147.184 μs | 24.71 | 4.49 | - | - | - | - |
| NetworkStreamCopyMultipleBuffersIntoSingle | 30 | 100 | 6.102 μs | 0.1082 μs | 0.1013 μs | 6.132 μs | 1.07 | 0.02 | - | - | - | - |
| SocketAsyncEventArgsBufferList | 30 | 100 | 5.841 μs | 0.0503 μs | 0.0470 μs | 5.828 μs | 1.03 | 0.01 | - | - | - | - |
| SocketAsyncEventArgsSendPackets | 30 | 100 | 15.770 μs | 0.0750 μs | 0.0702 μs | 15.782 μs | 2.77 | 0.02 | 0.1526 | - | - | 1008 B |
| | | | | | | | | | | | | |
| NetworkStreamSingleBuffer | 30 | 1000 | 11.514 μs | 0.1687 μs | 0.3168 μs | 11.399 μs | 1.00 | 0.00 | - | - | - | 4 B |
| NetworkStreamMultipleBuffers | 30 | 1000 | 152.884 μs | 0.7678 μs | 0.6412 μs | 152.861 μs | 13.05 | 0.25 | - | - | - | - |
| NetworkStreamCopyMultipleBuffersIntoSingle | 30 | 1000 | 12.394 μs | 0.2224 μs | 0.1971 μs | 12.401 μs | 1.06 | 0.03 | - | - | - | 4 B |
| SocketAsyncEventArgsBufferList | 30 | 1000 | 12.721 μs | 0.2435 μs | 0.2605 μs | 12.720 μs | 1.09 | 0.04 | - | - | - | - |
| SocketAsyncEventArgsSendPackets | 30 | 1000 | 32.859 μs | 0.1714 μs | 0.1338 μs | 32.898 μs | 2.80 | 0.07 | 0.1221 | - | - | 1008 B |
| | | | | | | | | | | | | |
| NetworkStreamSingleBuffer | 30 | 10000 | 59.785 μs | 0.4962 μs | 0.3874 μs | 59.783 μs | 1.00 | 0.00 | - | - | - | - |
| NetworkStreamMultipleBuffers | 30 | 10000 | 135.953 μs | 2.5654 μs | 2.1422 μs | 136.175 μs | 2.27 | 0.04 | - | - | - | 22 B |
| NetworkStreamCopyMultipleBuffersIntoSingle | 30 | 10000 | 196.229 μs | 2.5452 μs | 2.2562 μs | 195.844 μs | 3.28 | 0.05 | - | - | - | 80 B |
| SocketAsyncEventArgsBufferList | 30 | 10000 | 109.449 μs | 1.2758 μs | 1.1934 μs | 109.752 μs | 1.83 | 0.03 | - | - | - | 1 B |
| SocketAsyncEventArgsSendPackets | 30 | 10000 | 92.665 μs | 0.5370 μs | 0.4484 μs | 92.772 μs | 1.55 | 0.01 | - | - | - | 1008 B |
| | | | | | | | | | | | | |
| NetworkStreamSingleBuffer | 40 | 10 | 5.305 μs | 0.0299 μs | 0.0280 μs | 5.299 μs | 1.00 | 0.00 | - | - | - | - |
| NetworkStreamMultipleBuffers | 40 | 10 | 203.825 μs | 3.9899 μs | 3.9186 μs | 204.747 μs | 38.39 | 0.71 | - | - | - | - |
| NetworkStreamCopyMultipleBuffersIntoSingle | 40 | 10 | 5.624 μs | 0.1411 μs | 0.4161 μs | 5.677 μs | 1.03 | 0.17 | - | - | - | - |
| SocketAsyncEventArgsBufferList | 40 | 10 | 5.645 μs | 0.0229 μs | 0.0203 μs | 5.647 μs | 1.06 | 0.01 | - | - | - | - |
| SocketAsyncEventArgsSendPackets | 40 | 10 | 16.056 μs | 0.1253 μs | 0.1172 μs | 16.052 μs | 3.03 | 0.03 | 0.1831 | - | - | 1328 B |
| | | | | | | | | | | | | |
| NetworkStreamSingleBuffer | 40 | 100 | 5.908 μs | 0.1028 μs | 0.0961 μs | 5.869 μs | 1.00 | 0.00 | - | - | - | - |
| NetworkStreamMultipleBuffers | 40 | 100 | 203.884 μs | 0.4843 μs | 0.4293 μs | 203.878 μs | 34.47 | 0.59 | - | - | - | 3 B |
| NetworkStreamCopyMultipleBuffersIntoSingle | 40 | 100 | 6.277 μs | 0.1031 μs | 0.0964 μs | 6.305 μs | 1.06 | 0.02 | - | - | - | - |
| SocketAsyncEventArgsBufferList | 40 | 100 | 6.028 μs | 0.0736 μs | 0.0615 μs | 6.040 μs | 1.02 | 0.02 | - | - | - | - |
| SocketAsyncEventArgsSendPackets | 40 | 100 | 16.904 μs | 0.0710 μs | 0.0664 μs | 16.907 μs | 2.86 | 0.05 | 0.1831 | - | - | 1328 B |
| | | | | | | | | | | | | |
| NetworkStreamSingleBuffer | 40 | 1000 | 16.576 μs | 0.3294 μs | 0.7764 μs | 16.299 μs | 1.00 | 0.00 | - | - | - | 6 B |
| NetworkStreamMultipleBuffers | 40 | 1000 | 206.927 μs | 4.0291 μs | 3.9571 μs | 207.926 μs | 11.67 | 0.34 | - | - | - | 3 B |
| NetworkStreamCopyMultipleBuffersIntoSingle | 40 | 1000 | 17.132 μs | 0.2610 μs | 0.2442 μs | 17.132 μs | 0.97 | 0.02 | - | - | - | - |
| SocketAsyncEventArgsBufferList | 40 | 1000 | 14.682 μs | 0.2479 μs | 0.2198 μs | 14.719 μs | 0.83 | 0.02 | - | - | - | - |
| SocketAsyncEventArgsSendPackets | 40 | 1000 | 40.045 μs | 0.2153 μs | 0.1681 μs | 39.997 μs | 2.25 | 0.04 | 0.1831 | - | - | 1328 B |
| | | | | | | | | | | | | |
| NetworkStreamSingleBuffer | 40 | 10000 | 141.411 μs | 2.7909 μs | 2.7410 μs | 142.146 μs | 1.00 | 0.00 | - | - | - | 41 B |
| NetworkStreamMultipleBuffers | 40 | 10000 | 212.469 μs | 12.6034 μs | 37.1613 μs | 222.287 μs | 1.56 | 0.19 | - | - | - | - |
| NetworkStreamCopyMultipleBuffersIntoSingle | 40 | 10000 | 204.917 μs | 2.2431 μs | 1.9884 μs | 204.908 μs | 1.45 | 0.03 | - | - | - | 78 B |
| SocketAsyncEventArgsBufferList | 40 | 10000 | 137.543 μs | 2.2148 μs | 2.0717 μs | 137.099 μs | 0.97 | 0.02 | - | - | - | - |
| SocketAsyncEventArgsSendPackets | 40 | 10000 | 104.034 μs | 0.7048 μs | 0.5885 μs | 103.974 μs | 0.74 | 0.02 | 0.1221 | - | - | 1329 B |
| | | | | | | | | | | | | |
| NetworkStreamSingleBuffer | 50 | 10 | 5.248 μs | 0.0277 μs | 0.0259 μs | 5.244 μs | 1.00 | 0.00 | - | - | - | - |
| NetworkStreamMultipleBuffers | 50 | 10 | 253.654 μs | 1.4716 μs | 1.3046 μs | 253.975 μs | 48.29 | 0.30 | - | - | - | 1 B |
| NetworkStreamCopyMultipleBuffersIntoSingle | 50 | 10 | 5.750 μs | 0.0385 μs | 0.0360 μs | 5.762 μs | 1.10 | 0.01 | - | - | - | - |
| SocketAsyncEventArgsBufferList | 50 | 10 | 5.653 μs | 0.0228 μs | 0.0178 μs | 5.656 μs | 1.08 | 0.01 | - | - | - | - |
| SocketAsyncEventArgsSendPackets | 50 | 10 | 16.939 μs | 0.0670 μs | 0.0627 μs | 16.939 μs | 3.23 | 0.02 | 0.2441 | - | - | 1648 B |
| | | | | | | | | | | | | |
| NetworkStreamSingleBuffer | 50 | 100 | 5.923 μs | 0.1172 μs | 0.1990 μs | 5.977 μs | 1.00 | 0.00 | - | - | - | - |
| NetworkStreamMultipleBuffers | 50 | 100 | 248.751 μs | 0.7173 μs | 0.6710 μs | 248.575 μs | 42.65 | 2.45 | - | - | - | 1 B |
| NetworkStreamCopyMultipleBuffersIntoSingle | 50 | 100 | 6.573 μs | 0.4767 μs | 1.3982 μs | 7.340 μs | 1.03 | 0.26 | - | - | - | - |
| SocketAsyncEventArgsBufferList | 50 | 100 | 6.298 μs | 0.1252 μs | 0.3298 μs | 6.358 μs | 1.06 | 0.07 | - | - | - | - |
| SocketAsyncEventArgsSendPackets | 50 | 100 | 17.908 μs | 0.1571 μs | 0.1470 μs | 17.928 μs | 3.07 | 0.18 | 0.2441 | - | - | 1648 B |
| | | | | | | | | | | | | |
| NetworkStreamSingleBuffer | 50 | 1000 | 20.506 μs | 0.3646 μs | 0.3232 μs | 20.465 μs | 1.00 | 0.00 | - | - | - | 7 B |
| NetworkStreamMultipleBuffers | 50 | 1000 | 261.622 μs | 9.5744 μs | 28.2304 μs | 268.357 μs | 12.56 | 1.50 | - | - | - | - |
| NetworkStreamCopyMultipleBuffersIntoSingle | 50 | 1000 | 18.869 μs | 0.3746 μs | 0.8607 μs | 18.623 μs | 0.98 | 0.02 | - | - | - | - |
| SocketAsyncEventArgsBufferList | 50 | 1000 | 20.282 μs | 0.4011 μs | 0.6003 μs | 20.119 μs | 1.01 | 0.03 | - | - | - | - |
| SocketAsyncEventArgsSendPackets | 50 | 1000 | 39.699 μs | 0.1201 μs | 0.1179 μs | 39.687 μs | 1.94 | 0.03 | 0.2441 | - | - | 1648 B |
| | | | | | | | | | | | | |
| NetworkStreamSingleBuffer | 50 | 10000 | 177.767 μs | 3.5481 μs | 9.3471 μs | 179.053 μs | 1.00 | 0.00 | - | - | - | - |
| NetworkStreamMultipleBuffers | 50 | 10000 | 290.140 μs | 14.8617 μs | 43.8201 μs | 313.180 μs | 1.69 | 0.25 | - | - | - | - |
| NetworkStreamCopyMultipleBuffersIntoSingle | 50 | 10000 | 206.765 μs | 4.0923 μs | 6.3712 μs | 208.200 μs | 1.16 | 0.07 | - | - | - | - |
| SocketAsyncEventArgsBufferList | 50 | 10000 | 178.545 μs | 2.8085 μs | 2.3452 μs | 178.241 μs | 1.02 | 0.09 | - | - | - | - |
| SocketAsyncEventArgsSendPackets | 50 | 10000 | 150.540 μs | 0.9962 μs | 0.8831 μs | 150.545 μs | 0.86 | 0.08 | 0.2441 | - | - | 1649 B |

Benchmark code

```c#
[MemoryDiagnoser]
public class Program
{
const int Port = 9595;
const string Address = "127.0.0.1";

DummyServer _server;
Socket _socket;
SocketAsyncEventArgs _args;
NetworkStream _networkStream;
ManualResetEventSlim _mre;

[Params(10, 20, 30, 40, 50)]
public int NumBuffers { get; set; }

[Params(10, 100, 1000, 10000)]
public int BufferSize { get; set; }

byte[][] _buffers;
List<ArraySegment<byte>> _bufferList;
Memory<byte> _singleBuffer;

public void SetupServerAndConnect()
{
    _server = new DummyServer();
    _server.Start();
    _socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
    _socket.Connect(new IPEndPoint(IPAddress.Parse(Address), Port));
}

[GlobalCleanup]
public async Task Cleanup()
{
    _args?.Dispose();
    _args = null;
    _socket.Close();
    _socket = null;
    _networkStream = null;
    await _server.FinishTask;
}

[GlobalSetup(Target = nameof(NetworkStreamSingleBuffer))]
public void SetupNetworkStreamSingleBuffer()
{
    SetupServerAndConnect();
    _networkStream = new NetworkStream(_socket, ownsSocket: false);
    _singleBuffer = new byte[NumBuffers * BufferSize];
}

[GlobalSetup(Target = nameof(NetworkStreamMultipleBuffers))]
public void SetupNetworkStreamMultipleBuffers()
{
    SetupServerAndConnect();
    _networkStream = new NetworkStream(_socket, ownsSocket: false);
    _buffers = new byte[NumBuffers][];
    for (var i = 0; i < NumBuffers; i++)
        _buffers[i] = new byte[BufferSize];
}

[GlobalSetup(Target = nameof(NetworkStreamCopyMultipleBuffersIntoSingle))]
public void SetupNetworkStreamCopyMultipleBuffersIntoSingle()
{
    SetupServerAndConnect();
    _networkStream = new NetworkStream(_socket, ownsSocket: false);
    _buffers = new byte[NumBuffers][];
    for (var i = 0; i < NumBuffers; i++)
        _buffers[i] = new byte[BufferSize];
}

[GlobalSetup(Target = nameof(SocketAsyncEventArgsBufferList))]
public void SetupSocketAsyncEventArgsBufferList()
{
    SetupServerAndConnect();
    var bufferList = new List<ArraySegment<byte>>(NumBuffers);
    for (var i = 0; i < NumBuffers; i++)
        bufferList.Add(new byte[BufferSize]);
    _args = new SocketAsyncEventArgs
    {
        BufferList = bufferList
    };
    _mre = new ManualResetEventSlim();
    _args.Completed += (s, e) => _mre.Set();
}

[GlobalSetup(Target = nameof(SocketAsyncEventArgsSendPackets))]
public void SetupSocketAsyncEventArgsSendPackets()
{
    SetupServerAndConnect();
    _args = new SocketAsyncEventArgs
    {
        SendPacketsElements = new SendPacketsElement[NumBuffers]
    };
    for (var i = 0; i < NumBuffers; i++)
        _args.SendPacketsElements[i] = new SendPacketsElement(new byte[BufferSize]);
    _mre = new ManualResetEventSlim();
    _args.Completed += (s, e) => _mre.Set();
}

[Benchmark(Baseline = true)]
public async Task NetworkStreamSingleBuffer()
    => await _networkStream.WriteAsync(_singleBuffer);

[Benchmark]
public async Task NetworkStreamMultipleBuffers()
{
    for (var i = 0; i < NumBuffers; i++)
        await _networkStream.WriteAsync(_buffers[i]);
}

[Benchmark]
public async Task NetworkStreamCopyMultipleBuffersIntoSingle()
{
    var singleBuffer = ArrayPool<byte>.Shared.Rent(NumBuffers * BufferSize);
    for (var i = 0; i < NumBuffers; i++)
        Buffer.BlockCopy(_buffers[i], 0, singleBuffer, i * BufferSize, BufferSize);
    await _networkStream.WriteAsync(singleBuffer);
    ArrayPool<byte>.Shared.Return(singleBuffer);
}

[Benchmark]
public void SocketAsyncEventArgsBufferList()
{
    _mre.Reset();
    if (_socket.SendAsync(_args))
        _mre.Wait();
}

[Benchmark]
public void SocketAsyncEventArgsSendPackets()
{
    _mre.Reset();
    if (_socket.SendPacketsAsync(_args))
        _mre.Wait();
}

static void Main(string[] args)
    => new BenchmarkSwitcher(typeof(Program).GetTypeInfo().Assembly).Run(args);

class DummyServer
{
    TcpListener _listener;

    public Task FinishTask { get; private set; }

    public void Start()
    {
        _listener = new TcpListener(IPAddress.Parse(Address), Port);
        _listener.Start();
        FinishTask = Task.Run(Process);
    }

    void Process()
    {
        var client = _listener.AcceptTcpClient();
        var buf = new byte[8192];
        var stream = client.GetStream();
        while (stream.Read(buf, 0, buf.Length) != 0)
        {
            // Do nothing with the data
        }
    }
}

}
```

roji on 24 May 2020

Just to be clear, this isn't urgent and I don't have any immediate pressing need for using this (though I may at some point).

roji on 24 May 2020

SocketAsyncEventArgsBufferList is allocating:

https://github.com/dotnet/runtime/blob/d0e22812943f358b438f1e379ccf99027d34789e/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketPal.Unix.cs#L237-L238

The IOVector and GCHandles could be stackalloced (up to some bound).

tmds on 25 May 2020

The IOVector and GCHandles could be stackalloced (up to some bound).

As the Windows version of this does. Not sure why this one wasn't changed similarly.

stephentoub on 25 May 2020

@roji Linux defines UIO_FASTIOV to 8. This means up to 8 iovs (NumBuffers) will be stackalloced in the kernel. It is an interesting nr to try in the benchmarks.