Standard: Make SequenceReader available

[klemmchr]

Reading from a ReadOnlySequence<T> requires to keep track of position and consumed data. In core you can use the type SequenceReader that does that for you. In standard you don't have it which leads to three possibilities:

• Write your own implementation which does similar things like SequenceReader
• Copy the implementation of SequenceReader
• Converting the ReadOnlySequence<t> to T[] which defeats the purpose

All options are not convenient. Making SequenceReader available in standard would allow libraries to use the performance benefits of SequenceReader without the need to rely on third party implementations or crushing performance because of allocations.

[christiannagel]

What about option 4?

• Add a reference to the package System.Memory, and use SequenceReader

[klemmchr]

Is SequenceReader actually part of System.Memory? I just added a reference to it and it's not available. I mean, the docs state it, but it doesn't seem to be available.

[christiannagel]

According to the documentation on SequenceReader it should be in the assembly System.Memory.dll. But you're right, it's not in the latest available version on the NuGet server (4.5.3 which is from May-2019).

Checking the definition of SequenceReader in a .NET Core 3.0 Preview 8 console application, it shows it's in the assembly System.Memory version 4.2.0.0.

Is it available on a nightly feed NuGet server? I'm also not sure about the version numbers. It's included in 4.2.0 but not in 4.5.3?

[klemmchr]

Well, in this case this issue goes back to dotnet/corefx, doesn't it?

[terrajobst]

The short answer is: it belongs here. System.Memory is a NuGet package that adds downlevel support for span, but it cannot give you new APIs over .NET Standard 2.1.

[klemmchr]

Awesome how fast this got implemented. Love to see how Microsoft thrives for the open source community. Thanks people!

[GF-Huang]

Similar problem, I have install System.Memory NuGet package for .Net Framework 4.7.2 project, but SequenceReader not available.

[terrajobst]

Similar problem, I have install System.Memory NuGet package for .Net Framework 4.7.2 project, but SequenceReader not available.

That's by-design. We genrally don't expose platform types for downlevel platforms after we put them in-box as this has proven to be quite fragile. You'll need to target .NET Standard 2.1 or .NET Core 3.0.

[klemmchr]

Just curious, what is fragile regarding SequenceReader? I would have thought that Span is fragile but it is shipped in System.Memory.

[terrajobst]

Just curious, what is fragile regarding SequenceReader? I would have thought that Span is fragile but it is shipped in System.Memory.

It's nothing to do with this type; it has to do with shipping new types to old platforms while maintain type identity when your application rolls forward to newer versions of the platform where the type is now built-in.

[GF-Huang]

Depressing, ReadOnlySequence without SequenceReader is even harder to use than byte[] when separating HTTP header by \r\n. 😢

[GF-Huang]

Any suggests for parsing http header without SequenceReader? Be deeply grateful.

private async Task HandleTcpConnection(TcpClient client) {
    try {
        var stream = client.GetStream();
        var reader = PipeReader.Create(stream);

        while (true) {
            var result = await reader.ReadAsync();
            var buffer = result.Buffer;

            if (buffer.IsEmpty)
                break;

            var position = buffer.PositionOf((byte)'\r');
            if (position == null) {
                // need more data
                reader.AdvanceTo(buffer.Start, buffer.End);

            } else {
                // how to do here to ensure next byte is '\n'?
            }
        }

    } catch (Exception e) {
        // ...

    } finally {
        client.Close();
    }
}

[klemmchr]

You just might want to copy the implementation of SequenceReader to your project and use that instead. Makes things clearer.

[GF-Huang]

A ton of error to build the SequenceReader source, too many dependencies, may have to download the entire project to compile. And I haven't vs2019.

[darl0026]

Any suggests for parsing http header without SequenceReader? Be deeply grateful.

private async Task HandleTcpConnection(TcpClient client) {
    try {
        var stream = client.GetStream();
        var reader = PipeReader.Create(stream);

        while (true) {
            var result = await reader.ReadAsync();
            var buffer = result.Buffer;

            if (buffer.IsEmpty)
                break;

            var position = buffer.PositionOf((byte)'\r');
            if (position == null) {
                // need more data
                reader.AdvanceTo(buffer.Start, buffer.End);

            } else {
                // how to do here to ensure next byte is '\n'?
            }
        }

    } catch (Exception e) {
        // ...

    } finally {
        client.Close();
    }
}

This the exact scenario, I am trying to solve! I had to build my own TCPServer and need to check for a custom string terminator, not just one character!

[davidfowl]

Somebody can copy it once, make it work on .NET Standard 2.0, put it on nuget and everyone can use it

[darl0026]

@davidfowl Well instead of converting the bytes to string and then searching on a string. What about concatenating the byte arrays using Buffer.BlockCopy and then using a ReadonlySpan to read over that. Because that will give me things like IndexOf and Slice.

[davidfowl]

Sure, there's lots of ways to write this code, some more error prone than others. Just make sure to test the edge cases

[darl0026]

@davidfowl is that the most efficient way to combine byte arrays? Quick googling suggested it was. In any case it seems like being able to search for a string over a sequence of bytes will be valuable. It doesnt solve my problem of potentially getting part of the terminating string in the next read, but still seems very valuable.