Runtime: PipeReader Wrong Position

[klemmchr]

Executing this code

var pipe = new Pipe();

await pipe.Writer.WriteAsync(new ReadOnlyMemory<byte>(new byte[100]));
await pipe.Writer.FlushAsync();

while (true)
{
    var readResult = await pipe.Reader.ReadAsync();
    if (readResult.Buffer.Length <= 0)
        break;

    Console.WriteLine("Buffer length:" + readResult.Buffer.Length);
    Console.WriteLine("Buffer start:" + readResult.Buffer.Start.GetInteger());

    var position = readResult.Buffer.GetPosition(10);
    Console.WriteLine("End position:" + position.GetInteger());
    Console.WriteLine();

    pipe.Reader.AdvanceTo(position);
}

gives the following output:

Buffer length:100
Buffer start:0
End position:10

Buffer length:90
Buffer start:10
End position:20

Buffer length:80
Buffer start:20
End position:30

Buffer length:70
Buffer start:30
End position:40

Buffer length:60
Buffer start:40
End position:50

Buffer length:50
Buffer start:50
End position:60

Buffer length:40
Buffer start:60
End position:70

Buffer length:30
Buffer start:70
End position:80

Buffer length:20
Buffer start:80
End position:90

Buffer length:10
Buffer start:90
End position:100

When iterating the buffer length decreases, because we consumed 10 bytes. However the buffer start and end position are also increasing by 10. I would expect that if I advance the reader the buffer reduces in length (which it does) and the start position is 0 and the end position would be the buffer size.
If you want to read continuously from the stream in sequence you can't rely on the position this way.

[davidfowl]

Unfortunately GetInteger and GetObject aren't for public consumption this way. There are opaque objects that need to be used in conjunction with the ReadOnlySequence<T>.

[klemmchr]

What would you recommend to keep track of the position? Just counting by myself? In my particular case I need to read the first 4 bytes of the sequence to determine the packet length. Currently I'm using TryGet to get the actual memory out of the sequence. I'm unsure whether the SequencePosition parameter affects the start position of the operation. Is TryGet just reading out from the beginning of the sequence and only uses the SequencePosition as an internal reference or is it actually the starting point of the read operation?

[benaadams]

You are outputting the new positions tag (current + 10); not the end positions tag, so it ends up bring what the start is after you move it to that location (current + 10).

However; its an internal reference and the sequence can be made of many blocks so it will reset back to 0 after it crosses a block; so its not an absolute position.

SequencePosition is a composite structure that is the absolute position; and the parts looked at in isolation (GetObject() and GetInteger()) don't make much sense outside of a single buffer ReadOnlySequence.

[tmds]

Try SequenceReader. It has a .Consumed property that counts for you.

[davidfowl]

PS: I'm writing detailed docs on how to use these types together.

[klemmchr]

PS: I'm writing detailed docs on how to use these types together.

Thanks! That's what I'm missing currently. I'm was following your blogpost for Pipelines which covers the basics, but there's a lot more to it.

Edit: SequenceReader is only available in .NET Core 3.0, any advice what/how to use for .NET Standard 2.1?

[davidfowl]

Edit: SequenceReader is only available in .NET Core 3.0, any advice what/how to use for .NET Standard 2.1?

Only way to use it would be to copy the source.

@terrajobst any suggestions on what we can do here going forward?

[terrajobst]

@terrajobst any suggestions on what we can do here going forward?

File an issue in dotnet/standard for addition. We can still add APIs to .NET Standard 2.1, but given where we are all changes need to be taken to ship room, so I need a compelling argument beyond "it's in .NET Core 3.0 but not in .NET Standard 2.1".

[klemmchr]

Reading from a sequence without SequenceReader is not really pleasant. Having it in Standard would allow libraries to use it instead of relying on 3rd party/copied implementations.

[davidfowl]

@terrajobst Right, the argument is that its damn near impossible to write efficient code that works well without the SequenceReader. The options today lead you to writing something very similar or converting the ReadOnlySequence<T> to T[] (which defeats the purpose).

[davidfowl]

@chris579 can you file the issue? Lets see if we can get it in.

[klemmchr]

dotnet/standard#1493