Runtime: Stream Facade Pattern and Stream enhancements

[juliusfriedman]

Background and Motivation

Pipes are great and easy to use but what if I have an byte[] or even IEnumerable<byte>, should I not be able to combine multiple instances of these into something which represents a contiguous piece of memory?

Prior Art

SegmentStream

MediaFileStream

Proposed API

//Or StreamExtensions
public class System.IO.Stream
{
+    public virtual Stream Empty {get;} //not to confuse with Null, allows the stream to define it's preamble if any, can use Null by default until overridden in a derived class.
+    public static Stream Create(IEnumerable<byte> segments);
+    public static Stream Create(params byte[] segments);
+    public static Stream Create(params byte[][] segments);
+    public static Stream Create(IList<ArraySegment<byte>> segments);
+    public static Stream Create(params Span<byte>[] segments);? if this ever happens
+    public static Stream AsReadOnly(Stream stream);
+    public static Stream AsWriteOnly(Stream stream);
+    public static Stream WithoutSeek(Stream stream);
+    public static Stream WithCapacity(Stream stream, long capacity);
+    public static Stream WithLength(Stream stream, long length);
 }

//Or StreamExtensions
namespace System.IO {
//BufferedStream is also possible to use as the intermediate here and might be easier for users to comprehend... 
+ public class ReadOnlyStream : Stream { public ReadOnlyStream(Stream stream, int length = -1/*, int bufferSize = -1*/); }
+ public class WriteOnlyStream : Stream { public WriteOnlyStream (Stream stream, int length = -1/*, int bufferSize = -1*/); }
+ public class SeekableStream : Stream { public SeekableStream (Stream stream, int length = -1/*, int bufferSize  = -1,*/ bool canRead = true, bool canWrite = true); }
}

Usage Examples

C# ArraySegment<byte> something; byte[] somethingElse; ReadOnlySpan<byte> rom; var s = Stream.Create(something); s.Write(somethingElse); s.Write(rom); s.Seek(0, SeekOrigin.Begin); var r = s.AsReadOnly(); s.Seek(0, SeekOrigin.End); var w = s.AsWriteOnly(); var fp = WithLength(r, something.Length);

Alternative Designs

In System.IO.Buffers or as consensus arrives.

See prior art, Extension methods StreamExtensions

See also (as requested by @CyrusNajmabadi) Sketch Implementation

Risks

Low, we already have overloads of on Socket which accept IList<ArraySegment<byte>> and they do this but only for sockets.

How much code really will use this:

https://github.com/search?l=C%23&q=limitedstream&type=Code

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[juliusfriedman]

cc @davidfowl as he was the creator of pipes.

[GrabYourPitchforks]

An API to create a read-only Stream from a ReadOnlySequence<byte> might be useful. I tried searching briefly but couldn't find one. Seems like it'd be a weird omission.

[AlgorithmsAreCool]

Some thoughts...

I worry about encouraging creating a stream from an IEnumerable<byte>. While functional, it is a very high overhead pattern.

A stream from a single byte[] is just a MemoryStream, your API would likely just be a wrapper around that. But I do think it is a little more discoverable.

Span<T>[] isn't legal. I don't think there is any way to express an arbitrary collection of Spans in C#.

As for the last three, I'm not sure how useful they are in reality. If a stream supports seeking, why would I need a wrapper that disables seeking? Or Reading? You probaly need to offer up some concrete examples of when these are useful.

Also, davidfowl was the architect behind Pipelines.

[juliusfriedman]

Some thoughts...

I worry about encouraging creating a stream from an IEnumerable<byte>. While functional, it is a very high overhead pattern.

A stream from a single byte[] is just a MemoryStream, your API would likely just be a wrapper around that. But I do think it is a little more discoverable.

Span<T>[] isn't legal. I don't think there is any way to express an arbitrary collection of Spans in C#.

It might be one day... also technically there are a myriad of ways around that. It just comes down to how not to crash the CLR. You can pass an array of IntPtr around can't you?

As for the last three, I'm not sure how useful they are in reality. If a stream supports seeking, why would I need a wrapper that disables seeking? Or Reading? You probaly need to offer up some concrete examples of when these are useful.

Umm because I want to limit a stream to a certain length per-say and only make that portion visible. Do I need more examples?

Also, davidfowl was the architect behind Pipelines.

my apologies to @stephentoub then.

It depends on what implements it, doesn't it? I don't have that overhead in my class all of my enumerable instances are fast and have low overhead.

[AlgorithmsAreCool]

It depends on what implements it, doesn't it? I don't have that overhead in my class all of my enumerable instances are fast and have low overhead.

I just mean that looping to call MoveNext() for each byte is always going to incur more overhead than bulk operations that can read many bytes at once. For shorter streams it wouldn't matter much, but streams are typically used for for "large" data.

Umm because I want to limit a stream to a certain length per-say and only make that portion visible. Do I need more examples?

Ah, I see. I agree, that restricting access to a sub-section of a seekable stream is an interesting (albeit niche) use case, I now see that in your example code.

Btw, if these are intended to be extension methods you might want to annotate them as such to make that clear.

[CyrusNajmabadi]

Can you explain how your impl of these methods would work:

public static Stream AsReadOnly(Stream stream);
+    public static Stream AsWriteOnly(Stream stream);

I'm not really getting how this would work given arbitrary underlying implementations you can't control. Thanks!

[juliusfriedman]

Can you explain how your impl of these methods would work:

public static Stream AsReadOnly(Stream stream);
+    public static Stream AsWriteOnly(Stream stream);

I'm not really getting how this would work given arbitrary underlying implementations you can't control. Thanks!

I will attempt to explain, the indicated methods would use the ReadOnlyStream and WriteOnlyStream right?

What is it exactly your asking...

[CyrusNajmabadi]

Could you give a small sketch on how this would work? Thanks!

[juliusfriedman]

Sorry for my delayed response, I will do more than give a sketch, I make make a fork of the repo and add the implementation and well as some unit tests if that would be easier?

There are a lot of ways and of lot of potential things you would be able to expose e.g. this entire added API set would also all be able to live in System.IO.Concurrent I will make a prototype implementation if that is what your asking for...

I apologize for not being able to be as clear as required.

Thank you again for your time.

Here is an honest and IMHO clean attempt I took about an hour or two on while I had the time[which could probably use both review and critique for both performance characteristic and correctness] but I went for the shorthand here as impl's I were told are not typically required.

Hopefully one can easily see that Buffered is now a one liner over the impl as well,

c# Stream.Buffered(Stream stream) => new BufferedStream(stream); Stream.Buffered(Stream stream, int capacity) => new BufferedStream(stream, capacity);

With further options possible for ConcurrentStream such as ConcurrentFileStream, ConcurretMemoryStream etc easily available.

[CyrusNajmabadi]

Sorry for my delayed response, I will do more than give a sketch, I make make a fork of the repo and add the implementation and well as some unit tests if that would be easier?

I would prefer a sketch. However, if you can just show the code, that will likely help as well. However, it might be a lot to get through, and i can't necessarily promise that.

Here is an honest and IMHO clean attempt I took about an hour or two on

Ah, i see. So the underlying stream isn't affected. You are just creating a new stream that uses this as the underlying source. IMO, these woudl be better as just disparate classes that can be instantiated and passed in the data,

[GSPP]

What I needed a few times is a wrapper stream that ignores Flush. Flush is sometimes called rather generously for no functional reason. Depending on the concrete Stream it can have rather drastic performance implications such as causing network access, disk access or just a kernel call.