Runtime: Utf8JsonReader/JsonStreamReader: Add reading values as Stream

[andriysavin]

Consider the following (real) scenario: a web service returns a binary content (file) of arbitrary length (up to hundreds of MB) as a base64-encoded value of a JSON property of a JSON document. With current implementation I have to read whole property value into memory before I can start processing it. Needless to explain what the consequences of this are, especially related to LOH (though, memory pooling inside the library may make this less painful). What I'd like to have is ability to open a property value as a stream (or, maybe, some lower lever object such as pipe reader) and read the value in a streamed fashion as bytes without allocating huge amount of memory. I'm not sure if this should go into Utf8JsonReader or (planned?) JsonStreamReader, but you got the idea. The same should probably be considered for writers.

[scalablecory]

Utf8JsonReader is resumable, allowing you to use it with Stream.

Once Utf8JsonReader.Read() returns false, save the CurrentState and read more from a stream. Then, create a new Utf8JsonReader this time passing in the previous state.

Sample here: https://github.com/scalablecory/system-text-json-samples/blob/master/json-test/JsonParser.ParseSimpleAsync.cs

[davidfowl]

Dupe of https://github.com/dotnet/corefx/issues/39655

[andriysavin]

@davidfowl I don't think this is a dup of that issue. That issue is about reading JSON from a stream. My issue is about reading part of JSON as a stream. Or maybe you're suggesting that while reading from a stream at some point (e.g. we encountered property value) a portion of that stream could be returned as a stream?

[davidfowl]

Ah my bad you just want to read a specific property value as a stream. The reader deals with buffers not streams so in its current form it wouldn't make any sense to have an API like this.

You'd need to be reading from a stream in chunks already combined with some logic to identify the beginning of the value, then you'd have to manually parse the stream to find the end. The reader can't give you a Stream for the underlying value since it has to have buffered that data to begin with (in order to identify the end and have APIs like Value/ValueSequence function).

[andriysavin]

@davidfowl yeah, I felt that, and this is why I was also referring to JsonStreamReader which is mentioned in the roadmap. Today I can implement what I need myself as you described. But if this JsonStreamReader is still on the roadmap, it would be nice to have this feature implemented in it out of the box.

[scalablecory]

Ah, this makes sense! XmlReader does have a similar feature; it may be worth doing something similar here with JsonStreamReader.

[andriysavin]

@scalablecory I guess you're talking about ReadValueChunkAsync, which seems to be the closest to the idea. But returning a Stream would be even better and ready to use functionality.

[xania]

I guess everyone is going to write his owns extension method to read from the stream then, or am i the only one?

[layomia]

I guess everyone is going to write his owns extension method to read from the stream then, or am i the only one?

This scenario is covered by #30328.

[layomia]

Changes would need to be made to Utf8JsonReader to support this, which will affect performance for the common case. Closing as not feasible.