Node: Readable stream API

[ronag]

I thought I had a pretty good understanding of streams until I started digging into Readable.

As far as I understand there are 3+ ways to work with streams:

1. readable.pipe
2. readable.on('readable')/readable.read()
3. readable.pause()/readable.resume()
4. (stream.pipeline)

Since I have trouble following the implementations of 1 and 2 (even before digging). I have always only trusted and used 3. Furthermore I've enforced the rest of our team to use this together with some rules and helpers.

Though after reading the code and docs I have a feeling 3. is considered legacy?

While the docs are very good at describing how the different modes work. I have not been able to figure out from the docs what method is recommended/appropriate when and why.

Why do we have 3 modes (other than backwards compat)? Could we explicitly (doc) deprecate or possibly recommend one of them or alternatively why and when to use each one?

I'd be more than happy to make suggestions to improve the docs once I understand this myself.

Furthermore, I have a feeling that 98% of our users should not bother with any of this and just simply use stream.pipeline. I would suggest we earlier and more explicitly try to recommend and forward readers of the streams docs to pipeline and leave the rest explicitly for "advanced" users. Especially as pipeline takes very good care of and normalizes a lot of the complexities and inconsistencies that exist (some of which are either not documented at all or ambiguous). Maybe more explicitly splitting the docs into basic (98%) and advanced (2%) would be helpful?

[sam-github]

I agree, if there are multiple ways to do things, we should explain why, and if not recommend a way, give enough information that the best way for the specific user can be chosen.

I doubt we can deprecate any of this .... its been around too long, and all the modes are going to be used.

[ronag]

Regarding deprecation. I was more thinking of doc deprecation.

[ronag]

@benjamingr: Who can help us with some wisdom on this issue?

[benjamingr]

@nodejs/streams

[mcollina]

I would recommend people to use either pipeline or async iterators at this point, and leave the rest for advanced users. I'd be happy to have some recommendation at the top of our streams docs, and leave the rest for the advanced users.

(We cannot deprecate any of them because of backward compatibility).

[ronag]

I would recommend people to use either pipeline or async iterators at this point,

What is the performance overhead of async iterators?

and leave the rest for advanced users. I'd be happy to have some recommendation at the top of our streams docs, and leave the rest for the advanced users.

I'll prepare a proposal.

(We cannot deprecate any of them because of backward compatibility).

Not even doc deprecate in order to discourage more usage? I'm not sure doc deprecation has to mean that it will be removed, or does it?

[mcollina]

I would not deprecate them because there are so used in the wild with so many examples about it.
Moreover, they are useful in certain edge case scenarios.

Not even doc deprecate in order to discourage more usage? I'm not sure doc deprecation has to mean that it will be removed, or does it?

Doc-deprecation is the first step of our deprecation process, see https://github.com/nodejs/node/blob/37321a9e11f2198d03a525cddca20827636b786c/doc/api/deprecations.md for more details.

[benjamingr]

What is the performance overhead of async iterators?

It is huge but hardly impactful in cases they are used, it allocates a promise _and_ an object for every result. We can ask V8 to optimize them but honestly people haven't complained so far and performance has been adequate and libraries can optimize them and pool internally.