Reqwest: Convert response body to Stream

[amesgen]

Right now, there is no "direct" way to consume a response body as a Stream<Item = Result<Bytes>>. IMO this would be a good thing (either via some into_stream function, or a direct impl, like in 0.9). I went ahead and made the existing into_stream in Body public, but this also requires to make ImplStream public, which is probably suboptimal.

Related: How about enabling the stream feature by default, just as it is done in hyper 0.13?

[Arnavion]

You can make a stream in a slightly round-about way with:

async fn next_chunk(mut response: reqwest::Response) -> Option<(Result<bytes::Bytes, reqwest::Error>, reqwest::Response)> {
    let chunk = response.chunk().await;
    let chunk = chunk.transpose()?;
    Some((chunk, response))
}

let body = futures_util::stream::unfold(response, next_chunk);

But yes, it would be easier if it was just an into_stream on Response itself.

[amesgen]

Yeah, that's why I wrote that there is no "direct" way. ~Your solution has a minor drawback: body is lifetime-constrained by response, so if you want to return body from a function (for e.g. some function async fn(url: &str) -> Result<impl Stream<Item = Result<Bytes>>>), it is not easily possible without sth like owning_ref.~ I am tired...

Another way is to use async-stream which does not have this problems, but instead you occasionally get weird compiler errors (see e.g. here).

[Arnavion]

Yeah, I switched from the unfold method to async-stream for my own stuff too.

[bartlomieju]

@seanmonstar is this issue is something you would consider before releasing reqwest? Ability to "take" body as a stream is the only missing feature for us at Deno in regard to updating reqwest.

[seanmonstar]

What if we added Response::into_stream(self) -> impl Stream<Item = crate::Result<Bytes>>?

[bartlomieju]

What if we added Response::into_stream(self) -> impl Stream<Item = crate::Result<Bytes>>?

That'd be perfect!