Fs2: gzip compression is not streaming well

[svalaskevicius]

two issues (same problem / file :) )

1. gzip does not allow to pass SYNC_FLUSH to deflater

2. gunzip delays chunks by at least one in .through(_gunzip_validateTrailer(contentCrc32, inflater)) - which is particularly bad when inflating a real time stream (e.g. a streaming http response / long polling)

[mpilquist]

Thanks for the report. Can you clarify if this is with fs2.compress or fs2.compression?

[svalaskevicius]

Hi, SYNC_FLUSH is a problem for both, the 2nd problem is from the fs2.compression

[mpilquist]

cc @greg-a-atkinson @djspiewak

[djspiewak]

SYNC_FLUSH is probably an easy fix (and also totally sounds like something that we would have missed). I don't know enough about the compression implementation to speak to how easy it would be to fix the run-behind problem.

[svalaskevicius]

Yeah, I guess an easy workaround could be to allow to opt out of the trailer validation - for (hopefully) infinite streams of gzipped content it's not of much use anyways

Although it's likely there are plenty of usecases where this is not a solution (cannot think of them now though :) )...

[djspiewak]

Yeah, I guess an easy workaround would be to allow to opt out of the trailer validation - for (hopefully) infinite streams of gzipped content it's not of much use anyways

Seems like something that should be configurable for exactly this reason.

[greg-a-atkinson]

two issues (same problem / file :) )

1. gzip does not allow to pass SYNC_FLUSH to deflater
2. gunzip delays chunks by at least one in .through(_gunzip_validateTrailer(contentCrc32, inflater)) - which is particularly bad when inflating a real time stream (e.g. a streaming http response / long polling)

Hi, thanks for raising your query.

For the first point, try compression.gzip(deflateStrategy = Some(Deflater.SYNC_FLUSH)) for your use case? The deflate strategy defaults to Deflater.DEFLUAT_STRATEGY in the same way as the previous compress.gzip and Deflater itself.

Regarding the second point, I would agree that gzip is probably not the ideal choice for infinite streams or long-polling. Gzip, as per the RFC, is a wrapper around (usually) the deflate/inflate compression methods, where that wrapper adds a header with file attributes and a trailer with length and CRC details. For your use case, are you able to use compression.inflate instead, that will just inflate the compressed stream without the need for file identifying and validating attributes? If in your use case you are unable to control the source of the stream and unable to change it from a gzip stream to a deflate stream, it may be best to skip over the gzip header and treat it as a deflate/inflate stream?

[svalaskevicius]

hi @greg-a-atkinson , thanks for the reply.

1. SYNC_FLUSH is not a deflate strategy, but flush mode for function https://docs.oracle.com/javase/8/docs/api/java/util/zip/Deflater.html#deflate-byte:A-int-int-int-

2. thinking about it, deflate seems like a better choice in this case (avoids calculation of crc32 while consuming it). however, the issue raised is purely about gzip trailer check implementation - it's delaying outgoing data - which is avoidable for anything related to streaming chunks :) as trailer check if happening at the end of the stream, there should be no need to delay the data, and should be enough to simply "watch" is as it flows through, appending the check at the end?

in terms of priorities, I think 1 > 2, and the issue could be split if preferred?

[greg-a-atkinson]

Agreed, compression.deflate and by extension compression.gunzip should allow for the flush mode to be configured. Please review this PR for this to be the case:

https://github.com/functional-streams-for-scala/fs2/pull/1842

The second problem is trickier and will require some thought. The gzip trailer is the final 8 bytes of the stream, but the header and compressed payload provide no indication for when the trailer will arrive. It is only possibly to identify the trailer once the compressed payload has been fully decompressed and 8 bytes of the stream remain. The current implementation holds on to the most recent chunk so that if that chunk results in the end of the decompressed stream it is known that the the chuck also holds the trailer. This implementation is better suited to file gunzipping as opposed to infinite stream gunzipping where a trailer may not be reached. Nevertheless, an improved algorithm should be possible and would be preferrable.

[svalaskevicius]

Thanks!

Yes. I see why it is not trivial to fix the pt2.

Maybe worth to simply add a piece of documentation to note that this behaviour is expected and point to deflate as per your earlier suggestion?

[greg-a-atkinson]

This PR should improve real-time streaming with gunzip (the second issue) by not delaying chunks until the inflater is finish before searching for the trailer:

https://github.com/functional-streams-for-scala/fs2/pull/1843

[greg-a-atkinson]

Please upgrade to fs2 v2.4.1. Thanks for raising the issue.

[svalaskevicius]

Thanks!