Lz4: Decompress in streaming keeping memory low

[DiegoTUI]

Hi there,

I am building an application that receives TCP connections sending streams of data previously compressed by the sending party using lz4. I need to decompress the data and pipe it to the next stage of the application keeping the memory used as low as possible. I've thoroughly read the docs of this implementation as well as some issues addressing this issue, and it seems like the block is the minimum, unsplittable unit that I need to send to the decompressing functions. Since I don't have any control of how the data was compressed, and the maximum block size allowed in LZ4 is 4MB, my worst case scenario would be using 4MB of memory per stream of compressed data (i.e. per connection). This use of memory is not acceptable for my application. I have two questions:

1- Have I misread the docs/issues and there is actually a way of sending partial blocks to the decompression functions?
2- Any plans to add this feature in the near future?

Thanks!!

[Cyan4973]

Hi @DiegoTUI

Your analysis is correct.

Since I don't have any control of how the data was compressed, and the maximum block size allowed in LZ4 is 4MB, my worst case scenario would be using 4MB of memory per stream of compressed data (i.e. per connection).

Going into the details, it also depends if your user is sending LZ4 data using the official frame format, where the 4 MB maximum block size is defined. Otherwise, all bets are off, and it can even be worse, since the maximum API limit for a block size is close to 2 GB.

The only short term solution is to _send_ smaller blocks.
This could be enforced as a condition to use your interface.
Since the decoder needs to know the (maximum possible) decompressed size of a block to start decompression, it's possible to reject a block on the ground that it's too large.
In the case of the frame format, this information is also present right into the frame header.

But if you can't do that, I'm afraid there is currently no available solution.

Intra-block streaming would require a new, different, decoding engine.
The current one is hard-wired on the concept of block, which was designed to simplify its work btw.
A decoder able to stop decoding mid-block will have more states to manage, hence will be more complex, and likely slower. It will also duplicate code in the source file, resulting in increased maintenance burden.
So it's quite an intense feature to add.

At this stage, it's a remote possibility I would say.

[DiegoTUI]

Thanks for the explanation @Cyan4973!!! My colleague @alexfernandez has build our own implementation of the LZ4 algorithm allowing intra-block streaming (it uses the sequence as the unsplittable entity instead of the block). And you a re right, we use a lot more states than you do. It's written in Javascript, and of course, it performs quite worse than this implementation. I was trying to improve a bit the performance by binding the core parts to the C implementation, and that's why I raised the issue. So far, our implementation of LZ4 is private, but I believe we will release it soon. I will let you know when we do.

Thanks again!!

[alexfernandez]

Just a small point: my implementation does not require sequences to be passed whole, it can receive random bytes. It is indeed necessary to keep a lot of state and performance is going to be much worse. It does not matter much in pure JavaScript but can be a killer in C. Hope to publish it soon! :)