Lz4: Bad compression ratio

[aleksazr]

(you guys are probably aware of LZ4 limitations like this...)

I have a binary file of 5712 bytes.
It consists of dword bit-fields.

Compressed with RAR, it gets down to 600 bytes.
Zip gets to 2700.
But "lz4 -9 table.bin" actually slightly increases the size to 5731 bytes.

You can get the file from here:
http://users.beotel.net/~gwh/lz4/table.bin

[Cyan4973]

LZ4 will only work if there are some segments of at least 4 continuous bytes repeated across the files.
Considering the nature of the data (16-bit bit fields), it seems unlikely.

In contrast, zip, rar, zstd and other compressors in this category will be able to generate compression from the presence of individual bytes of higher probabilities, which are frequent in this file.

So yes, this is an expected outcome.

[degski]

@Cyan4973 Thanks for that information, that gives good guidance as to use LZ4 or something else depending on the data-type. Or to change the data-type to allow for LZ4 to compress it.

[sgf]

If lz4 has requirements or restrictions on the source data, can it be listed in readme or wiki? I don't seem to see the relevant introduction at the moment. Am I missed something?

[felixhandte]

LZ4 can operate on any byte-stream, so in one sense there are no restrictions on the input. But it will only be effective when the data is compressible using the tools available in LZ4. That tool (the language LZ4 uses to describe messages it encodes) is described here and particularly here.

Two quotes from the above:

There is no entropy encoder

The minimum length of a match [...] is 4

These together explain why this data is incompressible in LZ4: LZ4's only tool for shrinking the representation of a message is to find repetitions of four or more contiguous bytes within 64KB of each other (hence the name, LZ4). While this happens very frequently in things like text, it seems like this doesn't happen in this input.

This simple design was selected so that the encoder and especially decoder can run at ludicrous speeds. More complex compressors like zstd have more tools available, and will be able to compress this kind of data better. (Although integer compression, in general, is a hard topic that frequently challenges general-purpose compressors.)

Hope that helps.

[sgf]

it seems that zstd is more suitable for most situations.
Although LZ4 has higher performance, it may not be effective in some cases due to insufficient versatility.

[terrelln]

it seems that zstd is more suitable for most situations.

Yes. LZ4 is great when you need extremely fast compression or decompression, and are willing to trade off quite a bit of compressed size to get it. For most use cases the tradeoff that zstd offers is more compelling, since you save a lot of compressed size and it is still very fast.