Lz4: Low Compression Ratio

[Sweeper-Bot]

I am attempting to compress 16 bit binary values that have a large amount of repetition. With my current solution the values are compressing to 97%-100% of their original size. I am considering implementing an external dictionary or lowing the default acceleration value to <1, but I thought it would be wise to get some input from others who may be more experienced with LZ4 first. Is there a common cause for such a low ratio?

[felixhandte]

Hi @Sweeper-Bot, can you give us more details about your data? (Including a sample, maybe?) Do you mean that you're getting 0% to 3% reduction in size by applying compression?

I'd have to check to make sure, but I don't believe it's valid to select a negative acceleration factor.

LZ4 is a byte-level compressor, and will only compress pieces of the input for which it can find an identical >=5 byte match somewhere else in the stream. For packed 16 bit integers, this might happen only very rarely.

If you're not attached to LZ4 in particular, you might want to try Zstd. It might perform better on your data.

[Cyan4973]

While it's hard to tell from just the description, this type of data seems a bad fit for LZ4.
LZ4 only cares of repetitions which are at least 4+ bytes long (that's where the 4 comes from).
So that means at least 2 consecutive 16-bit integers which are exactly identical.
Depending on your source, this may be more or less likely.

Prefer to pass such data type through a filter before handing over the result to a compressor.
Common filters are :

• delta transform ( r[n] = s[n] - s[n-1] )
• zig zag encoding
• transpose (like blosc )

It will likely provide a better compression result, though by how much, it depends. The more "correlated" are consecutive values, the better the outcome.

[Sweeper-Bot]

Thanks for the input. I think I will be switching over to a different compression as was suggested.