Lz4: What is the best way to include lz4 frames into stream of zstd frames

[scherepanov]

HI,
I am about to engage to write near-real time (multithreaded) zstd/lz4 compressor
Idea is to use higher compression levels for zstd by default, but if it not keeping up switch to lower level. And if it is not holding again fell into lz4.
I should be able to sustain 1GB/sec prolonged, with spikes to 2GB/sec.
I measured multithreaded lz4 consuming 3GB/sec of input stream on recent server CPU, scaling with hyperthreading... it is the only program I know that almost linearly scaling with hyperthreaded cores.
But slower zstd provides better compression and will be used by default. And zstd does not scale as well as lz4 even on physical cores.
So...
What is the best way to add lz4 frame to stream of zstd frames. How compatible lz4 and zstd framing standards.
Ideally zstd framing standard should have frame type LZ4.
Suggestions how to dynamically switch between lz4 and zstd frames will be appreciated
Thank you

[terrelln]

I think it is out of scope to have zstd define a frame type for LZ4. However, zstd frames and lz4 frames are compatible. Both have different 4 byte magic numbers at the beginning of the frame, and each streaming decompressor (zstd and lz4) knows to stop at the end of its frame. At the beginning of each frame, you can read 4 bytes, and dispatch to either lz4 or zstd.

Additionally, they both have the same skippable frame magic numbers, if you need that.

[Cyan4973]

If you use LZ4 Frame format (which is default format of lz4frame api), you'll be able to write them into a single file or stream alongside ZSTD frames. They can be appended in no particular order, and as @terrelln underlined, you can even mix in a few skippable frames to transport custom information.

Then you just need a decoder which can understand both kind of frames.

It happens that the new experimental target zstd4, that you'll find in latest zstd release (in /programs directory), can do exactly that. It's a CLI solution though.
For API, there is currently no automated method, and it's still necessary to switch between frames manually, using the magic number as indication.

[scherepanov]

Zstd4 is a perfect idea. At least I will have a client that will be able to decode combined Lz4/zstd frame stream. No need to write my own combined lz4/zstd decoder, which is nice!
Please consider adding decoding capabilities for LZ4 to Zstd decoder, on API level, depending on frame magic number. This should be rather straightforward.

For coder, I would be nice if one API would allow to compress either in LZ4 or Zstd, depending on current needs. I can use two separate lz4 and zstd API's, it is not very handy but quite usable.

[Cyan4973]

We have now this request in our backlog.
It will take some time to make it a real option, but we'll get there ...