Zstd: API help

[jgarvin]

I want to stream to and from a file, but I want to control the input and output buffers, so my code will decide when the uncompressed buffer is full, then tell zstd to compress it as a whole. The simple API works fine for compression, but then on decompression I need to the size of the compressed frame. I don't see an obvious get_frame_size or get_compressed_size call. Is there an equivalent? I assume this information must be stored in the format?

It looks like the the buffer-less decompression functions might be what I want, since they give you a source size. But they require you to retrieve frame parameters, and do something with frameContentSize, which I'm not sure if it's the compressed size or the decompressed one, or if it's the size of the frame, or if it's the size of the user data in a skippable frame, or....?

For context, I have some file reading code that currently uses LZ4 with custom framing that I'm trying to port to zstd. I'd like to avoid custom framing with the port so that tools like the zstd command line utility will work on the files. The current framing is just 2 uint64s, a compressed size and an uncompressed size. Use of these is pretty simple, the code just calls lz4 compress/decompress with the sizes in the frame, either consuming or producing data in a loop. The code assumes it controls the source and destination buffers, and that a whole frame is compressed/decompressed at a time.

[terrelln]

I think what you want is the streaming API. It can compress and decompress to and from buffers of your choosing. You can either choose to compress the whole file into one frame, or compress it into multiple frames, though I would recommend one frame. Then you can decompress that frame, and the API will alert you when the frame has ended.

There are streaming compression and streaming decompression examples, and the documentation is in zstd.h.

[Cyan4973]

I confirm @terrelln statement : if you wish to keep your current compression process, that is compress input "one by one" using the simple API, and appending all these elements together, the decompression side will work fine with the streaming API (ZSTD_decompressStream).

You can feed ZSTD_decompressStream() with any amount of data you wish.
If you give too much, ZSTD_decompressStream() will stop at frame boundary (with a return code of 0) and tells how much bytes it generated into dst buffer. If dst buffer is not large enough, it will fill it as much as it can, and require to be called again to finish the work (return code > 0). If input is not large enough to decode a complete frame, it will decode as much as it can, and ask for more data (return code > 0).

On return == 0, you know that a complete frame has been decoded. Whatever input that has not been consumed is likely the beginning of the next frame. This way you can retrieve elements one by one without mixing them.

The buffer-less decompression functions could also work, but they are significantly more complex to use. I would recommend to stick with the "stable" streaming API. Buffer-less streaming feature tricky advantages and restrictions, and is rather reserved for experts already familiar with the interface.

[Cyan4973]

As this helped your topic @jgarvin ?

[jgarvin]

Hey thanks for the fast response sorry I didn't reply sooner. I'm using the streaming API for now but I don't think it's what I want. I'm trying to write a zero copy API, and the fact that the end stream and flush functions can have bytes left over according to the docs suggests to me that zstd is doing buffering internally. I want zstd to not keep a buffer itself so I can be sure end to end that no bytes are copied. Is there a buffer-less example? I didn't see one in examples/.

[Cyan4973]

Indeed, the regular streaming API features an internal buffer. That's why it is more flexible and safer to use.

For a zero-copy API, you can turn your eyes on the more advanced synchronous bufferless API, which is quite likely the one you want. Be aware though that it's more complex to use. Follow the instructions in zstd.h for proper usage.

There is no example available, because this advanced API is not part of the "stable API" section, and we want casual users to concentrate first of the "stable API".

[jgarvin]

I will take a look, but I'm curious if I can just use the simple API. Since I'm already handling buffering and only invoking the compressor when my buffer is full, what is the difference between me compressing with the simple API with my own buffering and someone using the streaming API? Is that buffering the only thing the streaming API is providing, or is there additional setup/teardown work that gets done every ZSTD_compress call?

[Cyan4973]

The simple API generates independent frames. One invocation generates one frame.
The streaming APIs, both in buffered and bufferless modes, use previous data to better compress future data. All data provided will be part of the same frame.
The most visible difference with simple API will be compression ratio.

[jgarvin]

OK, I think that makes sense, but I'm wondering: what is the difference between me doing my own buffering and using the context API, and using the streaming API? Does it have the same issue where the compression ratio will suffer because it won't carry state between frames?

My expectation is that if you have a compression algorithm that can compress a buffer, the way you make it streaming is by accumulating a buffer, compressing it and framing it, accumulating a new buffer, compressing it and framing it, ad nauseum. But the header comments on ZSTD_decompressedSize suggest that when using streaming the sizes are not in the frame. So the streaming API is for when you want to stream, but not keep anything in a buffer... Except we established earlier it buffers internally, so I'm left not really understanding what its use case is and what advantage it has over the simple compress, frame, compress, frame, etc technique.

[terrelln]

Each zstd frame is made up of a series of blocks. Each zstd frame is independent, but the blocks in the frame are dependent on the data in the last window size bytes.

When you compress a bunch of independent frames, the compression ratio will suffer because the current frame can't match data in the previous frame.

The streaming API produces 1 frame if ZSTD_endStream() is only called once. It produces the frame header, and then compresses 1 block at a time. The blocks are allowed to reference data in the previous blocks. Then at the end of the input data, you call ZSTD_endStream(), which will end the frame.

The streaming API maintains internal buffers for a few reasons. It always needs to be able to access the last window size bytes (input for compression, output for decompression). It collects your input until it has a full block of data to (de)compress. It collects the output so you can provide an output buffer of any size.

[Cyan4973]

what is the difference between me doing my own buffering and using the context API, and using the streaming API?

If by "context API", you mean ZSTD_compressContinue(), then there is no difference.
ZSTD_compressStream() does just that : internally, it's only a client of the bufferless streaming API. The benefit is that it takes care of all the hassle : user can feed input of any size, provide destination buffer of any size, flush whenever they want, etc.
You can also create your own version, it's possible and could have different properties, possibly more suitable for your application. You would have more control over buffers too.

the header comments on ZSTD_decompressedSize suggest that when using streaming the sizes are not in the frame

Indeed, when using streaming, the encoder doesn't know the size of data to compress : it will learn it, when ZSTD_endStream() will be invoked. At which time, it's too late to "change" the header, which was already sent.
(note : an exception to this rule is when a pledgedSrcSize argument is provided).

we established earlier it buffers internally,

It buffers only up to window size.
The streamed content itself can be many times that size. ex : for level 1, window size is 512 KB, but the streaming interface has no problem compressing ~200MB silesia corpus : it only buffers the last 512 KB, not the full 200 MB.

Furthermore, as explained by @terrelln, the interface works block by block. Each block is <= 128 KB. When it reaches that size, the API generates and output a compressed block.
It doesn't matter if the input is buffered or not, because the compressed block is nonetheless produced. The first block will also generate the frame header. So by the time we reach the 2nd block, it's already too late : frame header is produced, it cannot be "retrofitted" to add some "decompressed size" field.

I'm left not really understanding what its use case is and what advantage it has over the simple compress

The streaming API guarantees it can handle input of _any_ size in a bounded memory budget.
The "simple" API requires access to the whole input, and must be provided enough destination buffer space to successfully compress or decompress.
When data to handle is large, the "simple" API memory requirement can become unpractical (think about GB or TB-level data).

[Cyan4973]

Have these answers helped you in your implementation ?

[jgarvin]

Thanks for the great answers. I haven't had a chance to return to looking
at zstd because of other work but I think things are clear enough you can
go ahead and close. I can always refer to this later and I'm sure there are
others that will appreciate the Google result ;)

On Mar 21, 2017 2:56 PM, "Yann Collet" [e-mail] wrote:

Have these answers helped you in your implementation ?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/facebook/zstd/issues/521#issuecomment-288199229, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAGPsn7LdEKxlBHMa52VusJ3nODAB036ks5roCttgaJpZM4LuQTT
.