Zstd: skippable blocks for streaming compression

[AbdulrahmanAltabba]

I have a scenario where I continuously receive streamed data and I want to compress this data in multiple frames with a pre-given FrameSize. I am using the C-API function ZSTD_CCtx_setPledgedSrcSize() for this purpose. The problem arise when the data stream stops or requests closing the file and starting a new file in a time point when the the current frame size is not yet reached. I can't close the frame in this case. I thought I could do padding with meaningless data till the end of the frame. However, this would not be clean when decompressing the file. If one could use something like skippable blocks, then I could fill the rest of the frame with these and the decompressor would skip them automatically. This could solve my problem. Otherwise I would appreciate any help to overcome this issue.

Thank you

[terrelln]

I am using the C-API function ZSTD_CCtx_setPledgedSrcSize() for this purpose

Why are you setting a pledged source size when it might not be accurate? Wouldn't a better solution be to not set the pledged source size, then you can have a frame of whatever size you please?

If you need to know the source size on the decompression side, you could communicate it out-of-band. That would allow you to over/under-estimate because you can have a protocol that adjusts it.

[pizzard]

after further investigation we found our analysis to be incorrect. we will look at pledged size further to see if they can work.

[pizzard]

I forgot to report. Setting the pledged size actually works as expected, if you use it correctly. Having blocks with larger pledged size than actual content decompress correctly to the end and then throw an unexpected end of input error (which can be caught and handled). So this can be closed.