Zstd: Stability of parallel compression

[marxin]

Just to be sure. When using the parallel compression, is the compressed stream stable?
I mean can it vary when compression is repeated or is it deterministic?

[Cyan4973]

Compression is reproducible.

[terrelln]

The single-threaded output will be different than the multi-threaded output. However, both are deterministic, and the multi-threaded output produces the same compressed data no matter how many threads you use.

[marxin]

Great, good job made!

[marxin]

One further question: is the compressed output dependant on a number of threads that are used for compression?

[terrelln]

One further question: is the compressed output dependant on a number of threads that are used for compression?

No. Single-threaded output is different than multi-threaded output, but the multi-threaded output is the same for any number of threads.

[animalize]

the multi-threaded output is the same for any number of threads.

When:

• use ZSTD_e_end end directive
• output buffer size >= ZSTD_compressBound()

The job number is calculated by ZSTDMT_computeNbJobs() function.

Then in this case, different thread number may produce different compressed data, is this intended?

[terrelln]

Then in this case, different thread number may produce different compressed data, is this intended?

No. I'd call that a bug. I've opened Issue #2327 to track it.

If you need to work around this bug, don't start your streaming job with ZSTD_e_end. Pass at least one byte of input with ZSTD_e_continue before calling ZSTD_e_end.