Zstd: Multi thread decompression

[nand28]

Are there any plans to support multi thread decompression? What other ways can the decompression speed be improved other than reusing statically allocated context and avoiding to time spent on allocation?

[Cyan4973]

There is a long-term plan on the topic of multi-threaded decompression, but we can't commit to any release date yet.

[nand28]

Goal is to reduce the time spent for decompression. Do you have any suggestions?

[Cyan4973]

It comes at the cost of increase cpu usage. But yes, it's possible.

A "simple" method is already available and used in pzstd, where input is split into multiple independent frames, which can then be decompressed in parallel. This comes at the cost of some compression ratio, since inter-frame correlation is lost in the process. Also, this is only applicable if input size is "large" enough, so that it makes sense to split it into multiple independent frames.
Any 3rd party application is free to use the same method if they wish. The concatenation of multiple frames is still a valid stream for zstd, so it can be decoded by zstd CLI and API later on, even by applications unaware of the splitting strategy.

Multi-threaded decompression of a single frame is a topic too, but a much more complex one. As stated, we plan to work on it, but as it's complex and time is scarce, we can't provide at this time any release date for such a feature.

[mirh]

I thought it was already supported?
You just need to multi-thread compress with -T beforehand.