Zstd: Is a Zstd plug-in dll for 7-zip possible?

[Hoggyy]

Hi,

For people not aware, Tino has lost interest in his "7-zip Zstd" fork of 7zip. Which is sad, because I use Zstd many times a day with it. But unfortunately. I don't know much about coding anymore. And it still seems like Igor is still not interested in incorporating Zstd into his mainstream program (outside of plugging in a dll into a "Codecs" folder).

I tried plugging in a couple dll's I found in an executable I downloaded from here, into the regular 7-zip's "Codecs" folder, but 7-zip will not recognize them.

Is there any hope for a plug-in dll that could work with the regular 7-zip? Or is anyone here interested in taking over the 7-zip Zstd project (even if it's to only support Zstd, rather than all of the other codecs)?

..... Ideally, it would also be nice if there was a 7z.dll that I could plug into Directory Opus, as well - for entering and dealing with 7-zip archives as if they were standard folders. (Just like the 7z.dll found within the "Codecs.7z" archive on Tino's 7-zip Zstd project page.) But I don't know how do-able that might be.

Many thanks for any help or consideration. I know there would _at least_ be several of us 7-zip Zstd users that would be EXTREMELY grateful.

[DenisAnisimov]

You can try this one - https://sourceforge.net/p/sevenzip/discussion/45797/thread/6db98beb/

[Cyan4973]

Your new 7zip plugin is now listed on the Zstandard homepage.

[Hoggyy]

In order to get the compression I'm looking for (equal to Tino's), I need to limit the thread count to 1 - which I think probably negates the reported speed increase with v1.3.4.
Denis was saying that's because native Zstandard multi-threading is still in the unstable state. So is that getting close to stable, at this point?

(I hoping to see that when I soon get the AMD Threadripper 16/32-core, that it tears through even large 7-zips at level 22 like it was tissue paper. =) )

[Cyan4973]

I need to limit the thread count to 1 - which I think probably negates the reported speed increase with v1.3.4.

There were several speed-oriented improvements mentioned in the release note :

• general speed improvements for compression and decompression : that was for single-threaded performance, so it's still applicable
• faster compression with asynchronous mode : this is specific to the cli (command line interface), and is therefore not applicable to the library
• better combination of long range mode + multi-threading : this one is indeed MT specific. But I suspect long range mode isn't used in the 7zip plugin anyway, so it's a non-event.
• faster compression levels : this is accessible from the stable API, so yes it's available. Just, I'm not sure if the plugin is able to expose them. Plus, I don't believe this capability to be useful in the context of archiving application.

Zstandard multi-threading is still in the unstable state. So is that getting close to stable, at this point?

Multi-threading feature has been available for a long time now, and works well.
It's not going to change, the format is stable.

But accessing Multi-threading is done through an API (Application Programming Interface).
This API exists, but is labelled "experimental", because it is tied to a larger package, named "new advanced API", which offers much more capabilities.
However, publishing a new API in the "stable" category is a very risky move, with serious long term consequences, because nothing published there can ever change in the future. Hence, we pay a lot of attention to this decision.

Expect this new API to be labelled "stable" sometimes this year.
This is just an "intention" so far, not a hard commitment.

[Hoggyy]

So out of curiosity, if either of you know, was Tino using the experimental API, or did he write his own multi-threading that pertains to Zstandard? (It almost looks like he wrote his own, if I'm understanding somewhat, of what is in his 'C/zstdmt' folder of his source code.)

Thank you guys very much for being patient with me. And as a matter of fact, it was Zstandard(7-zip) that first prompted me to try and learn some Javascript (for creating Directory Opus functions using 7-Zip Zstandard). And now it was Tino's leaving that seems to have kickstarted a bit of desire to try and understand some C/C++ and current-day compiling... At least just enough to be dangerous.. Cobble up some Frankenstein code. :)

[Cyan4973]

If I remember correctly, Tino made his own code, based on pzstd design.
That's because pzstd was already available and well-tested when Tino started his own implementation. The newer zstdmt implementation, integrated within libzstd, was completed only a few months later.

Both MT implementations generate frames compatible with zstd decoder.
But they are nonetheless different.

pzstd, and by extension Tino's version, generate multiple appended frames. This is easier to parallelize. But a consequence is that each frame is fully independent, therefore damaging compression ratio in the process (by a small amount).

In contrast, zstdmt, the version integrated within libzstd, generates a single frame, like the single-threaded variant. That means, it's better able to preserve compression ratio. There's also a small advantage for headers, since the frame knows up-front how much data it contains, whereas in the multiple-frames approach, each frame only knows its own data.

However, zstdmt is also more complex. So it's quite normal for a 3rd party implementation to start with a simpler design such as pzstd.

[denfromufa]

btw, the 7z-zstd i was using and listed on homepage of zstd is now deleted both on github and the homepage:

https://github.com/mcmilk/7-Zip-zstd/

[denfromufa]

This is what I was using:

7z1701-zstd-x64.zip

[Hoggyy]

It sucks that Tino's entire page on Github is offline now.

Now I have to pay extra special attention to backing up and safeguarding his "codecs.7z" archive.
I use the "7z.dll" to replace the version in Directory Opus - in order to access and alter archives made in any of the extended codecs, as if they were standard directories.

Any chance that anyone else is able to make newer versions happen, with newer versions - at least for Zstandard and the other mainstream 7zip codecs?

@DenisAnisimov - Are you able to make that happen at all, even if you charge for it, like you do for your standard "TC4Shell"?

[DenisAnisimov]

Several users asked me to add support for the --long option in the 7-Zip plug-in. But the ZSTD_CCtx_setParameter function is in the "ADVANCED AND EXPERIMENTAL FUNCTIONS" section and it is recommended that I do not use this function with dynamic linking. Is this requirement rigid in the context of single ZSTD_p_windowLog parameter? Perhaps the function and this parameter is already stable and I can use them with dynamic linking?

[terrelln]

@DenisAnisimov we are planning on stabilizing the ZSTD_CCtx_setParameter() function in the next release, or the one after that, and then it will be safe to use.

[mcmilk]

Hello,

I just updated the release on this site: https://github.com/mcmilk/7-Zip-zstd/

Maybe this issue could be closed also ?
Sorry for the delay ...

[Cyan4973]

Sounds great @mcmilk ! Happy to see you back !