Zstd: How can I compress a directory?

[g666gle]

I need to compress a directory using zstd and I saw somewhere that there was a work around. Any ideas?

[terrelln]

zstd is a compressor, not an archiver. You can compress each file in a directory individually using zstd -r directory/. If you want to compress the whole directory you should use tar or libarchive.

For example, you can compress a directory with tar using zstd with the command line:

tar --zstd -cf directory.tar.zst directory/

If you have an older version of tar without native zstd support you can use:

tar --use-compress-program zstd -cf directory.tar.zst directory/

[g666gle]

How would you decompress the "dir_name.tar.zst" directory?

[terrelln]

tar --zstd -xf directory.tar.zst

You can google how to use tar for more detailed instructions.

[terrelln]

I'm going to close this issue, feel free to reopen if you have further questions.

[whit-colm]

How could I set the compression level on it? I know there's a GZIP variable but I don't know if it applies here. I tried the --options suggested by jsonn in libarchive/#957 but it says options isn't a valid argument. I'm using tar v1.32 if that helps. Thanks!

[terrelln]

Since zstd-1.3.8 we default to the compression level set in the ZSTD_CLEVEL environment variable if set

[sergeevabc]

$ zstd --version
*** zstd command line interface 64-bits v1.3.8, by Yann Collet ***

$ tar --version
tar (GNU tar) 1.31

$ tar --zstd -cf logs.tar.zst logs/
tar (child): zstd: Cannot exec: No such file or directory
tar (child): Error is not recoverable: exiting now
/Unix/tar: directory: Cannot write: Broken pipe
/Unix/tar: Child returned status 2
/Unix/tar: Error is not recoverable: exiting now

:(

[terrelln]

I suspect that tar can't find zstd in your path. Where did you install zstd? Does tar --use-compress-program zstd also fail?

[sergeevabc]

@terrelln, required files tar.exe and zstd.exe are placed and launched in the same folder.
tar --use-compress-program zstd also fails with the same aforementioned error.
The only successful workaround so far is tar cf - logs | zstd > logs.tar.zst.
I have packed the following tree in test.zip for you to replicate.

C:\Test                      
|-- Logs                     
|   |-- Tor                  
|   |   |-- freenode.Tor.log 
|   |   `-- server.Tor.log       
|   |-- freenode.log         
|   `-- server.log           
|-- msys-2.0.dll             
|-- msys-iconv-2.dll         
|-- msys-intl-8.dll          
|-- tar.exe                  
`-- zstd.exe

$ sha1sum test.zip
6de8be4fe94f6b95a1fe6832926bb6f30fae4f71 *test.zip                 

[terrelln]

I believe that this is a problem with your setup. I didn't even know that tar worked on Windows.

I suspect that the $PATH that tar gets doesn't align with Windows path, so it can't find zstd.

Does it work if you provide the full path the the zstd executable (with or without .exe), or with zstd.exe?

In either way, this isn't a bug in zstd, and isn't a problem with tar on Unix systems, but maybe a deficiency of tar on Windows, if they officially support that.

[sergeevabc]

@terrelln, have you tried to reproduce the case with the package I provided?

[terrelln]

Sadly, I don't have a Windows machine, so I won't be able to reproduce it. But, I'm confident it is a problem related to tar on Windows, not specific to zstd, so I'm not the right person to fix it.

[sergeevabc]

Workaround to create:
7z.exe -ttar a dummy "c:\Test\Logs" -so | zstd.exe --long -o logs.tar.zst

Extracting as usual:
tar.exe -xf logs.tar.zst

[alibitek]

When doing tar --zstd -cf directory.tar.zst directory/ is there an environment variable to specify the number of compression threads?
Aka what this option does: -T# : spawns # compression threads (default: 1, 0==# cores)

[Cyan4973]

Nope.
This feature has already been requested, and is currently part of our backlog.

[jboero]

I manage my way around it using pzstd instead of zstd. Much faster and you can control parallel threads via pzstd params or just use one for each core by default.

tar --use-compress-program pzstd -cf directory.tar.zstd directory/

extract:

tar --use-compress-program pzstd -xf directory.tar.zstd

[Cyan4973]

Next release will have ZSTD_NBTHREADS for this purpose :
https://github.com/facebook/zstd/blob/dev/programs/README.md#passing-parameters-through-environment-variables