Zstd: [request for help] Undo zstd -r folder -o output.zst

[twnaing]

This is not the issue of the zstd itself. This mess is created by me.

I didn't read enough the manual and run the following command

$ zstd -r folder -o output.zst

The following command gave me back a single file called output

$ unzstd output.zst 

The output file has all the contents of the files under the folder concatenated.

Is there some tools or programs to un-concatenate the single file into multiple original files?

This is the only backup file I have and I need the backup.

[Cyan4973]

arf, no, unfortunately, all compressed frames are just stored back to back in the same file output.zst.

While there would be a way, at least in theory, to separate each frame, and therefore find the boundaries of each file, another problem is that none of these frames contain the file name, nor the position in directory tree. So you would end up with a bunch of nameless files.
Depending on your use case, that might be a start, or that might be useless.

The proper way to archive is to combine zstd with tar, which is in charge of preserving file metadata.

[twnaing]

@Cyan4973 Thank you for the information.

Is there any tools or programs that can help me to separate each frame?

[Cyan4973]

None that I can think of off the top of my head.
By default, the CLI will just decompress all frames back-to-back into the same decompressed file, which is not the outcome you are looking for.

Most likely, you'll have to program it yourself, using the lz4frame.h decompression API.

_edit_ : duh, I confused that part.
Indeed, for zstd frames, use the ZSTD_decompressStream() API.

[twnaing]

@Cyan4973 thank you again. I think I am out of my depth to program using lz4frame.h

[felixhandte]

I think @Cyan4973 probably meant zstd frame, not lz4 frame.

This is not necessarily all that difficult, although as mentioned you will not be able to recover filenames. Each compressed file will be one zstd frame. So you can use ZSTD_findFrameCompressedSize(), for example, to partition the compressed file into separate individually compressed files, which you can then decompress. (Alternatively, a hacky version would just split the compressed file at every instance of "\x2b\xb5\x2f\xfd", which is the magic value that signals the start of a new frame.)

[terrelln]

@twnaing using the API you can call ZSTD_findFrameCompressedSize() to find the size of the first frame, decompress only that, and repeat the process at the second frame.

[Cyan4973]

I think @Cyan4973 probably meant zstd frame, not lz4 frame.

Right, I confused that part.

[twnaing]

Thank you all for the input. I will try to read on how to write a C program. It will take a while for me, I think.

[twnaing]

Meanwhile, what I should have run is described in https://github.com/facebook/zstd/issues/1526#issuecomment-464374334