Duplicati: Terrible memory leak in 7Z compression

[mnaiman]

During testing I found out that Duplicati.Server.exe is taking 1,5GB RAM.
After memory profiling and futher digging it show structures around LZMA.

So I ran same backup with ZIP compression (Duplicati never exceeded 100MB RAM) and same backup with 7Z compression (In half of backup memory went from 100MB->1,5GB and kept 1,5GB even when backup was finished).

Backup was easy from one local folder to another one (3x500MB files + system32-lot of small files, but I think it doesnt matter).

Up to now, I dont get if it is flaw in Dupllicati or Managed-LZMA library.
But Managed-LZMA library is not very well maintained (repo look abandoned) why duplicati is using it?
Compression is called 7z, but it doesnt produce 7z compatible files, but raw LZMA.

Why real 7Z SDK is not used? http://www.7-zip.org/sdk.html
It is maintained and it will produce 7z compatible files for sure.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[mnaiman]

Ok, it is not problem of Duplicati, but Managed-LZMA is flawed.

There is even unsafe code inside library and my debugging of the library itself ended with AccessViolationException, which indicates why there are memory leaks.

[mnaiman]

Im going to migrate all of my backups to ZIP now. If anybody interested in same approach, I have implemented recompress functionality to RecoveryTool of Duplicati without loosing backup history.
Of course at least temporary backup of remote storage is recommended.

[mnaiman]

I tried to decompress all my 10 remote storages and I found few simmilar erros:

Backup_Daily-b05c343d034cb4af39e115d8609f62e53.dblock.7z.aes - error: System.IO.InvalidDataException: Found invalid data while decoding.
   at master._7zip.Legacy.Lzma2DecoderStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at ManagedLzma._7zip.Decoder.FileBufferedDecoderStream.Fetch()
   at ManagedLzma._7zip.Decoder.FileBufferedDecoderStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at master._7zip.Utilities.UnpackSubStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at Duplicati.Library.Utility.Utility.CopyStream(Stream source, Stream target, Boolean tryRewindSource, Byte[] buf) in D:\dupgit\duplicati\Duplicati\Library\Utility\Utility.cs:line 91

When file is checksum is performed on file and compared to Backup_Daily-verification.json it is correct.
So corruption has to happened during compression of backup not during upload or because of for example media corruption.
I think it has connection to this bug (no prove), so my advice is to mark 7z module as experimental.

[joshmule]

Is this still an issue 2+ years later? Could we avoid this brokenness by using the xz / lzma2 libraries from the xz project itself? See https://tukaani.org/xz/

[ts678]

Identical post later at Is 7z safe to use yet? got an answer from lead author from a backend-storage view, however the 2016 issue here is coming from a different angle. I don't know if OP has any updates on this.

I assume use of https://github.com/weltkante/managed-lzma is the OP issue, and last update was 2 years ago (so not abandoned but not very well maintained). Duplicati's version looks like a Duplicati build from 5 year old source while waiting for a binary release. Though 2 years old, that's at nuget.org if we want it...

OP had a different package in mind though, and I have no ideas about performance, or suitability for use. Duplicati's design, I believe, compresses not at the file level but at the individual file block level, which by default is 100 KB, which might be too small to get best compression regardless of how well large files do.

Choosing sizes in Duplicati gets into the two sizes. In Duplicati, I believe compression is at the --blocksize level, and blocks from some files might not be compressed at all, as per the --compression-extension-file.

https://github.com/duplicati/duplicati/blob/55c76ca583726c7cc37eb7de3ba9f759982e9c5f/Duplicati/Library/Main/Operation/Backup/StreamBlockSplitter.cs#L114-L114

How the backup process works

[BlueBlock]

What is the benefit of using LZMA than over ZIP?