Zstd: Error 36 : Decoding error : Unsupported frame parameter

[rxrbln]

While adding zstd support for T2 Linux source mirrors and distributed binary packages I noticed that when I compress a file on x86-64 with --ultra -22 and decompress the file on i686 I get: Error 36 : Decoding error : Unsupported frame parameter

The same file compresses without error on x86-64.

[terrelln]

The maximum window size is smaller on 32-bit systems than it is on 64-bit systems. If you compress a large file (> 256 KB) with -22 --ultra you will get a window log of 27 on 64-bit systems. However, on 32-bit systems the maximum window log is 25, so the decompresser will complain.

It should work if you use -20 --ultra, if it doesn't there is a different issue.

The zstd format document specifies that decompressors should support window sizes of at least 8 MB. If you don't use --ultra, the window size is always less than 8 MB. So I would recommend not compressing with --ultra if you are worried about compatibility with 32-bit platforms, or even with other zstd decompressors.

[rxrbln]

Ok, fair enough - though I find it a bit irritating that a 32-bit build supports smaller window sizes. Given that there are still plenty of 32-bit machines (ARM, MIPS ;-) with enough RAM maybe 32-bit support can be made great again?

In any case a more obvious error diagnostic would help, too. Looking at the source this may be a candidate for: frameParameter_unsupportedBy32bits - "Frame parameter unsupported in 32-bits mode" ?

[Cyan4973]

frameParameter_unsupportedBy32bits error code was created for this purpose.
Somehow, over the course of development, the more generic frameParameter_unsupported took over, presumably because it is triggered earlier.

Short term action would be to re-enable frameParameter_unsupportedBy32bits, for a more explicit error message.

Longer term objective is to support long distances in 32-bits more.
But it will require care, as it makes the code more complex, and a naive implementation will significantly impact decoding speed in all situations for the sake of a rarely-used capability, which is not a good deal.

[Cyan4973]

OK, now I remember.
The change happened when finalizing the format specifications.
The question is wider than just --ultra -22.

According to the specification, a frame is allowed to ask for a very large amount of memory.
For example, it could tell "hello, I'm a frame of 2 TB, and back references can happen anywhere in my content, so please allocate 2 TB".
On its side, a decoder is allowed to answer "2 TB ? not even in your dreams".
Refusing a memory requirement is an important defence mechanism : a remote attacker could abuse the specification to make the target allocate way too much memory for its health.

The only recommendation is that a decoder _should_ be able to accept memory requirements up to 8 MB, but even that limit could be reduced if the decoder is not willing to be "generic" and target some known specific usage.

Anyway, bottom line, beyond 8 MB, all bets are off. It's implementation dependent, or even parameter dependent.
And it happens that, for some complex technical reason, the 32-bits version of the reference decoder is unable to accept frames with back-references > 32 MB. So it's its default limit.
The 64-bits version has a default limit of 128 MB, which corresponds to --ultra -22. It could be higher, it's just that a limit is needed to avoid crazy scenarios.

This leads me to believe that an even better solution would be to have a specific error code for this case : the decoder refuses a frame because it requires too much memory.

It could even be complemented with an advanced option, where the user could tell which maximum back-reference distance shall be tolerated in received frames, to ensure a maximum memory budget.

[Cyan4973]

First sketch implemented in dev branch

[Cyan4973]

error code and string has been made clearer in dev branch