Zstd: Hints for "Decoding error (37) : should consume entire input"

[klauspost]

I have an issue, where I get the error in the topic, and I was wondering if you have some hints as to why it could occur.

It is probably a problem with my compression: https://github.com/klauspost/compress/issues/111

I have truncated the input (it is not a complete stream), but the error shows after just one frame: https://download.klauspost.com/broken-stream.zst

I have a hard time seeing how I could have extra data at the end of a block and still be able to decode it. But I am probably doing something wrong :/

[klauspost]

Weird, the size is off by 3.

I am pretty sure I keep to the "Block_Size is the size of the block excluding the header" at all times... weird.

[Cyan4973]

A classical issue when using streaming compression is to forget checking that all data has been flushed.
This can happen when the output buffer is already full, so streaming cannot flush more.

Another possibility could be that the last block is missing the isLastBlock bit in the header, so the decoder is looking for yet another block, but there is none.

Since the encoder is your implementation, it can be prone to very different scenarios than the ones we experienced in the reference encoder, so I'm not sure how much the comparison can help.

the error shows after just one frame: https://download.klauspost.com/broken-stream.zst

Actually, the frame scanner cannot even finish the first frame :

/zstd -l broken-stream.zst
Frames  Skips  Compressed  Uncompressed  Ratio  Check  Filename
Error while reading block header
Error while parsing "broken-stream.zst"
     0      0     1.39 MB                        None  broken-stream.zst

./zstd -v --list broken-stream.zst
Error while reading block header
Error while parsing "broken-stream.zst"
broken-stream.zst
# Zstandard Frames: 0
Window Size: 0.25 MB (262144 B)
Compressed Size: 1.39 MB (1457664 B)
Check: None

This is in line with above potential scenarios.

[klauspost]

It is truncated from the beginning of a 200MB file, so it should not complete fully. And everything is in a single frame, so no "last" block.

I made a debug build with level 5 logging, here is what it prints:

e:\dev\zstd-1.4.0\build\VS2010\bin\Win32_Debug>zstd.exe -d broken-stream.zst
zstd_decompress.c: ZSTD_createDStream
zstd_decompress.c: ZSTD_initDStream_usingDict
zstd_decompress.c: ZSTD_decompressStream
zstd_decompress.c: input size : 18
zstd_decompress.c: stage zdss_init => transparent reset
zstd_decompress.c: stage zdss_loadHeader (srcSize : 18)
zstd_decompress.c: header size : 5
zstd_decompress.c: stage zdss_loadHeader (srcSize : 13)
zstd_decompress.c: header size : 6
zstd_decompress.c: stage zdss_loadHeader (srcSize : 12)
zstd_decompress.c: header size : 0
zstd_decompress.c: Consume header
zstd_decompress.c: ZSTD_decompressBegin_usingDDict
zstd_decompress.c: Control max memory usage (256 KB <= max 131072 KB)
zstd_decompress.c: inBuff  : from 0 to 131072
zstd_decompress.c: outBuff : from 0 to 393232
zstd_decompress.c: stage zdss_read
zstd_decompress.c: neededInSize = 3
zstd_decompress.c: ZSTD_decompressContinue (srcSize:3)
zstd_decompress.c: stage zdss_read
zstd_decompress.c: neededInSize = 51924
broken-stream.zst    : 0 MB...     zstd_decompress.c: ZSTD_decompressStream
zstd_decompress.c: input size : 51918
zstd_decompress.c: ZSTD_decompressContinue (srcSize:51924)
zstd_decompress.c: ZSTD_decompressContinue: case ZSTDds_decompressBlock
zstd_decompress.c: ZSTD_decompressContinue: case bt_compressed
zstd_decompress_block.c: ZSTD_decompressBlock_internal (size : 51924)
zstd_decompress_block.c: ZSTD_decodeLiteralsBlock : 35299
zstd_decompress_block.c: ZSTD_decodeSeqHeaders
zstd_decompress_block.c: ZSTD_decompressSequences
zstd_decompress_block.c: ZSTD_decompressSequences_body
zstd_decompress_block.c: ZSTD_decompressSequences_body: after decode loop, remaining nbSeq : 0
zstd_decompress.c: ZSTD_decompressContinue: decoded size from block : 131099 
broken-stream.zst : Decoding error (37) : should consume entire input 51918 != 51915
````
The error in last line shows that values of:

        DISPLAYLEVEL(1, "%s : Decoding error (37) : should consume entire input %d != %d\n",
                        srcFileName,inBuff.size, inBuff.pos);

```

So it does complete the first block. I can't really explain why you see something else.

Any hints as to what it sees as the problem with -l?

[felixhandte]

One thing I'll note is that your first block decompresses to 131099 bytes, which is larger than the 131072 B (128 KB) maximum block size in the zstd spec.

[klauspost]

There is is! Awesome! Thanks a bunch for the sharp 👀