Zstd: Empty string decompression is broken with option combination

[andrew-aladev]

Hello. I was making light options combination test and found a bug. This is an extracted test:

#include <zstd.h>
#include <stdlib.h>

int main () {
  ZSTD_CCtx* cctx = ZSTD_createCCtx();
  if (cctx == NULL) exit(1);

  size_t r = ZSTD_CCtx_setParameter(cctx, ZSTD_c_contentSizeFlag, 0);
  if (ZSTD_isError(r)) { ZSTD_freeCCtx(cctx); exit(2); }

  char comp[100];
  size_t comp_size = ZSTD_compress2(cctx, comp, 100, NULL, 0);
  ZSTD_freeCCtx(cctx);
  if (ZSTD_isError(comp_size)) exit(3);

  ZSTD_DCtx* dctx = ZSTD_createDCtx();
  if (dctx == NULL) exit(4);

  r = ZSTD_DCtx_setParameter(dctx, ZSTD_d_windowLogMax, 20);
  if (ZSTD_isError(r)) { ZSTD_freeDCtx(dctx); exit(5); }

  char decomp[100];
  size_t decomp_size = ZSTD_decompressDCtx(dctx, decomp, 100, comp, comp_size); // works
  if (ZSTD_isError(decomp_size)) { ZSTD_freeDCtx(dctx); exit(6); }

  ZSTD_inBuffer comp_buf = { comp, comp_size, 0 };
  ZSTD_outBuffer decomp_buf = { decomp, 100, 0 };
  r = ZSTD_decompressStream(dctx, &decomp_buf, &comp_buf); // broken
  ZSTD_freeDCtx(dctx);
  if (ZSTD_isError(r)) exit(7);

  return 0;
}

Please run gcc fit.c -lzstd -pthread -o fit && ./fit ; echo $?. It returns 7.

Compressor has contentSizeFlag: 0 and decompressor can have any valid windowLogMax value. This combination is not able to compress and decompress empty string using ZSTD_decompressStream. It throws ZSTD_error_frameParameter_windowTooLarge.

We can fix this test by setting contentSizeFlag to 1 or windowLogMax to 0.

tests/fuzzer.c looks like a monster, but actually it is very weak. Such bugs are his job. I can recommend to rewrite it based on all (reasonable) options combinations. My example combinations generator written in ruby is here.

[andrew-aladev]

I see the problem here. Please review ZSTD_adjustCParams_internal.

• srcSize can be ZSTD_CONTENTSIZE_UNKNOWN when not known.
• note : for the time being, srcSize==0 means "unknown" too, for compatibility with older convention.

So this is the problem for functions calling ZSTD_adjustCParams_internal. Some functions may want legacy behaviour, some functions doesn't.

See

size_t ZSTD_compress2(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize)

srcSize in this function can't work as ZSTD_CONTENTSIZE_UNKNOWN. This issue can be fixed by core developers only, it requires large refactoring.

There is a workaround for everyone received this bug. You can just return empty string if srcSize is zero. Empty string is a valid for decompressor.

Another workaround is to force minimal window log for zero input. This workaround is required for streaming functions.

if (srcSize === 0) {
  r = ZSTD_CCtx_setParameter(cctx, ZSTD_c_windowLog, 10);
  if (ZSTD_isError(r)) { ZSTD_freeCCtx(cctx); exit(2); }
}

Or you can simply force contentSizeFlag to be always true. This is the easiest workaround.

[felixhandte]

@andrew-aladev,

Thanks for reporting this! Yeah, this is a result of the slow migration we've been making between different API versions, away from 0 meaning unknown and towards it meaning 0. We will get a fix for this together, but maybe not immediately.

[terrelln]

@andrew-aladev the issue should be fixed now.

Whenever we've reached the library internals srcSize == 0 should mean 0, anywhere it doesn't is a bug. Only the specific endpoints which are documented to treat 0 as unknown should have that behavior.

[andrew-aladev]

I've created new release for ruby-zstds, content size with zstd 1.4.5 works perfect, perforamnce was automatically improved +10%. zstd 1.4.5 is excellent release. Thank you.