Lz4: ASAN issues in the new decompression loop in 1.9.0

[ppodolsky]

Hi, guys!
I have run large set of tests using your brand new decompression, and AddressSanitizer had found troubles with https://github.com/lz4/lz4/blob/dev/lib/lz4.c#L1684
Probably 16 should be replaced with length variable.
Here is ASAN output, worth to note that code in lz4.c is shifted someway because I patched it.

==544083==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x606000005040 at pc 0x000000721886 bp 0x7fffaaddbb50 sp 0x7fffaaddb300
READ of size 16 at 0x606000005040 thread T0
    #0 0x721885 in __asan_memcpy .../asan_interceptors_memintrinsics.cc:23:3
    #1 0x1577e51 in LZ4_decompress_generic .../lz4/lz4.c:1681:17
    #2 0x1577e51 in LZ4_decompress_fast .../lz4/lz4.c:1986

SUMMARY: AddressSanitizer: heap-buffer-overflow .../asan_interceptors_memintrinsics.cc:23:3 in __asan_memcpy
Shadow bytes around the buggy address:
  0x0c0c7fff89b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0c7fff89c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0c7fff89d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0c7fff89e0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c0c7fff89f0: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
=>0x0c0c7fff8a00: 00 00 00 00 00 00 00 00[fa]fa fa fa fd fd fd fd
  0x0c0c7fff8a10: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c0c7fff8a20: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
  0x0c0c7fff8a30: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd
  0x0c0c7fff8a40: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c0c7fff8a50: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc

[Cyan4973]

Thanks for report @ppodolsky ,

we apparently have an issue within LZ4_decompress_fast(),
I'm on it.

[ppodolsky]

Did you found the issue? I can try to make a reproducible example, but it can take a while

[Cyan4973]

Thanks @ppodolsky, issue is now reliably reproduced.
Fix should be comparatively simpler,
what really matters is that the CI test suite is augmented to always catch similar issues in the future.

[ppodolsky]

Great news, thank you