Zstd: 1.4.7 dictionary tests fail on i586 and s390x

[bmwiedemann]

Describe the bug
dictionary tests fail on i586 and s390x

To Reproduce
Steps to reproduce the behavior:

1. on openSUSE or Debian do
2. osc checkout Archiving/zstd && cd $_
3. osc build --vm-type=kvm -j1 --clean openSUSE_Factory i586

Expected behavior
tests should pass the same way they did in 1.4.5

Screenshots and charts
build log:
``` ===> dictionary tests

• test with raw dict (content only)

• Create first dictionary
  ! Warning : data size of samples too small for target dictionary size
  ! Samples should be about 100x larger than target dictionary size
  Loading /home/abuild/rpmbuild/BUILD/zstd-1.4.7/tests/../programs/zstdcli.c... ^M^M ^MTrying 5 different sets of parameters
  WARNING: The maximum dictionary size 112640 is too large compared to the source size 688542! size(source)/size(dictionary) = 6.112766, but it should be >= 10! This may lead to a subpar dictionary! We recommend training on sources at least 10x, and preferably 100x the size of the dictionary!
  HUF_writeCTable error
  Failed to select dictionary
  HUF_writeCTable error
  Failed to select dictionary
  HUF_writeCTable error
  Failed to select dictionary
  HUF_writeCTable error
  Failed to select dictionary
  HUF_writeCTable error
  Failed to select dictionary

^Mdictionary training failed : Error (generic)
make: * [Makefile:330: test-zstd] Error 1
```

Desktop:

• OS: openSUSE-Tumbleweed
• Version 20201215
• Compiler gcc10
• Flags -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type -flto=auto -ffat-lto-objects
• Build system: Makefile

Additional context

How openSUSE builds

Current build logs can be found at
https://build.opensuse.org/package/live_build_log/Archiving/zstd/openSUSE_Factory/i586

[ismail]

On openSUSE we compile both the library and the tests with some flags and turns out, if you do

export CFLAGS="-O2 -fstack-protector"
export CXXFLAGS="-O2 -fstack-protector"
make -C tests test-zstd

test will crash on i586 and s390x, removing -fstack-protector fixes the crash, didn't do a git bisect yet.

[ismail]

Did a git bisect and https://github.com/facebook/zstd/commit/a90779397ad6372429188253eb00ed4c39f34912 is the commit that caused this.

[Cyan4973]

Thanks for the report @bmwiedemann .
The issue is confirmed on 32-bit systems, and also confirmed the exact commit involved (thanks @ismail !).

Investigation ongoing...

[animalize]

Will there be a v1.4.8 hotfix for this?

[Cyan4973]

OK, so this was an alignment issue on one internal buffer. Thanks @terrelln for the fix !

In many cases, this issue is inconsequential, either because unalignment is not possible due to compiler decision,
or simply because the cpu doesn't care about unalignment.
In this case, x86 32-bit doesn't care about alignment, but the code does, and returns an alignment error.
That's because it could become an issue when alignment _must_ be strictly respected, such as for MIPS or older ARM designs for example.

Even though the "damage" seems limited (x64 and arm64 targets are safe), it probably deserves a hotfix.