Zstd: make check fails on Windows (MSYS2 + mingw-w64)

[vtorri]

file ../programs/zstd
../programs/zstd: PE32+ executable (console) x86-64, for MS Windows
ZSTD=" ../programs/zstd" ./playTests.sh

Starting playTests.sh isWindows=true ZSTD=' ../programs/zstd'

===> simple tests
test : basic compression

tmp : 32.50% ( 65537 => 21298 bytes, tmp.zst)
test : basic decompression
tmp.zst : 65537 bytes
test : too large compression level => auto-fix
Warning : compression level higher than max, reduced to 19
tmp : 31.54% ( 65537 => 20669 bytes, tmp.zst)
error: numeric value too large
test : --fast aka negative compression levels
tmp : 55.93% ( 65537 => 36656 bytes, tmp.zst)
tmp : 56.31% ( 65537 => 36902 bytes, tmp.zst)
tmp :100.02% ( 65537 => 65551 bytes, tmp.zst)
error: numeric value too large
Incorrect parameters
Usage :
zstd.exe [args] [FILE(s)] [-o file]

FILE : a filename
with no FILE, or when FILE is - , read standard input
Arguments :
-# : # compression level (1-19, default: 3)
-d : decompression
-D file: use file as Dictionary
-o file: result stored into file (only if 1 input file)
-f : overwrite output without prompting and (de)compress links
--rm : remove source file(s) after successful de/compression
-k : preserve source file(s) (default)
-h/-H : display help/long help and exit
test : too large numeric argument
error: numeric value too large
test : set compression level with environment variable ZSTD_CLEVEL
tmp : 33.15% ( 65537 => 21726 bytes, tmp.zst)
tmp : 61.55% ( 65537 => 40341 bytes, tmp.zst)
tmp : 33.15% ( 65537 => 21726 bytes, tmp.zst)
tmp : 32.50% ( 65537 => 21298 bytes, tmp.zst)
Ignore environment variable setting ZSTD_CLEVEL=-: not a valid integer value
tmp : 32.50% ( 65537 => 21298 bytes, tmp.zst)
Ignore environment variable setting ZSTD_CLEVEL=a: not a valid integer value
tmp : 32.50% ( 65537 => 21298 bytes, tmp.zst)
Ignore environment variable setting ZSTD_CLEVEL=+a: not a valid integer value
tmp : 32.50% ( 65537 => 21298 bytes, tmp.zst)
Ignore environment variable setting ZSTD_CLEVEL=3a7: not a valid integer value
tmp : 32.50% ( 65537 => 21298 bytes, tmp.zst)
Ignore environment variable setting ZSTD_CLEVEL=50000000000: numeric value too large
tmp : 32.50% ( 65537 => 21298 bytes, tmp.zst)
test : override ZSTD_CLEVEL with command line option
tmp : 56.31% ( 65537 => 36902 bytes, tmp.zst)
test : compress to stdout
tmp : 32.50% ( 65537 => 21298 bytes, /stdout)
test : compress to named file
tmp : 32.50% ( 65537 => 21298 bytes, tmpCompressed)
test : -o must be followed by filename (must fail)
error : command must be followed by argument
test : force write, correct order
tmp : 32.50% ( 65537 => 21298 bytes, tmpCompressed)
test : forgotten argument
error : command must be followed by argument
test : implied stdout when input is stdin
bob
test : compressed data to terminal
(▒/▒X!bob
▒▒▒,should have refused : compressed data to terminal
make[1]: * [Makefile:333: test-zstd] Error 1
make[1] : on quitte le répertoire « /home/vtorri/tmp/zstd-1.4.4/tests »
make: * [Makefile:77: shortest] Error 2

[Cyan4973]

I believe the issue is mintty, the default shell of msys2, rather than msys2 itself.

mintty doesn't identify itself as a console from C library level, while it does at shell level. Hence tests based on detecting console property are run by playTests.sh, but fail.

In order to pass playTests.sh on mintty, the current work around is to set the environment variable isTerminal=false, which will disable console detection tests.

[vtorri]

you can also write these tests in C instead of shell. A bit more work, of course

also unit tests with 'check' library for example

[bimbashrestha]

These are tests for the zstd cli. I suppose there might be a way to do it in c, but it will be quite a bit of work from what I can tell.

Our unit tests are currently mostly in fuzzer.c. I looked into libraries like unity, ctest and check for unit testing this week but adopting one would require rewriting all our current tests (check and ctest also have some compatibility issues with clang on Windows I think).

We are currently scoping out our testing landscape to get a better sense of what we have (it's become slightly unwieldy over the years) but that will most likely just involve reorganization without the adoption of a new test framework:)

[terrelln]

Our unit tests are currently mostly in fuzzer.c. I looked into libraries like unity, ctest and check for unit testing this week but adopting one would require rewriting all our current tests (check and ctest also have some compatibility issues with clang on Windows I think).

If we were going to rewrite our tests, it would be with gtest. I would love to see it done, but it would be a lot of work.

[vtorri]

btww, do you know https://coveralls.io/ ?

[terrelln]

Similar to #1878 we would be happy to accept patches to fix this, along with integration into Appveyor CI, otherwise it will break again in a few months.

[vtorri]

with MSYS2 :
echo $OS --> Windows_NT
uname --> MINGW64_NT-6.1-7601

isn't the test in playtest.sh line 80 wrong when msys2 is used ?