zstd stops reading data and loops in one thread

[tobiasjakobi]

Describe the bug
I have a simple tar+zstd pipe here, zstd is 1.4.4-r4, which is the most recent version my distro (Gentoo Linux x86-64) supplies, and apparantly also the most recent version available here. For some input data the zstd process stops reading data.

zstd is launched with the following arguments:
--compress --threads=0 --ultra -22 -o output_file

When this happens, I see one thread creating a huge amount of load, and the other three (system has 4 cores) idling.

I have rebuild zstd without optimizations and debug symbols (-O0 -ggdb). You can find a gdb backtrace and threads info below. My current guess is that this is multi-threading related.

Sorry, but I cannot share the input data.

Backtrace
https://gist.github.com/tobiasjakobi/73e919fd313a2214605e558cfa1fd564

Threads
https://gist.github.com/tobiasjakobi/6e8297bebc7cb77de7abe0735fae8378

Expected behavior
zstd reading all data and eventually finishing.

Desktop (please complete the following information):

• OS: Gentoo Linux x86-64
• zstd v1.4.4
• gcc-9.3.0
• see above for build flags

[tobiasjakobi]

OK, I disabled multi-threading by setting threads to one, but the issue is still present. Backtrace identical, here's the thread info:
Id Target Id Frame

• 1 Thread 0x7efd92203b80 (LWP 1698) "zstd" 0x00007efd923e2f25 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  2 Thread 0x7efd92202700 (LWP 1699) "zstd" MEM_readST (ptr=0x7efd3468b5a8) at ../lib/common/mem.h:226

[Cyan4973]

Setting --threads=1 doesn't actually "disable" multi-threading.
It merely sets the nb of compression workers to 1.
This is designed so that the compressed result remains identical whatever the nb of threads.

--single-thread will effectively disable multi-threading, so the code path will be different (no more pthread invocations). The resulting compressed file will also be _slightly_ different.

http://www.manpagez.com/man/1/zstd/

[tobiasjakobi]

Thanks for the headsup, I retried with --single-thread and cannot reproduce the issue. So I guess my initial guess (MT related) still holds.

[Cyan4973]

1) What's the size of the input file tested ?
2) Does it happen specifically with --ultra -22, or could it also happen with other settings, such as --ultra -21 or -19 for example ?
3) Could you try latest version of zstd in dev branch , and check if it reproduces the same problem ?

[Cyan4973]

Hi @tobiasjakobi ,
could we have some follow-up on the questions asked on this topic ?

MT-related bugs are among the most difficult ones to debug, hence without further element, and without an ability to reproduce the issue on our side, it will be difficult to fix.

[tobiasjakobi]

Hey Yann,

sorry, I've been pretty busy the last couple of days and didn't have any time to check if the issue also happens with other settings. I try to provide some findings the coming weekend.

-- Tobias

[tobiasjakobi]

OK, so I've tested this some more. First of all, it looks like I didn't wait long enough. It takes several minutes for zstd to recover from that state, but it then continues to compress.

Passing --ultra -21 as argument makes things a lot better. With the dev branch I see the same behaviour.

My guess is, that this input data has chunks that differ vastly in how well they compress. And that might not mix well with how zstd distributes data to the compression threads.

It seems like that I'm able to give zstd a hint about how large the input data is. The problem here is, this hint is currently (at least in v1.4.4) limited to 32bit values. The input data is around 8GB.

So sorry for the noise, closing this now.