Runtime: .NET Core 3.0 RTM Freeze on Ubuntu 16.04 OpenVZ

Created on 25 Sep 2019 · 14Comments · Source: dotnet/runtime

Today I installed .NET Core 3.0 RTM (3.0.100) on my ubuntu 16.04.6 x86-64 vps, when I trying to build a hello world console project, the dotnet process with following arguments freeze:

dotnet exec /usr/share/dotnet/sdk/3.0.100/MSBuild.dll -maxcpucount -verbosity:m -nologo -target:Restore -verbosity:diag -distributedlogger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,/usr/share/dotnet/sdk/3.0.100/dotnet.dll*Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingLogger,/usr/share/dotnet/sdk/3.0.100/dotnet.dll

The previous version (.NET Core 2.2) doesn't have this issue.

Then I trying to build coreclr from source to find why it happends.
I clone coreclr repository, checkout release/3.1 branch, then run build.sh, and it freeze again.
The command line:

/home/ubuntu/tmp/coreclr/.dotnet/dotnet msbuild /home/ubuntu/tmp/coreclr/src/.nuget/optdata/optdata.csproj /t:DumpPgoDataPackageVersion /p:__BuildArch=x64 /p:__BuildType=Debug /p:__BuildOS=Linux /nodeReuse:false /p:RestoreDuringBuild=true /p:PgoDataPackageVersionOutputFile=/home/ubuntu/tmp/coreclr/bin/obj/Linux.x64.Debug/optdataversion.txt /nologo

I attach the process with gdb and run thread apply all bt, here is the output messages, looks like .NET Core 3.0 have some deadlock bug on ubuntu 16.04, both RTM and preview 7.

Thread 19 (Thread 0x7f1ed5ea0700 (LWP 12863)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
dotnet/coreclr#1  0x00007f1ed45a8422 in CorUnix::InternalEnterCriticalSection(CorUnix::CPalThread*, _CRITICAL_SECTION*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#2  0x00007f1ed4200435 in CrstBase::Enter() () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#3  0x00007f1ed4349e47 in ThreadSuspend::LockThreadStore(ThreadSuspend::SUSPEND_REASON) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#4  0x00007f1ed42796cb in ThreadStore::AddThread(Thread*, int) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#5  0x00007f1ed42797c2 in SetupUnstartedThread(int) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#6  0x00007f1ed429e020 in ThreadpoolMgr::CreateUnimpersonatedThread(unsigned int (*)(void*), void*, int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#7  0x00007f1ed429d000 in ThreadpoolMgr::MaybeAddWorkingWorker() () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#8  0x00007f1ed42a07dd in ThreadpoolMgr::GateThreadStart(void*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#9  0x00007f1ed45b743d in CorUnix::CPalThread::ThreadEntry(void*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#10 0x00007f1ed5be26ba in start_thread (arg=0x7f1ed5ea0700) at pthread_create.c:333
dotnet/coreclr#11 0x00007f1ed4e7341d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 18 (Thread 0x7f1d8a1fa700 (LWP 12856)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
dotnet/coreclr#1  0x00007f1ed45ad2b5 in CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#2  0x00007f1ed45acec4 in CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#3  0x00007f1ed45b14be in CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* const*, int, unsigned int, int, int) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#4  0x00007f1ed45b1759 in WaitForSingleObjectEx () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#5  0x00007f1ed4346d74 in CLREventBase::WaitEx(unsigned int, WaitMode, PendingSync*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#6  0x00007f1ed446da9d in WKS::gc_heap::wait_for_gc_done(int) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#7  0x00007f1ed44a1311 in WKS::GCHeap::WaitUntilGCComplete(bool) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#8  0x00007f1ed429e567 in ThreadpoolMgr::WorkerThreadStart(void*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#9  0x00007f1ed45b743d in CorUnix::CPalThread::ThreadEntry(void*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#10 0x00007f1ed5be26ba in start_thread (arg=0x7f1d8a1fa700) at pthread_create.c:333
dotnet/coreclr#11 0x00007f1ed4e7341d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 17 (Thread 0x7f1d8abfb700 (LWP 12847)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
dotnet/coreclr#1  0x00007f1ed45ad2b5 in CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#2  0x00007f1ed45acec4 in CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#3  0x00007f1ed45b14be in CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* const*, int, unsigned int, int, int) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#4  0x00007f1ed45b1759 in WaitForSingleObjectEx () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#5  0x00007f1ed4346d74 in CLREventBase::WaitEx(unsigned int, WaitMode, PendingSync*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#6  0x00007f1ed434a11b in Thread::RareDisablePreemptiveGC() () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#7  0x00007f1ed42798e6 in Thread::OnThreadTerminate(int) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#8  0x00007f1ed429e8e3 in ThreadpoolMgr::WorkerThreadStart(void*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#9  0x00007f1ed45b743d in CorUnix::CPalThread::ThreadEntry(void*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#10 0x00007f1ed5be26ba in start_thread (arg=0x7f1d8abfb700) at pthread_create.c:333
dotnet/coreclr#11 0x00007f1ed4e7341d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 16 (Thread 0x7f1d8b5fc700 (LWP 12845)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
dotnet/coreclr#1  0x00007f1ed45ad2b5 in CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#2  0x00007f1ed45acec4 in CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#3  0x00007f1ed45b14be in CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* const*, int, unsigned int, int, int) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#4  0x00007f1ed45b1759 in WaitForSingleObjectEx () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#5  0x00007f1ed4346d74 in CLREventBase::WaitEx(unsigned int, WaitMode, PendingSync*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#6  0x00007f1ed446da9d in WKS::gc_heap::wait_for_gc_done(int) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#7  0x00007f1ed44a1311 in WKS::GCHeap::WaitUntilGCComplete(bool) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#8  0x00007f1ed429e567 in ThreadpoolMgr::WorkerThreadStart(void*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#9  0x00007f1ed45b743d in CorUnix::CPalThread::ThreadEntry(void*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#10 0x00007f1ed5be26ba in start_thread (arg=0x7f1d8b5fc700) at pthread_create.c:333
dotnet/coreclr#11 0x00007f1ed4e7341d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 15 (Thread 0x7f1d8bffd700 (LWP 12841)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
dotnet/coreclr#1  0x00007f1ed45ad2b5 in CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#2  0x00007f1ed45acec4 in CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#3  0x00007f1ed45b14be in CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* const*, int, unsigned int, int, int) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#4  0x00007f1ed45b1831 in WaitForMultipleObjectsEx () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#5  0x00007f1ed427ca04 in Thread::DoAppropriateWaitWorker(int, void**, int, unsigned int, WaitMode) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#6  0x00007f1ed4277ef9 in Thread::DoAppropriateWait(int, void**, int, unsigned int, WaitMode, PendingSync*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#7  0x00007f1ed44000dc in WaitHandleNative::CorWaitOneNative(void*, int) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#8  0x00007f1e5aa7942d in ?? ()
dotnet/coreclr#9  0x00007f1d8bffcc40 in ?? ()
dotnet/coreclr#10 0x00007f1d8c1bc8a8 in ?? ()
dotnet/coreclr#11 0x00007f1d8c1d20b8 in ?? ()
dotnet/coreclr#12 0x00007f1d00000001 in ?? ()
dotnet/coreclr#13 0x00007f1d8c1d20a0 in ?? ()
dotnet/coreclr#14 0x00007f1d8c1bc8a8 in ?? ()
dotnet/coreclr#15 0x00007f1d8bffcc90 in ?? ()
dotnet/coreclr#16 0x00007f1e5aa79314 in ?? ()
dotnet/coreclr#17 0x00007f1d8c1d20a0 in ?? ()
dotnet/coreclr#18 0x00007f1d8c1bc8a8 in ?? ()
dotnet/coreclr#19 0x00007f1d8bffcce0 in ?? ()
dotnet/coreclr#20 0x00007f1e5aa82ebf in ?? ()
dotnet/coreclr#21 0x00007f1d8bffcca0 in ?? ()
dotnet/coreclr#22 0x00007f1d8c1bc888 in ?? ()
dotnet/coreclr#23 0x00007f1d8c1bc858 in ?? ()
dotnet/coreclr#24 0x0000000200000000 in ?? ()
dotnet/runtime#3858 0x0000000200000002 in ?? ()
dotnet/runtime#3859 0x00000000ffffffff in ?? ()
dotnet/coreclr#27 0x00007f1d8c1bc858 in ?? ()
dotnet/runtime#3860 0x0000000000000009 in ?? ()
dotnet/runtime#3861 0x00007f1d8bffcd70 in ?? ()
dotnet/runtime#3862 0x00007f1e5aa81e80 in ?? ()
dotnet/runtime#3863 0xa1a168411a874f10 in ?? ()
dotnet/coreclr#32 0x1318fdefacd62aa6 in ?? ()
dotnet/runtime#3864 0x97fce06f1e159f73 in ?? ()
dotnet/runtime#3865 0xa4460fe33eb93df9 in ?? ()
dotnet/coreclr#35 0x46000000ac000000 in ?? ()
dotnet/coreclr#36 0x00007f1d8c1bc858 in ?? ()
dotnet/runtime#3866 0x00000000ffffffff in ?? ()
dotnet/coreclr#38 0x0000000000000000 in ?? ()

Thread 14 (Thread 0x7f1e59140700 (LWP 12840)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
dotnet/coreclr#1  0x00007f1ed45ad2b5 in CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#2  0x00007f1ed45acec4 in CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#3  0x00007f1ed45b14be in CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* const*, int, unsigned int, int, int) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#4  0x00007f1ed45b1759 in WaitForSingleObjectEx () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#5  0x00007f1ed4346d74 in CLREventBase::WaitEx(unsigned int, WaitMode, PendingSync*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#6  0x00007f1ed434a11b in Thread::RareDisablePreemptiveGC() () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#7  0x00007f1ed427cc85 in Thread::DoAppropriateWaitWorker(int, void**, int, unsigned int, WaitMode) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#8  0x00007f1ed4277ef9 in Thread::DoAppropriateWait(int, void**, int, unsigned int, WaitMode, PendingSync*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#9  0x00007f1ed44000dc in WaitHandleNative::CorWaitOneNative(void*, int) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#10 0x00007f1e5aa7942d in ?? ()
dotnet/coreclr#11 0x00007f1e5913f6e0 in ?? ()
dotnet/coreclr#12 0x0000000000000000 in ?? ()

Thread 13 (Thread 0x7f1e59b41700 (LWP 12839)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
dotnet/coreclr#1  0x00007f1ed45ad2b5 in CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#2  0x00007f1ed45acec4 in CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#3  0x00007f1ed45b14be in CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* const*, int, unsigned int, int, int) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#4  0x00007f1ed45b1759 in WaitForSingleObjectEx () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#5  0x00007f1ed4346d74 in CLREventBase::WaitEx(unsigned int, WaitMode, PendingSync*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#6  0x00007f1ed434a11b in Thread::RareDisablePreemptiveGC() () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#7  0x00007f1ed427cc85 in Thread::DoAppropriateWaitWorker(int, void**, int, unsigned int, WaitMode) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#8  0x00007f1ed4277ef9 in Thread::DoAppropriateWait(int, void**, int, unsigned int, WaitMode, PendingSync*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#9  0x00007f1ed44000dc in WaitHandleNative::CorWaitOneNative(void*, int) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#10 0x00007f1e5aa7942d in ?? ()
dotnet/coreclr#11 0x00007f1e59b406d0 in ?? ()
dotnet/coreclr#12 0x0000000000000000 in ?? ()

Thread 12 (Thread 0x7f1ebd0c1700 (LWP 12837)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
dotnet/coreclr#1  0x00007f1ed45ad2ab in CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#2  0x00007f1ed45acec4 in CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#3  0x00007f1ed45b14be in CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* const*, int, unsigned int, int, int) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#4  0x00007f1ed45b1759 in WaitForSingleObjectEx () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#5  0x00007f1ed4346d74 in CLREventBase::WaitEx(unsigned int, WaitMode, PendingSync*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#6  0x00007f1ed434af73 in ThreadSuspend::SuspendRuntime(ThreadSuspend::SUSPEND_REASON) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#7  0x00007f1ed434c60e in ThreadSuspend::SuspendEE(ThreadSuspend::SUSPEND_REASON) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#8  0x00007f1ed44a2e53 in GCToEEInterface::SuspendEE(SUSPEND_REASON) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#9  0x00007f1ed44793b8 in WKS::GCHeap::GarbageCollectGeneration(unsigned int, gc_reason) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#10 0x00007f1ed447b440 in WKS::gc_heap::try_allocate_more_space(alloc_context*, unsigned long, unsigned int, int) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#11 0x00007f1ed449d080 in WKS::GCHeap::Alloc(gc_alloc_context*, unsigned long, unsigned int) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#12 0x00007f1ed42de450 in AllocateSzArray(MethodTable*, int, GC_ALLOC_FLAGS, int) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#13 0x00007f1ed42ebc8b in JIT_NewArr1(CORINFO_CLASS_STRUCT_*, long) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#14 0x00007f1e5bfd8819 in ?? ()
dotnet/coreclr#15 0x0000000000001000 in ?? ()
dotnet/coreclr#16 0x00007f1d8c3f5860 in ?? ()
dotnet/coreclr#17 0x0000000000000000 in ?? ()

Thread 11 (Thread 0x7f1ebdac2700 (LWP 12836)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
dotnet/coreclr#1  0x00007f1ed45ad2b5 in CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#2  0x00007f1ed45acec4 in CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#3  0x00007f1ed45b14be in CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* const*, int, unsigned int, int, int) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#4  0x00007f1ed45b1759 in WaitForSingleObjectEx () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#5  0x00007f1ed4346d74 in CLREventBase::WaitEx(unsigned int, WaitMode, PendingSync*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#6  0x00007f1ed434a11b in Thread::RareDisablePreemptiveGC() () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#7  0x00007f1ed427cc85 in Thread::DoAppropriateWaitWorker(int, void**, int, unsigned int, WaitMode) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#8  0x00007f1ed4277ef9 in Thread::DoAppropriateWait(int, void**, int, unsigned int, WaitMode, PendingSync*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#9  0x00007f1ed44000dc in WaitHandleNative::CorWaitOneNative(void*, int) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#10 0x00007f1e5aa7942d in ?? ()
dotnet/coreclr#11 0x00007f1ebdac16d0 in ?? ()
dotnet/coreclr#12 0x0000000000000000 in ?? ()

Thread 10 (Thread 0x7f1ebed8f700 (LWP 12834)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
dotnet/coreclr#1  0x00007f1ed45ad2ab in CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#2  0x00007f1ed45acec4 in CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#3  0x00007f1ed45b1c89 in SleepEx () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#4  0x00007f1ed42e233a in __SwitchToThread(unsigned int, unsigned int) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#5  0x00007f1ed429dd35 in ThreadpoolMgr::GetRecycledMemory(ThreadpoolMgr::MemType) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#6  0x00007f1ed43fcdd2 in DelegateInfo::MakeDelegateInfo(Object**, Object**, Object**) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#7  0x00007f1ed43fe2ea in ThreadPoolNative::CorRegisterWaitForSingleObject(Object*, Object*, unsigned int, bool, Object*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#8  0x00007f1e5aa7711a in ?? ()
dotnet/coreclr#9  0x00007f1d8c1c2ca0 in ?? ()
dotnet/coreclr#10 0x00007f1d8c1c2d20 in ?? ()
dotnet/coreclr#11 0x00007f1d8c1c2ce0 in ?? ()
dotnet/coreclr#12 0x00007f1d8c1c2ce0 in ?? ()
dotnet/coreclr#13 0x0000000100000001 in ?? ()
dotnet/coreclr#14 0x00007f1d8c1c2ca0 in ?? ()
dotnet/coreclr#15 0x00007f1d8c1c2c20 in ?? ()
dotnet/coreclr#16 0x00007f1d8c1c2c68 in ?? ()
dotnet/coreclr#17 0x00007f1d8c1c2c88 in ?? ()
dotnet/coreclr#18 0x000000000000322a in ?? ()
dotnet/coreclr#19 0x00007f1ebed8e380 in ?? ()
dotnet/coreclr#20 0x00007f1e5aa773ee in ?? ()
dotnet/coreclr#21 0x00007f1d8c1c2ca0 in ?? ()
dotnet/coreclr#22 0x00007f1d8c1c2c88 in ?? ()
dotnet/coreclr#23 0x00007f1ebed8e3c0 in ?? ()
dotnet/coreclr#24 0x00007f1e5b945064 in ?? ()
dotnet/runtime#3858 0x00007f1ebed8e390 in ?? ()
dotnet/runtime#3859 0x00007f1d8c1bc2c8 in ?? ()
dotnet/coreclr#27 0x00007f1d8c1bc2c8 in ?? ()
dotnet/runtime#3860 0x00007f1d00000001 in ?? ()
dotnet/runtime#3861 0x00000000000002c0 in ?? ()
dotnet/runtime#3862 0x00007f1d8c1c2a90 in ?? ()
dotnet/runtime#3863 0x00007f1ebed8e4e0 in ?? ()
dotnet/coreclr#32 0x00007f1e5b93dfce in ?? ()
dotnet/runtime#3864 0x0000000000000001 in ?? ()
dotnet/runtime#3865 0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7f1ed5ee9700 (LWP 12833)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
dotnet/coreclr#1  0x00007f1ed45ad2ab in CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#2  0x00007f1ed45acec4 in CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#3  0x00007f1ed45b1c89 in SleepEx () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#4  0x00007f1ed42a0f8d in ThreadpoolMgr::TimerThreadFire() () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#5  0x00007f1ed42a0e55 in ThreadpoolMgr::TimerThreadStart(void*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#6  0x00007f1ed45b743d in CorUnix::CPalThread::ThreadEntry(void*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#7  0x00007f1ed5be26ba in start_thread (arg=0x7f1ed5ee9700) at pthread_create.c:333
dotnet/coreclr#8  0x00007f1ed4e7341d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 8 (Thread 0x7f1ec1a3d700 (LWP 12832)):
#0  0x00007f1ed5beb76d in accept () at ../sysdeps/unix/syscall-template.S:84
dotnet/coreclr#1  0x00007f1ed44dffdd in IpcStream::DiagnosticsIpc::Accept(void (*)(char const*, unsigned int)) const ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#2  0x00007f1ed42c47c0 in DiagnosticsServerThread(void*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#3  0x00007f1ed45b743d in CorUnix::CPalThread::ThreadEntry(void*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#4  0x00007f1ed5be26ba in start_thread (arg=0x7f1ec1a3d700) at pthread_create.c:333
dotnet/coreclr#5  0x00007f1ed4e7341d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 7 (Thread 0x7f1ec243e700 (LWP 12831)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
dotnet/coreclr#1  0x00007f1ed45ad2ab in CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#2  0x00007f1ed45acec4 in CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#3  0x00007f1ed45b14be in CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* const*, int, unsigned int, int, int) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#4  0x00007f1ed45b1831 in WaitForMultipleObjectsEx () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#5  0x00007f1ed42dcc4e in FinalizerThread::WaitForFinalizerEvent(CLREvent*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#6  0x00007f1ed42dcd62 in FinalizerThread::FinalizerThreadWorker(void*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#7  0x00007f1ed4280225 in ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#8  0x00007f1ed42808dd in ManagedThreadBase::FinalizerBase(void (*)(void*)) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#9  0x00007f1ed42dcf9c in FinalizerThread::FinalizerThreadStart(void*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#10 0x00007f1ed45b743d in CorUnix::CPalThread::ThreadEntry(void*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#11 0x00007f1ed5be26ba in start_thread (arg=0x7f1ec243e700) at pthread_create.c:333
dotnet/coreclr#12 0x00007f1ed4e7341d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 6 (Thread 0x7f1e5a546700 (LWP 12830)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
dotnet/coreclr#1  0x00007f1ed45ad2b5 in CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#2  0x00007f1ed45acec4 in CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#3  0x00007f1ed45b14be in CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* const*, int, unsigned int, int, int) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#4  0x00007f1ed45b1831 in WaitForMultipleObjectsEx () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#5  0x00007f1ed44d71a6 in DebuggerRCThread::MainLoop() () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#6  0x00007f1ed44d704b in DebuggerRCThread::ThreadProc() () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#7  0x00007f1ed44d6d69 in DebuggerRCThread::ThreadProcStatic(void*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#8  0x00007f1ed45b743d in CorUnix::CPalThread::ThreadEntry(void*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#9  0x00007f1ed5be26ba in start_thread (arg=0x7f1e5a546700) at pthread_create.c:333
dotnet/coreclr#10 0x00007f1ed4e7341d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 5 (Thread 0x7f1ed1403700 (LWP 12829)):
#0  0x00007f1ed5bebc7d in open64 () at ../sysdeps/unix/syscall-template.S:84
dotnet/coreclr#1  0x00007f1ed44e024f in TwoWayPipe::WaitForConnection() () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#2  0x00007f1ed44d9af2 in DbgTransportSession::TransportWorker() () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#3  0x00007f1ed44d8a09 in DbgTransportSession::TransportWorkerStatic(void*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#4  0x00007f1ed45b743d in CorUnix::CPalThread::ThreadEntry(void*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#5  0x00007f1ed5be26ba in start_thread (arg=0x7f1ed1403700) at pthread_create.c:333
dotnet/coreclr#6  0x00007f1ed4e7341d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 4 (Thread 0x7f1ed200f700 (LWP 12828)):
#0  0x00007f1ed4e6774d in poll () at ../sysdeps/unix/syscall-template.S:84
dotnet/coreclr#1  0x00007f1ed45af30e in CorUnix::CPalSynchronizationManager::ReadBytesFromProcessPipe(int, unsigned char*, int) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#2  0x00007f1ed45aeee8 in CorUnix::CPalSynchronizationManager::ReadCmdFromProcessPipe(int, CorUnix::CPalSynchronizationManager::SynchWorkerCmd*, void**, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#3  0x00007f1ed45aeb64 in CorUnix::CPalSynchronizationManager::WorkerThread(void*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#4  0x00007f1ed45b743d in CorUnix::CPalThread::ThreadEntry(void*) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#5  0x00007f1ed5be26ba in start_thread (arg=0x7f1ed200f700) at pthread_create.c:333
dotnet/coreclr#6  0x00007f1ed4e7341d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 3 (Thread 0x7f1ed2a10700 (LWP 12827)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
dotnet/coreclr#1  0x00007f1ed3a5287c in ?? () from /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
dotnet/coreclr#2  0x00007f1ed5be26ba in start_thread (arg=0x7f1ed2a10700) at pthread_create.c:333
dotnet/coreclr#3  0x00007f1ed4e7341d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 2 (Thread 0x7f1ed3411700 (LWP 12826)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
dotnet/coreclr#1  0x00007f1ed3a5287c in ?? () from /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
dotnet/coreclr#2  0x00007f1ed5be26ba in start_thread (arg=0x7f1ed3411700) at pthread_create.c:333
dotnet/coreclr#3  0x00007f1ed4e7341d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 1 (Thread 0x7f1ed600e740 (LWP 12825)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
dotnet/coreclr#1  0x00007f1ed45ad2b5 in CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#2  0x00007f1ed45acec4 in CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#3  0x00007f1ed45b14be in CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* const*, int, unsigned int, int, int) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#4  0x00007f1ed45b1831 in WaitForMultipleObjectsEx () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#5  0x00007f1ed427ca04 in Thread::DoAppropriateWaitWorker(int, void**, int, unsigned int, WaitMode) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#6  0x00007f1ed4277ef9 in Thread::DoAppropriateWait(int, void**, int, unsigned int, WaitMode, PendingSync*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#7  0x00007f1ed44000dc in WaitHandleNative::CorWaitOneNative(void*, int) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#8  0x00007f1e5aa7942d in ?? ()
dotnet/coreclr#9  0x00007ffec2b34b90 in ?? ()
dotnet/coreclr#10 0x00007f1d8c213740 in ?? ()
dotnet/coreclr#11 0x00007f1d8c213758 in ?? ()
dotnet/coreclr#12 0x00007f1d00000001 in ?? ()
dotnet/coreclr#13 0x00007f1d8c213740 in ?? ()
dotnet/coreclr#14 0x00007f1d8c1bc8a8 in ?? ()
dotnet/coreclr#15 0x00007ffec2b34be0 in ?? ()
dotnet/coreclr#16 0x00007f1e5aa79314 in ?? ()
dotnet/coreclr#17 0x00007f1d8c213740 in ?? ()
dotnet/coreclr#18 0x00007f1d8c1bc8a8 in ?? ()
dotnet/coreclr#19 0x00007ffec2b34c30 in ?? ()
dotnet/coreclr#20 0x00007f1e5aa82ebf in ?? ()
dotnet/coreclr#21 0x00007ffec2b34bf0 in ?? ()
dotnet/coreclr#22 0x00007f1d8c1bc88c in ?? ()
dotnet/coreclr#23 0x00007f1d8c1bc858 in ?? ()
dotnet/coreclr#24 0x0000000000000000 in ?? ()

area-System.Threading untriaged

Source

303248153

👍1

Most helpful comment

The reason is sched_getcpu() may return a value greater than or equal to CPU_COUNT(&cpuSet) on my ubuntu 16.04 vps which using openvz.

I wrote a simple c program to verify it:

#define _GNU_SOURCE
#include <sched.h>
#include <unistd.h>
#include <stdio.h>

int main() {
    while (1) {
        cpu_set_t cpuSet;
        int st = sched_getaffinity(getpid(), sizeof(cpu_set_t), &cpuSet);
        int nproc = CPU_COUNT(&cpuSet);
        for (int x = 0; x < nproc; ++x) {
            printf("affinity is %d for cpu %d\n", CPU_ISSET(x, &cpuSet), x);
        }

        printf("cpuid: %d\n", sched_getcpu());
        sleep(1);
    }
    return 0;
}

output:

affinity is 1 for cpu 0
cpuid: 0
affinity is 1 for cpu 0
cpuid: 0
affinity is 1 for cpu 0
cpuid: 1
affinity is 1 for cpu 0
cpuid: 1
affinity is 1 for cpu 0
cpuid: 1
affinity is 1 for cpu 0
cpuid: 1
affinity is 1 for cpu 0
cpuid: 1
affinity is 1 for cpu 0
cpuid: 1
affinity is 1 for cpu 0
cpuid: 5
affinity is 1 for cpu 0
cpuid: 5

In .NET Core 2.2, GetRecycleMemoryInfo will use GetCurrentProcessorNumber()%NumberOfProcessors for both windows and pal, but in .NET Core 3.0, GetRecycleMemoryInfo will only use GetCurrentProcessorNumber() for pal, so return pRecycledListPerProcessor[processorNumber][memType] will access out of range element and get an invalid RecycledListInfo instance.

Link:
https://github.com/dotnet/coreclr/blob/v2.2.0/src/vm/win32threadpool.h
https://github.com/dotnet/coreclr/blob/v3.0.0/src/vm/win32threadpool.h

The fix should be replace processorNumber = GetCurrentProcessorNumber() to processorNumber = GetCurrentProcessorNumber() % NumberOfProcessors in win32threadpool.h.

Confirmed the fix works on my environment, should I create a pull request?

303248153 on 26 Sep 2019

👍8

All 14 comments

Looks like a gc bug, most threads are waiting for gc done (enter cooperative mode), but the gc thread is blocking at ThreadSuspend::SuspendRuntime.

303248153 on 25 Sep 2019

Thank you for collecting the dump and stack traces. The problem is Thread 10:

Thread 10 (Thread 0x7f1ebed8f700 (LWP 12834)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
dotnet/coreclr#1  0x00007f1ed45ad2ab in CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#2  0x00007f1ed45acec4 in CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#3  0x00007f1ed45b1c89 in SleepEx () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#4  0x00007f1ed42e233a in __SwitchToThread(unsigned int, unsigned int) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#5  0x00007f1ed429dd35 in ThreadpoolMgr::GetRecycledMemory(ThreadpoolMgr::MemType) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#6  0x00007f1ed43fcdd2 in DelegateInfo::MakeDelegateInfo(Object**, Object**, Object**) () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so
dotnet/coreclr#7  0x00007f1ed43fe2ea in ThreadPoolNative::CorRegisterWaitForSingleObject(Object*, Object*, unsigned int, bool, Object*) ()
   from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview7-27912-14/libcoreclr.so

This thread is in cooperative mode and the GC is waiting for it to get to a safe spot. Would you be able to find out why it is not making progress? Is it stuck spinning inside ThreadpoolMgr::GetRecycledMemory ?

jkotas on 25 Sep 2019

I was able to reproduce the problem by copying a custom built coreclr (tag v3.0.0) to .dotnet/shared/Microsoft.NETCore.App/3.0.0-preview6-27804-01 and run ./build.sh.

When use release build, gdb gives following frames when freeze occurs (see thread 14, looks like the same location cause this problem):

(gdb) thread apply all bt

Thread 16 (Thread 0x7f76b92c7700 (LWP 9141)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
dotnet/coreclr#1  0x00007f7735b65b45 in CorUnix::CPalSynchronizationManager::ThreadNativeWait (ptnwdNativeWaitData=<optimized out>, dwTimeout=<optimized out>, ptwrWakeupReason=<optimized out>,
    pdwSignaledObject=<optimized out>) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/synchmanager.cpp:484
dotnet/coreclr#2  0x00007f7735b65764 in CorUnix::CPalSynchronizationManager::BlockThread (this=0x882a40, pthrCurrent=0x97d010, dwTimeout=10, fAlertable=false, fIsSleep=<optimized out>,
    ptwrWakeupReason=0x7f76b92c3af8, pdwSignaledObject=0xfffffffffffffdfc) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/synchmanager.cpp:302
dotnet/coreclr#3  0x00007f7735b6a13b in CorUnix::InternalWaitForMultipleObjectsEx (pThread=0x97d010, nCount=<optimized out>, lpHandles=<optimized out>, bWaitAll=<optimized out>, dwMilliseconds=<optimized out>,
    bAlertable=<optimized out>, bPrioritize=<optimized out>) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/wait.cpp:640
dotnet/coreclr#4  0x00007f7735b6a3b9 in WaitForSingleObjectEx (hHandle=<error reading variable: Cannot access memory at address 0x1073>, dwMilliseconds=10, bAlertable=0)
    at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/wait.cpp:139
dotnet/coreclr#5  0x00007f77358e2784 in CLREventWaitHelper2 (handle=0x97d27c, dwMilliseconds=<optimized out>, alertable=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/synch.cpp:377
dotnet/coreclr#6  CLREventWaitHelper(void*, unsigned int, int)::$_1::operator()(CLREventWaitHelper(void*, unsigned int, int)::Param*) const (this=<optimized out>, pParam=<optimized out>)
    at /home/ubuntu/tmp/coreclr/src/vm/synch.cpp:402
dotnet/coreclr#7  CLREventWaitHelper (dwMilliseconds=10, alertable=<optimized out>, handle=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/synch.cpp:404
dotnet/coreclr#8  CLREventBase::WaitEx (this=<optimized out>, dwMilliseconds=10, mode=<optimized out>, syncState=0x0) at /home/ubuntu/tmp/coreclr/src/vm/synch.cpp:471
dotnet/coreclr#9  0x00007f77358e70d3 in ThreadSuspend::SuspendRuntime (reason=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/threadsuspend.cpp:4187
dotnet/coreclr#10 0x00007f77358e8ed5 in ThreadSuspend::SuspendEE (reason=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/threadsuspend.cpp:6512
dotnet/coreclr#11 0x00007f7735a4c693 in GCToEEInterface::SuspendEE (reason=(SUSPEND_FOR_GC_PREP | unknown: 9949816)) at /home/ubuntu/tmp/coreclr/src/vm/gcenv.ee.cpp:25
dotnet/coreclr#12 0x00007f7735a1f87d in WKS::GCHeap::GarbageCollectGeneration (this=<optimized out>, gen=0, reason=reason_alloc_soh) at /home/ubuntu/tmp/coreclr/src/gc/gc.cpp:36526
dotnet/coreclr#13 0x00007f7735a218a0 in WKS::gc_heap::try_allocate_more_space (acontext=<optimized out>, size=<optimized out>, flags=<optimized out>, gen_number=<optimized out>)
    at /home/ubuntu/tmp/coreclr/src/gc/gc.cpp:13915
dotnet/coreclr#14 0x00007f7735a46330 in WKS::gc_heap::allocate_more_space (size=<optimized out>, flags=0, alloc_generation_number=0, acontext=<optimized out>) at /home/ubuntu/tmp/coreclr/src/gc/gc.cpp:14350
dotnet/coreclr#15 WKS::gc_heap::allocate (jsize=<optimized out>, acontext=<optimized out>, flags=<optimized out>) at /home/ubuntu/tmp/coreclr/src/gc/gc.cpp:14381
dotnet/coreclr#16 WKS::GCHeap::Alloc (this=<optimized out>, context=<optimized out>, size=<optimized out>, flags=<optimized out>) at /home/ubuntu/tmp/coreclr/src/gc/gc.cpp:35808
dotnet/coreclr#17 0x00007f773587bbfe in Alloc (size=32, flags=GC_ALLOC_NO_FLAGS) at /home/ubuntu/tmp/coreclr/src/vm/gchelpers.cpp:240
dotnet/coreclr#18 AllocateString (cchStringLength=5) at /home/ubuntu/tmp/coreclr/src/vm/gchelpers.cpp:996
dotnet/coreclr#19 0x00007f77358883e6 in FramedAllocateString (stringLength=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/jithelpers.cpp:2884
dotnet/coreclr#20 0x00007f76bbf558c3 in ?? ()
dotnet/coreclr#21 0x0000000000000000 in ?? ()

Thread 15 (Thread 0x7f76b9cc8700 (LWP 9140)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
dotnet/coreclr#1  0x00007f7735b65b82 in CorUnix::CPalSynchronizationManager::ThreadNativeWait (ptnwdNativeWaitData=<optimized out>, dwTimeout=<optimized out>, ptwrWakeupReason=<optimized out>,
    pdwSignaledObject=<optimized out>) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/synchmanager.cpp:479
dotnet/coreclr#2  0x00007f7735b65764 in CorUnix::CPalSynchronizationManager::BlockThread (this=0x882a40, pthrCurrent=0xa2fd00, dwTimeout=4294967295, fAlertable=false, fIsSleep=<optimized out>,
    ptwrWakeupReason=0x7f76b9cc7bc8, pdwSignaledObject=0xfffffffffffffe00) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/synchmanager.cpp:302
dotnet/coreclr#3  0x00007f7735b6a13b in CorUnix::InternalWaitForMultipleObjectsEx (pThread=0xa2fd00, nCount=<optimized out>, lpHandles=<optimized out>, bWaitAll=<optimized out>, dwMilliseconds=<optimized out>,
    bAlertable=<optimized out>, bPrioritize=<optimized out>) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/wait.cpp:640
dotnet/coreclr#4  0x00007f7735b6a3b9 in WaitForSingleObjectEx (hHandle=<error reading variable: Cannot access memory at address 0x3>, dwMilliseconds=4294967295, bAlertable=0)
---Type <return> to continue, or q <return> to quit---
    at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/wait.cpp:139
dotnet/coreclr#5  0x00007f77358e2784 in CLREventWaitHelper2 (handle=0xa2ff6c, dwMilliseconds=<optimized out>, alertable=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/synch.cpp:377
dotnet/coreclr#6  CLREventWaitHelper(void*, unsigned int, int)::$_1::operator()(CLREventWaitHelper(void*, unsigned int, int)::Param*) const (this=<optimized out>, pParam=<optimized out>)
    at /home/ubuntu/tmp/coreclr/src/vm/synch.cpp:402
dotnet/coreclr#7  CLREventWaitHelper (dwMilliseconds=4294967295, alertable=<optimized out>, handle=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/synch.cpp:404
dotnet/coreclr#8  CLREventBase::WaitEx (this=<optimized out>, dwMilliseconds=4294967295, mode=<optimized out>, syncState=0x0) at /home/ubuntu/tmp/coreclr/src/vm/synch.cpp:471
dotnet/coreclr#9  0x00007f77358e61eb in Thread::RareDisablePreemptiveGC (this=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/threadsuspend.cpp:2576
dotnet/coreclr#10 0x00007f7735810ff6 in Thread::DisablePreemptiveGC (this=0xa480a0) at /home/ubuntu/tmp/coreclr/src/vm/threads.h:2036
dotnet/coreclr#11 GCHolderBase::EnterInternalCoop (pThread=0xa480a0, conditional=<error reading variable: access outside bounds of object referenced via synthetic pointer>, this=<optimized out>)
    at /home/ubuntu/tmp/coreclr/src/vm/threads.h:5800
dotnet/coreclr#12 GCCoop::GCCoop (this=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/threads.h:5920
dotnet/coreclr#13 Thread::OnThreadTerminate (this=0xa480a0, holdingLock=0) at /home/ubuntu/tmp/coreclr/src/vm/threads.cpp:2912
dotnet/coreclr#14 0x00007f77358380d3 in ThreadpoolMgr::WorkerThreadStart (lpArgs=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/win32threadpool.cpp:2192
dotnet/coreclr#15 0x00007f7735b7092d in CorUnix::CPalThread::ThreadEntry (pvParam=0xa2fd00) at /home/ubuntu/tmp/coreclr/src/pal/src/thread/thread.cpp:1807
dotnet/coreclr#16 0x00007f77371a66ba in start_thread (arg=0x7f76b9cc8700) at pthread_create.c:333
dotnet/coreclr#17 0x00007f773643741d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 14 (Thread 0x7f76ba6c9700 (LWP 9138)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
dotnet/coreclr#1  0x00007f7735b65b45 in CorUnix::CPalSynchronizationManager::ThreadNativeWait (ptnwdNativeWaitData=<optimized out>, dwTimeout=<optimized out>, ptwrWakeupReason=<optimized out>,
    pdwSignaledObject=<optimized out>) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/synchmanager.cpp:484
dotnet/coreclr#2  0x00007f7735b65764 in CorUnix::CPalSynchronizationManager::BlockThread (this=0x882a40, pthrCurrent=0xb63140, dwTimeout=1, fAlertable=false, fIsSleep=<optimized out>,
    ptwrWakeupReason=0x7f76ba6c5e10, pdwSignaledObject=0xfffffffffffffdfc) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/synchmanager.cpp:302
dotnet/coreclr#3  0x00007f7735b6a8e9 in CorUnix::InternalSleepEx (pThread=<optimized out>, dwMilliseconds=1, bAlertable=0) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/wait.cpp:855
dotnet/coreclr#4  SleepEx (dwMilliseconds=1, bAlertable=0) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/wait.cpp:286
dotnet/coreclr#5  0x00007f773587ec7a in EESleepEx (dwMilliseconds=<optimized out>, bAlertable=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/hosting.cpp:379
dotnet/coreclr#6  __DangerousSwitchToThread (dwSleepMSec=<optimized out>, goThroughOS=0, dwSwitchCount=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/hosting.cpp:455
dotnet/coreclr#7  __SwitchToThread (dwSleepMSec=<optimized out>, dwSwitchCount=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/hosting.cpp:400
dotnet/coreclr#8  0x00007f7735837525 in ThreadpoolMgr::RecycledListInfo::AcquireLock (this=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/win32threadpool.h:697
dotnet/coreclr#9  ThreadpoolMgr::RecycledListInfo::Remove (this=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/win32threadpool.h:649
dotnet/coreclr#10 ThreadpoolMgr::GetRecycledMemory (memType=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/win32threadpool.cpp:1674
dotnet/coreclr#11 0x00007f7735998962 in DelegateInfo::MakeDelegateInfo (state=0x7f76ba6c5f08, waitEvent=0x7f76ba6c5f00, registeredWaitHandle=0x7f76ba6c5f10) at /home/ubuntu/tmp/coreclr/src/vm/comthreadpool.cpp:102
dotnet/coreclr#12 0x00007f773599a04a in ThreadPoolNative::CorRegisterWaitForSingleObject (waitObjectUNSAFE=<optimized out>, stateUNSAFE=<optimized out>, timeout=4294967295, executeOnlyOnce=<optimized out>,
    registeredWaitObjectUNSAFE=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/comthreadpool.cpp:435
dotnet/coreclr#13 0x00007f76bc0071fa in ?? ()
dotnet/coreclr#14 0x00007f75ec98d500 in ?? ()
dotnet/coreclr#15 0x00007f75ec98d598 in ?? ()
dotnet/coreclr#16 0x00007f75ec98d558 in ?? ()
dotnet/coreclr#17 0x00007f75ec98d558 in ?? ()
dotnet/coreclr#18 0x0000000100000001 in ?? ()
---Type <return> to continue, or q <return> to quit---
dotnet/coreclr#19 0x00007f75ec98d220 in ?? ()
dotnet/coreclr#20 0x0000000000000002 in ?? ()
dotnet/coreclr#21 0x00007f75ec98d4b0 in ?? ()
dotnet/coreclr#22 0x00000000ffffffff in ?? ()
dotnet/coreclr#23 0x00007f75ec98d3d8 in ?? ()
dotnet/coreclr#24 0x00007f76ba6c60f0 in ?? ()
dotnet/runtime#3858 0x00007f76bc0074ce in ?? ()
dotnet/runtime#3859 0x00007f75ec98d220 in ?? ()
dotnet/coreclr#27 0x00000000ffffffff in ?? ()
dotnet/runtime#3860 0x00007f76ba6c6170 in ?? ()
dotnet/runtime#3861 0x00007f76c0370075 in ?? ()
dotnet/runtime#3862 0x00007f76ba6c6100 in ?? ()
dotnet/runtime#3863 0x0000000000000000 in ?? ()

Thread 13 (Thread 0x7f76bb0ca700 (LWP 9137)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
dotnet/coreclr#1  0x00007f7735b65b82 in CorUnix::CPalSynchronizationManager::ThreadNativeWait (ptnwdNativeWaitData=<optimized out>, dwTimeout=<optimized out>, ptwrWakeupReason=<optimized out>,
    pdwSignaledObject=<optimized out>) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/synchmanager.cpp:479
dotnet/coreclr#2  0x00007f7735b65764 in CorUnix::CPalSynchronizationManager::BlockThread (this=0x882a40, pthrCurrent=0xb5f580, dwTimeout=4294967295, fAlertable=true, fIsSleep=<optimized out>,
    ptwrWakeupReason=0x7f76bb0c9318, pdwSignaledObject=0xfffffffffffffe00) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/synchmanager.cpp:302
dotnet/coreclr#3  0x00007f7735b6a13b in CorUnix::InternalWaitForMultipleObjectsEx (pThread=0xb5f580, nCount=<optimized out>, lpHandles=<optimized out>, bWaitAll=<optimized out>, dwMilliseconds=<optimized out>,
    bAlertable=<optimized out>, bPrioritize=<optimized out>) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/wait.cpp:640
dotnet/coreclr#4  0x00007f7735b6a491 in WaitForMultipleObjectsEx (nCount=2, lpHandles=0x7f76bb0c9750, bWaitAll=0, dwMilliseconds=4294967295, bAlertable=1)
    at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/wait.cpp:202
dotnet/coreclr#5  0x00007f77358144ee in Thread::DoAppropriateAptStateWait (numWaiters=2, pHandles=0x7f76bb0c9750, bWaitAll=0, timeout=<optimized out>, this=<optimized out>, mode=<optimized out>)
    at /home/ubuntu/tmp/coreclr/src/vm/threads.cpp:3313
dotnet/coreclr#6  Thread::DoAppropriateWaitWorker (this=0xb5eab0, countHandles=<optimized out>, handles=<optimized out>, waitAll=0, millis=<optimized out>, mode=WaitMode_Alertable)
    at /home/ubuntu/tmp/coreclr/src/vm/threads.cpp:3451
dotnet/coreclr#7  0x00007f773580f4b9 in Thread::DoAppropriateWait(int, void**, int, unsigned int, WaitMode, PendingSync*)::$_0::operator()(Thread::DoAppropriateWait(int, void**, int, unsigned int, WaitMode, PendingSync*)::__EEParam*) const (this=<optimized out>, __pEEParam=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/threads.cpp:3147
dotnet/coreclr#8  Thread::DoAppropriateWait (this=0xb5f7ec, countHandles=2, handles=0x7, waitAll=-1, millis=11925248, mode=(WaitMode_Alertable | WaitMode_IgnoreSyncCtx), syncState=<optimized out>)
    at /home/ubuntu/tmp/coreclr/src/vm/threads.cpp:3162
dotnet/coreclr#9  0x00007f773599c3e5 in WaitHandleNative::CorWaitMultipleNative (handleArray=<optimized out>, numHandles=2, waitForAll=<optimized out>, timeout=-1)
    at /home/ubuntu/tmp/coreclr/src/vm/comwaithandle.cpp:57
dotnet/coreclr#10 0x00007f76bc009c08 in ?? ()
dotnet/coreclr#11 0x00000000000002b8 in ?? ()
dotnet/coreclr#12 0x00000000000002b4 in ?? ()
dotnet/coreclr#13 0x00007f76bb0c9760 in ?? ()
dotnet/coreclr#14 0x00007f75ec06aba0 in ?? ()
dotnet/coreclr#15 0x00007f75ec06bd18 in ?? ()
---Type <return> to continue, or q <return> to quit---
dotnet/coreclr#16 0x00000000b3cc2330 in ?? ()
dotnet/coreclr#17 0x00007f75ec06bcf0 in ?? ()
dotnet/coreclr#18 0x00007f76bb0c9750 in ?? ()
dotnet/coreclr#19 0x0000000000000002 in ?? ()
dotnet/coreclr#20 0x00007f75ec06bd00 in ?? ()
dotnet/coreclr#21 0x0000000000000002 in ?? ()
dotnet/coreclr#22 0x00007f75ec06bcd8 in ?? ()
dotnet/coreclr#23 0x0000000000000002 in ?? ()
dotnet/coreclr#24 0x0000000200000000 in ?? ()
dotnet/runtime#3858 0x00007f76bb0c9750 in ?? ()
dotnet/runtime#3859 0x00007f75ec06bcd8 in ?? ()
dotnet/coreclr#27 0x00007f7500000002 in ?? ()
dotnet/runtime#3860 0x00007f75ec06a768 in ?? ()
dotnet/runtime#3861 0x00007f75ec06a7f0 in ?? ()
dotnet/runtime#3862 0x0000000000000001 in ?? ()
dotnet/runtime#3863 0x00007f76bb0c9890 in ?? ()
dotnet/coreclr#32 0x00007f75ec06bcc8 in ?? ()
dotnet/runtime#3864 0x00007f76bb0c9840 in ?? ()
dotnet/runtime#3865 0x00007f76bc009942 in ?? ()
dotnet/coreclr#35 0x00007f76bb0c9810 in ?? ()
dotnet/coreclr#36 0x00007f75ec06bcd8 in ?? ()
dotnet/runtime#3866 0x00007f7500000002 in ?? ()
dotnet/coreclr#38 0x00000000ec06a768 in ?? ()
dotnet/runtime#3867 0x00007f75ec06a768 in ?? ()
dotnet/runtime#3868 0x00007f76bb0c9890 in ?? ()
dotnet/coreclr#41 0x00007f76bb0c9880 in ?? ()
dotnet/coreclr#42 0x00007f76be19d764 in ?? ()
dotnet/runtime#3869 0x00007f76bb0c9850 in ?? ()
dotnet/runtime#3870 0x0000000000000000 in ?? ()

Thread 12 (Thread 0x7f771ef42700 (LWP 9135)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
dotnet/coreclr#1  0x00007f7735b65b45 in CorUnix::CPalSynchronizationManager::ThreadNativeWait (ptnwdNativeWaitData=<optimized out>, dwTimeout=<optimized out>, ptwrWakeupReason=<optimized out>,
    pdwSignaledObject=<optimized out>) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/synchmanager.cpp:484
dotnet/coreclr#2  0x00007f7735b65764 in CorUnix::CPalSynchronizationManager::BlockThread (this=0x882a40, pthrCurrent=0xa8a4e0, dwTimeout=30000, fAlertable=true, fIsSleep=<optimized out>,
    ptwrWakeupReason=0x7f771ef412d8, pdwSignaledObject=0xfffffffffffffdfc) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/synchmanager.cpp:302
dotnet/coreclr#3  0x00007f7735b6a13b in CorUnix::InternalWaitForMultipleObjectsEx (pThread=0xa8a4e0, nCount=<optimized out>, lpHandles=<optimized out>, bWaitAll=<optimized out>, dwMilliseconds=<optimized out>,
    bAlertable=<optimized out>, bPrioritize=<optimized out>) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/wait.cpp:640
dotnet/coreclr#4  0x00007f7735b6a491 in WaitForMultipleObjectsEx (nCount=1, lpHandles=0x7f771ef41698, bWaitAll=1, dwMilliseconds=30000, bAlertable=1) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/wait.cpp:202
dotnet/coreclr#5  0x00007f7735814334 in Thread::DoAppropriateAptStateWait (numWaiters=1, pHandles=0x7f771ef41698, bWaitAll=1, timeout=<optimized out>, this=<optimized out>, mode=<optimized out>)
---Type <return> to continue, or q <return> to quit---
    at /home/ubuntu/tmp/coreclr/src/vm/threads.cpp:3313
dotnet/coreclr#6  Thread::DoAppropriateWaitWorker (this=0xa8f520, countHandles=<optimized out>, handles=<optimized out>, waitAll=1, millis=<optimized out>, mode=WaitMode_Alertable)
    at /home/ubuntu/tmp/coreclr/src/vm/threads.cpp:3451
dotnet/coreclr#7  0x00007f773580f4b9 in Thread::DoAppropriateWait(int, void**, int, unsigned int, WaitMode, PendingSync*)::$_0::operator()(Thread::DoAppropriateWait(int, void**, int, unsigned int, WaitMode, PendingSync*)::__EEParam*) const (this=<optimized out>, __pEEParam=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/threads.cpp:3147
dotnet/coreclr#8  Thread::DoAppropriateWait (this=0xa8a74c, countHandles=1, handles=0x1, waitAll=-1, millis=11052832, mode=(WaitMode_Alertable | WaitMode_IgnoreSyncCtx | WaitMode_InDeadlock | unknown: 4294967288),
    syncState=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/threads.cpp:3162
dotnet/coreclr#9  0x00007f773599c14c in WaitHandleNative::CorWaitOneNative (handle=<optimized out>, timeout=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/comwaithandle.cpp:32
dotnet/coreclr#10 0x00007f76bc00954d in ?? ()
dotnet/coreclr#11 0x00007f771ef41700 in ?? ()
dotnet/coreclr#12 0x0000000000000000 in ?? ()

Thread 11 (Thread 0x7f771f943700 (LWP 9133)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
dotnet/coreclr#1  0x00007f7735b65b82 in CorUnix::CPalSynchronizationManager::ThreadNativeWait (ptnwdNativeWaitData=<optimized out>, dwTimeout=<optimized out>, ptwrWakeupReason=<optimized out>,
    pdwSignaledObject=<optimized out>) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/synchmanager.cpp:479
dotnet/coreclr#2  0x00007f7735b65764 in CorUnix::CPalSynchronizationManager::BlockThread (this=0x882a40, pthrCurrent=0xa426f0, dwTimeout=4294967295, fAlertable=false, fIsSleep=<optimized out>,
    ptwrWakeupReason=0x7f771f942bc8, pdwSignaledObject=0xfffffffffffffe00) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/synchmanager.cpp:302
dotnet/coreclr#3  0x00007f7735b6a13b in CorUnix::InternalWaitForMultipleObjectsEx (pThread=0xa426f0, nCount=<optimized out>, lpHandles=<optimized out>, bWaitAll=<optimized out>, dwMilliseconds=<optimized out>,
    bAlertable=<optimized out>, bPrioritize=<optimized out>) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/wait.cpp:640
dotnet/coreclr#4  0x00007f7735b6a3b9 in WaitForSingleObjectEx (hHandle=<error reading variable: Cannot access memory at address 0xd>, dwMilliseconds=4294967295, bAlertable=0)
    at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/wait.cpp:139
dotnet/coreclr#5  0x00007f77358e2784 in CLREventWaitHelper2 (handle=0xa4295c, dwMilliseconds=<optimized out>, alertable=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/synch.cpp:377
dotnet/coreclr#6  CLREventWaitHelper(void*, unsigned int, int)::$_1::operator()(CLREventWaitHelper(void*, unsigned int, int)::Param*) const (this=<optimized out>, pParam=<optimized out>)
    at /home/ubuntu/tmp/coreclr/src/vm/synch.cpp:402
dotnet/coreclr#7  CLREventWaitHelper (dwMilliseconds=4294967295, alertable=<optimized out>, handle=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/synch.cpp:404
dotnet/coreclr#8  CLREventBase::WaitEx (this=<optimized out>, dwMilliseconds=4294967295, mode=<optimized out>, syncState=0x0) at /home/ubuntu/tmp/coreclr/src/vm/synch.cpp:471
dotnet/coreclr#9  0x00007f77358e61eb in Thread::RareDisablePreemptiveGC (this=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/threadsuspend.cpp:2576
dotnet/coreclr#10 0x00007f7735810ff6 in Thread::DisablePreemptiveGC (this=0xa41c20) at /home/ubuntu/tmp/coreclr/src/vm/threads.h:2036
dotnet/coreclr#11 GCHolderBase::EnterInternalCoop (pThread=0xa41c20, conditional=<error reading variable: access outside bounds of object referenced via synthetic pointer>, this=<optimized out>)
    at /home/ubuntu/tmp/coreclr/src/vm/threads.h:5800
dotnet/coreclr#12 GCCoop::GCCoop (this=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/threads.h:5920
dotnet/coreclr#13 Thread::OnThreadTerminate (this=0xa41c20, holdingLock=0) at /home/ubuntu/tmp/coreclr/src/vm/threads.cpp:2912
dotnet/coreclr#14 0x00007f77358380d3 in ThreadpoolMgr::WorkerThreadStart (lpArgs=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/win32threadpool.cpp:2192
dotnet/coreclr#15 0x00007f7735b7092d in CorUnix::CPalThread::ThreadEntry (pvParam=0xa426f0) at /home/ubuntu/tmp/coreclr/src/pal/src/thread/thread.cpp:1807
dotnet/coreclr#16 0x00007f77371a66ba in start_thread (arg=0x7f771f943700) at pthread_create.c:333
dotnet/coreclr#17 0x00007f773643741d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 10 (Thread 0x7f7720344700 (LWP 9132)):
#0  0x00007f77371af51d in read () at ../sysdeps/unix/syscall-template.S:84
---Type <return> to continue, or q <return> to quit---
dotnet/coreclr#1  0x00007f77301c6d5f in ?? () from /home/ubuntu/tmp/coreclr/.dotnet/shared/Microsoft.NETCore.App/3.0.0-preview6-27804-01/System.Native.so
dotnet/coreclr#2  0x00007f77371a66ba in start_thread (arg=0x7f7720344700) at pthread_create.c:333
dotnet/coreclr#3  0x00007f773643741d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 9 (Thread 0x7f77374ad700 (LWP 9131)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
dotnet/coreclr#1  0x00007f7735b65b45 in CorUnix::CPalSynchronizationManager::ThreadNativeWait (ptnwdNativeWaitData=<optimized out>, dwTimeout=<optimized out>, ptwrWakeupReason=<optimized out>,
    pdwSignaledObject=<optimized out>) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/synchmanager.cpp:484
dotnet/coreclr#2  0x00007f7735b65764 in CorUnix::CPalSynchronizationManager::BlockThread (this=0x882a40, pthrCurrent=0x8a3230, dwTimeout=1, fAlertable=false, fIsSleep=<optimized out>,
    ptwrWakeupReason=0x7f77374accd0, pdwSignaledObject=0xfffffffffffffdfc) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/synchmanager.cpp:302
dotnet/coreclr#3  0x00007f7735b6a8e9 in CorUnix::InternalSleepEx (pThread=<optimized out>, dwMilliseconds=1, bAlertable=0) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/wait.cpp:855
dotnet/coreclr#4  SleepEx (dwMilliseconds=1, bAlertable=0) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/wait.cpp:286
dotnet/coreclr#5  0x00007f773587ec7a in EESleepEx (dwMilliseconds=<optimized out>, bAlertable=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/hosting.cpp:379
dotnet/coreclr#6  __DangerousSwitchToThread (dwSleepMSec=<optimized out>, goThroughOS=0, dwSwitchCount=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/hosting.cpp:455
dotnet/coreclr#7  __SwitchToThread (dwSleepMSec=<optimized out>, dwSwitchCount=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/hosting.cpp:400
dotnet/coreclr#8  0x00007f7735837525 in ThreadpoolMgr::RecycledListInfo::AcquireLock (this=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/win32threadpool.h:697
dotnet/coreclr#9  ThreadpoolMgr::RecycledListInfo::Remove (this=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/win32threadpool.h:649
dotnet/coreclr#10 ThreadpoolMgr::GetRecycledMemory (memType=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/win32threadpool.cpp:1674
dotnet/coreclr#11 0x00007f7735985264 in ThreadpoolMgr::MakeWorkRequest (function=<optimized out>, context=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/win32threadpool.h:367
dotnet/coreclr#12 UnManagedPerAppDomainTPCount::QueueUnmanagedWorkRequest (this=0x7f7735e52640 <PerAppDomainTPCountList::s_unmanagedTPCount>,
    function=0x7f773583b570 <ThreadpoolMgr::AsyncTimerCallbackCompletion(void*)>, context=0x88e720) at /home/ubuntu/tmp/coreclr/src/vm/threadpoolrequest.cpp:356
dotnet/coreclr#13 0x00007f773583ad80 in ThreadpoolMgr::QueueUserWorkItem (Function=<optimized out>, Flags=0, UnmanagedTPRequest=1, Context=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/win32threadpool.cpp:855
dotnet/coreclr#14 ThreadpoolMgr::FireTimers () at /home/ubuntu/tmp/coreclr/src/vm/win32threadpool.cpp:4710
dotnet/coreclr#15 0x00007f773583a9c1 in ThreadpoolMgr::TimerThreadFire () at /home/ubuntu/tmp/coreclr/src/vm/win32threadpool.cpp:4598
dotnet/coreclr#16 0x00007f773583a895 in ThreadpoolMgr::TimerThreadStart (p=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/win32threadpool.cpp:4569
dotnet/coreclr#17 0x00007f7735b7092d in CorUnix::CPalThread::ThreadEntry (pvParam=0x8a3230) at /home/ubuntu/tmp/coreclr/src/pal/src/thread/thread.cpp:1807
dotnet/coreclr#18 0x00007f77371a66ba in start_thread (arg=0x7f77374ad700) at pthread_create.c:333
dotnet/coreclr#19 0x00007f773643741d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 8 (Thread 0x7f77231fd700 (LWP 9130)):
#0  0x00007f77371af76d in accept () at ../sysdeps/unix/syscall-template.S:84
dotnet/coreclr#1  0x00007f7735a8d8ed in IpcStream::DiagnosticsIpc::Accept (this=<optimized out>, callback=0x7f773585f970 <DiagnosticsServerThread(void*)::$_2::__invoke(char const*, unsigned int)>)
    at /home/ubuntu/tmp/coreclr/src/debug/debug-pal/unix/diagnosticsipc.cpp:126
dotnet/coreclr#2  0x00007f773585f310 in DiagnosticsServerThread (lpThreadParameter=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/diagnosticserver.cpp:51
dotnet/coreclr#3  0x00007f7735b7092d in CorUnix::CPalThread::ThreadEntry (pvParam=0x8a2880) at /home/ubuntu/tmp/coreclr/src/pal/src/thread/thread.cpp:1807
dotnet/coreclr#4  0x00007f77371a66ba in start_thread (arg=0x7f77231fd700) at pthread_create.c:333
dotnet/coreclr#5  0x00007f773643741d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 7 (Thread 0x7f7723bfe700 (LWP 9129)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
---Type <return> to continue, or q <return> to quit---
dotnet/coreclr#1  0x00007f7735b65b45 in CorUnix::CPalSynchronizationManager::ThreadNativeWait (ptnwdNativeWaitData=<optimized out>, dwTimeout=<optimized out>, ptwrWakeupReason=<optimized out>,
    pdwSignaledObject=<optimized out>) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/synchmanager.cpp:484
dotnet/coreclr#2  0x00007f7735b65764 in CorUnix::CPalSynchronizationManager::BlockThread (this=0x882a40, pthrCurrent=0x8a18d0, dwTimeout=10000, fAlertable=false, fIsSleep=<optimized out>,
    ptwrWakeupReason=0x7f7723bfdb28, pdwSignaledObject=0xfffffffffffffdfc) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/synchmanager.cpp:302
dotnet/coreclr#3  0x00007f7735b6a13b in CorUnix::InternalWaitForMultipleObjectsEx (pThread=0x8a18d0, nCount=<optimized out>, lpHandles=<optimized out>, bWaitAll=<optimized out>, dwMilliseconds=<optimized out>,
    bAlertable=<optimized out>, bPrioritize=<optimized out>) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/wait.cpp:640
dotnet/coreclr#4  0x00007f7735b6a491 in WaitForMultipleObjectsEx (nCount=1, lpHandles=0x7f7735e4f458 <FinalizerThread::MHandles+8>, bWaitAll=0, dwMilliseconds=10000, bAlertable=0)
    at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/wait.cpp:202
dotnet/coreclr#5  0x00007f773587925e in FinalizerThread::WaitForFinalizerEvent (event=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/finalizerthread.cpp:200
dotnet/coreclr#6  0x00007f7735879372 in FinalizerThread::FinalizerThreadWorker (args=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/finalizerthread.cpp:289
dotnet/coreclr#7  0x00007f7735817d05 in ManagedThreadBase_DispatchInner (pCallState=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/threads.cpp:7410
dotnet/coreclr#8  ManagedThreadBase_DispatchMiddle (pCallState=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/threads.cpp:7454
dotnet/coreclr#9  ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)::$_6::operator()(ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)::TryArgs*) const::{lambda(Param*)#1}::operator()(Param*) const (
    this=<optimized out>, pParam=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/threads.cpp:7618
dotnet/coreclr#10 ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)::$_6::operator()(ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)::TryArgs*) const (this=<optimized out>, pArgs=<optimized out>)
    at /home/ubuntu/tmp/coreclr/src/vm/threads.cpp:7620
dotnet/coreclr#11 ManagedThreadBase_DispatchOuter (pCallState=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/threads.cpp:7646
dotnet/coreclr#12 0x00007f77358183bd in ManagedThreadBase_NoADTransition (pTarget=<optimized out>, filterType=FinalizerThread) at /home/ubuntu/tmp/coreclr/src/vm/threads.cpp:7690
dotnet/coreclr#13 ManagedThreadBase::FinalizerBase (pTarget=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/threads.cpp:7716
dotnet/coreclr#14 0x00007f77358795ac in FinalizerThread::FinalizerThreadStart (args=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/finalizerthread.cpp:411
dotnet/coreclr#15 0x00007f7735b7092d in CorUnix::CPalThread::ThreadEntry (pvParam=0x8a18d0) at /home/ubuntu/tmp/coreclr/src/pal/src/thread/thread.cpp:1807
dotnet/coreclr#16 0x00007f77371a66ba in start_thread (arg=0x7f7723bfe700) at pthread_create.c:333
dotnet/coreclr#17 0x00007f773643741d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 6 (Thread 0x7f76bbacf700 (LWP 9128)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
dotnet/coreclr#1  0x00007f7735b65b82 in CorUnix::CPalSynchronizationManager::ThreadNativeWait (ptnwdNativeWaitData=<optimized out>, dwTimeout=<optimized out>, ptwrWakeupReason=<optimized out>,
    pdwSignaledObject=<optimized out>) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/synchmanager.cpp:479
dotnet/coreclr#2  0x00007f7735b65764 in CorUnix::CPalSynchronizationManager::BlockThread (this=0x882a40, pthrCurrent=0x8d0810, dwTimeout=4294967295, fAlertable=false, fIsSleep=<optimized out>,
    ptwrWakeupReason=0x7f76bbaceca8, pdwSignaledObject=0xfffffffffffffe00) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/synchmanager.cpp:302
dotnet/coreclr#3  0x00007f7735b6a13b in CorUnix::InternalWaitForMultipleObjectsEx (pThread=0x8d0810, nCount=<optimized out>, lpHandles=<optimized out>, bWaitAll=<optimized out>, dwMilliseconds=<optimized out>,
    bAlertable=<optimized out>, bPrioritize=<optimized out>) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/wait.cpp:640
dotnet/coreclr#4  0x00007f7735b6a491 in WaitForMultipleObjectsEx (nCount=3, lpHandles=0x7f76bbacee50, bWaitAll=0, dwMilliseconds=4294967295, bAlertable=0)
    at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/wait.cpp:202
dotnet/coreclr#5  0x00007f7735a83d06 in DebuggerRCThread::MainLoop (this=<optimized out>) at /home/ubuntu/tmp/coreclr/src/debug/ee/rcthread.cpp:1137
dotnet/coreclr#6  0x00007f7735a83bab in DebuggerRCThread::ThreadProc (this=0x885ae0) at /home/ubuntu/tmp/coreclr/src/debug/ee/rcthread.cpp:942
dotnet/coreclr#7  0x00007f7735a838c9 in DebuggerRCThread::ThreadProcStatic () at /home/ubuntu/tmp/coreclr/src/debug/ee/rcthread.cpp:1526
dotnet/coreclr#8  0x00007f7735b7092d in CorUnix::CPalThread::ThreadEntry (pvParam=0x8d0810) at /home/ubuntu/tmp/coreclr/src/pal/src/thread/thread.cpp:1807
dotnet/coreclr#9  0x00007f77371a66ba in start_thread (arg=0x7f76bbacf700) at pthread_create.c:333
dotnet/coreclr#10 0x00007f773643741d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
---Type <return> to continue, or q <return> to quit---

Thread 5 (Thread 0x7f773298c700 (LWP 9127)):
#0  0x00007f77371afc7d in open64 () at ../sysdeps/unix/syscall-template.S:84
dotnet/coreclr#1  0x00007f7735a8db5f in TwoWayPipe::WaitForConnection (this=0x88ce50) at /home/ubuntu/tmp/coreclr/src/debug/debug-pal/unix/twowaypipe.cpp:88
dotnet/coreclr#2  0x00007f7735a86696 in DbgTransportSession::TransportWorker (this=0x88cd70) at /home/ubuntu/tmp/coreclr/src/debug/ee/../shared/dbgtransportsession.cpp:1320
dotnet/coreclr#3  0x00007f7735a85569 in DbgTransportSession::TransportWorkerStatic (pvContext=0x88ce5c) at /home/ubuntu/tmp/coreclr/src/debug/ee/../shared/dbgtransportsession.cpp:1236
dotnet/coreclr#4  0x00007f7735b7092d in CorUnix::CPalThread::ThreadEntry (pvParam=0x85b7f0) at /home/ubuntu/tmp/coreclr/src/pal/src/thread/thread.cpp:1807
dotnet/coreclr#5  0x00007f77371a66ba in start_thread (arg=0x7f773298c700) at pthread_create.c:333
dotnet/coreclr#6  0x00007f773643741d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 4 (Thread 0x7f7733598700 (LWP 9126)):
#0  0x00007f773642b74d in poll () at ../sysdeps/unix/syscall-template.S:84
dotnet/coreclr#1  0x00007f7735b67f83 in CorUnix::CPalSynchronizationManager::ReadBytesFromProcessPipe (this=0x882a40, iTimeout=-1, pRecvBuf=<optimized out>, iBytes=1)
    at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/synchmanager.cpp:2215
dotnet/coreclr#2  0x00007f7735b674e3 in CorUnix::CPalSynchronizationManager::ReadCmdFromProcessPipe (iPollTimeout=-1, this=<optimized out>, pswcWorkerCmd=<optimized out>, pshridMarshaledData=<optimized out>,
    pdwData=<optimized out>) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/synchmanager.cpp:1993
dotnet/coreclr#3  CorUnix::CPalSynchronizationManager::WorkerThread (pArg=<optimized out>) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/synchmanager.cpp:1696
dotnet/coreclr#4  0x00007f7735b7092d in CorUnix::CPalThread::ThreadEntry (pvParam=0x876cf0) at /home/ubuntu/tmp/coreclr/src/pal/src/thread/thread.cpp:1807
dotnet/coreclr#5  0x00007f77371a66ba in start_thread (arg=0x7f7733598700) at pthread_create.c:333
dotnet/coreclr#6  0x00007f773643741d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 3 (Thread 0x7f7733f99700 (LWP 9125)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
dotnet/coreclr#1  0x00007f7734fdb87c in ?? () from /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
dotnet/coreclr#2  0x00007f77371a66ba in start_thread (arg=0x7f7733f99700) at pthread_create.c:333
dotnet/coreclr#3  0x00007f773643741d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 2 (Thread 0x7f773499a700 (LWP 9124)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
dotnet/coreclr#1  0x00007f7734fdb87c in ?? () from /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
dotnet/coreclr#2  0x00007f77371a66ba in start_thread (arg=0x7f773499a700) at pthread_create.c:333
dotnet/coreclr#3  0x00007f773643741d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 1 (Thread 0x7f77375d2740 (LWP 9123)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
dotnet/coreclr#1  0x00007f7735b65b82 in CorUnix::CPalSynchronizationManager::ThreadNativeWait (ptnwdNativeWaitData=<optimized out>, dwTimeout=<optimized out>, ptwrWakeupReason=<optimized out>,
    pdwSignaledObject=<optimized out>) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/synchmanager.cpp:479
dotnet/coreclr#2  0x00007f7735b65764 in CorUnix::CPalSynchronizationManager::BlockThread (this=0x882a40, pthrCurrent=0x8762d0, dwTimeout=4294967295, fAlertable=true, fIsSleep=<optimized out>,
    ptwrWakeupReason=0x7ffce20d51b8, pdwSignaledObject=0xfffffffffffffe00) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/synchmanager.cpp:302
dotnet/coreclr#3  0x00007f7735b6a13b in CorUnix::InternalWaitForMultipleObjectsEx (pThread=0x8762d0, nCount=<optimized out>, lpHandles=<optimized out>, bWaitAll=<optimized out>, dwMilliseconds=<optimized out>,
---Type <return> to continue, or q <return> to quit---
    bAlertable=<optimized out>, bPrioritize=<optimized out>) at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/wait.cpp:640
dotnet/coreclr#4  0x00007f7735b6a491 in WaitForMultipleObjectsEx (nCount=1, lpHandles=0x7ffce20d5578, bWaitAll=1, dwMilliseconds=4294967295, bAlertable=1)
    at /home/ubuntu/tmp/coreclr/src/pal/src/synchmgr/wait.cpp:202
dotnet/coreclr#5  0x00007f7735814334 in Thread::DoAppropriateAptStateWait (numWaiters=1, pHandles=0x7ffce20d5578, bWaitAll=1, timeout=<optimized out>, this=<optimized out>, mode=<optimized out>)
    at /home/ubuntu/tmp/coreclr/src/vm/threads.cpp:3313
dotnet/coreclr#6  Thread::DoAppropriateWaitWorker (this=0x85cfd0, countHandles=<optimized out>, handles=<optimized out>, waitAll=1, millis=<optimized out>, mode=WaitMode_Alertable)
    at /home/ubuntu/tmp/coreclr/src/vm/threads.cpp:3451
dotnet/coreclr#7  0x00007f773580f4b9 in Thread::DoAppropriateWait(int, void**, int, unsigned int, WaitMode, PendingSync*)::$_0::operator()(Thread::DoAppropriateWait(int, void**, int, unsigned int, WaitMode, PendingSync*)::__EEParam*) const (this=<optimized out>, __pEEParam=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/threads.cpp:3147
dotnet/coreclr#8  Thread::DoAppropriateWait (this=0x87653c, countHandles=1, handles=0x5, waitAll=-1, millis=8873216, mode=WaitMode_IgnoreSyncCtx, syncState=<optimized out>)
    at /home/ubuntu/tmp/coreclr/src/vm/threads.cpp:3162
dotnet/coreclr#9  0x00007f773599c14c in WaitHandleNative::CorWaitOneNative (handle=<optimized out>, timeout=<optimized out>) at /home/ubuntu/tmp/coreclr/src/vm/comwaithandle.cpp:32
dotnet/coreclr#10 0x00007f76bc00954d in ?? ()
dotnet/coreclr#11 0x00007ffce20d55e0 in ?? ()
dotnet/coreclr#12 0x0000000000000000 in ?? ()

When using debug build, some assertion failed instead of freeze:

Commencing CoreCLR Repo build
__DistroRid: linux-x64
__RuntimeId: linux-x64
Setting up directories for build
Checking prerequisites...

Assert failure(PID 8651 [0x000021cb], Thread: 8664 [0x21d8]): !"Heap contamination detected! HeapFree was called on a heap other than the one that memory was allocated from.\n" "Possible cause: you used new (executable) to allocate the memory, but didn't use DeleteExecutable() to free it."
    File: /home/ubuntu/tmp/coreclr/src/vm/hosting.cpp Line: 268
    Image: /home/ubuntu/tmp/coreclr/.dotnet/dotnet

Failed to get IBC data package version.

303248153 on 25 Sep 2019

tag v3.0.0 vs. 3.0.0-preview6-27804-01

CoreCLR is coupled with the rest of the stack. Tag v3.0.0 and preview6 are many months apart. It is quite possible that CoreCLR from v3.0.0 tag is not compatible with preview6 and the crashes you are seeing are caused by mismatched bits. Are you able to reproduce the problems with exactly matching bits?

When using debug build, some assertion failed instead of freeze:

Any change you can get this under debugger and capture the stacktrace of the crash?

jkotas on 25 Sep 2019

After I set tools.dotnet to "3.0.100", and overwrite custom built coreclr to .dotnet/shared/Microsoft.NETCore.App/3.0.0, no assertion failure occurs, both debug and release build will just freeze. Looks like you're right about the mismatched bits.

To invesgate the freeze issue, I modify src/vm/win32threadpool.h to make it output messages around acquire and release operation:

        FORCEINLINE LPVOID Remove()
        {
            LIMITED_METHOD_CONTRACT;

            if(root == NULL) return NULL; // No need for acquiring the lock, there's nothing to remove.

        printf("remove lock acquiring %p %d\n", GetThread(), lock != 0);
            AcquireLock();
        printf("remove lock acquired %p %d\n", GetThread(), lock != 0);

            Entry* ret = (Entry*)root;

            if(ret)
            {
                root   = ret->next;
                count -= 1;
            }

            ReleaseLock();
        printf("remove lock released %p %d\n", GetThread(), lock != 0);

            return ret;
        }

        FORCEINLINE void Insert( LPVOID mem )
        {
            LIMITED_METHOD_CONTRACT;

        printf("insert lock acquiring %p %d\n", GetThread(), lock != 0);
        AcquireLock();
        printf("insert lock acquired %p %d\n", GetThread(), lock != 0);

            Entry* entry = (Entry*)mem;

            entry->next = root;

            root   = entry;
            count += 1;

            ReleaseLock();
        printf("insert lock released %p %d\n", GetThread(), lock != 0);
        }

Then I saw following messages when freeze occurs:

insert lock acquiring 00007F538400A470 0
insert lock acquired 00007F538400A470 1
insert lock released 00007F538400A470 0
insert lock acquiring 0000000000E984E0 0
insert lock acquired 0000000000E984E0 1
insert lock released 0000000000E984E0 0
remove lock acquiring 00007F5384000A10 1
remove lock acquiring 00007FBF60000A10 1
remove lock acquiring 0000000001F44E60 1

Looks like somewhere set the lock member to 1, or some weird out of order execution bug occurs :/

303248153 on 26 Sep 2019

After add address of this to message:

insert lock acquiring 00007FBF5800A470 0 this 0000000001FE40C0
insert lock acquired 00007FBF5800A470 1 this 0000000001FE40C0
insert lock released 00007FBF5800A470 0 this 0000000001FE40C0
insert lock acquiring 00000000020ABCD0 0 this 0000000001FE40F0
insert lock acquired 00000000020ABCD0 1 this 0000000001FE40F0
insert lock released 00000000020ABCD0 0 this 0000000001FE40F0
remove lock acquiring 00007FBF58000A10 0 this 0000000001FE40F0
remove lock acquired 00007FBF58000A10 1 this 0000000001FE40F0
remove lock released 00007FBF58000A10 0 this 0000000001FE40F0
insert lock acquiring 00000000020ABCD0 0 this 0000000001FE40F0
insert lock acquired 00000000020ABCD0 1 this 0000000001FE40F0
insert lock released 00000000020ABCD0 0 this 0000000001FE40F0
remove lock acquiring 00007FBF58000A10 0 this 0000000001FE40F0
remove lock acquired 00007FBF58000A10 1 this 0000000001FE40F0
remove lock released 00007FBF58000A10 0 this 0000000001FE40F0
insert lock acquiring 00000000020ABCD0 0 this 0000000001FE40F0
insert lock acquired 00000000020ABCD0 1 this 0000000001FE40F0
insert lock released 00000000020ABCD0 0 this 0000000001FE40F0
remove lock acquiring 00007F3728000A10 1 this 0000000001DBFD60
remove lock acquiring 00000000020C1AB0 0 this 0000000001FE40F0
remove lock acquired 00000000020C1AB0 1 this 0000000001FE40F0
remove lock released 00000000020C1AB0 0 this 0000000001FE40F0
insert lock acquiring 00000000020ABCD0 0 this 0000000001FE40F0
insert lock acquired 00000000020ABCD0 1 this 0000000001FE40F0
insert lock released 00000000020ABCD0 0 this 0000000001FE40F0
remove lock acquiring 00000000020ABCD0 0 this 0000000001FE40F0
remove lock acquired 00000000020ABCD0 1 this 0000000001FE40F0
remove lock released 00000000020ABCD0 0 this 0000000001FE40F0
insert lock acquiring 00007FBF5800A470 0 this 0000000001FE40F0
insert lock acquired 00007FBF5800A470 1 this 0000000001FE40F0
insert lock released 00007FBF5800A470 0 this 0000000001FE40F0
remove lock acquiring 00007FBF58000A10 0 this 0000000001FE40F0
remove lock acquired 00007FBF58000A10 1 this 0000000001FE40F0
remove lock released 00007FBF58000A10 0 this 0000000001FE40F0
insert lock acquiring 00000000020ABCD0 0 this 0000000001FE40F0
insert lock acquired 00000000020ABCD0 1 this 0000000001FE40F0
insert lock released 00000000020ABCD0 0 this 0000000001FE40F0
remove lock acquiring 00007FBF58000A10 0 this 0000000001FE40F0
remove lock acquired 00007FBF58000A10 1 this 0000000001FE40F0
remove lock released 00007FBF58000A10 0 this 0000000001FE40F0
insert lock acquiring 00000000020ABCD0 0 this 0000000001FE40F0
insert lock acquired 00000000020ABCD0 1 this 0000000001FE40F0
insert lock released 00000000020ABCD0 0 this 0000000001FE40F0
remove lock acquiring 00000000020ABCD0 0 this 0000000001FE40F0
remove lock acquired 00000000020ABCD0 1 this 0000000001FE40F0
remove lock released 00000000020ABCD0 0 this 0000000001FE40F0
insert lock acquiring 00007FBF5800A470 0 this 0000000001FE40F0
insert lock acquired 00007FBF5800A470 1 this 0000000001FE40F0
insert lock released 00007FBF5800A470 0 this 0000000001FE40F0
remove lock acquiring 00007FBF5800A470 0 this 0000000001FE40C0
remove lock acquired 00007FBF5800A470 1 this 0000000001FE40C0
remove lock released 00007FBF5800A470 0 this 0000000001FE40C0
insert lock acquiring 00000000020ABCD0 0 this 0000000001FE40C0
insert lock acquired 00000000020ABCD0 1 this 0000000001FE40C0
insert lock released 00000000020ABCD0 0 this 0000000001FE40C0
remove lock acquiring 00000000020C1AB0 0 this 0000000001FE40C0
remove lock acquired 00000000020C1AB0 1 this 0000000001FE40C0
remove lock released 00000000020C1AB0 0 this 0000000001FE40C0
insert lock acquiring 00000000020ABCD0 0 this 0000000001FE40C0
insert lock acquired 00000000020ABCD0 1 this 0000000001FE40C0
insert lock released 00000000020ABCD0 0 this 0000000001FE40C0
remove lock acquiring 00000000020BDD30 0 this 0000000001FE40C0
remove lock acquired 00000000020BDD30 1 this 0000000001FE40C0
remove lock released 00000000020BDD30 0 this 0000000001FE40C0
insert lock acquiring 00000000020ABCD0 0 this 0000000001FE40C0
insert lock acquired 00000000020ABCD0 1 this 0000000001FE40C0
insert lock released 00000000020ABCD0 0 this 0000000001FE40C0
remove lock acquiring 00000000020BED30 0 this 0000000001FE40C0
remove lock acquired 00000000020BED30 1 this 0000000001FE40C0
remove lock released 00000000020BED30 0 this 0000000001FE40C0
insert lock acquiring 00007FBF5800A470 0 this 0000000001FE40C0
insert lock acquired 00007FBF5800A470 1 this 0000000001FE40C0
insert lock released 00007FBF5800A470 0 this 0000000001FE40C0
remove lock acquiring 00000000020BDD30 0 this 0000000001FE40C0
remove lock acquired 00000000020BDD30 1 this 0000000001FE40C0
remove lock released 00000000020BDD30 0 this 0000000001FE40C0
insert lock acquiring 00007FBF5800A470 0 this 0000000001FE40C0
insert lock acquired 00007FBF5800A470 1 this 0000000001FE40C0
insert lock released 00007FBF5800A470 0 this 0000000001FE40C0
remove lock acquiring 00007FBF5800A470 0 this 0000000001FE40C0
remove lock acquired 00007FBF5800A470 1 this 0000000001FE40C0
remove lock released 00007FBF5800A470 0 this 0000000001FE40C0
insert lock acquiring 00007FC0A4016B30 0 this 0000000001FE40C0
insert lock acquired 00007FC0A4016B30 1 this 0000000001FE40C0
insert lock released 00007FC0A4016B30 0 this 0000000001FE40C0
remove lock acquiring 0000000001F94E10 1 this 0000000001DBFD20
remove lock acquiring 00007FBF58000A10 1 this 0000000001FE41B0

so some thread using RecycledListInfo instance at incorrect memory address?

303248153 on 26 Sep 2019

The reason is sched_getcpu() may return a value greater than or equal to CPU_COUNT(&cpuSet) on my ubuntu 16.04 vps which using openvz.

I wrote a simple c program to verify it:

#define _GNU_SOURCE
#include <sched.h>
#include <unistd.h>
#include <stdio.h>

int main() {
    while (1) {
        cpu_set_t cpuSet;
        int st = sched_getaffinity(getpid(), sizeof(cpu_set_t), &cpuSet);
        int nproc = CPU_COUNT(&cpuSet);
        for (int x = 0; x < nproc; ++x) {
            printf("affinity is %d for cpu %d\n", CPU_ISSET(x, &cpuSet), x);
        }

        printf("cpuid: %d\n", sched_getcpu());
        sleep(1);
    }
    return 0;
}

output:

affinity is 1 for cpu 0
cpuid: 0
affinity is 1 for cpu 0
cpuid: 0
affinity is 1 for cpu 0
cpuid: 1
affinity is 1 for cpu 0
cpuid: 1
affinity is 1 for cpu 0
cpuid: 1
affinity is 1 for cpu 0
cpuid: 1
affinity is 1 for cpu 0
cpuid: 1
affinity is 1 for cpu 0
cpuid: 1
affinity is 1 for cpu 0
cpuid: 5
affinity is 1 for cpu 0
cpuid: 5

Link:
https://github.com/dotnet/coreclr/blob/v2.2.0/src/vm/win32threadpool.h
https://github.com/dotnet/coreclr/blob/v3.0.0/src/vm/win32threadpool.h

The fix should be replace processorNumber = GetCurrentProcessorNumber() to processorNumber = GetCurrentProcessorNumber() % NumberOfProcessors in win32threadpool.h.

Confirmed the fix works on my environment, should I create a pull request?

303248153 on 26 Sep 2019

👍8

Confirmed the fix works on my environment, should I create a pull request?

Yes, that would be great! Thank you a lot for tracking this down.

cc @janvorli Regression introduced by dotnet/coreclr#23824

jkotas on 26 Sep 2019

Until this fix lands in a release, a workaround can be performed by using LD_PRELOAD to override the sched_getcpu library function. From my comment on https://github.com/dotnet/coreclr/issues/27955#issuecomment-559853972:

As a workaround, I defined a sched_getcpu method that always returns 0 (which is fine given my VPS only has one vCPU):

// coreclr-27955-workaround.c
int sched_getcpu(void) {
    return 0;
}

Compiled it:

gcc -shared -fPIC coreclr-27955-workaround.c -o libcoreclr-27955-workaround.so
sudo cp libcoreclr-27955-workaround.so /usr/local/lib

Then ran my app with the LD_PRELOAD environment variable set:

LD_PRELOAD=/usr/local/lib/libcoreclr-27955-workaround.so ASPNETCORE_ENVIRONMENT=Production ASPNETCORE_URLS=http://*:54561 ./Foo

It worked!

Daniel15 on 29 Nov 2019

👍4

It worked!

@Daniel15, thanks, it definitely does! it also helped me run dotnet core 3.1 hwapp on FreeBSD using Linux®️ Binary Compatibility emulation: https://cirrus-ci.com/build/6435715873505280. Note that we can skip the cp step and still use absolute path to wherever the workaround .so is.

cc @wfurt, the self-contained repro is here, independent of qemu, libvirt and other solutions we were exploring. the "catch" is that all the build-time dependencies .so files (lttng-ust, krb5 etc.) for running linux-x64 dotnet(1) in compatibility mode (to crossgen FreeBSD assemblies) must be compiled for Linux (e.g. we can download and extract debpkgs from web using curl). needs more work to finalize the reproducible build environment.

am11 on 7 Feb 2020

@am11 I'm glad you found the workaround useful :)

For what it's worth, for https://dnstools.ws/ I'm deploying this workaround to all servers as part of an Ansible playbook:

- name: Temporary hack for https://github.com/dotnet/coreclr/issues/27955
  get_url:
    url: https://d.ls/dotnet/bugs/libcoreclr-27955-workaround.so
    dest: /usr/local/lib/libcoreclr-27955-workaround.so
    mode: "0755"
    checksum: sha256:be021161c98f69367745cd7d821b8175b6466eb5fa5921ecee3dcb6f9ff8f150

https://github.com/Daniel15/dnstools/blob/ab2fbd4202530722e6587f3813a0deb5f228e723/ansible/roles/dnstools-worker/tasks/main.yml#L63-L70

Then in my systemd file I have:

# Workaround for https://github.com/dotnet/coreclr/issues/27955
Environment=LD_PRELOAD=/usr/local/lib/libcoreclr-27955-workaround.so

https://github.com/Daniel15/dnstools/blob/master/src/DnsTools.Worker/deployment/dnstools-worker.service#L15

Daniel15 on 7 Feb 2020

Fixed in 3.1.2

jkotas on 1 Mar 2020

🎉1

The helloworld app in Linux chroot of FreeBSD still failed when invoking the compiler (at dotnet exec /path/csc.dll..., during the build). Tested with 3.1.2, without the workaround. Here is the truss output: https://api.cirrus-ci.com/v1/task/6667788790005760/logs/emulate%20dotnet.log. With the sched_getcpu workaround, it continues to work.

am11 on 2 Mar 2020

FreeBSD hang is likely a different underlying problem. Could you please open a new issue on it?

jkotas on 2 Mar 2020

👍1

Was this page helpful?

0 / 5 - 0 ratings