Runtime: DoStackSnapshot after ICorProfilerInfo10::SuspendRuntime E_FAIL(-es) almost always for simple CPU consuming app (w/o switches)

[k15tfu]

Hi!

After discussing some issues with async stack walking, I decided to try ICorProfilerInfo10::SuspendRuntime() followed by DoStackSnapshot(). It works good on all examples I have tested (Yay!), but I found the following:

DoStackSnapshot() returns E_FAIL for 90% samples if the program calculates something in a loop (prime numbers, for ex.) and doesn't use anything else:

using System;
using System.Threading;

namespace calculate_prime
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Current ManagedThreadId: {0}", Thread.CurrentThread.ManagedThreadId);
            Console.WriteLine(getNthPrime(30000));
        }

        private static uint getNthPrime(uint n)
        {
            for (uint i = 0; ; ++i)
                if (isPrime(i))
                    if (n-- == 0)
                        return i;
        }

        private static bool isPrime(uint n)
        {
            for (uint i = 2; i < n / 2; ++i)
                if (n % i == 0)
                    return false;
            return true;
        }
    }
}

In this case, DoStackSnapshot() got called 1694 times (counting the main thread only), but failed in 1527 cases with E_FAIL.

I tried to debug it a bit, and it seems for me that in (almost?) all of these cases there was a RedirectedThreadFrame on the top, and StackFrameIterator::Init() (here https://github.com/dotnet/coreclr/blob/aed84886b8d777168f53fae4da15c7af5f7572da/src/vm/stackwalk.cpp#L956) failed to initialize, because StackFrameIterator::NextRaw() failed to step out of it there: https://github.com/dotnet/coreclr/blob/aed84886b8d777168f53fae4da15c7af5f7572da/src/vm/stackwalk.cpp#L2666

clrstack at that moment:

(lldb) clrstack
OS Thread Id: 0x522e (1)
        Child SP               IP Call Site
00007FFE1D00AD80 00007f71a70699f3 [RedirectedThreadFrame: 00007ffe1d00ad80]
00007FFE1D00C550 00007F712BEE99C1 calculate_prime.Program.isPrime(UInt32)
00007FFE1D00C580 00007F712BEE9900 calculate_prime.Program.getNthPrime(UInt32) [/home/user/p/test/hello_csharp/calculate_prime/Program.cs @ 17]
00007FFE1D00C5B0 00007F712BEC7FCE calculate_prime.Program.Main(System.String[]) [/home/user/p/test/hello_csharp/calculate_prime/Program.cs @ 11]
00007FFE1D00CA70 00007f71a5089cc3 [GCFrame: 00007ffe1d00ca70]
00007FFE1D00D318 00007f71a5089cc3 [GCFrame: 00007ffe1d00d318]
(lldb) bt
* thread dotnet/coreclr#1, name = 'dotnet'
  * frame #0: 0x00007f71a70699f3 libpthread.so.0`__pthread_cond_wait at futex-internal.h:88
    frame dotnet/coreclr#1: 0x00007f71a70699d8 libpthread.so.0`__pthread_cond_wait at pthread_cond_wait.c:502
    frame dotnet/coreclr#2: 0x00007f71a70698f8 libpthread.so.0`__pthread_cond_wait(cond=0x0000000001524228, mutex=0x0000000001524200) at pthread_cond_wait.c:655
    frame dotnet/coreclr#3: 0x00007f71a55d0c75 libcoreclr.so`CorUnix::CPalSynchronizationManager::ThreadNativeWait(ptnwdNativeWaitData=0x0000000001524200, dwTimeout=4294967295, ptwrWakeupReason=0x00007ffe1d00a4b4, pdwSignaledObject=0x00007ffe1d00a4b0) at synchmanager.cpp:479:28
    frame dotnet/coreclr#4: 0x00007f71a55cff4e libcoreclr.so`CorUnix::CPalSynchronizationManager::BlockThread(this=0x0000000001533d00, pthrCurrent=0x0000000001523fc0, dwTimeout=4294967295, fAlertable=false, fIsSleep=false, ptwrWakeupReason=0x00007ffe1d00a7b0, pdwSignaledObject=0x00007ffe1d00a7dc) at synchmanager.cpp:302:22
    frame dotnet/coreclr#5: 0x00007f71a55e20e7 libcoreclr.so`CorUnix::InternalWaitForMultipleObjectsEx(pThread=0x0000000001523fc0, nCount=1, lpHandles=0x00007ffe1d00a988, bWaitAll=NO, dwMilliseconds=4294967295, bAlertable=NO, bPrioritize=NO) at wait.cpp:640:45
    frame dotnet/coreclr#6: 0x00007f71a55e2c94 libcoreclr.so`::WaitForSingleObjectEx(hHandle=0x0000000000000098, dwMilliseconds=4294967295, bAlertable=NO) at wait.cpp:139:13
    frame dotnet/coreclr#7: 0x00007f71a501b940 libcoreclr.so`CLREventWaitHelper2(handle=0x0000000000000098, dwMilliseconds=4294967295, alertable=NO) at synch.cpp:377:12
    frame dotnet/coreclr#8: 0x00007f71a501b7ea libcoreclr.so`CLREventWaitHelper(this=0x00007ffe1d00aa80, pParam=0x00007ffe1d00aa90)::$_1::operator()(CLREventWaitHelper(void*, unsigned int, int)::Param*) const at synch.cpp:402:26
    frame dotnet/coreclr#9: 0x00007f71a501a06d libcoreclr.so`CLREventWaitHelper(handle=0x0000000000000098, dwMilliseconds=4294967295, alertable=NO) at synch.cpp:404:5
    frame dotnet/coreclr#10: 0x00007f71a5019fbe libcoreclr.so`CLREventBase::WaitEx(this=0x0000000001585130, dwMilliseconds=4294967295, mode=WaitMode_None, syncState=0x0000000000000000) at synch.cpp:471:20
    frame dotnet/coreclr#11: 0x00007f71a5019dff libcoreclr.so`CLREventBase::Wait(this=0x0000000001585130, dwMilliseconds=4294967295, alertable=NO, syncState=0x0000000000000000) at synch.cpp:417:12
    frame dotnet/coreclr#12: 0x00007f71a51b3d75 libcoreclr.so`GCEvent::Impl::Wait(this=0x0000000001585130, timeout=4294967295, alertable=false) at gcenv.os.cpp:1215:24
    frame dotnet/coreclr#13: 0x00007f71a51b3135 libcoreclr.so`GCEvent::Wait(this=0x00000000014d6610, timeout=4294967295, alertable=false) at gcenv.os.cpp:1293:20
    frame dotnet/coreclr#14: 0x00007f71a52dd310 libcoreclr.so`WKS::GCHeap::WaitUntilGCComplete(this=0x00000000014e1120, bConsiderGCStart=false) at gcee.cpp:312:40
    frame dotnet/coreclr#15: 0x00007f71a5021327 libcoreclr.so`Thread::RareDisablePreemptiveGC(this=0x0000000001589130) at threadsuspend.cpp:2576:60
    frame dotnet/coreclr#16: 0x00007f71a4cb24fe libcoreclr.so`Thread::DisablePreemptiveGC(this=0x0000000001589130) at threads.h:2036:13
    frame dotnet/coreclr#17: 0x00007f71a5022869 libcoreclr.so`Thread::PulseGCMode(this=0x0000000001589130) at threadsuspend.cpp:2961:9
    frame dotnet/coreclr#18: 0x00007f71a5027788 libcoreclr.so`HandleGCSuspensionForInterruptedThread(interruptedContext=0x00007ffe1d00bad0) at threadsuspend.cpp:6714:18
    frame dotnet/coreclr#19: 0x00007f71a551dcb3 libcoreclr.so`inject_activation_handler(code=34, siginfo=0x00007ffe1d00c0f0, context=0x00007ffe1d00bfc0) at signal.cpp:650:13
    frame dotnet/coreclr#20: 0x00007f71a706e890 libpthread.so.0`___lldb_unnamed_symbol1$$libpthread.so.0 + 1
    frame dotnet/coreclr#21: 0x00007f712bee99c1
    frame dotnet/coreclr#22: 0x00007f712bee9900
    frame dotnet/coreclr#23: 0x00007f712bec7fce
    frame dotnet/coreclr#24: 0x00007f71a5089cc3 libcoreclr.so`CallDescrWorkerInternal at unixasmmacrosamd64.inc:866
    frame dotnet/runtime#3858: 0x00007f71a4e78fb2 libcoreclr.so`CallDescrWorkerWithHandler(pCallDescrData=0x00007ffe1d00c938, fCriticalCall=NO) at callhelpers.cpp:70:5
    frame dotnet/runtime#3859: 0x00007f71a4e7a4e8 libcoreclr.so`MethodDescCallSite::CallTargetWorker(this=0x00007ffe1d00caa0, pArguments=0x00007ffe1d00ca30, pReturnValue=0x0000000000000000, cbReturnValue=0) at callhelpers.cpp:604:9
    frame dotnet/coreclr#27: 0x00007f71a4cbeb83 libcoreclr.so`MethodDescCallSite::Call(this=0x00007ffe1d00caa0, pArguments=0x00007ffe1d00ca30) at callhelpers.h:468:9
    frame dotnet/runtime#3860: 0x00007f71a50b675e libcoreclr.so`RunMain(this=0x00007ffe1d00cbf8, pParam=0x00007ffe1d00ccf0)::$_1::operator()(RunMain(MethodDesc*, short, int*, REF<PtrArray>*)::Param*) const::'lambda'(RunMain(MethodDesc*, short, int*, REF<PtrArray>*)::Param*)::operator()(RunMain(MethodDesc*, short, int*, REF<PtrArray>*)::Param*) const at assembly.cpp:1556:25
    frame dotnet/runtime#3861: 0x00007f71a50b3b19 libcoreclr.so`RunMain(this=0x00007ffe1d00cce0, __EXparam=0x00007ffe1d00ccf0)::$_1::operator()(RunMain(MethodDesc*, short, int*, REF<PtrArray>*)::Param*) const at assembly.cpp:1571:5
    frame dotnet/runtime#3862: 0x00007f71a50b392c libcoreclr.so`RunMain(pFD=0x00007f712c1197d8, numSkipArgs=1, piRetVal=0x00007ffe1d00cdec, stringArgs=0x00007ffe1d00d2d8) at assembly.cpp:1571:5
    frame dotnet/runtime#3863: 0x00007f71a50b3dc2 libcoreclr.so`Assembly::ExecuteMainMethod(this=0x00000000016e2100, stringArgs=0x00007ffe1d00d2d8, waitForOtherThreads=YES) at assembly.cpp:1681:18
    frame dotnet/coreclr#32: 0x00007f71a4cb8095 libcoreclr.so`CorHost2::ExecuteAssembly(this=0x00000000014adca0, dwAppDomainId=1, pwzAssemblyPath=u"/home/user/p/test/hello_csharp/calculate_prime/bin/Debug/netcoreapp3.0/calculate_prime.dll", argc=0, argv=0x0000000000000000, pReturnValue=0x00007ffe1d00d490) at corhost.cpp:460:39
    frame dotnet/runtime#3864: 0x00007f71a4c4cccf libcoreclr.so`::coreclr_execute_assembly(hostHandle=0x00000000014adca0, domainId=1, argc=0, argv=0x0000000000000000, managedAssemblyPath="/home/user/p/test/hello_csharp/calculate_prime/bin/Debug/netcoreapp3.0/calculate_prime.dll", exitCode=0x00007ffe1d00d490) at unixinterface.cpp:412:24
    frame dotnet/runtime#3865: 0x00007f71a5c931c3 libhostpolicy.so`___lldb_unnamed_symbol111$$libhostpolicy.so + 755
    frame dotnet/coreclr#35: 0x00007f71a5c93a1c libhostpolicy.so`corehost_main + 204
    frame dotnet/coreclr#36: 0x00007f71a5ef7b10 libhostfxr.so`___lldb_unnamed_symbol383$$libhostfxr.so + 1536
    frame dotnet/runtime#3866: 0x00007f71a5ef66a3 libhostfxr.so`___lldb_unnamed_symbol381$$libhostfxr.so + 691
    frame dotnet/coreclr#38: 0x00007f71a5ef16f3 libhostfxr.so`hostfxr_main_startupinfo + 147
    frame dotnet/runtime#3867: 0x000000000040c441 dotnet`___lldb_unnamed_symbol106$$dotnet + 785
    frame dotnet/runtime#3868: 0x000000000040c9fe dotnet`___lldb_unnamed_symbol107$$dotnet + 142
    frame dotnet/coreclr#41: 0x00007f71a6149b97 libc.so.6`__libc_start_main(main=(dotnet`___lldb_unnamed_symbol107$$dotnet), argc=2, argv=0x00007ffe1d00d9d8, init=<unavailable>, fini=<unavailable>, rtld_fini=<unavailable>, stack_end=0x00007ffe1d00d9c8) at libc-start.c:310
    frame dotnet/coreclr#42: 0x0000000000402f40 dotnet`_start + 41

my debug output:

StackFrameIterator::Init m_crawl.pFrame = 00007FFE1D00AD80
StackFrameIterator::ProcessCurrentFrame m_crawl.pFrame 00007FFE1D00AD80 ...
StackFrameIterator::Filter m_crawl.pFrame 00007FFE1D00AD80 ...
StackFrameIterator::Filter m_crawl.pFrame 00007FFE1D00AD80 frameState SFITER_INITIAL_NATIVE_CONTEXT <-- first iteration
StackFrameIterator::Filter m_crawl.pFrame 00007FFE1D00AD80 fStop 0 NextRaw() = ...
StackFrameIterator::ProcessCurrentFrame m_crawl.pFrame 00007FFE1D00AD80 ...
StackFrameIterator::ProcessCurrentFrame m_crawl.pFrame 00007FFE1D00AD80 m_crawl.pFunc = 0000000000000000
StackFrameIterator::Filter m_crawl.pFrame 00007FFE1D00AD80 frameState SFITER_FRAME_FUNCTION <-- second iteration
StackFrameIterator::Filter m_crawl.pFrame 00007FFE1D00AD80 fStop 0 NextRaw() = ...
StackFrameIterator::NextRaw m_crawl.pFrame 00007FFE1D00AD80 uFrameAttribs 8 m_crawl.isFirst = 1 adr = 00007F712BEE99C1 <-- and now it will fail to avoid deadlock
StackFrameIterator::Filter fStop 0 NextRaw() = 2
DoStackSnapshotHelper pThreadToSnapshot=0000000001589130[curr 0]} pctxSeed=0000000000000000 hr=80004005 <-- E_FAIL

[tommcdon]

@davmason

[k15tfu]

@tommcdon @davmason Honestly ICorProfilerInfo2::DoStackSnapshot support on Linux/MacOs was announced few months ago in https://github.com/dotnet/coreclr/issues/15136 and we planed to implement a performance profiler for .NET Core on Linux/MacOs and release it shortly, but unfortunetly we see that sometimes it gets inappropriate results, so we are not sure that it's ready for community-wide use. Could you please (re-)prioritize it to get it fixed in the near future, or maybe in 3.1 this December?

[davmason]

@k15tfu the issue here is that we have some logic to detect if a thread has been hijacked and abort if it has because it could potentially deadlock.

My gut feeling here is that there is no deadlock if the hijack happens because of ICorProfilerInfo10::SuspendRuntime, so it should be safe to relax that check and do the stack walk anyways. I have to test and prove myself right though.

[davmason]

@k15tfu if you have time, can you check that the change in dotnet/coreclr#26985 fixes this issue for you? When I run your sample program locally it works now.

[k15tfu]

@davmason Yes, it looks good now. Thank you.

[davmason]

Fixed in master with dotnet/coreclr#26985 and ported to 3.1 with dotnet/coreclr#27300

[k15tfu]

@davmason One more thing. SuspendRuntime sometimes returns error codes:

• CORPROF_E_RUNTIME_UNINITIALIZED -- probably because I call it at a very early stage. What's the moment when I can start doing this? The first AppDomainCreationStarted or so?
• CORPROF_E_SUSPENSION_IN_PROGRESS -- Does it happen because of GC? If so, is it indended to do DoStackSnapshot() for all threads in this case, or I should just skip this sample and go futher?

[davmason]

@davmason One more thing. SuspendRuntime sometimes returns error codes:

* CORPROF_E_RUNTIME_UNINITIALIZED -- probably because I call it at a very early stage. What's the moment when I can start doing this? The first AppDomainCreationStarted or so?

We return CORPROF_E_RUNTIME_UNINITIALIZED when the runtime isn't initialized yet. The specific check we do is for g_fEEStarted which gets set to true shortly after setting up the default appdomain. If you start sampling at the first appdomain created notification you'll probably get a few CORPROF_E_RUNTIME_UNINITIALIZED before you can start sampling.

There's nothing wrong with calling SuspendRuntime before initialization though, it won't cause any issues. I don't see any reason to try to synchronize it, just call SuspendRuntime and once it stops returning CORPROF_E_RUNTIME_UNINITIALIZED your app is up and running. Just ignore the error before that.

* CORPROF_E_SUSPENSION_IN_PROGRESS -- Does it happen because of GC? If so, is it indended to do DoStackSnapshot() for all threads in this case, or I should just skip this sample and go futher?

That means something else is already attempting to suspend the runtime. It could be the GC, or we have various other things that can cause a runtime suspension, although they are far more rare than GCs. You can either ignore that sample or block in RuntimeSuspendStarted when you get notified about the other suspension to do your sampling. That would effectively hijack the suspension. I don't think there should be any issue with hijacking the suspension that way, other than making the GC/whatever else take longer than it would normally.

[k15tfu]

@davmason FYI, sometimes after successful SuspendRuntime, DoStackSnapshotreturns the following suspicious error codes:

• CORPROF_E_STACKSNAPSHOT_UNSAFE because the thread is in a forbid suspend region (https://github.com/dotnet/coreclr/blob/30583ce8b3816f6855b14f6f855db604db23cfc4/src/vm/proftoeeinterfaceimpl.cpp#L8026). Probably this is fine.
• E_NOTIMPL because the target thread not in pre-emptive mode and thread suspension is not supported on Linux (https://github.com/dotnet/coreclr/blob/30583ce8b3816f6855b14f6f855db604db23cfc4/src/vm/proftoeeinterfaceimpl.cpp#L8108). That's odd.
• E_FAIL because Thread::StackWalkFramesEx fails (i.e. returns SWA_FAILED) at the first iter.IsValid right after the iter.Init(this, pStartFrame, pRD, flags) (https://github.com/dotnet/coreclr/blob/30583ce8b3816f6855b14f6f855db604db23cfc4/src/vm/stackwalk.cpp#L967).

Is it supposed to work this way, or something goes wrong?

[davmason]

@k15tfu Are you seeing these errors after blocking in RuntimeSuspendStarted? I misspoke earlier in this thread, you should block in RuntimeSuspendFinished, not RuntimeSuspendStarted.

The RuntimeSuspendStarted event is fired after we set the runtime to trap returning threads but before we do the logic to hijack methods in tight loops/managed code so you would see the errors you're describing above when a thread is running managed code and is in a tight loop or otherwise needs hijacked.

If you are seeing this in the normal case (i.e. after calling SuspendRuntime) then it will need some investigation.

[k15tfu]

@davmason No, I see it in the normal case after successful SuspendRuntime.

[davmason]

@k15tfu can you get native call stacks for the threads that give you E_FAIL or E_NOTIMPL? If you run under a debugger and then set a breakpoint for when you get E_FAIL it should hopefully let you examine the state of the thread you're trying to walk.

[k15tfu]

@davmason

1. E_FAIL: DoStackSnapshot called for tid=0x7f529c003de0.

(lldb) thread list
Process 32328 stopped
...
  thread dotnet/coreclr#16: tid = 32510, 0x00007f536030aed9 libpthread.so.0`__pthread_cond_timedwait at futex-internal.h:142, name = 'dotnet'
...
(lldb) clrthreads
ThreadCount:      4
UnstartedThread:  0
BackgroundThread: 4
PendingThread:    0
DeadThread:       0
Hosted Runtime:   no
                                                                                                        Lock
 DBG   ID OSID ThreadOBJ           State GC Mode     GC Alloc Context                  Domain           Count Apt Exception
   1    1 7e48 0000000000ADA630    20220 Preemptive  00007F52A8000618:00007F52A8001FD0 0000000000ABE920 0     Ukn
  13    2 7efb 0000000000B04680    21220 Preemptive  0000000000000000:0000000000000000 0000000000ABE920 0     Ukn (Finalizer)
  15    3 7efd 00007F529C000C50  1020220 Preemptive  0000000000000000:0000000000000000 0000000000ABE920 0     Ukn (Threadpool Worker)
  16    4 7efe 00007F529C003DE0  1021220 Preemptive  0000000000000000:0000000000000000 0000000000ABE920 0     Ukn (Threadpool Worker)
(lldb) thread select 16
* thread dotnet/coreclr#16, name = 'dotnet'
    frame #0: 0x00007f536030aed9 libpthread.so.0`__pthread_cond_timedwait at futex-internal.h:142
(lldb) clrstack
OS Thread Id: 0x7efe (16)
        Child SP               IP Call Site
(lldb) bt
* thread dotnet/coreclr#16, name = 'dotnet'
  * frame #0: 0x00007f536030aed9 libpthread.so.0`__pthread_cond_timedwait at futex-internal.h:142
    frame dotnet/coreclr#1: 0x00007f536030aebc libpthread.so.0`__pthread_cond_timedwait at pthread_cond_wait.c:533
    frame dotnet/coreclr#2: 0x00007f536030ae00 libpthread.so.0`__pthread_cond_timedwait(cond=0x00007f529c004f88, mutex=0x00007f529c004f60, abstime=0x00007f52a7527b70) at pthread_cond_wait.c:667
    frame dotnet/coreclr#3: 0x00007f535ec2c7fb libcoreclr.so`CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) + 299
    frame dotnet/coreclr#4: 0x00007f535ec2c414 libcoreclr.so`CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) + 388
    frame dotnet/coreclr#5: 0x00007f535ec30a2f libcoreclr.so`CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* const*, int, unsigned int, int, int) + 1743
    frame dotnet/coreclr#6: 0x00007f535ec30c64 libcoreclr.so`PAL_WaitForSingleObjectPrioritized + 84
    frame dotnet/coreclr#7: 0x00007f535e9cd2c0 libcoreclr.so`CLRLifoSemaphore::WaitForSignal(unsigned int) + 32
    frame dotnet/coreclr#8: 0x00007f535e9cd6d7 libcoreclr.so`CLRLifoSemaphore::Wait(unsigned int, unsigned int, unsigned int) + 615
    frame dotnet/coreclr#9: 0x00007f535e92438e libcoreclr.so`ThreadpoolMgr::WorkerThreadStart(void*) + 910
    frame dotnet/coreclr#10: 0x00007f535ec3699d libcoreclr.so`CorUnix::CPalThread::ThreadEntry(void*) + 349
    frame dotnet/coreclr#11: 0x00007f53603046db libpthread.so.0`start_thread(arg=0x00007f52a7528700) at pthread_create.c:463
    frame dotnet/coreclr#12: 0x00007f535f4ea88f libc.so.6`__GI___clone at clone.S:95

I also have 2 more threads for which DoStackSnapshot() returns E_FAIL. They have the following callstacks:

(lldb) thread select 15
* thread dotnet/coreclr#15, name = 'dotnet'
    frame #0: 0x00007f536030a9f3 libpthread.so.0`__pthread_cond_wait at futex-internal.h:88
(lldb) clrstack
OS Thread Id: 0x7efd (15)
        Child SP               IP Call Site
(lldb) bt
* thread dotnet/coreclr#15, name = 'dotnet'
  * frame #0: 0x00007f536030a9f3 libpthread.so.0`__pthread_cond_wait at futex-internal.h:88
    frame dotnet/coreclr#1: 0x00007f536030a9d8 libpthread.so.0`__pthread_cond_wait at pthread_cond_wait.c:502
    frame dotnet/coreclr#2: 0x00007f536030a8f8 libpthread.so.0`__pthread_cond_wait(cond=0x00007f529c005cc0, mutex=0x00007f529c005c98) at pthread_cond_wait.c:655
    frame dotnet/coreclr#3: 0x00007f535ec363d9 libcoreclr.so`CorUnix::InternalCreateThread(CorUnix::CPalThread*, _SECURITY_ATTRIBUTES*, unsigned int, unsigned int (*)(void*), void*, unsigned int, CorUnix::PalThreadType, unsigned long*, void**) + 729
    frame dotnet/coreclr#4: 0x00007f535ec360b0 libcoreclr.so`CreateThread + 128
    frame dotnet/coreclr#5: 0x00007f535e900591 libcoreclr.so`Thread::CreateUtilityThread(Thread::StackSizeBucket, unsigned int (*)(void*), void*, char16_t const*, unsigned int, unsigned int*) + 81
    frame dotnet/coreclr#6: 0x00007f535e923773 libcoreclr.so`ThreadpoolMgr::EnsureGateThreadRunning() + 115
    frame dotnet/coreclr#7: 0x00007f535ea683d0 libcoreclr.so`UnManagedPerAppDomainTPCount::QueueUnmanagedWorkRequest(unsigned int (*)(void*), void*) + 272
    frame dotnet/coreclr#8: 0x00007f535e9270c0 libcoreclr.so`ThreadpoolMgr::FireTimers() + 416
    frame dotnet/coreclr#9: 0x00007f535e926d01 libcoreclr.so`ThreadpoolMgr::TimerThreadFire() + 145
    frame dotnet/coreclr#10: 0x00007f535e926bd5 libcoreclr.so`ThreadpoolMgr::TimerThreadStart(void*) + 181
    frame dotnet/coreclr#11: 0x00007f535ec3699d libcoreclr.so`CorUnix::CPalThread::ThreadEntry(void*) + 349
    frame dotnet/coreclr#12: 0x00007f53603046db libpthread.so.0`start_thread(arg=0x00007f535c066700) at pthread_create.c:463
    frame dotnet/coreclr#13: 0x00007f535f4ea88f libc.so.6`__GI___clone at clone.S:95
(lldb) thread select 13
* thread dotnet/coreclr#13, name = 'dotnet'
    frame #0: 0x00007f536030aed9 libpthread.so.0`__pthread_cond_timedwait at futex-internal.h:142
(lldb) bt
* thread dotnet/coreclr#13, name = 'dotnet'
  * frame #0: 0x00007f536030aed9 libpthread.so.0`__pthread_cond_timedwait at futex-internal.h:142
    frame dotnet/coreclr#1: 0x00007f536030aebc libpthread.so.0`__pthread_cond_timedwait at pthread_cond_wait.c:533
    frame dotnet/coreclr#2: 0x00007f536030ae00 libpthread.so.0`__pthread_cond_timedwait(cond=0x0000000000b05778, mutex=0x0000000000b05750, abstime=0x00007f52e4ac2a10) at pthread_cond_wait.c:667
    frame dotnet/coreclr#3: 0x00007f535ec2c7fb libcoreclr.so`CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) + 299
    frame dotnet/coreclr#4: 0x00007f535ec2c414 libcoreclr.so`CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) + 388
    frame dotnet/coreclr#5: 0x00007f535ec30a2f libcoreclr.so`CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* const*, int, unsigned int, int, int) + 1743
    frame dotnet/coreclr#6: 0x00007f535ec30db1 libcoreclr.so`WaitForMultipleObjectsEx + 81
    frame dotnet/coreclr#7: 0x00007f535e962d3e libcoreclr.so`FinalizerThread::WaitForFinalizerEvent(CLREvent*) + 142
    frame dotnet/coreclr#8: 0x00007f535e962e52 libcoreclr.so`FinalizerThread::FinalizerThreadWorker(void*) + 98
    frame dotnet/coreclr#9: 0x00007f535e905225 libcoreclr.so`ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 341
    frame dotnet/coreclr#10: 0x00007f535e9058dd libcoreclr.so`ManagedThreadBase::FinalizerBase(void (*)(void*)) + 45
    frame dotnet/coreclr#11: 0x00007f535e96308c libcoreclr.so`FinalizerThread::FinalizerThreadStart(void*) + 188
    frame dotnet/coreclr#12: 0x00007f535ec3699d libcoreclr.so`CorUnix::CPalThread::ThreadEntry(void*) + 349
    frame dotnet/coreclr#13: 0x00007f53603046db libpthread.so.0`start_thread(arg=0x00007f52e4ac3700) at pthread_create.c:463
    frame dotnet/coreclr#14: 0x00007f535f4ea88f libc.so.6`__GI___clone at clone.S:95

1. E_NOTIMPL: DoStackSnapshot called for tid=0xd25630.

(lldb) clrthreads
ThreadCount:      4
UnstartedThread:  0
BackgroundThread: 3
PendingThread:    0
DeadThread:       0
Hosted Runtime:   no
                                                                                                        Lock
 DBG   ID OSID ThreadOBJ           State GC Mode     GC Alloc Context                  Domain           Count Apt Exception
   1    1  256 0000000000D25630    20020 Preemptive  00007F28980272C0:00007F2898027FD0 0000000000D09920 1     Ukn System.IO.InvalidDataException 00007f289801fa88 (nested exceptions)
  13    2  265 0000000000D4F680    21220 Preemptive  0000000000000000:0000000000000000 0000000000D09920 0     Ukn (Finalizer)
  15    3  267 00007F288C000C50  1020220 Preemptive  0000000000000000:0000000000000000 0000000000D09920 0     Ukn (Threadpool Worker)
  16    4  268 00007F288C003DE0  1021220 Preemptive  0000000000000000:0000000000000000 0000000000D09920 0     Ukn (Threadpool Worker)
(lldb) thread select 1
* thread dotnet/coreclr#1, name = 'dotnet', stop reason = signal SIGSTOP
    frame #0: 0x00007f29505779f3 libpthread.so.0`__pthread_cond_wait at futex-internal.h:88
(lldb) bt
* thread dotnet/coreclr#1, name = 'dotnet', stop reason = signal SIGSTOP
  * frame #0: 0x00007f29505779f3 libpthread.so.0`__pthread_cond_wait at futex-internal.h:88
    frame dotnet/coreclr#1: 0x00007f29505779d8 libpthread.so.0`__pthread_cond_wait at pthread_cond_wait.c:502
    frame dotnet/coreclr#2: 0x00007f29505778f8 libpthread.so.0`__pthread_cond_wait(cond=0x0000000000ca4ab8, mutex=0x0000000000ca4a90) at pthread_cond_wait.c:655
    frame dotnet/coreclr#3: 0x00007f294ee99805 libcoreclr.so`CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) + 309
    frame dotnet/coreclr#4: 0x00007f294ee99414 libcoreclr.so`CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) + 388
    frame dotnet/coreclr#5: 0x00007f294ee9da2f libcoreclr.so`CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* const*, int, unsigned int, int, int) + 1743
    frame dotnet/coreclr#6: 0x00007f294ee9dcd9 libcoreclr.so`WaitForSingleObjectEx + 89
    frame dotnet/coreclr#7: 0x00007f294ec39e84 libcoreclr.so`CLREventBase::WaitEx(unsigned int, WaitMode, PendingSync*) + 228
    frame dotnet/coreclr#8: 0x00007f294ec3d22b libcoreclr.so`Thread::RareDisablePreemptiveGC() + 427
    frame dotnet/coreclr#9: 0x00007f294ebb82b1 libcoreclr.so`TheUMEntryPrestubWorker + 81
    frame dotnet/coreclr#10: 0x00007f294ec70a13 libcoreclr.so`TheUMEntryPrestub + 71
    frame dotnet/coreclr#11: 0x00007f2896be33ed System.Globalization.Native.so`InvokeCallbackForDateTimePattern + 205
    frame dotnet/coreclr#12: 0x00007f2896be2e2f System.Globalization.Native.so`GlobalizationNative_EnumCalendarInfo + 319
    frames dotnet/coreclr#13 - dotnet/runtime#3956 do not have symbols
    frame dotnet/runtime#3957: 0x00007f294ec6fb2f libcoreclr.so`CallDescrWorkerInternal + 124
    frame dotnet/coreclr#205: 0x00007f294eb9ff85 libcoreclr.so`CallDescrWorkerWithHandler(CallDescrData*, int) + 117
    frame dotnet/runtime#3958: 0x00007f294eba0a60 libcoreclr.so`MethodDescCallSite::CallTargetWorker(unsigned long const*, unsigned long*, int) + 1264
    frame dotnet/coreclr#207: 0x00007f294ec844c3 libcoreclr.so`RunMain(MethodDesc*, short, int*, PtrArray**) + 819
    frame dotnet/runtime#3959: 0x00007f294ec847a0 libcoreclr.so`Assembly::ExecuteMainMethod(PtrArray**, int) + 272
    frame dotnet/runtime#3960: 0x00007f294eaece51 libcoreclr.so`CorHost2::ExecuteAssembly(unsigned int, char16_t const*, int, char16_t const**, unsigned int*) + 577
    frame dotnet/runtime#3961: 0x00007f294eac5c64 libcoreclr.so`coreclr_execute_assembly + 212
    frame dotnet/coreclr#211: 0x00007f294f1a02f3 libhostpolicy.so`___lldb_unnamed_symbol111$$libhostpolicy.so + 755
    frame dotnet/coreclr#212: 0x00007f294f1a06b4 libhostpolicy.so`___lldb_unnamed_symbol112$$libhostpolicy.so + 68
    frame dotnet/coreclr#213: 0x00007f294f1a0d8c libhostpolicy.so`corehost_main + 204
    frame dotnet/runtime#3962: 0x00007f294f405b10 libhostfxr.so`___lldb_unnamed_symbol383$$libhostfxr.so + 1536
    frame dotnet/coreclr#215: 0x00007f294f4046a3 libhostfxr.so`___lldb_unnamed_symbol381$$libhostfxr.so + 691
    frame dotnet/runtime#3963: 0x00007f294f3ff6f3 libhostfxr.so`hostfxr_main_startupinfo + 147
    frame dotnet/coreclr#217: 0x000000000040c441 dotnet`___lldb_unnamed_symbol106$$dotnet + 785
    frame dotnet/runtime#3964: 0x000000000040c9fe dotnet`___lldb_unnamed_symbol107$$dotnet + 142
    frame dotnet/coreclr#219: 0x00007f294f657b97 libc.so.6`__libc_start_main(main=(dotnet`___lldb_unnamed_symbol107$$dotnet), argc=2, argv=0x00007ffda5100af8, init=<unavailable>, fini=<unavailable>, rtld_fini=<unavailable>, stack_end=0x00007ffda5100ae8) at libc-start.c:310
    frame dotnet/coreclr#220: 0x0000000000402f40 dotnet`_start + 41
(lldb) clrstack
OS Thread Id: 0x256 (1)
        Child SP               IP Call Site
00007FFDA50EA6E8 00007f29505779f3 [InlinedCallFrame: 00007ffda50ea6e8] Interop+Globalization.EnumCalendarInfo(EnumCalendarInfoCallback, System.String, System.Globalization.CalendarId, System.Globalization.CalendarDataType, IntPtr)
00007FFDA50EA6E8 00007f28d58857d2 [InlinedCallFrame: 00007ffda50ea6e8] Interop+Globalization.EnumCalendarInfo(EnumCalendarInfoCallback, System.String, System.Globalization.CalendarId, System.Globalization.CalendarDataType, IntPtr)
00007FFDA50EA6E0 00007F28D58857D2 ILStubClass.IL_STUB_PInvoke(EnumCalendarInfoCallback, System.String, System.Globalization.CalendarId, System.Globalization.CalendarDataType, IntPtr)
00007FFDA50EA780 00007F28D58856AF System.Globalization.CalendarData.EnumCalendarInfo(System.String, System.Globalization.CalendarId, System.Globalization.CalendarDataType, EnumCalendarsData ByRef)
00007FFDA50EA7C0 00007F28D58854D0 System.Globalization.CalendarData.EnumDatePatterns(System.String, System.Globalization.CalendarId, System.Globalization.CalendarDataType, System.String[] ByRef) [/_/src/System.Private.CoreLib/shared/System/Globalization/CalendarData.Unix.cs @ 139]
00007FFDA50EA810 00007F28D58845DD System.Globalization.CalendarData.LoadCalendarDataFromSystem(System.String, System.Globalization.CalendarId) [/_/src/System.Private.CoreLib/shared/System/Globalization/CalendarData.Unix.cs @ 50]
00007FFDA50EA850 00007F28D5884096 System.Globalization.CalendarData..ctor(System.String, System.Globalization.CalendarId, Boolean) [/_/src/System.Private.CoreLib/shared/System/Globalization/CalendarData.cs @ 108]
00007FFDA50EA880 00007F28D5883FFC System.Globalization.CultureData.GetCalendar(System.Globalization.CalendarId) [/_/src/System.Private.CoreLib/shared/System/Globalization/CultureData.cs @ 1929]
00007FFDA50EA8C0 00007F28D5883F10 System.Globalization.CultureData.ShortDates(System.Globalization.CalendarId) [/_/src/System.Private.CoreLib/shared/System/Globalization/CultureData.cs @ 1754]
00007FFDA50EA8E0 00007F28D5883EB7 System.Globalization.CultureData.DateSeparator(System.Globalization.CalendarId) [/_/src/System.Private.CoreLib/shared/System/Globalization/CultureData.cs @ 2177]
00007FFDA50EA900 00007F28D5882B87 System.Globalization.DateTimeFormatInfo.InitializeOverridableProperties(System.Globalization.CultureData, System.Globalization.CalendarId) [/_/src/System.Private.CoreLib/shared/System/Globalization/DateTimeFormatInfo.cs @ 299]
00007FFDA50EA930 00007F28D5882777 System.Globalization.DateTimeFormatInfo..ctor(System.Globalization.CultureData, System.Globalization.Calendar) [/_/src/System.Private.CoreLib/shared/System/Globalization/DateTimeFormatInfo.cs @ 269]00007FFDA50EA960 00007F28D5881A95 System.Globalization.CultureInfo.get_DateTimeFormat() [/_/src/System.Private.CoreLib/shared/System/Globalization/CultureInfo.cs @ 855]
00007FFDA50EA9A0 00007F28D58817F1 System.Globalization.CultureInfo.GetFormat(System.Type) [/_/src/System.Private.CoreLib/shared/System/Globalization/CultureInfo.cs @ 786]
00007FFDA50EA9E0 00007F28D588170E System.Globalization.DateTimeFormatInfo.get_CurrentInfo() [/_/src/System.Private.CoreLib/shared/System/Globalization/DateTimeFormatInfo.cs @ 353]
00007FFDA50EAA20 00007F28D5881659 System.Globalization.DateTimeFormatInfo.GetInstance(System.IFormatProvider)
00007FFDA50EAA80 00007F28D588143E System.Globalization.TimeSpanFormat.TryFormat(System.TimeSpan, System.Span`1<Char>, Int32 ByRef, System.ReadOnlySpan`1<Char>, System.IFormatProvider) [/_/src/System.Private.CoreLib/shared/System/Globalization/TimeSpanFormat.cs @ 94]
00007FFDA50EAB40 00007F28D5880FCA System.TimeSpan.TryFormat(System.Span`1<Char>, Int32 ByRef, System.ReadOnlySpan`1<Char>, System.IFormatProvider) [/_/src/System.Private.CoreLib/shared/System/TimeSpan.cs @ 456]
00007FFDA50EAB90 00007F28D5880BDC System.Text.StringBuilder.AppendFormatHelper(System.IFormatProvider, System.String, System.ParamsArray)
00007FFDA50EAC60 00007F28D587DAC0 System.TimeZoneInfo..ctor(Byte[], System.String, Boolean)
00007FFDA50EAD30 00007F28D587D108 System.TimeZoneInfo.GetTimeZoneFromTzData(Byte[], System.String) [/_/src/System.Private.CoreLib/shared/System/TimeZoneInfo.Unix.cs @ 638]
00007FFDA50EADA0 00007F28D5873E46 System.TimeZoneInfo.GetLocalTimeZoneFromTzFile() [/_/src/System.Private.CoreLib/shared/System/TimeZoneInfo.Unix.cs @ 621]
00007FFDA50EADD0 00007F28D5873DF3 System.TimeZoneInfo.GetLocalTimeZone(CachedData) [/_/src/System.Private.CoreLib/shared/System/TimeZoneInfo.Unix.cs @ 202]
00007FFDA50EADF0 00007F28D5873CF1 System.TimeZoneInfo+CachedData.CreateLocal() [/_/src/System.Private.CoreLib/shared/System/TimeZoneInfo.cs @ 80]
00007FFDA50EAE50 00007F28D5873C60 System.TimeZoneInfo+CachedData.get_Local() [/_/src/System.Private.CoreLib/shared/System/TimeZoneInfo.cs @ 107]
00007FFDA50EAE70 00007F28D587280C System.TimeZoneInfo.get_Local() [/_/src/System.Private.CoreLib/shared/System/TimeZoneInfo.cs @ 894]
00007FFDA50EAE80 00007F28D587279D System.TimeZoneInfo.GetDateTimeNowUtcOffsetFromUtc(System.DateTime, Boolean ByRef) [/_/src/System.Private.CoreLib/shared/System/TimeZoneInfo.Unix.cs @ 733]
00007FFDA50EAEB0 00007F28D5872615 System.DateTime.get_Now() [/_/src/System.Private.CoreLib/shared/System/DateTime.cs @ 1111]
00007FFDA50EAF00 00007F28D587211B recursive_sampling.Program.Touch(System.Object, System.Object, System.Object, System.Object, System.Object) [/home/user/p/test/hello_csharp/recursive_sampling/Program.cs @ 16]
00007FFDA50EAF70 00007F28D58711DE recursive_sampling.Program.InvokeWithStackDepth(UInt32, InvokeDelegate, System.Object, System.Object, System.Object, System.Object, System.Object) [/home/user/p/test/hello_csharp/recursive_sampling/Program.cs @ 70]
00007FFDA50EB010 00007F28D58717E4 recursive_sampling.Program+<>c__DisplayClass7_0.<InvokeWithStackDepth>b__0(System.Object, System.Object, System.Object, System.Object, System.Object)
00007FFDA50EB060 00007F28D5871879 recursive_sampling.Program.InvokeInTry(InvokeDelegate, System.Object, System.Object, System.Object, System.Object, System.Object) [/home/user/p/test/hello_csharp/recursive_sampling/Program.cs @ 38] 00007FFDA50EB0D0 00007F28D58712BF recursive_sampling.Program.InvokeWithStackDepth(UInt32, InvokeDelegate, System.Object, System.Object, System.Object, System.Object, System.Object) [/home/user/p/test/hello_csharp/recursive_sampling/Program.cs @ 72]
00007FFDA50EB170 00007F28D58717E4 recursive_sampling.Program+<>c__DisplayClass7_0.<InvokeWithStackDepth>b__0(System.Object, System.Object, System.Object, System.Object, System.Object)
00007FFDA50EB1C0 00007F28D58715D7 recursive_sampling.Program.InvokeInCatch(InvokeDelegate, System.Object, System.Object, System.Object, System.Object, System.Object) [/home/user/p/test/hello_csharp/recursive_sampling/Program.cs @ 55]
00007FFDA50EC480 00007f294ec5ea9b [HelperMethodFrame: 00007ffda50ec480]
00007FFDA50EC600 00007F28D587158F recursive_sampling.Program.InvokeInCatch(InvokeDelegate, System.Object, System.Object, System.Object, System.Object, System.Object) [/home/user/p/test/hello_csharp/recursive_sampling/Program.cs @ 51]
00007FFDA50EC670 00007F28D58712BF recursive_sampling.Program.InvokeWithStackDepth(UInt32, InvokeDelegate, System.Object, System.Object, System.Object, System.Object, System.Object) [/home/user/p/test/hello_csharp/recursive_sampling/Program.cs @ 72]
00007FFDA50EC710 00007F28D58717E4 recursive_sampling.Program+<>c__DisplayClass7_0.<InvokeWithStackDepth>b__0(System.Object, System.Object, System.Object, System.Object, System.Object)
00007FFDA50EC760 00007F28D5871B8F recursive_sampling.Program+<>c__DisplayClass6_0.<InvokeInLambda>g__Wrapper|0(ActionDelegate)
00007FFDA50EC790 00007F28D5871AA9 recursive_sampling.Program.InvokeInLambda(InvokeDelegate, System.Object, System.Object, System.Object, System.Object, System.Object) [/home/user/p/test/hello_csharp/recursive_sampling/Program.cs @ 63]
00007FFDA50EC7F0 00007F28D58712BF recursive_sampling.Program.InvokeWithStackDepth(UInt32, InvokeDelegate, System.Object, System.Object, System.Object, System.Object, System.Object) [/home/user/p/test/hello_csharp/recursive_sampling/Program.cs @ 72]
00007FFDA50EC890 00007F28D58717E4 recursive_sampling.Program+<>c__DisplayClass7_0.<InvokeWithStackDepth>b__0(System.Object, System.Object, System.Object, System.Object, System.Object)
00007FFDA50EC8E0 00007F28D5871978 recursive_sampling.Program.InvokeDirectly(InvokeDelegate, System.Object, System.Object, System.Object, System.Object, System.Object) [/home/user/p/test/hello_csharp/recursive_sampling/Program.cs @ 30]
00007FFDA50EC930 00007F28D58712BF recursive_sampling.Program.InvokeWithStackDepth(UInt32, InvokeDelegate, System.Object, System.Object, System.Object, System.Object, System.Object) [/home/user/p/test/hello_csharp/recursive_sampling/Program.cs @ 72]
00007FFDA50EC9D0 00007F28D58717E4 recursive_sampling.Program+<>c__DisplayClass7_0.<InvokeWithStackDepth>b__0(System.Object, System.Object, System.Object, System.Object, System.Object)
00007FFDA50ECA20 00007F28D5871879 recursive_sampling.Program.InvokeInTry(InvokeDelegate, System.Object, System.Object, System.Object, System.Object, System.Object) [/home/user/p/test/hello_csharp/recursive_sampling/Program.cs @ 38]
(and so on)
00007FFDA50FCBA0 00007F28D5871879 recursive_sampling.Program.InvokeInTry(InvokeDelegate, System.Object, System.Object, System.Object, System.Object, System.Object) [/home/user/p/test/hello_csharp/recursive_sampling/Program.cs @ 38] 00007FFDA50FCC10 00007F28D58712BF recursive_sampling.Program.InvokeWithStackDepth(UInt32, InvokeDelegate, System.Object, System.Object, System.Object, System.Object, System.Object) [/home/user/p/test/hello_csharp/recursive_sampling/Program.cs @ 72]
00007FFDA50FCCB0 00007F28D58717E4 recursive_sampling.Program+<>c__DisplayClass7_0.<InvokeWithStackDepth>b__0(System.Object, System.Object, System.Object, System.Object, System.Object)
00007FFDA50FCD00 00007F28D58715D7 recursive_sampling.Program.InvokeInCatch(InvokeDelegate, System.Object, System.Object, System.Object, System.Object, System.Object) [/home/user/p/test/hello_csharp/recursive_sampling/Program.cs @ 55]
00007FFDA50FDFC0 00007f294ec5ea9b [HelperMethodFrame: 00007ffda50fdfc0]
00007FFDA50FE140 00007F28D587158F recursive_sampling.Program.InvokeInCatch(InvokeDelegate, System.Object, System.Object, System.Object, System.Object, System.Object) [/home/user/p/test/hello_csharp/recursive_sampling/Program.cs @ 51]
00007FFDA50FE1B0 00007F28D58712BF recursive_sampling.Program.InvokeWithStackDepth(UInt32, InvokeDelegate, System.Object, System.Object, System.Object, System.Object, System.Object) [/home/user/p/test/hello_csharp/recursive_sampling/Program.cs @ 72]
00007FFDA50FE250 00007F28D58717E4 recursive_sampling.Program+<>c__DisplayClass7_0.<InvokeWithStackDepth>b__0(System.Object, System.Object, System.Object, System.Object, System.Object)
00007FFDA50FE2A0 00007F28D5871B8F recursive_sampling.Program+<>c__DisplayClass6_0.<InvokeInLambda>g__Wrapper|0(ActionDelegate)
00007FFDA50FE2D0 00007F28D5871AA9 recursive_sampling.Program.InvokeInLambda(InvokeDelegate, System.Object, System.Object, System.Object, System.Object, System.Object) [/home/user/p/test/hello_csharp/recursive_sampling/Program.cs @ 63]
00007FFDA50FE330 00007F28D58712BF recursive_sampling.Program.InvokeWithStackDepth(UInt32, InvokeDelegate, System.Object, System.Object, System.Object, System.Object, System.Object) [/home/user/p/test/hello_csharp/recursive_sampling/Program.cs @ 72]
00007FFDA50FE3D0 00007F28D58717E4 recursive_sampling.Program+<>c__DisplayClass7_0.<InvokeWithStackDepth>b__0(System.Object, System.Object, System.Object, System.Object, System.Object)
00007FFDA50FE420 00007F28D5871978 recursive_sampling.Program.InvokeDirectly(InvokeDelegate, System.Object, System.Object, System.Object, System.Object, System.Object) [/home/user/p/test/hello_csharp/recursive_sampling/Program.cs @ 30]
00007FFDA50FE470 00007F28D58712BF recursive_sampling.Program.InvokeWithStackDepth(UInt32, InvokeDelegate, System.Object, System.Object, System.Object, System.Object, System.Object) [/home/user/p/test/hello_csharp/recursive_sampling/Program.cs @ 72]
00007FFDA50FE510 00007F28D58717E4 recursive_sampling.Program+<>c__DisplayClass7_0.<InvokeWithStackDepth>b__0(System.Object, System.Object, System.Object, System.Object, System.Object)
00007FFDA50FE560 00007F28D5871879 recursive_sampling.Program.InvokeInTry(InvokeDelegate, System.Object, System.Object, System.Object, System.Object, System.Object) [/home/user/p/test/hello_csharp/recursive_sampling/Program.cs @ 38] 00007FFDA50FE5D0 00007F28D58712BF recursive_sampling.Program.InvokeWithStackDepth(UInt32, InvokeDelegate, System.Object, System.Object, System.Object, System.Object, System.Object) [/home/user/p/test/hello_csharp/recursive_sampling/Program.cs @ 72]
00007FFDA50FE670 00007F28D58717E4 recursive_sampling.Program+<>c__DisplayClass7_0.<InvokeWithStackDepth>b__0(System.Object, System.Object, System.Object, System.Object, System.Object)
00007FFDA50FE6C0 00007F28D58715D7 recursive_sampling.Program.InvokeInCatch(InvokeDelegate, System.Object, System.Object, System.Object, System.Object, System.Object) [/home/user/p/test/hello_csharp/recursive_sampling/Program.cs @ 55]
00007FFDA50FF980 00007f294ec5ea9b [HelperMethodFrame: 00007ffda50ff980]
00007FFDA50FFB00 00007F28D587158F recursive_sampling.Program.InvokeInCatch(InvokeDelegate, System.Object, System.Object, System.Object, System.Object, System.Object) [/home/user/p/test/hello_csharp/recursive_sampling/Program.cs @ 51]
00007FFDA50FFB70 00007F28D58712BF recursive_sampling.Program.InvokeWithStackDepth(UInt32, InvokeDelegate, System.Object, System.Object, System.Object, System.Object, System.Object) [/home/user/p/test/hello_csharp/recursive_sampling/Program.cs @ 72]
00007FFDA50FFC10 00007F28D585B7C5 recursive_sampling.Program.Main(System.String[]) [/home/user/p/test/hello_csharp/recursive_sampling/Program.cs @ 83]
00007FFDA50FFF98 00007f294ec6fb2f [GCFrame: 00007ffda50fff98]
00007FFDA5100480 00007f294ec6fb2f [GCFrame: 00007ffda5100480]

[davmason]

Thanks @k15tfu for the callstacks. I haven't forgotten about this, I've just been busy with some other things lately. I hope to get to this relatively soon

[k15tfu]

@davmason Friendly ping.

[k15tfu]

@davmason @noahfalk btw, can we port this fix to 3.0 branch?

I can prepare & check PR. But what repo I should use (dotnet/runtime or dotnet/coreclr)? What repo will you use to release a bugfix?

[davmason]

@k15tfu it is unlikely to be ported to 3.0 since 3.1 is the LTS release. We expect that most customers will be using 3.1 and there is no reason to stay on 3.0. The only type of fixes that are ported to 3.0 are critical ones

[k15tfu]

@davmason I didn't notice that there are only two weeks left. Okay =)

[davmason]

Hi @k15tfu,

Sorry for taking so long to get to this. I had to build a repro and investigate.

The E_FAILs are due to not having any managed frames on the callstacks. There are certain cases where a managed thread might not have any managed frames on the callstack, for example a threadpool thread waiting with no work to be done, or the finalizer thread when it is not actively running finalizers. If you call DoStackSnapshot it will return E_FAIL because it can't find any managed frames to process. This is why the stacks you paste above in item 1 are returning E_FAIL.

I wasn't able to recreate item 2 you listed above. How often do you see E_NOTIMPL. Based on the code, that means that a thread is in cooperative mode, which should never happen after calling SuspendRuntime.