Runtime: System.Net.Mail.Tests.SmtpClientTest.TestZeroTimeout hangs in CI

Created on 30 Aug 2019  路  35Comments  路  Source: dotnet/runtime

https://dnceng.visualstudio.com/public/_build/results?buildId=332322&view=ms.vss-test-web.build-test-results-tab&runId=9745396&resultId=175895&paneView=attachments

Configuration: netcoreapp-Linux-Release-x64-(Alpine.38.Amd64.Open)ubuntu.1604.amd64.[email protected]/dotnet-buildtools/prereqs:alpine-3.8-helix-09ca40b-20190620184125

  Discovering: System.Net.Mail.Functional.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Mail.Functional.Tests (found 137 test cases)
  Starting:    System.Net.Mail.Functional.Tests (parallel test collections = on, max threads = 2)
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.TestZeroTimeout', Elapsed: 00:02:10
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.TestZeroTimeout', Elapsed: 00:04:10
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.TestZeroTimeout', Elapsed: 00:06:10
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.TestZeroTimeout', Elapsed: 00:08:10
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.TestZeroTimeout', Elapsed: 00:10:10
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.TestZeroTimeout', Elapsed: 00:12:10
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.TestZeroTimeout', Elapsed: 00:14:10

@davidsh,聽@wfurt,聽@scalablecory,聽@eiriktsarpalis,聽@karelz

area-System.Net
Source
picture ViktorHofer

Most helpful comment

It failed again and it produced a coredump:

https://helix.dot.net/api/2019-06-17/jobs/5335d74d-89b3-4543-9c31-2236a792653d/workitems/System.Net.Mail.Functional.Tests/files/core.1000.2870

https://dev.azure.com/dnceng/public/_build/results?buildId=408989&view=ms.vss-test-web.build-test-results-tab&runId=12790968&resultId=176542&paneView=attachments

picture safern on 31 Oct 2019
🎉2 ❤1

All 35 comments

@tmds just added this in https://github.com/dotnet/corefx/pull/37462/files#diff-a1e1c77e4753d51b3aa361a5130872f1R319.

We should disable it.

stephentoub picture stephentoub on 30 Aug 2019

did we collect core file @ViktorHofer?
And yes, disabling this for now seems like best step.
However it may be hard to improver the test if it does not fail locally.

wfurt picture wfurt on 30 Aug 2019

did we collect core file

you mean a crash dump? I don't know if dumps are collected for work items that are hanging by terminating them. Also I don't know if crash dumps are already working for this configuration (alpine on docker).

cc @MattGal who might be able to answer some of these.

ViktorHofer picture ViktorHofer on 30 Aug 2019

It's not a test bug. The test is is working, it's showing an issue that was there, and unfortunately not fixed.
A coredump would be nice indeed.
I'll run the test in a loop and see if I can reproduce the hang.

tmds picture tmds on 30 Aug 2019

I'll run the test in a loop and see if I can reproduce the hang.

I ran the test project 1000 times completely, and 1000 times filtered to TestZeroTimeout. I couldn't reproduce the hang.

If we can get a coredump from CI it would help figure out the issue.

tmds picture tmds on 2 Sep 2019

can we re-enable the test to get a coredump?

tmds picture tmds on 12 Sep 2019

As said, I don't think we collect a dump when tests are hanging.

ViktorHofer picture ViktorHofer on 12 Sep 2019

If I change the test so it calls Environment.FailFast for cases it is now hanging. Will that produce a coredump?

tmds picture tmds on 12 Sep 2019

Yes

ViktorHofer picture ViktorHofer on 12 Sep 2019

It failed again and it produced a coredump:

https://helix.dot.net/api/2019-06-17/jobs/5335d74d-89b3-4543-9c31-2236a792653d/workitems/System.Net.Mail.Functional.Tests/files/core.1000.2870

https://dev.azure.com/dnceng/public/_build/results?buildId=408989&view=ms.vss-test-web.build-test-results-tab&runId=12790968&resultId=176542&paneView=attachments

safern picture safern on 31 Oct 2019
🎉2 ❤1

https://helix.dot.net/api/2019-06-17/jobs/5335d74d-89b3-4543-9c31-2236a792653d/workitems/System.Net.Mail.Functional.Tests/files/core.1000.2870

I tried to use dotnet dump analyze but the clrstack and clrthread commands are hanging. Not sure if it is supposed to work.

I'll have a look at debugging this using libsosplugin.so next week.

tmds picture tmds on 31 Oct 2019

@jkotas @danmosemsft can you help Tom? I tried it myself and didn't get very far with WinDbg.

ViktorHofer picture ViktorHofer on 31 Oct 2019

This is Linux right? Windbg won't work, as far as I know.

danmosemsft picture danmosemsft on 31 Oct 2019

I tried to use dotnet dump analyze but the clrstack and clrthread commands are hanging. Not sure if it is supposed to work.

@mikem8361 could you please take a look at this? dump is at https://helix.dot.net/api/2019-06-17/jobs/5335d74d-89b3-4543-9c31-2236a792653d/workitems/System.Net.Mail.Functional.Tests/files/core.1000.2870

danmosemsft picture danmosemsft on 31 Oct 2019

It seems to work for me of course. The only thing I can think of is that dotnet-dump is trying to download symbols from the ms symbol servers (is on by default). You can disable this with setsymbolserver -disable before you run clrstack.

> clrthreads
ThreadCount:      18
UnstartedThread:  0
BackgroundThread: 9
PendingThread:    0
DeadThread:       8
Hosted Runtime:   no
                                                                                                        Lock
 DBG   ID OSID ThreadOBJ           State GC Mode     GC Alloc Context                  Domain           Count Apt Exception
   3    1  b36 0000000000E1E810  2020020 Preemptive  00007F15984E6E58:00007F15984E7FD0 0000000000E04B30 0     Ukn
  13    2  b3a 0000000000E43360    21220 Preemptive  0000000000000000:0000000000000000 0000000000E04B30 0     Ukn (Finalizer)
   7    3  b3c 00007F158C0009F0  1020220 Preemptive  0000000000000000:0000000000000000 0000000000E04B30 0     Ukn (Threadpool Worker)
  10    4  b3d 00007F158C0019B0  1021220 Preemptive  00007F1598AECB20:00007F1598AEDFD0 0000000000E04B30 0     Ukn (Threadpool Worker)
XXXX    5    0 0000000000ECEE50    31820 Preemptive  0000000000000000:0000000000000000 0000000000E04B30 0     Ukn
XXXX    6    0 00007F1578002270    31820 Preemptive  0000000000000000:0000000000000000 0000000000E04B30 0     Ukn
  11    7  b42 00007F1580000A60  3021220 Preemptive  00007F15984D19A0:00007F15984D1FD0 0000000000E04B30 0     Ukn (Threadpool Worker)
   9    8  b43 00000000010A3AE0  2021220 Preemptive  00007F1598AF2360:00007F1598AF3FD0 0000000000E04B30 0     Ukn
   0    9  b44 0000000001254DA0    21220 Preemptive  00007F1598ADA1D8:00007F1598ADBFD0 0000000000E04B30 0     Ukn System.ExecutionEngineException 00007f1597fff1c8
   2   10  b45 0000000001256400  2021220 Preemptive  00007F159870B7D8:00007F159870BAF8 0000000000E04B30 0     Ukn
XXXX   11    0 00007F157C017440    31820 Preemptive  0000000000000000:0000000000000000 0000000000E04B30 0     Ukn
XXXX   12    0 00007F157C01EEF0    31820 Preemptive  0000000000000000:0000000000000000 0000000000E04B30 0     Ukn
XXXX   13    0 00007F1580002E80  1031820 Preemptive  0000000000000000:0000000000000000 0000000000E04B30 0     Ukn (Threadpool Worker)
   1   14  b52 00007F1580005C50    21220 Preemptive  00007F1598A56DD8:00007F1598A57FD0 0000000000E04B30 0     Ukn
XXXX   15    0 00007F157C01FAD0    31820 Preemptive  0000000000000000:0000000000000000 0000000000E04B30 0     Ukn
XXXX   16    0 00007F157C01C9E0    31820 Preemptive  0000000000000000:0000000000000000 0000000000E04B30 0     Ukn
XXXX   17    0 00007F157C022740    31820 Preemptive  0000000000000000:0000000000000000 0000000000E04B30 0     Ukn
   8   18  b3f 00007F1568000AF0    20220 Preemptive  00007F1598AEE1E8:00007F1598AEFFD0 0000000000E04B30 0     Ukn
>
> clrstack -all
OS Thread Id: 0xb36
        Child SP               IP Call Site
00007FFD6604D080 00007f16388a9a35 [HelperMethodFrame: 00007ffd6604d080] System.Threading.WaitHandle.WaitOneCore(IntPtr, Int32)
00007FFD6604D1D0 00007F15BDA5B75C System.Threading.WaitHandle.WaitOneNoCheck(Int32) [/_/src/System.Private.CoreLib/shared/System/Threading/WaitHandle.cs @ 145]
00007FFD6604D270 00007F15BD7E8680 Xunit.ConsoleClient.ConsoleRunner.ExecuteAssembly(System.Object, Xunit.XunitProjectAssembly, Boolean, Boolean, System.Nullable`1<Boolean>, System.Nullable`1<Int32>, Boolean, Boolean, System.Nullable`1<Xunit.AppDomainSupport>, Boolean, Boolean, Xunit.XunitFilters, Boolean) [/_/src/Microsoft.DotNet.XUnitConsoleRunner/src/ConsoleRunner.cs @ 399]
00007FFD6604D390 00007F15BD7DA362 Xunit.ConsoleClient.ConsoleRunner.RunProject(Xunit.XunitProject, Boolean, System.Nullable`1<Boolean>, System.Nullable`1<Boolean>, System.Nullable`1<Int32>, Boolean, Boolean, System.Nullable`1<Xunit.AppDomainSupport>, Boolean, Boolean, Boolean) [/_/src/Microsoft.DotNet.XUnitConsoleRunner/src/ConsoleRunner.cs @ 294]
00007FFD6604D530 00007F15BD5A014A Xunit.ConsoleClient.ConsoleRunner.EntryPoint(System.String[]) [/_/src/Microsoft.DotNet.XUnitConsoleRunner/src/ConsoleRunner.cs @ 81]
00007FFD6604D6B0 00007F15BD39BBD2 Xunit.ConsoleClient.Program.Main(System.String[]) [/_/src/Microsoft.DotNet.XUnitConsoleRunner/src/Program.cs @ 16]
00007FFD6604DA08 00007f16370a420f [GCFrame: 00007ffd6604da08]
00007FFD6604DEF0 00007f16370a420f [GCFrame: 00007ffd6604def0]
OS Thread Id: 0xb3a
        Child SP               IP Call Site
00007F1634EC7D40 00007f16388a9de2 [DebuggerU2MCatchHandlerFrame: 00007f1634ec7d40]
OS Thread Id: 0xb3c
        Child SP               IP Call Site
OS Thread Id: 0xb3d
        Child SP               IP Call Site
00007F16316E1488 00007f16388acbad [InlinedCallFrame: 00007f16316e1488] Interop+Sys.ReceiveMessage(System.Runtime.InteropServices.SafeHandle, MessageHeader*, System.Net.Sockets.SocketFlags, Int64*)
00007F16316E1488 00007f15bdebb41b [InlinedCallFrame: 00007f16316e1488] Interop+Sys.ReceiveMessage(System.Runtime.InteropServices.SafeHandle, MessageHeader*, System.Net.Sockets.SocketFlags, Int64*)
00007F16316E1470 00007F15BDEBB41B ILStubClass.IL_STUB_PInvoke(System.Runtime.InteropServices.SafeHandle, MessageHeader*, System.Net.Sockets.SocketFlags, Int64*)
00007F16316E1580 00007F15BDEBB1F0 System.Net.Sockets.SocketPal.Receive(System.Net.Sockets.SafeSocketHandle, System.Net.Sockets.SocketFlags, System.Span`1<Byte>, Byte[], Int32 ByRef, System.Net.Sockets.SocketFlags ByRef, Error ByRef)
00007F16316E16E0 00007F15BDEBAE28 System.Net.Sockets.SocketPal.TryCompleteReceiveFrom(System.Net.Sockets.SafeSocketHandle, System.Span`1<Byte>, System.Collections.Generic.IList`1<System.ArraySegment`1<Byte>>, System.Net.Sockets.SocketFlags, Byte[], Int32 ByRef, Int32 ByRef, System.Net.Sockets.SocketFlags ByRef, System.Net.Sockets.SocketError ByRef)
00007F16316E17B0 00007F15BDEBAAC3 System.Net.Sockets.SocketPal.TryCompleteReceiveFrom(System.Net.Sockets.SafeSocketHandle, System.Span`1<Byte>, System.Net.Sockets.SocketFlags, Byte[], Int32 ByRef, Int32 ByRef, System.Net.Sockets.SocketFlags ByRef, System.Net.Sockets.SocketError ByRef)
00007F16316E1820 00007F15BDEB9F0F System.Net.Sockets.SocketAsyncContext.ReceiveFrom(System.Memory`1<Byte>, System.Net.Sockets.SocketFlags ByRef, Byte[], Int32 ByRef, Int32, Int32 ByRef)
00007F16316E1900 00007F15BDEB9D87 System.Net.Sockets.SocketAsyncContext.Receive(System.Memory`1<Byte>, System.Net.Sockets.SocketFlags ByRef, Int32, Int32 ByRef)
00007F16316E1960 00007F15BDEB9AFE System.Net.Sockets.SocketPal.Receive(System.Net.Sockets.SafeSocketHandle, Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, Int32 ByRef)
00007F16316E1A20 00007F15BDEB983A System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError ByRef)
00007F16316E1B40 00007F15BDEB9462 System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags)
00007F16316E1B90 00007F15BDEB9231 System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32)
00007F16316E1CA0 00007F15BDEB8ECF System.Net.DelegatedStream.Read(Byte[], Int32, Int32)
00007F16316E1D00 00007F15BDEB8DE3 System.Net.BufferedReadStream.Read(Byte[], Int32, Int32)
00007F16316E1D50 00007F15BDEB882B System.Net.Mail.SmtpReplyReaderFactory.ReadLines(System.Net.Mail.SmtpReplyReader, Boolean)
00007F16316E1F10 00007F15BDEB85C1 System.Net.Mail.SmtpReplyReaderFactory.ReadLine(System.Net.Mail.SmtpReplyReader)
00007F16316E1F80 00007F15BDEB8533 System.Net.Mail.SmtpReplyReader.ReadLine()
00007F16316E1FC0 00007F15BDEA6D89 System.Net.Mail.SmtpConnection.GetConnection(System.String, Int32)
00007F16316E2200 00007F15BDEA5368 System.Net.Mail.SmtpTransport.GetConnection(System.String, Int32)
00007F16316E2270 00007F15BDEA513C System.Net.Mail.SmtpClient.GetConnection()
00007F16316E2290 00007F15BDD98C8A System.Net.Mail.SmtpClient.Send(System.Net.Mail.MailMessage)
00007F16316E25B0 00007F15BDF027F9 System.Net.Mail.Tests.SmtpClientTest+<>c__DisplayClass27_0.<TestZeroTimeout>b__1()
00007F16316E25D0 00007F15BDC360CE Xunit.Assert.RecordException(System.Action) [C:\Dev\xunit\xunit\src\xunit.assert\Asserts\Record.cs @ 28]
00007F16316E2610 00007F15BDC36032 Xunit.Assert.Throws[[System.__Canon, System.Private.CoreLib]](System.Action) [C:\Dev\xunit\xunit\src\xunit.assert\Asserts\ExceptionAsserts.cs @ 27]
00007F16316E2660 00007F15BDF025CF System.Net.Mail.Tests.SmtpClientTest+<>c.<TestZeroTimeout>b__27_0()
00007F16316E26F0 00007F15BDA5C209 System.Threading.Tasks.Task.InnerInvoke() [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2376]
00007F16316E2730 00007F15BDA5C191 System.Threading.Tasks.Task+<>c.<.cctor>b__274_0(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2355]
00007F16316E2760 00007F15BDECD17F System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs @ 289]
00007F16316E27E0 00007F15BDA5BF05 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2316]
00007F16316E28A0 00007F15BDA5BC40 System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2254]
00007F16316E28D0 00007F15BDECD0DF System.Threading.Tasks.Task.ExecuteFromThreadPool(System.Threading.Thread)
00007F16316E28F0 00007F15BDA95249 System.Threading.ThreadPoolWorkQueue.Dispatch() [/_/src/System.Private.CoreLib/shared/System/Threading/ThreadPool.cs @ 662]
00007F16316E2960 00007F15BDA95059 System.Threading._ThreadPoolWaitCallback.PerformWaitCallback() [/_/src/System.Private.CoreLib/src/System/Threading/ThreadPool.CoreCLR.cs @ 29]
00007F16316E2CF0 00007f16370a420f [DebuggerU2MCatchHandlerFrame: 00007f16316e2cf0]
OS Thread Id: 0xb42
        Child SP               IP Call Site
00007F162F70C5F0 00007f16388a9de2 [HelperMethodFrame: 00007f162f70c5f0] System.Threading.WaitHandle.WaitOneCore(IntPtr, Int32)
00007F162F70C740 00007F15BDA5B75C System.Threading.WaitHandle.WaitOneNoCheck(Int32) [/_/src/System.Private.CoreLib/shared/System/Threading/WaitHandle.cs @ 145]
00007F162F70C7E0 00007F15BDA96CEF System.Threading.WaitHandle.WaitOne(Int32) [/_/src/System.Private.CoreLib/shared/System/Threading/WaitHandle.cs @ 120]
00007F162F70C820 00007F15BDA96C48 Xunit.DelegatingLongRunningTestDetectionSink.WaitForStopEvent(Int32) [C:\Dev\xunit\xunit\src\xunit.runner.utility\Sinks\DelegatingSinks\DelegatingLongRunningTestDetectionSink.cs @ 169]
00007F162F70C850 00007F15BDA96A2B Xunit.DelegatingLongRunningTestDetectionSink.ThreadWorker() [C:\Dev\xunit\xunit\src\xunit.runner.utility\Sinks\DelegatingSinks\DelegatingLongRunningTestDetectionSink.cs @ 153]
00007F162F70C880 00007F15BDA96872 Xunit.Sdk.XunitWorkerThread+<>c.<QueueUserWorkItem>b__4_0(System.Object) [C:\Dev\xunit\xunit\src\common\XunitWorkerThread.cs @ 92]
00007F162F70C8C0 00007F15BDA9592D System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute() [/_/src/System.Private.CoreLib/shared/System/Threading/ThreadPool.cs @ 878]
00007F162F70C8F0 00007F15BDA95266 System.Threading.ThreadPoolWorkQueue.Dispatch() [/_/src/System.Private.CoreLib/shared/System/Threading/ThreadPool.cs @ 676]
00007F162F70C960 00007F15BDA95059 System.Threading._ThreadPoolWaitCallback.PerformWaitCallback() [/_/src/System.Private.CoreLib/src/System/Threading/ThreadPool.CoreCLR.cs @ 29]
00007F162F70CCF0 00007f16370a420f [DebuggerU2MCatchHandlerFrame: 00007f162f70ccf0]
OS Thread Id: 0xb43
        Child SP               IP Call Site
00007F1630716370 00007f16388a9a35 [HelperMethodFrame: 00007f1630716370] System.Threading.WaitHandle.WaitOneCore(IntPtr, Int32)
00007F16307164C0 00007F15BDA5B75C System.Threading.WaitHandle.WaitOneNoCheck(Int32) [/_/src/System.Private.CoreLib/shared/System/Threading/WaitHandle.cs @ 145]
00007F1630716560 00007F15BDA5D0E1 Xunit.Sdk.MessageBus.ReporterWorker() [C:\Dev\xunit\xunit\src\xunit.execution\Sdk\MessageBus.cs @ 81]
00007F1630716580 00007F15BDA5C2D2 Xunit.Sdk.XunitWorkerThread+<>c.<QueueUserWorkItem>b__5_0(System.Object) [C:\Dev\xunit\xunit\src\common\XunitWorkerThread.cs @ 37]
00007F16307165C0 00007F15BDA5C251 System.Threading.Tasks.Task.InnerInvoke() [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2376]
00007F1630716600 00007F15BDA5C191 System.Threading.Tasks.Task+<>c.<.cctor>b__274_0(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2355]
00007F1630716630 00007F15BDA5B925 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs @ 172]
00007F16307166D0 00007F15BDA5BEE5 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2312]
00007F1630716790 00007F15BDA5BC40 System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2254]
00007F16307167C0 00007F15BDA5BB9C System.Threading.Tasks.ThreadPoolTaskScheduler+<>c.<.cctor>b__10_0(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/ThreadPoolTaskScheduler.cs @ 37]
00007F16307167F0 00007F15BDA5BAC1 System.Threading.ThreadHelper.ThreadStart_Context(System.Object) [/_/src/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs @ 50]
00007F1630716840 00007F15BDA5B925 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs @ 172]
00007F16307168E0 00007F15BDA5B54E System.Threading.ThreadHelper.ThreadStart(System.Object) [/_/src/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs @ 83]
00007F1630716C40 00007f16370a420f [GCFrame: 00007f1630716c40]
00007F1630716D10 00007f16370a420f [DebuggerU2MCatchHandlerFrame: 00007f1630716d10]
OS Thread Id: 0xb44
        Child SP               IP Call Site
00007F162FF0BDB8 00007f1637ae3377 [GCFrame: 00007f162ff0bdb8]
00007F162FF0C080 00007f1637ae3377 [HelperMethodFrame_1OBJ: 00007f162ff0c080] System.Environment.FailFast(System.String)
00007F162FF0C1B0 00007F15BDF01C69 System.Net.Mail.Tests.SmtpClientTest.TestZeroTimeout()
00007F162FF0C488 00007f16370a420f [DebuggerU2MCatchHandlerFrame: 00007f162ff0c488]
00007F162FF0C598 00007f16370a420f [HelperMethodFrame_PROTECTOBJ: 00007f162ff0c598] System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean, Boolean)
00007F162FF0C710 00007F15BD7D6697 System.Reflection.MethodBase.Invoke(System.Object, System.Object[]) [/_/src/System.Private.CoreLib/shared/System/Reflection/MethodBase.cs @ 50]
00007F162FF0C740 00007F15BDC08B99 Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib]].CallTestMethod(System.Object) [C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs @ 150]
00007F162FF0C770 00007F15BDC087A9 Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1+<<InvokeTestMethodAsync>b__1>d[[System.__Canon, System.Private.CoreLib]].MoveNext() [C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs @ 257]
00007F162FF0C990 00007F15BDC08136 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1+<<InvokeTestMethodAsync>b__1>d[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<<InvokeTestMethodAsync>b__1>d<System.__Canon> ByRef) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 63]
00007F162FF0CA20 00007F15BDC08055 System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1+<<InvokeTestMethodAsync>b__1>d[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<<InvokeTestMethodAsync>b__1>d<System.__Canon> ByRef)
00007F162FF0CA60 00007F15BDC07FD0 Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1[[System.__Canon, System.Private.CoreLib]].<InvokeTestMethodAsync>b__1()
00007F162FF0CAD0 00007F15BDC07D89 Xunit.Sdk.ExecutionTimer+<AggregateAsync>d__4.MoveNext() [C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\ExecutionTimer.cs @ 48]
00007F162FF0CB40 00007F15BDC07853 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExecutionTimer+<AggregateAsync>d__4, xunit.execution.dotnet]](<AggregateAsync>d__4 ByRef) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 63]
00007F162FF0CBB0 00007F15BDC077AB System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Xunit.Sdk.ExecutionTimer+<AggregateAsync>d__4, xunit.execution.dotnet]](<AggregateAsync>d__4 ByRef)
00007F162FF0CBD0 00007F15BDC0775C Xunit.Sdk.ExecutionTimer.AggregateAsync(System.Func`1<System.Threading.Tasks.Task>)
00007F162FF0CC30 00007F15BDC076C5 Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1[[System.__Canon, System.Private.CoreLib]].<InvokeTestMethodAsync>b__0()
00007F162FF0CC80 00007F15BDC0746D Xunit.Sdk.ExceptionAggregator+<RunAsync>d__9.MoveNext() [C:\Dev\xunit\xunit\src\xunit.core\Sdk\ExceptionAggregator.cs @ 90]
00007F162FF0CD00 00007F15BDC07353 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__9, xunit.core]](<RunAsync>d__9 ByRef) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 63]
00007F162FF0CD70 00007F15BDC072AB System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__9, xunit.core]](<RunAsync>d__9 ByRef)
00007F162FF0CD90 00007F15BDC0725C Xunit.Sdk.ExceptionAggregator.RunAsync(System.Func`1<System.Threading.Tasks.Task>)
00007F162FF0CDF0 00007F15BDC06B76 Xunit.Sdk.TestInvoker`1+<InvokeTestMethodAsync>d__48[[System.__Canon, System.Private.CoreLib]].MoveNext() [C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs @ 241]
00007F162FF0CEF0 00007F15BDC06416 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestInvoker`1+<InvokeTestMethodAsync>d__48[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<InvokeTestMethodAsync>d__48<System.__Canon> ByRef) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 63]
00007F162FF0CF80 00007F15BDC06335 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Decimal, System.Private.CoreLib]].Start[[Xunit.Sdk.TestInvoker`1+<InvokeTestMethodAsync>d__48[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<InvokeTestMethodAsync>d__48<System.__Canon> ByRef)
00007F162FF0CFC0 00007F15BDC062B0 Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib]].InvokeTestMethodAsync(System.Object)
00007F162FF0D040 00007F15BDC060C2 Xunit.Sdk.XunitTestInvoker.InvokeTestMethodAsync(System.Object)
00007F162FF0D090 00007F15BDC04A29 Xunit.Sdk.TestInvoker`1+<<RunAsync>b__47_0>d[[System.__Canon, System.Private.CoreLib]].MoveNext() [C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs @ 206]
00007F162FF0D1F0 00007F15BDC04116 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestInvoker`1+<<RunAsync>b__47_0>d[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<<RunAsync>b__47_0>d<System.__Canon> ByRef) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 63]
00007F162FF0D280 00007F15BDC04035 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Decimal, System.Private.CoreLib]].Start[[Xunit.Sdk.TestInvoker`1+<<RunAsync>b__47_0>d[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<<RunAsync>b__47_0>d<System.__Canon> ByRef)
00007F162FF0D2C0 00007F15BDC03FB0 Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib]].<RunAsync>b__47_0()
00007F162FF0D340 00007F15BDC03D45 Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.Decimal, System.Private.CoreLib]].MoveNext()
00007F162FF0D3E0 00007F15BDC03C13 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.Decimal, System.Private.CoreLib]], xunit.core]](<RunAsync>d__10`1<System.Decimal> ByRef) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 63]
00007F162FF0D450 00007F15BDC03B6B System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Decimal, System.Private.CoreLib]].Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.Decimal, System.Private.CoreLib]], xunit.core]](<RunAsync>d__10`1<System.Decimal> ByRef)
00007F162FF0D470 00007F15BDC03ADC Xunit.Sdk.ExceptionAggregator.RunAsync[[System.Decimal, System.Private.CoreLib]](System.Func`1<System.Threading.Tasks.Task`1<System.Decimal>>)
00007F162FF0D4D0 00007F15BDC03A1E Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib]].RunAsync()
00007F162FF0D500 00007F15BDC0336E Xunit.Sdk.XunitTestRunner.InvokeTestMethodAsync(Xunit.Sdk.ExceptionAggregator)
00007F162FF0D590 00007F15BDC03055 Xunit.Sdk.XunitTestRunner+<InvokeTestAsync>d__4.MoveNext() [C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\XunitTestRunner.cs @ 67]
00007F162FF0D600 00007F15BDC02EB3 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.XunitTestRunner+<InvokeTestAsync>d__4, xunit.execution.dotnet]](<InvokeTestAsync>d__4 ByRef) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 63]
00007F162FF0D670 00007F15BDC02E0F System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.XunitTestRunner+<InvokeTestAsync>d__4, xunit.execution.dotnet]](<InvokeTestAsync>d__4 ByRef)
00007F162FF0D6A0 00007F15BDC02DB0 Xunit.Sdk.XunitTestRunner.InvokeTestAsync(Xunit.Sdk.ExceptionAggregator)
00007F162FF0D710 00007F15BDC02D0A Xunit.Sdk.TestRunner`1+<>c__DisplayClass43_0[[System.__Canon, System.Private.CoreLib]].<RunAsync>b__0() [C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestRunner.cs @ 149]
00007F162FF0D730 00007F15BDC025CE Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.__Canon, System.Private.CoreLib]].MoveNext() [C:\Dev\xunit\xunit\src\xunit.core\Sdk\ExceptionAggregator.cs @ 107]
00007F162FF0D810 00007F15BDC02286 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.__Canon, System.Private.CoreLib]], xunit.core]](<RunAsync>d__10`1<System.__Canon> ByRef) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 63]
00007F162FF0D8A0 00007F15BDC021A5 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.__Canon, System.Private.CoreLib]], xunit.core]](<RunAsync>d__10`1<System.__Canon> ByRef)
00007F162FF0D8E0 00007F15BDC020E8 Xunit.Sdk.ExceptionAggregator.RunAsync[[System.__Canon, System.Private.CoreLib]](System.Func`1<System.Threading.Tasks.Task`1<System.__Canon>>)
00007F162FF0D960 00007F15BDC01882 Xunit.Sdk.TestRunner`1+<RunAsync>d__43[[System.__Canon, System.Private.CoreLib]].MoveNext() [C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestRunner.cs @ 149]
00007F162FF0DB00 00007F15BDC013D6 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestRunner`1+<RunAsync>d__43[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__43<System.__Canon> ByRef) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 63]
00007F162FF0DB90 00007F15BDC012F5 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestRunner`1+<RunAsync>d__43[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__43<System.__Canon> ByRef)
00007F162FF0DBD0 00007F15BDC00E5A Xunit.Sdk.TestRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync()
00007F162FF0DC50 00007F15BDBFF571 Xunit.Sdk.XunitTestCaseRunner.RunTestAsync() [C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\XunitTestCaseRunner.cs @ 139]
00007F162FF0DD00 00007F15BDBFEACC Xunit.Sdk.TestCaseRunner`1+<RunAsync>d__19[[System.__Canon, System.Private.CoreLib]].MoveNext() [C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestCaseRunner.cs @ 82]
00007F162FF0DE30 00007F15BDBFE766 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestCaseRunner`1+<RunAsync>d__19[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__19<System.__Canon> ByRef) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 63]
00007F162FF0DEC0 00007F15BDBFE685 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestCaseRunner`1+<RunAsync>d__19[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__19<System.__Canon> ByRef)
00007F162FF0DF00 00007F15BDBFE5EA Xunit.Sdk.TestCaseRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync()
00007F162FF0DF70 00007F15BDBFE193 Xunit.Sdk.XunitTestCase.RunAsync(Xunit.Abstractions.IMessageSink, Xunit.Sdk.IMessageBus, System.Object[], Xunit.Sdk.ExceptionAggregator, System.Threading.CancellationTokenSource)
00007F162FF0DFF0 00007F15BDBFE046 Xunit.Sdk.XunitTestMethodRunner.RunTestCaseAsync(Xunit.Sdk.IXunitTestCase)
00007F162FF0E050 00007F15BDBFDC8A Xunit.Sdk.TestMethodRunner`1+<RunTestCasesAsync>d__32[[System.__Canon, System.Private.CoreLib]].MoveNext() [C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestMethodRunner.cs @ 136]
00007F162FF0E0A0 00007F15BDBFD646 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestMethodRunner`1+<RunTestCasesAsync>d__32[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestCasesAsync>d__32<System.__Canon> ByRef) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 63]
00007F162FF0E130 00007F15BDBFD565 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestMethodRunner`1+<RunTestCasesAsync>d__32[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestCasesAsync>d__32<System.__Canon> ByRef)
00007F162FF0E170 00007F15BDBFD4CA Xunit.Sdk.TestMethodRunner`1[[System.__Canon, System.Private.CoreLib]].RunTestCasesAsync()
00007F162FF0E1F0 00007F15BDBFCE2D Xunit.Sdk.TestMethodRunner`1+<RunAsync>d__31[[System.__Canon, System.Private.CoreLib]].MoveNext() [C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestMethodRunner.cs @ 106]
00007F162FF0E310 00007F15BDBFCB96 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestMethodRunner`1+<RunAsync>d__31[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__31<System.__Canon> ByRef) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 63]
00007F162FF0E3A0 00007F15BDBFCAB5 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestMethodRunner`1+<RunAsync>d__31[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__31<System.__Canon> ByRef)
00007F162FF0E3E0 00007F15BDBFCA1A Xunit.Sdk.TestMethodRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync()
00007F162FF0E450 00007F15BDBFC5AE Xunit.Sdk.XunitTestClassRunner.RunTestMethodAsync(Xunit.Abstractions.ITestMethod, Xunit.Abstractions.IReflectionMethodInfo, System.Collections.Generic.IEnumerable`1<Xunit.Sdk.IXunitTestCase>, System.Object[])
00007F162FF0E4F0 00007F15BDBF826C Xunit.Sdk.TestClassRunner`1+<RunTestMethodsAsync>d__38[[System.__Canon, System.Private.CoreLib]].MoveNext() [C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestClassRunner.cs @ 213]
00007F162FF0E560 00007F15BDEFCEA9 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Xunit.Sdk.TestClassRunner`1+<RunTestMethodsAsync>d__38[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]].ExecutionContextCallback(System.Object) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 290]
00007F162FF0E5B0 00007F15BDA5B925 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs @ 172]
00007F162FF0E650 00007F15BDEFC817 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Xunit.Sdk.TestClassRunner`1+<RunTestMethodsAsync>d__38[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]].MoveNext(System.Threading.Thread) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 365]
00007F162FF0E6E0 00007F15BDEFC6D9 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Xunit.Sdk.TestClassRunner`1+<RunTestMethodsAsync>d__38[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]].MoveNext()
00007F162FF0E700 00007F15BDD42B7E System.Threading.Tasks.AwaitTaskContinuation+<>c.<.cctor>b__17_0(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 661]
00007F162FF0E730 00007F15BDD42A64 System.Threading.Tasks.AwaitTaskContinuation.RunCallback(System.Threading.ContextCallback, System.Object, System.Threading.Tasks.Task ByRef) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 687]
00007F162FF0E7A0 00007F15BDD42704 System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation.Run(System.Threading.Tasks.Task, Boolean) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 401]
00007F162FF0E830 00007F15BDC30EC5 System.Threading.Tasks.Task.RunContinuations(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 3250]
00007F162FF0E8D0 00007F15BDA6BBEC System.Threading.Tasks.Task.FinishContinuations() [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 3212]
00007F162FF0E8F0 00007F15BDC32675 System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib]].TrySetResult(System.__Canon) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Future.cs @ 399]
00007F162FF0E920 00007F15BDEFB0FA System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.__Canon>, System.__Canon) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 443]
00007F162FF0E950 00007F15BDC0FE48 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetResult(System.__Canon) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 426]
00007F162FF0E980 00007F15BDBFD083 Xunit.Sdk.TestMethodRunner`1+<RunAsync>d__31[[System.__Canon, System.Private.CoreLib]].MoveNext()
00007F162FF0EAA0 00007F15BDEFC699 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Xunit.Sdk.TestMethodRunner`1+<RunAsync>d__31[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]].ExecutionContextCallback(System.Object) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 290]
00007F162FF0EAF0 00007F15BDA5B925 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs @ 172]
00007F162FF0EB90 00007F15BDEFC407 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Xunit.Sdk.TestMethodRunner`1+<RunAsync>d__31[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]].MoveNext(System.Threading.Thread) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 365]
00007F162FF0EC20 00007F15BDEFC2C9 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Xunit.Sdk.TestMethodRunner`1+<RunAsync>d__31[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]].MoveNext()
00007F162FF0EC40 00007F15BDD42B7E System.Threading.Tasks.AwaitTaskContinuation+<>c.<.cctor>b__17_0(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 661]
00007F162FF0EC70 00007F15BDD42A64 System.Threading.Tasks.AwaitTaskContinuation.RunCallback(System.Threading.ContextCallback, System.Object, System.Threading.Tasks.Task ByRef) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 687]
00007F162FF0ECE0 00007F15BDD42704 System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation.Run(System.Threading.Tasks.Task, Boolean) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 401]
00007F162FF0ED70 00007F15BDC30EC5 System.Threading.Tasks.Task.RunContinuations(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 3250]
00007F162FF0EE10 00007F15BDA6BBEC System.Threading.Tasks.Task.FinishContinuations() [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 3212]
00007F162FF0EE30 00007F15BDC32675 System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib]].TrySetResult(System.__Canon) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Future.cs @ 399]
00007F162FF0EE60 00007F15BDBFDEA0 Xunit.Sdk.TestMethodRunner`1+<RunTestCasesAsync>d__32[[System.__Canon, System.Private.CoreLib]].MoveNext()
00007F162FF0EEB0 00007F15BDEFC289 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Xunit.Sdk.TestMethodRunner`1+<RunTestCasesAsync>d__32[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]].ExecutionContextCallback(System.Object) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 290]
00007F162FF0EF00 00007F15BDA5B925 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs @ 172]
00007F162FF0EFA0 00007F15BDEFBFF7 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Xunit.Sdk.TestMethodRunner`1+<RunTestCasesAsync>d__32[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]].MoveNext(System.Threading.Thread) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 365]
00007F162FF0F030 00007F15BDEFBEB9 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Xunit.Sdk.TestMethodRunner`1+<RunTestCasesAsync>d__32[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]].MoveNext()
00007F162FF0F050 00007F15BDD42B7E System.Threading.Tasks.AwaitTaskContinuation+<>c.<.cctor>b__17_0(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 661]
00007F162FF0F080 00007F15BDD42A64 System.Threading.Tasks.AwaitTaskContinuation.RunCallback(System.Threading.ContextCallback, System.Object, System.Threading.Tasks.Task ByRef) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 687]
00007F162FF0F0F0 00007F15BDD42704 System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation.Run(System.Threading.Tasks.Task, Boolean) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 401]
00007F162FF0F180 00007F15BDC30EC5 System.Threading.Tasks.Task.RunContinuations(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 3250]
00007F162FF0F220 00007F15BDA6BBEC System.Threading.Tasks.Task.FinishContinuations() [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 3212]
00007F162FF0F240 00007F15BDC32675 System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib]].TrySetResult(System.__Canon) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Future.cs @ 399]
00007F162FF0F270 00007F15BDEFB0FA System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.__Canon>, System.__Canon) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 443]
00007F162FF0F2A0 00007F15BDC0FE48 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetResult(System.__Canon) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 426]
00007F162FF0F2D0 00007F15BDBFEE1A Xunit.Sdk.TestCaseRunner`1+<RunAsync>d__19[[System.__Canon, System.Private.CoreLib]].MoveNext()
00007F162FF0F400 00007F15BDEFBE79 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Xunit.Sdk.TestCaseRunner`1+<RunAsync>d__19[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]].ExecutionContextCallback(System.Object) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 290]
00007F162FF0F450 00007F15BDA5B925 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs @ 172]
00007F162FF0F4F0 00007F15BDEFBBE7 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Xunit.Sdk.TestCaseRunner`1+<RunAsync>d__19[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]].MoveNext(System.Threading.Thread) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 365]
00007F162FF0F580 00007F15BDEFBAA9 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Xunit.Sdk.TestCaseRunner`1+<RunAsync>d__19[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]].MoveNext()
00007F162FF0F5A0 00007F15BDD42B7E System.Threading.Tasks.AwaitTaskContinuation+<>c.<.cctor>b__17_0(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 661]
00007F162FF0F5D0 00007F15BDD42A64 System.Threading.Tasks.AwaitTaskContinuation.RunCallback(System.Threading.ContextCallback, System.Object, System.Threading.Tasks.Task ByRef) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 687]
00007F162FF0F640 00007F15BDD42704 System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation.Run(System.Threading.Tasks.Task, Boolean) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 401]
00007F162FF0F6D0 00007F15BDC30EC5 System.Threading.Tasks.Task.RunContinuations(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 3250]
00007F162FF0F770 00007F15BDA6BBEC System.Threading.Tasks.Task.FinishContinuations() [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 3212]
00007F162FF0F790 00007F15BDC32675 System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib]].TrySetResult(System.__Canon) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Future.cs @ 399]
00007F162FF0F7C0 00007F15BDEFB0FA System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.__Canon>, System.__Canon) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 443]
00007F162FF0F7F0 00007F15BDC0FE48 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetResult(System.__Canon) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 426]
00007F162FF0F820 00007F15BDC01D00 Xunit.Sdk.TestRunner`1+<RunAsync>d__43[[System.__Canon, System.Private.CoreLib]].MoveNext()
00007F162FF0F9C0 00007F15BDEFB519 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Xunit.Sdk.TestRunner`1+<RunAsync>d__43[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]].ExecutionContextCallback(System.Object) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 290]
00007F162FF0FA10 00007F15BDA5B925 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs @ 172]
00007F162FF0FAB0 00007F15BDEFB287 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Xunit.Sdk.TestRunner`1+<RunAsync>d__43[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]].MoveNext(System.Threading.Thread) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 365]
00007F162FF0FB40 00007F15BDEFB149 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Xunit.Sdk.TestRunner`1+<RunAsync>d__43[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]].MoveNext()
00007F162FF0FB60 00007F15BDD42B7E System.Threading.Tasks.AwaitTaskContinuation+<>c.<.cctor>b__17_0(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 661]
00007F162FF0FB90 00007F15BDD42A64 System.Threading.Tasks.AwaitTaskContinuation.RunCallback(System.Threading.ContextCallback, System.Object, System.Threading.Tasks.Task ByRef) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 687]
00007F162FF0FC00 00007F15BDD42704 System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation.Run(System.Threading.Tasks.Task, Boolean) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 401]
00007F162FF0FC90 00007F15BDC30EC5 System.Threading.Tasks.Task.RunContinuations(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 3250]
00007F162FF0FD30 00007F15BDA6BBEC System.Threading.Tasks.Task.FinishContinuations() [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 3212]
00007F162FF0FD50 00007F15BDC32675 System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib]].TrySetResult(System.__Canon) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Future.cs @ 399]
00007F162FF0FD80 00007F15BDEFB0FA System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.__Canon>, System.__Canon) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 443]
00007F162FF0FDB0 00007F15BDC0FE48 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetResult(System.__Canon) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 426]
00007F162FF0FDE0 00007F15BDC0276A Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.__Canon, System.Private.CoreLib]].MoveNext()
00007F162FF0FEC0 00007F15BDEFB0A9 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.__Canon, System.Private.CoreLib]], xunit.core]].ExecutionContextCallback(System.Object) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 290]
00007F162FF0FF10 00007F15BDA5B925 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs @ 172]
00007F162FF0FFB0 00007F15BDEFAE17 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.__Canon, System.Private.CoreLib]], xunit.core]].MoveNext(System.Threading.Thread) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 328]
00007F162FF10040 00007F15BDEFACD9 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.__Canon, System.Private.CoreLib]], xunit.core]].MoveNext()
00007F162FF10060 00007F15BDD42B7E System.Threading.Tasks.AwaitTaskContinuation+<>c.<.cctor>b__17_0(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 661]
00007F162FF10090 00007F15BDD42A64 System.Threading.Tasks.AwaitTaskContinuation.RunCallback(System.Threading.ContextCallback, System.Object, System.Threading.Tasks.Task ByRef) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 687]
00007F162FF10100 00007F15BDD42704 System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation.Run(System.Threading.Tasks.Task, Boolean) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 401]
00007F162FF10190 00007F15BDC30EC5 System.Threading.Tasks.Task.RunContinuations(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 3250]
00007F162FF10230 00007F15BDA6BBEC System.Threading.Tasks.Task.FinishContinuations() [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 3212]
00007F162FF10250 00007F15BDC32675 System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib]].TrySetResult(System.__Canon) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Future.cs @ 399]
00007F162FF10280 00007F15BDC031DB Xunit.Sdk.XunitTestRunner+<InvokeTestAsync>d__4.MoveNext()
00007F162FF102F0 00007F15BDEFACA2 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Xunit.Sdk.XunitTestRunner+<InvokeTestAsync>d__4, xunit.execution.dotnet]].ExecutionContextCallback(System.Object) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 290]
00007F162FF10330 00007F15BDA5B925 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs @ 172]
00007F162FF103D0 00007F15BDEFAA62 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Xunit.Sdk.XunitTestRunner+<InvokeTestAsync>d__4, xunit.execution.dotnet]].MoveNext(System.Threading.Thread) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 365]
00007F162FF10440 00007F15BDEFA979 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Xunit.Sdk.XunitTestRunner+<InvokeTestAsync>d__4, xunit.execution.dotnet]].MoveNext()
00007F162FF10460 00007F15BDD42B7E System.Threading.Tasks.AwaitTaskContinuation+<>c.<.cctor>b__17_0(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 661]
00007F162FF10490 00007F15BDD42A64 System.Threading.Tasks.AwaitTaskContinuation.RunCallback(System.Threading.ContextCallback, System.Object, System.Threading.Tasks.Task ByRef) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 687]
00007F162FF10500 00007F15BDD42704 System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation.Run(System.Threading.Tasks.Task, Boolean) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 401]
00007F162FF10590 00007F15BDC30EC5 System.Threading.Tasks.Task.RunContinuations(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 3250]
00007F162FF10630 00007F15BDA6BBEC System.Threading.Tasks.Task.FinishContinuations() [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 3212]
00007F162FF10650 00007F15BDEFA2AC System.Threading.Tasks.Task`1[[System.Decimal, System.Private.CoreLib]].TrySetResult(System.Decimal) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Future.cs @ 402]
00007F162FF10690 00007F15BDEFA1DE System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Decimal, System.Private.CoreLib]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.Decimal>, System.Decimal) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 443]
00007F162FF106C0 00007F15BDC122A4 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Decimal, System.Private.CoreLib]].SetResult(System.Decimal) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 426]
00007F162FF106F0 00007F15BDC03E0E Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.Decimal, System.Private.CoreLib]].MoveNext()
00007F162FF10790 00007F15BDEFA93E System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Decimal, System.Private.CoreLib],[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.Decimal, System.Private.CoreLib]], xunit.core]].ExecutionContextCallback(System.Object) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 290]
00007F162FF107B0 00007F15BDA5B925 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs @ 172]
00007F162FF10850 00007F15BDEFA7C3 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Decimal, System.Private.CoreLib],[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.Decimal, System.Private.CoreLib]], xunit.core]].MoveNext(System.Threading.Thread) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 365]
00007F162FF10890 00007F15BDEFA709 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Decimal, System.Private.CoreLib],[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.Decimal, System.Private.CoreLib]], xunit.core]].MoveNext()
00007F162FF108B0 00007F15BDD42B7E System.Threading.Tasks.AwaitTaskContinuation+<>c.<.cctor>b__17_0(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 661]
00007F162FF108E0 00007F15BDD42A64 System.Threading.Tasks.AwaitTaskContinuation.RunCallback(System.Threading.ContextCallback, System.Object, System.Threading.Tasks.Task ByRef) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 687]
00007F162FF10950 00007F15BDD42704 System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation.Run(System.Threading.Tasks.Task, Boolean) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 401]
00007F162FF109E0 00007F15BDC30EC5 System.Threading.Tasks.Task.RunContinuations(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 3250]
00007F162FF10A80 00007F15BDA6BBEC System.Threading.Tasks.Task.FinishContinuations() [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 3212]
00007F162FF10AA0 00007F15BDEFA2AC System.Threading.Tasks.Task`1[[System.Decimal, System.Private.CoreLib]].TrySetResult(System.Decimal) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Future.cs @ 402]
00007F162FF10AE0 00007F15BDEFA1DE System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Decimal, System.Private.CoreLib]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.Decimal>, System.Decimal) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 443]
00007F162FF10B10 00007F15BDC122A4 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Decimal, System.Private.CoreLib]].SetResult(System.Decimal) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 426]
00007F162FF10B40 00007F15BDC04E6E Xunit.Sdk.TestInvoker`1+<<RunAsync>b__47_0>d[[System.__Canon, System.Private.CoreLib]].MoveNext()
00007F162FF10CA0 00007F15BDEFA6C9 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Decimal, System.Private.CoreLib],[Xunit.Sdk.TestInvoker`1+<<RunAsync>b__47_0>d[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]].ExecutionContextCallback(System.Object) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 290]
00007F162FF10CF0 00007F15BDA5B925 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs @ 172]
00007F162FF10D90 00007F15BDEFA437 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Decimal, System.Private.CoreLib],[Xunit.Sdk.TestInvoker`1+<<RunAsync>b__47_0>d[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]].MoveNext(System.Threading.Thread) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 365]
00007F162FF10E20 00007F15BDEFA2F9 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Decimal, System.Private.CoreLib],[Xunit.Sdk.TestInvoker`1+<<RunAsync>b__47_0>d[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]].MoveNext()
00007F162FF10E40 00007F15BDD42B7E System.Threading.Tasks.AwaitTaskContinuation+<>c.<.cctor>b__17_0(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 661]
00007F162FF10E70 00007F15BDD42A64 System.Threading.Tasks.AwaitTaskContinuation.RunCallback(System.Threading.ContextCallback, System.Object, System.Threading.Tasks.Task ByRef) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 687]
00007F162FF10EE0 00007F15BDD42704 System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation.Run(System.Threading.Tasks.Task, Boolean) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 401]
00007F162FF10F70 00007F15BDC30EC5 System.Threading.Tasks.Task.RunContinuations(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 3250]
00007F162FF11010 00007F15BDA6BBEC System.Threading.Tasks.Task.FinishContinuations() [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 3212]
00007F162FF11030 00007F15BDEFA2AC System.Threading.Tasks.Task`1[[System.Decimal, System.Private.CoreLib]].TrySetResult(System.Decimal) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Future.cs @ 402]
00007F162FF11070 00007F15BDEFA1DE System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Decimal, System.Private.CoreLib]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.Decimal>, System.Decimal) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 443]
00007F162FF110A0 00007F15BDC122A4 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Decimal, System.Private.CoreLib]].SetResult(System.Decimal) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 426]
00007F162FF110D0 00007F15BDC06C99 Xunit.Sdk.TestInvoker`1+<InvokeTestMethodAsync>d__48[[System.__Canon, System.Private.CoreLib]].MoveNext()
00007F162FF111D0 00007F15BDEFA189 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Decimal, System.Private.CoreLib],[Xunit.Sdk.TestInvoker`1+<InvokeTestMethodAsync>d__48[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]].ExecutionContextCallback(System.Object) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 290]
00007F162FF11220 00007F15BDA5B925 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs @ 172]
00007F162FF112C0 00007F15BDEF9EF7 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Decimal, System.Private.CoreLib],[Xunit.Sdk.TestInvoker`1+<InvokeTestMethodAsync>d__48[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]].MoveNext(System.Threading.Thread) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 365]
00007F162FF11350 00007F15BDEF9DB9 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Decimal, System.Private.CoreLib],[Xunit.Sdk.TestInvoker`1+<InvokeTestMethodAsync>d__48[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]].MoveNext()
00007F162FF11370 00007F15BDEF940E System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation+<>c.<.cctor>b__8_0(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs @ 360]
00007F162FF113A0 00007F15BDEF652B Xunit.Sdk.AsyncTestSyncContext+<>c__DisplayClass7_0.<Post>b__1(System.Object) [C:\Dev\xunit\xunit\src\xunit.execution\Sdk\AsyncTestSyncContext.cs @ 75]
00007F162FF113F0 00007F15BDBEEB0D Xunit.Sdk.MaxConcurrencySyncContext.RunOnSyncContext(System.Threading.SendOrPostCallback, System.Object) [C:\Dev\xunit\xunit\src\xunit.execution\Sdk\MaxConcurrencySyncContext.cs @ 107]
00007F162FF11430 00007F15BDBEE8D3 Xunit.Sdk.MaxConcurrencySyncContext+<>c__DisplayClass11_0.<WorkerThreadProc>b__0(System.Object)
00007F162FF11470 00007F15BDA5B925 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs @ 172]
00007F162FF11510 00007F15BDBEE852 Xunit.Sdk.ExecutionContextHelper.Run(System.Object, System.Action`1<System.Object>) [C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Utility\ExecutionContextHelper.cs @ 111]
00007F162FF11550 00007F15BDBE8D9A Xunit.Sdk.MaxConcurrencySyncContext.WorkerThreadProc() [C:\Dev\xunit\xunit\src\xunit.execution\Sdk\MaxConcurrencySyncContext.cs @ 89]
00007F162FF11580 00007F15BDA5C2D2 Xunit.Sdk.XunitWorkerThread+<>c.<QueueUserWorkItem>b__5_0(System.Object) [C:\Dev\xunit\xunit\src\common\XunitWorkerThread.cs @ 37]
00007F162FF115C0 00007F15BDA5C251 System.Threading.Tasks.Task.InnerInvoke() [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2376]
00007F162FF11600 00007F15BDA5C191 System.Threading.Tasks.Task+<>c.<.cctor>b__274_0(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2355]
00007F162FF11630 00007F15BDA5B925 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs @ 172]
00007F162FF116D0 00007F15BDA5BEE5 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2312]
00007F162FF11790 00007F15BDA5BC40 System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2254]
00007F162FF117C0 00007F15BDA5BB9C System.Threading.Tasks.ThreadPoolTaskScheduler+<>c.<.cctor>b__10_0(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/ThreadPoolTaskScheduler.cs @ 37]
00007F162FF117F0 00007F15BDA5BAC1 System.Threading.ThreadHelper.ThreadStart_Context(System.Object) [/_/src/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs @ 50]
00007F162FF11840 00007F15BDA5B925 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs @ 172]
00007F162FF118E0 00007F15BDA5B54E System.Threading.ThreadHelper.ThreadStart(System.Object) [/_/src/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs @ 83]
00007F162FF11C40 00007f16370a420f [GCFrame: 00007f162ff11c40]
00007F162FF11D10 00007f16370a420f [DebuggerU2MCatchHandlerFrame: 00007f162ff11d10]
OS Thread Id: 0xb45
        Child SP               IP Call Site
00007F15BD2672F8 00007f16388a9a35 [HelperMethodFrame: 00007f15bd2672f8] System.Threading.WaitHandle.WaitMultipleIgnoringSyncContext(IntPtr*, Int32, Boolean, Int32)
00007F15BD267440 00007F15BDBEA8B4 System.Threading.WaitHandle.WaitMultiple(System.ReadOnlySpan`1<System.Threading.WaitHandle>, Boolean, Int32) [/_/src/System.Private.CoreLib/shared/System/Threading/WaitHandle.cs @ 302]
00007F15BD267500 00007F15BDBE92BA System.Threading.WaitHandle.WaitMultiple(System.Threading.WaitHandle[], Boolean, Int32)
00007F15BD267550 00007F15BDBE8D23 Xunit.Sdk.MaxConcurrencySyncContext.WorkerThreadProc() [C:\Dev\xunit\xunit\src\xunit.execution\Sdk\MaxConcurrencySyncContext.cs @ 85]
00007F15BD267580 00007F15BDA5C2D2 Xunit.Sdk.XunitWorkerThread+<>c.<QueueUserWorkItem>b__5_0(System.Object) [C:\Dev\xunit\xunit\src\common\XunitWorkerThread.cs @ 37]
00007F15BD2675C0 00007F15BDA5C251 System.Threading.Tasks.Task.InnerInvoke() [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2376]
00007F15BD267600 00007F15BDA5C191 System.Threading.Tasks.Task+<>c.<.cctor>b__274_0(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2355]
00007F15BD267630 00007F15BDA5B925 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs @ 172]
00007F15BD2676D0 00007F15BDA5BEE5 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2312]
00007F15BD267790 00007F15BDA5BC40 System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2254]
00007F15BD2677C0 00007F15BDA5BB9C System.Threading.Tasks.ThreadPoolTaskScheduler+<>c.<.cctor>b__10_0(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/ThreadPoolTaskScheduler.cs @ 37]
00007F15BD2677F0 00007F15BDA5BAC1 System.Threading.ThreadHelper.ThreadStart_Context(System.Object) [/_/src/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs @ 50]
00007F15BD267840 00007F15BDA5B925 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs @ 172]
00007F15BD2678E0 00007F15BDA5B54E System.Threading.ThreadHelper.ThreadStart(System.Object) [/_/src/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs @ 83]
00007F15BD267C40 00007f16370a420f [GCFrame: 00007f15bd267c40]
00007F15BD267D10 00007f16370a420f [DebuggerU2MCatchHandlerFrame: 00007f15bd267d10]
OS Thread Id: 0xb52
        Child SP               IP Call Site
00007F15975E8588 00007f1637babea3 [InlinedCallFrame: 00007f15975e8588] Interop+Sys.WaitForSocketEvents(IntPtr, SocketEvent*, Int32*)
00007F15975E8588 00007f15bded69e8 [InlinedCallFrame: 00007f15975e8588] Interop+Sys.WaitForSocketEvents(IntPtr, SocketEvent*, Int32*)
00007F15975E8580 00007F15BDED69E8 ILStubClass.IL_STUB_PInvoke(IntPtr, SocketEvent*, Int32*)
00007F15975E8660 00007F15BDED65A5 System.Net.Sockets.SocketAsyncEngine.EventLoop()
00007F15975E8760 00007F15BDED6513 System.Net.Sockets.SocketAsyncEngine+<>c.<.ctor>b__23_0(System.Object)
00007F15975E8780 00007F15BDA5C251 System.Threading.Tasks.Task.InnerInvoke() [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2376]
00007F15975E87C0 00007F15BDA5BEC2 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2305]
00007F15975E8880 00007F15BDA5BC40 System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2254]
00007F15975E88B0 00007F15BDA5BB9C System.Threading.Tasks.ThreadPoolTaskScheduler+<>c.<.cctor>b__10_0(System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/ThreadPoolTaskScheduler.cs @ 37]
00007F15975E88E0 00007F15BDA5B594 System.Threading.ThreadHelper.ThreadStart(System.Object) [/_/src/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs @ 83]
00007F15975E8C40 00007f16370a420f [GCFrame: 00007f15975e8c40]
00007F15975E8D10 00007f16370a420f [DebuggerU2MCatchHandlerFrame: 00007f15975e8d10]
OS Thread Id: 0xb3f
        Child SP               IP Call Site
>
mikem8361 picture mikem8361 on 31 Oct 2019

@mikem8361 setsymbolserver -disable helps, now I get a different error:

$ dotnet dump analyze core.1000.2870 
Loading core dump: core.1000.2870 ...
Ready to process analysis commands. Type 'help' to list available commands or 'help [command]' to get detailed help on a command.
Type 'quit' or 'exit' to exit the session.
> setsymbolserver -disable                                                                                                                                                                                                                   
> clrstack                                                                                                                                                                                                                                   
Failed to load data access module, 0x80004005
Can not load or initialize libmscordaccore.so. The target runtime may not be initialized.

Is there a way to make this use libmscordaccore.so from my .NET Core 3.0 installation (at /usr/lib64/dotnet/shared/Microsoft.NETCore.App/3.0.0/libmscordaccore.so)?

tmds picture tmds on 5 Nov 2019

You can try adding just that directory to the symbol search path with setsymbolserver -directory /usr/lib64/dotnet/shared/Microsoft.NETCore.App/3.0.0.

mikem8361 picture mikem8361 on 5 Nov 2019

@mikem8361 I still get the same error. Why doesn't this fail for you?

tmds picture tmds on 6 Nov 2019

I loaded your core dump on my ubuntu 16.04 x64 machine. The DAC was downloaded from the symbol server for the "5.0.0" version of the runtime which has the path below. So on your machine that you are loading the dump you need to point setsymbolserver -directory to that version. Or the above link to the core dump that Dan provided isn't your dump.

/home/helixbot/work/5335d74d-89b3-4543-9c31-2236a792653d/Payload/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.so
mikem8361 picture mikem8361 on 6 Nov 2019

I'd like to understand a bit better how this works.

The DAC was downloaded from the symbol server for the "5.0.0" version of the runtime which has the path below.

When was this DAC uploaded to the symbol server? Did this happen as part of the corefx CI?

So on your machine that you are loading the dump you need to point setsymbolserver -directory to that version.

So the libmscordaccore.so in my Microsoft.NETCore.App/3.0.0 is not compatible? Does this rely on a version check? Or based on finding some symbols?
Where can I find/build a compatible version?
I've tried waiting a bit longer for the download to finish, but got this exception:

Unhandled Exception: System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled. ---> System.Net.Sockets.SocketException: Operation canceled
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
   at System.Net.Security.SslStreamInternal.<FillBufferAsync>g__InternalFillBufferAsync|38_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
   at System.Net.Security.SslStreamInternal.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.FillAsync()
   at System.Net.Http.HttpConnection.CopyToExactLengthAsync(Stream destination, UInt64 length, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.ContentLengthReadStream.CompleteCopyToAsync(Task copyTask, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.ContentLengthReadStream.CompleteCopyToAsync(Task copyTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.SymbolStore.SymbolStores.HttpSymbolStore.GetFileStream(Uri requestUri, CancellationToken token)
   at Microsoft.SymbolStore.SymbolStores.HttpSymbolStore.GetFileInner(SymbolStoreKey key, CancellationToken token)
   at Microsoft.SymbolStore.SymbolStores.SymbolStore.GetFile(SymbolStoreKey key, CancellationToken token)
   at Microsoft.SymbolStore.SymbolStores.SymbolStore.GetFile(SymbolStoreKey key, CancellationToken token)
   at SOS.SymbolReader.GetSymbolStoreFile(SymbolStoreKey key) in /_/src/SOS/SOS.NETCore/SymbolReader.cs:line 1009
   at SOS.SymbolReader.GetSymbolFile(SymbolStoreKey key) in /_/src/SOS/SOS.NETCore/SymbolReader.cs:line 327
   at SOS.SymbolReader.LoadNativeSymbols(SymbolFileCallback callback, IntPtr parameter, String moduleFilePath, UInt64 address, Int32 size, ReadMemoryDelegate readMemory) in /_/src/SOS/SOS.NETCore/SymbolReader.cs:line 300
   at SOS.LLDBServices.LoadNativeSymbols2(IntPtr self, Boolean runtimeOnly, ModuleLoadCallback callback) in /_/src/SOS/SOS.Hosting/LLDBServices.cs:line 180
   at SOS.SOSHost.ExecuteCommand(String command, String arguments) in /_/src/SOS/SOS.Hosting/SOSHost.cs:line 318
   at Microsoft.Diagnostics.Tools.Dump.SOSCommand.Invoke() in /_/src/Tools/dotnet-dump/Commands/SOSCommand.cs:line 60
tmds picture tmds on 6 Nov 2019

The DAC (libmscordaccore.so) has to match the runtime (libcoreclr.so) exactly. The DBI/DAC modules are published as part of our official build pipeline along with all the symbol files. Not part of any CI job.

Do you have a network connection on this machine. That exception is because it can't get to http://msdl.microsoft.com/download/symbols/ and times out. Or your connection is so slow that it times out (4 minutes) before the download finishes.

mikem8361 picture mikem8361 on 7 Nov 2019

In the end, dotnet-dump/SOS needs to be pointed at the exactly matching runtime bits locally with setsymbolserver -directory or have network connectivity to the above symbol server.

mikem8361 picture mikem8361 on 7 Nov 2019

Or your connection is so slow that it times out (4 minutes) before the download finishes.

This seems to be the case. It needs to be at least 300KB/s to be fast enough. I've created an issue to increase the timeout: https://github.com/dotnet/diagnostics/issues/602.

It managed to download the symbols today.

tmds picture tmds on 8 Nov 2019

These are the involved objects:

> dumpobj 0x00007f1598ac9dd8
Name:        System.Net.Mail.SmtpConnection
MethodTable: 00007f15bde82d90
EEClass:     00007f15bde7a898
Size:        136(0x88) bytes
File:        /home/helixbot/work/5335d74d-89b3-4543-9c31-2236a792653d/Payload/shared/Microsoft.NETCore.App/5.0.0/System.Net.Mail.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007f15bdddde10  400018d        8 ...ail.BufferBuilder  0 instance 00007f1598ac9e60 _bufferBuilder
00007f15bd406040  400018e       78       System.Boolean  1 instance                0 _isConnected
00007f15bd406040  400018f       79       System.Boolean  1 instance                0 _isClosed
00007f15bd406040  4000190       7a       System.Boolean  1 instance                0 _isStreamOpen
00007f15bd483380  4000191       10  System.EventHandler  0 instance 00007f1598aca088 _onCloseHandler
00007f15bde32a20  4000192       18 ...ail.SmtpTransport  0 instance 00007f1598ac8e80 _parent
00007f15bdd13930  4000193       20 ...t.Mail.SmtpClient  0 instance 00007f1598ac8dd8 _client
00007f15bde0b200  4000194       28 ...ets.NetworkStream  0 instance 00007f1598aec790 _networkStream
00007f15bde8dfa0  4000195       30 ...Sockets.TcpClient  0 instance 00007f1598ac9f98 _tcpClient
00007f15bd410f78  4000196       38        System.String  0 instance 0000000000000000 _host
00007f15bd40a060  4000197       70         System.Int32  1 instance                0 _port
00007f15be0a1650  4000198       40 ...eplyReaderFactory  0 instance 00007f1598aec7e0 _responseReader
00007f15bde31120  4000199       48 ...CredentialsByHost  0 instance 0000000000000000 _credentials
00007f15bd44ad08  400019a       50      System.String[]  0 instance 0000000000000000 _extensions
00007f15be0a0768  400019b       58 ...on.ChannelBinding  0 instance 0000000000000000 _channelBindingToken
00007f15bd406040  400019c       7b       System.Boolean  1 instance                0 _enableSsl
00007f15bde37708  400019d       60 ...ificateCollection  0 instance 0000000000000000 _clientCertificates
00007f15bd406040  400019e       7c       System.Boolean  1 instance                0 _serverSupportsEai
00007f15bd406040  400019f       7d       System.Boolean  1 instance                0 _dsnEnabled
00007f15bd406040  40001a0       7e       System.Boolean  1 instance                0 _serverSupportsStartTls
00007f15bd406040  40001a1       7f       System.Boolean  1 instance                0 _sawNegotiate
00007f15bde82c38  40001a2       74         System.Int32  1 instance                0 _supportedAuth
00007f15bde33998  40001a3       68 ...nticationModule[]  0 instance 00007f1598ac8ec8 _authenticationModules
00007f15bd50c378  400018c      160 ...g.ContextCallback  0   static 00007f1598884d68 s_AuthenticateCallback
00007f15bd44be88  40001a4      168        System.Char[]  0   static 00007f1598884da8 s_authExtensionSplitters
> dumpobj 0x00007f1598ac8e80
Name:        System.Net.Mail.SmtpTransport
MethodTable: 00007f15bde32a20
EEClass:     00007f15bde44280
Size:        72(0x48) bytes
File:        /home/helixbot/work/5335d74d-89b3-4543-9c31-2236a792653d/Payload/shared/Microsoft.NETCore.App/5.0.0/System.Net.Mail.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007f15bde33998  40001f0        8 ...nticationModule[]  0 instance 00007f1598ac8ec8 _authenticationModules
00007f15bde82d90  40001f1       10 ...il.SmtpConnection  0 instance 00007f1598ac9dd8 _connection
00007f15bdd13930  40001f2       18 ...t.Mail.SmtpClient  0 instance 00007f1598ac8dd8 _client
00007f15bde31120  40001f3       20 ...CredentialsByHost  0 instance 0000000000000000 _credentials
00007f15bde35648  40001f4       28 ...System.Net.Mail]]  0 instance 00007f1598ac8ef8 _failedRecipientExceptions
00007f15bd406040  40001f5       38       System.Boolean  1 instance                0 _identityRequired
00007f15bd406040  40001f6       39       System.Boolean  1 instance                0 _shouldAbort
00007f15bd406040  40001f7       3a       System.Boolean  1 instance                0 _enableSsl
00007f15bde37708  40001f8       30 ...ificateCollection  0 instance 0000000000000000 _clientCertificates
> dumpobj 0x00007f1598ac8dd8
Name:        System.Net.Mail.SmtpClient
MethodTable: 00007f15bdd13930
EEClass:     00007f15bdd0dbd0
Size:        168(0xa8) bytes
File:        /home/helixbot/work/5335d74d-89b3-4543-9c31-2236a792653d/Payload/shared/Microsoft.NETCore.App/5.0.0/System.Net.Mail.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007f15bd410f78  4000156        8        System.String  0 instance 00007f1598866d88 _host
00007f15bd40a060  4000157       88         System.Int32  1 instance            37147 _port
00007f15bd40a060  4000158       8c         System.Int32  1 instance                0 _timeout
00007f15bd406040  4000159       98       System.Boolean  1 instance                1 _inCall
00007f15bd406040  400015a       99       System.Boolean  1 instance                0 _cancelled
00007f15bd406040  400015b       9a       System.Boolean  1 instance                1 _timedOut
00007f15bd410f78  400015c       10        System.String  0 instance 00007f1598ac8f58 _targetName
00007f15bdb03758  400015d       90         System.Int32  1 instance                0 _deliveryMethod
00007f15bdd137d8  400015e       94         System.Int32  1 instance                0 _deliveryFormat
00007f15bd410f78  400015f       18        System.String  0 instance 0000000000000000 _pickupDirectoryLocation
00007f15bde32a20  4000160       20 ...ail.SmtpTransport  0 instance 00007f1598ac8e80 _transport
00007f15bddb4620  4000161       28 ....Mail.MailMessage  0 instance 0000000000000000 _message
00007f15bddddb80  4000162       30 ...t.Mail.MailWriter  0 instance 0000000000000000 _writer
00007f15bddb6e10  4000163       38 ...AddressCollection  0 instance 0000000000000000 _recipients
00007f15bdc78db8  4000164       40 ...endOrPostCallback  0 instance 00007f1598ac8f18 _onSendCompletedDelegate
00007f15bde82738  4000165       48 ...m.Threading.Timer  0 instance 00007f1598ac9d28 _timer
00007f15bde31d80  4000166       50 ...ontextAwareResult  0 instance 0000000000000000 _operationCompletedResult
00007f15bde32718  4000167       58 ...el.AsyncOperation  0 instance 0000000000000000 _asyncOp
00007f15bd410f78  4000169       60        System.String  0 instance 00007f1598ac9078 clientDomain
00007f15bd406040  400016a       9b       System.Boolean  1 instance                0 _disposed
00007f15bde3a3d0  400016b       68 ....Net.ServicePoint  0 instance 0000000000000000 _servicePoint
00007f15bde35580  400016c       70 ...ecipientException  0 instance 0000000000000000 _failedRecipientException
00007f15be0d36f8  400016d       78 ...letedEventHandler  0 instance 0000000000000000 SendCompleted
00007f15bd406040  400016e       9c       System.Boolean  1 instance                0 _useDefaultCredentials
00007f15bde31120  400016f       80 ...CredentialsByHost  0 instance 0000000000000000 _customCredentials
00007f15bde07050  4000168       a0 System.AsyncCallback  0   static 00007f1598820060 s_contextSafeCompleteCallback
> dumpobj 0x00007f1598ac9f98
Name:        System.Net.Sockets.TcpClient
MethodTable: 00007f15bde8dfa0
EEClass:     00007f15bde9aef0
Size:        48(0x30) bytes
File:        /home/helixbot/work/5335d74d-89b3-4543-9c31-2236a792653d/Payload/shared/Microsoft.NETCore.App/5.0.0/System.Net.Sockets.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007f15bde84d28  4000206       18         System.Int32  1 instance               23 _family
00007f15bde86ba0  4000207        8 ...et.Sockets.Socket  0 instance 00007f1598ac9fc8 _clientSocket
00007f15bde0b200  4000208       10 ...ets.NetworkStream  0 instance 00007f1598aec790 _dataStream
00007f15bd40a060  4000209       1c         System.Int32  1 instance                0 _disposed
00007f15bd406040  400020a       20       System.Boolean  1 instance                1 _active
> dumpobj 0x00007f1598ac9fc8
Name:        System.Net.Sockets.Socket
MethodTable: 00007f15bde86ba0
EEClass:     00007f15bde7db48
Size:        112(0x70) bytes
File:        /home/helixbot/work/5335d74d-89b3-4543-9c31-2236a792653d/Payload/shared/Microsoft.NETCore.App/5.0.0/System.Net.Sockets.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007f15bde87500  400012c        8 ....SafeSocketHandle  0 instance 00007f1598aec450 _handle
00007f15bde836e0  400012d       10  System.Net.EndPoint  0 instance 00007f1598aec550 _rightEndPoint
00007f15bde836e0  400012e       18  System.Net.EndPoint  0 instance 0000000000000000 _remoteEndPoint
00007f15bd406040  400012f       5c       System.Boolean  1 instance                1 _isConnected
00007f15bd406040  4000130       5d       System.Boolean  1 instance                0 _isDisconnected
00007f15bd406040  4000131       5e       System.Boolean  1 instance                1 _willBlock
00007f15bd406040  4000132       5f       System.Boolean  1 instance                1 _willBlockInternal
00007f15bd406040  4000133       60       System.Boolean  1 instance                0 _isListening
00007f15bd406040  4000134       61       System.Boolean  1 instance                0 _nonBlockingConnectInProgress
00007f15bde836e0  4000135       20  System.Net.EndPoint  0 instance 0000000000000000 _nonBlockingConnectRightEndPoint
00007f15bde84d28  4000136       40         System.Int32  1 instance               23 _addressFamily
00007f15bde868f8  4000137       44         System.Int32  1 instance                1 _socketType
00007f15bde86a48  4000138       48         System.Int32  1 instance                6 _protocolType
00007f15be0d48e0  4000139       28 ...s.Socket+CacheSet  0 instance 0000000000000000 _caches
00007f15bd406040  400013a       62       System.Boolean  1 instance                0 _receivingPacketInformation
00007f15bd40a060  400013c       4c         System.Int32  1 instance               -1 _closeTimeout
00007f15bd40a060  400013d       50         System.Int32  1 instance                0 _disposed
00007f15bd40db38  400013f       38        System.IntPtr  1 instance 0000000000000000 _lastReceiveHandle
00007f15bd40a060  4000140       54         System.Int32  1 instance                0 _lastReceiveThread
00007f15bd40a060  4000141       58         System.Int32  1 instance                0 _lastReceiveTick
00007f15be0d2640  4000149       30 ...t+CachedEventArgs  0 instance 0000000000000000 _cachedTaskEventArgs
00007f15bd27ca58  400013b       50        System.Object  0   static 00007f159886f7c8 s_internalSyncObject
00007f15bd406040  400013e       d8       System.Boolean  1   static                1 s_initialized
00007f15bde07050  4000142       58 System.AsyncCallback  0   static 00007f15984d2540 s_multipleAddressConnectCallback
00007f15bde89970  4000143       60 ...tem.Net.Sockets]]  0   static 00007f159886f4b0 AcceptCompletedHandler
00007f15bde89970  4000144       68 ...tem.Net.Sockets]]  0   static 00007f159886f508 ReceiveCompletedHandler
00007f15bde89970  4000145       70 ...tem.Net.Sockets]]  0   static 00007f159886f548 SendCompletedHandler
00007f15bde8a098  4000146       78 ...tem.Net.Sockets]]  0   static 00007f159886f588 s_rentedSocketSentinel
00007f15bde8a2f0  4000147       80 ...ketAsyncEventArgs  0   static 00007f159886f680 s_rentedInt32Sentinel
00007f15bd5138b0  4000148       88 ...Private.CoreLib]]  0   static 00007f159886f780 s_zeroTask
> dumpobj 0x00007f1598ac9d40
Name:        System.Threading.TimerQueueTimer
MethodTable: 00007f15bdcccde8
EEClass:     00007f15bdcd5ad8
Size:        96(0x60) bytes
File:        /home/helixbot/work/5335d74d-89b3-4543-9c31-2236a792653d/Payload/shared/Microsoft.NETCore.App/5.0.0/System.Private.CoreLib.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007f15be0a0408  4000813        8 ...eading.TimerQueue  0 instance 00007f159887a9f8 _associatedTimerQueue
00007f15bdcccde8  4000814       10 ...g.TimerQueueTimer  0 instance 0000000000000000 _next
00007f15bdcccde8  4000815       18 ...g.TimerQueueTimer  0 instance 0000000000000000 _prev
00007f15bd406040  4000816       54       System.Boolean  1 instance                0 _short
00007f15bd40b3e0  4000817       40         System.Int64  1 instance                0 _startTicks
00007f15bd40aa20  4000818       48        System.UInt32  1 instance       4294967295 _dueTime
00007f15bd40aa20  4000819       4c        System.UInt32  1 instance       4294967295 _period
00007f15bde83230  400081a       20 ...ing.TimerCallback  0 instance 00007f1598ac9ce8 _timerCallback
00007f15bd27ca58  400081b       28        System.Object  0 instance 0000000000000000 _state
00007f15bd709578  400081c       30 ....ExecutionContext  0 instance 00007f15982753f8 _executionContext
00007f15bd40a060  400081d       50         System.Int32  1 instance                0 _callbacksRunning
00007f15bd406040  400081e       55       System.Boolean  1 instance                0 _canceled
00007f15bd27ca58  400081f       38        System.Object  0 instance 0000000000000000 _notifyWhenNoCallbacksRunning
00007f15bd50c378  4000820      590 ...g.ContextCallback  0   static 00007f1598a44ad0 s_callCallbackInContext
> dumpobj 0x00007f1598aec790
Name:        System.Net.Sockets.NetworkStream
MethodTable: 00007f15bde0b200
EEClass:     00007f15bde22328
Size:        56(0x38) bytes
File:        /home/helixbot/work/5335d74d-89b3-4543-9c31-2236a792653d/Payload/shared/Microsoft.NETCore.App/5.0.0/System.Net.Sockets.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007f15be0f78e8  400137f        8 ...eam+ReadWriteTask  0 instance 0000000000000000 _activeReadWriteTask
00007f15bd506538  4001380       10 ...ing.SemaphoreSlim  0 instance 0000000000000000 _asyncActiveSemaphore
00007f15bd4dbb80  400137e     1178     System.IO.Stream  0   static 00007f1598154f78 Null
00007f15bde86ba0  4000109       18 ...et.Sockets.Socket  0 instance 00007f1598ac9fc8 _streamSocket
00007f15bd406040  400010a       2c       System.Boolean  1 instance                1 _ownsSocket
00007f15bd406040  400010b       2d       System.Boolean  1 instance                1 _readable
00007f15bd406040  400010c       2e       System.Boolean  1 instance                1 _writeable
00007f15bd40a060  400010d       20         System.Int32  1 instance               -1 _closeTimeout
00007f15bd406040  400010e       2f       System.Boolean  1 instance                0 _disposed
00007f15bd40a060  400010f       24         System.Int32  1 instance               -1 _currentReadTimeout
00007f15bd40a060  4000110       28         System.Int32  1 instance               -1 _currentWriteTimeout

From this, the timeout expired (SmtpClient._timedOut = true).
This calls SmtpClient.Abort:

        private void Abort()
        {
            try
            {
                _transport.Abort();
            }
            catch
            {
            }
        }

Unfortunately, this one will eat all exceptions.
This one calls Transport.Abort:

        internal void Abort()
        {
            lock (this)
            {
                if (_connection != null)
                {
                    _connection.Abort();
                }
                else
                {
                    _shouldAbort = true;
                }
            }
        }

Which calls SmtpConnection.Abort (from Transport.Abort, or in GetConnection via _shouldAbort = true):

        internal void Abort()
        {
            if (!_isClosed)
            {
                lock (this)
                {
                    if (!_isClosed && _tcpClient != null)
                    {
                        //free CBT buffer
                        if (_channelBindingToken != null)
                        {
                            _channelBindingToken.Close();
                        }

                        // must destroy manually since sending a QUIT here might not be
                        // interpreted correctly by the server if it's in the middle of a
                        // DATA command or some similar situation.  This may send a RST
                        // but this is ok in this situation.  Do not reuse this connection
                        _tcpClient.LingerState = new LingerOption(true, 0);
                        _networkStream?.Close();
                        _tcpClient.Dispose();
                    }
                    _isClosed = true;
                }
            }
            _isConnected = false;
        }

In this code, _tcpClient is never null, and _channelBindingToken is always null.
SmtpConnection._isClosed is still false. The NetworkStream, TcpClient and Socket don't have their disposed flag set.
So either this Abort was never called, or it threw at _tcpClient.LingerState = new LingerOption(true, 0);.

If we add a bool SmtpConnection._aborted and set it in SmtpConnection.Abort we can know which it is, when we get a new coredump.

@ViktorHofer shall I do that? Or can you make something more from the dump?

tmds picture tmds on 8 Nov 2019

@viktorhofer is not familiar with this code as far as I know, he just reported the failure. So I'd go ahead with your plan @tmds 馃檪

danmosemsft picture danmosemsft on 8 Nov 2019

I'm seeing a test failure that looks similar in CI leg https://dev.azure.com/dnceng/public/_build/results?buildId=425615 from https://github.com/dotnet/corefx/pull/42420.

Executed on a0018IL
+ ./RunTests.sh --runtime-path /home/helixbot/work/dcf819c8-dfc6-479d-a334-3d619501162f/Payload
----- start Wed Nov 13 03:31:48 UTC 2019 =============== To repro directly: =====================================================
pushd .
/home/helixbot/work/dcf819c8-dfc6-479d-a334-3d619501162f/Payload/dotnet exec --runtimeconfig System.Net.Mail.Functional.Tests.runtimeconfig.json xunit.console.dll System.Net.Mail.Functional.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=nonnetcoreapptests -notrait category=nonlinuxtests -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
~/work/dcf819c8-dfc6-479d-a334-3d619501162f/Work/b800be86-82cb-4c38-9238-5e20f5502eef/Exec ~/work/dcf819c8-dfc6-479d-a334-3d619501162f/Work/b800be86-82cb-4c38-9238-5e20f5502eef/Exec
  Discovering: System.Net.Mail.Functional.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Mail.Functional.Tests (found 144 test cases)
  Starting:    System.Net.Mail.Functional.Tests (parallel test collections = on, max threads = 2)
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.TestZeroTimeout', Elapsed: 00:02:10
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.TestZeroTimeout', Elapsed: 00:04:10
Process terminated. TestZeroTimeout
   at System.Environment.FailFast(System.String)
   at System.Net.Mail.Tests.SmtpClientTest.TestZeroTimeout()
   at System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean, Boolean)
   at System.Reflection.MethodBase.Invoke(System.Object, System.Object[])
   at Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].CallTestMethod(System.Object)
   at Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1+<<InvokeTestMethodAsync>b__1>d[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1+<<InvokeTestMethodAsync>b__1>d[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<<InvokeTestMethodAsync>b__1>d<System.__Canon> ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1+<<InvokeTestMethodAsync>b__1>d[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<<InvokeTestMethodAsync>b__1>d<System.__Canon> ByRef)
   at Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<InvokeTestMethodAsync>b__1()
   at Xunit.Sdk.ExecutionTimer+<AggregateAsync>d__4.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExecutionTimer+<AggregateAsync>d__4, xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<AggregateAsync>d__4 ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Xunit.Sdk.ExecutionTimer+<AggregateAsync>d__4, xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<AggregateAsync>d__4 ByRef)
   at Xunit.Sdk.ExecutionTimer.AggregateAsync(System.Func`1<System.Threading.Tasks.Task>)
   at Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<InvokeTestMethodAsync>b__0()
   at Xunit.Sdk.ExceptionAggregator+<RunAsync>d__9.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__9, xunit.core, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__9 ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__9, xunit.core, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__9 ByRef)
   at Xunit.Sdk.ExceptionAggregator.RunAsync(System.Func`1<System.Threading.Tasks.Task>)
   at Xunit.Sdk.TestInvoker`1+<InvokeTestMethodAsync>d__48[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestInvoker`1+<InvokeTestMethodAsync>d__48[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<InvokeTestMethodAsync>d__48<System.__Canon> ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Decimal, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Xunit.Sdk.TestInvoker`1+<InvokeTestMethodAsync>d__48[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<InvokeTestMethodAsync>d__48<System.__Canon> ByRef)
   at Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].InvokeTestMethodAsync(System.Object)
   at Xunit.Sdk.XunitTestInvoker.InvokeTestMethodAsync(System.Object)
   at Xunit.Sdk.TestInvoker`1+<<RunAsync>b__47_0>d[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestInvoker`1+<<RunAsync>b__47_0>d[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<<RunAsync>b__47_0>d<System.__Canon> ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Decimal, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Xunit.Sdk.TestInvoker`1+<<RunAsync>b__47_0>d[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<<RunAsync>b__47_0>d<System.__Canon> ByRef)
   at Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<RunAsync>b__47_0()
   at Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.Decimal, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.Decimal, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.core, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__10`1<System.Decimal> ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Decimal, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.Decimal, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.core, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__10`1<System.Decimal> ByRef)
   at Xunit.Sdk.ExceptionAggregator.RunAsync[[System.Decimal, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.Func`1<System.Threading.Tasks.Task`1<System.Decimal>>)
   at Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].RunAsync()
   at Xunit.Sdk.XunitTestRunner.InvokeTestMethodAsync(Xunit.Sdk.ExceptionAggregator)
   at Xunit.Sdk.XunitTestRunner+<InvokeTestAsync>d__4.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.XunitTestRunner+<InvokeTestAsync>d__4, xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<InvokeTestAsync>d__4 ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Xunit.Sdk.XunitTestRunner+<InvokeTestAsync>d__4, xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<InvokeTestAsync>d__4 ByRef)
   at Xunit.Sdk.XunitTestRunner.InvokeTestAsync(Xunit.Sdk.ExceptionAggregator)

Full failure log: https://helix.dot.net/api/2019-06-17/jobs/dcf819c8-dfc6-479d-a334-3d619501162f/workitems/System.Net.Mail.Functional.Tests/console

layomia picture layomia on 13 Nov 2019

As part of fixing this issue, https://github.com/dotnet/corefx/pull/42557 should be reverted, too.

stephentoub picture stephentoub on 13 Nov 2019

This failed here:
https://dev.azure.com/dnceng/public/_build/results?buildId=432120&view=ms.vss-test-web.build-test-results-tab&runId=13685330&resultId=176979&paneView=attachments
with a dump:
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-109-merge-ec88cd5945b2414080/System.Net.Mail.Functional.Tests/core.1000.2978?sv=2018-03-28&sr=c&sig=UoH%2BhlFBrmG6aZOoS3xmJ1kCviJZM9JAOk1vYe67ZM8%3D&se=2019-11-29T15%3A51%3A32Z&sp=rl

stephentoub picture stephentoub on 19 Nov 2019

Failed again here:
https://dev.azure.com/dnceng/public/_build/results?buildId=433585&view=ms.vss-test-web.build-test-results-tab&runId=13748274&resultId=176977&paneView=attachments
with a dump here:
https://helix.dot.net/api/2019-06-17/jobs/6f20c57d-10e0-4aa6-9740-85f01db2a599/workitems/System.Net.Mail.Functional.Tests/files/core.1000.2181
@tmds, do you have everything you need for this? It's causing a bunch of our runs to fail so I want to disable it again.

stephentoub picture stephentoub on 21 Nov 2019

@tmds, do you have everything you need for this? It's causing a bunch of our runs to fail so I want to disable it again.

Yes. I had same timeout issues yesterday loading the coredump (https://github.com/dotnet/diagnostics/issues/602) but managed to get it just now.

> dumpobj 0x00007f6728afbaa0                                                                                                                                                                                                                 
Name:        System.Net.Mail.SmtpConnection
MethodTable: 00007f67507714f0
EEClass:     00007f675076a918
Size:        144(0x90) bytes
File:        /home/helixbot/work/ec88cd59-45b2-4140-8048-9b5a47cc74ad/Payload/shared/Microsoft.NETCore.App/5.0.0/System.Net.Mail.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007f675077f4c0  400018e        8 ...ail.BufferBuilder  0 instance 00007f6728afbb30 _bufferBuilder
00007f674fd060d0  400018f       78       System.Boolean  1 instance                0 _isConnected
00007f674fd060d0  4000190       79       System.Boolean  1 instance                0 _isClosed
00007f674fd060d0  4000191       7a       System.Boolean  1 instance                0 _isStreamOpen
00007f674fd834b0  4000192       10  System.EventHandler  0 instance 00007f6728afbd58 _onCloseHandler
00007f6750603df0  4000193       18 ...ail.SmtpTransport  0 instance 00007f6728ad2f80 _parent
00007f67505e8ce0  4000194       20 ...t.Mail.SmtpClient  0 instance 00007f6728ad2ed8 _client
00007f67506e62b0  4000195       28 ...ets.NetworkStream  0 instance 00007f6728afc870 _networkStream
00007f675077cb40  4000196       30 ...Sockets.TcpClient  0 instance 00007f6728afbc68 _tcpClient
00007f674fd10f78  4000197       38        System.String  0 instance 0000000000000000 _host
00007f674fd0a0f0  4000198       70         System.Int32  1 instance                0 _port
00007f6750790518  4000199       40 ...eplyReaderFactory  0 instance 00007f6728afc8c0 _responseReader
00007f67505e7b08  400019a       48 ...CredentialsByHost  0 instance 0000000000000000 _credentials
00007f674fd4ae70  400019b       50      System.String[]  0 instance 0000000000000000 _extensions
00007f675077f590  400019c       58 ...on.ChannelBinding  0 instance 0000000000000000 _channelBindingToken
00007f674fd060d0  400019d       7b       System.Boolean  1 instance                0 _enableSsl
00007f67506097e8  400019e       60 ...ificateCollection  0 instance 0000000000000000 _clientCertificates
00007f674fd060d0  400019f       7c       System.Boolean  1 instance                1 _aborted
00007f674fd060d0  40001a0       7d       System.Boolean  1 instance                0 _serverSupportsEai
00007f674fd060d0  40001a1       7e       System.Boolean  1 instance                0 _dsnEnabled
00007f674fd060d0  40001a2       7f       System.Boolean  1 instance                0 _serverSupportsStartTls
00007f674fd060d0  40001a3       80       System.Boolean  1 instance                0 _sawNegotiate
00007f6750771398  40001a4       74         System.Int32  1 instance                0 _supportedAuth
00007f6750604e38  40001a5       68 ...nticationModule[]  0 instance 00007f6728ad2fc8 _authenticationModules
00007f674fe0e8a0  400018d      160 ...g.ContextCallback  0   static 00007f6728876810 s_AuthenticateCallback
00007f674fd4c028  40001a6      168        System.Char[]  0   static 00007f6728876850 s_authExtensionSplitters
> dumpobj 00007f6728afc870                                                                                                                                                                                                                   
Name:        System.Net.Sockets.NetworkStream
MethodTable: 00007f67506e62b0
EEClass:     00007f67506f9120
Size:        56(0x38) bytes
File:        /home/helixbot/work/ec88cd59-45b2-4140-8048-9b5a47cc74ad/Payload/shared/Microsoft.NETCore.App/5.0.0/System.Net.Sockets.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007f67509e9620  4001377        8 ...eam+ReadWriteTask  0 instance 0000000000000000 _activeReadWriteTask
00007f674fe07ba8  4001378       10 ...ing.SemaphoreSlim  0 instance 0000000000000000 _asyncActiveSemaphore
00007f674fddd120  4001376     1168     System.IO.Stream  0   static 00007f672815a4e0 Null
00007f6750775740  4000109       18 ...et.Sockets.Socket  0 instance 00007f6728afbc98 _streamSocket
00007f674fd060d0  400010a       2c       System.Boolean  1 instance                1 _ownsSocket
00007f674fd060d0  400010b       2d       System.Boolean  1 instance                1 _readable
00007f674fd060d0  400010c       2e       System.Boolean  1 instance                1 _writeable
00007f674fd0a0f0  400010d       20         System.Int32  1 instance               -1 _closeTimeout
00007f674fd060d0  400010e       2f       System.Boolean  1 instance                0 _disposed
00007f674fd0a0f0  400010f       24         System.Int32  1 instance               -1 _currentReadTimeout
00007f674fd0a0f0  4000110       28         System.Int32  1 instance               -1 _currentWriteTimeout

https://github.com/dotnet/corefx/blob/e99ec129cfd594d53f4390bf97d1d736cff6f860/src/System.Net.Mail/src/System/Net/Mail/SmtpConnection.cs#L155-L183

From the dump it Abort was called (_aborted = 1) bit _isClosed is still false.
The NetworkStream doesn't have _disposed set.
So it looks like _tcpClient.LingerState = new LingerOption(true, 0); throws an exception, which causes the Socket to not get closed. This exception gets swallowed by the try-catch in SmtpClient.Abort.

tmds picture tmds on 21 Nov 2019

@tmds, this test is still disabled against this issue:
https://github.com/dotnet/runtime/blob/04f22268efb4434b7cd81e8ae2c3354be99fd4a7/src/libraries/System.Net.Mail/tests/Functional/SmtpClientTest.cs#L311-L315
Should the [ActiveIssue] be removed?

stephentoub picture stephentoub on 23 Jan 2020

Yes, it should be removed. This issue can stay open. We're identifying what types of exceptions are at the root (via https://github.com/dotnet/runtime/pull/428).
I'll make a PR to remove it.

tmds picture tmds on 23 Jan 2020
❤1

Failed again here: https://dev.azure.com/dnceng/public/_build/results?buildId=505562&view=ms.vss-test-web.build-test-results-tab&runId=16050798&resultId=177375&paneView=attachments

safern picture safern on 3 Feb 2020

I guess we can close this in favor of: https://github.com/dotnet/runtime/issues/31719 which has all the builds and coredumps information.

We should just make sure to include the test name in the title for discoverability.

safern picture safern on 6 Feb 2020

Closing in favor of #31719

ViktorHofer picture ViktorHofer on 6 Feb 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

Timovzl picture Timovzl  路  3Comments
noahfalk picture noahfalk  路  3Comments
yahorsi picture yahorsi  路  3Comments
jamesqo picture jamesqo  路  3Comments
EgorBo picture EgorBo  路  3Comments