runtime 🚀 - System.Xml.XmlSchemaSet.Tests crashed on Linux in CI

This happened in official build also IIRC.

danmosemsft on 6 Mar 2018

@stephentoub @danmosemsft do you perhaps know if this is Ubuntu specific or has it happened also on other platforms?

XmlSchemaValidatorApi has similar project structure and same configuration file so would expect both of them fail at some point - pretty weird only one of them is failing.

@dotnet/dnceng, do you perhaps know if there are any known issues this may fall into? This suggest it might be a problem with infra but hard to say at this point:

2018-03-04 16:03:36,476: INFO: proc(54): run_and_log_output: Output: ----- end 16:03:36 ----- exit code 134 ----------------------------------------------------------
2018-03-04 16:03:36,478: INFO: proc(54): run_and_log_output: Output: Unable to find executable corerun

@stephentoub I'm slightly confused by the link to the CI: I do not see any failures in there and specifically "Ubuntu.1804.Amd64.Open:Release-x64" shows as "passed"

krwq on 6 Mar 2018

@krwq Have you tried getting a repo via the repro tool to see if CoreRun was passed in with the payloads?

Chrisboh on 6 Mar 2018

@Chrisboh - it shows as "passed" and shows different output to one @stephentoub has pasted.

krwq on 6 Mar 2018

https://github.com/dotnet/corefx/pull/27763#issuecomment-370912225 hit it on OpenSUSE

danmosemsft on 7 Mar 2018

Failed again here:
https://mc.dot.net/#/user/dotnet-maestro-bot/pr~2Fjenkins~2Fdotnet~2Fcorefx~2Fmaster~2F/test~2Ffunctional~2Fcli~2F/f31d24f0f1ff748ee84a9f67a7b4e4e2c80dc544/workItem/System.Xml.XmlSchema.XmlSchemaValidatorApi.Tests/wilogs

This does appear to be infrastructure related rather than test related. Every time I've seen this, I've seen it with the "Unable to find executable corerun" that @krwq pointed out. And since it doesn't happen every time, there's likely some race condition happening in the deployment of the bits that's causing this, which means the repro tool is unlikely to be particularly helpful.

stephentoub on 7 Mar 2018

The "unable to find corerun" happens when it's trying to examine the coredump using gdb - gdb wants an executable that matches the coredump. See runnertemplate.unix.txt. So it's not related to the failure. I opened https://github.com/dotnet/corefx/issues/27818

The question is why it's always XmlSchemaValidatorApi.Tests. There is a core dump being created, which might help? I can only suggest trying a repro machine.

danmosemsft on 7 Mar 2018

The "unable to find corerun" happens when it's trying to examine the coredump using gdb

Ah, ok, nevermind my comment then.

stephentoub on 7 Mar 2018

Closing this bug since @danmosemsft has opened the issue to track the infra problem

krwq on 7 Mar 2018

The infra problem is only in cleaning up after these tests quit. It's not the cause.

danmosemsft on 7 Mar 2018

any numbers on how often does it happen? I'd like to try finding the issue (my current guess would be file which might not be closed - not sure why would that be an issue though since unlinking should still work) - here is my lucky guess for now:

https://github.com/dotnet/corefx/blob/master/src/System.Private.Xml/tests/XmlSchema/XmlSchemaSet/TC_SchemaSet_Reprocess.cs#L638

krwq on 7 Mar 2018

@krwq here's every time this has happened across all jobs and which queue it happened on, since it started on 2/14. There seems to be no specific pattern to it in terms of OS. (75 times)

Finished QueueName
2018-02-14 06:20:29.6500000 Debian.87.Amd64.Open
2018-02-15 00:52:26.4630000 OpenSuse.423.Amd64.Open
2018-02-15 20:22:48.4140000 Debian.87.Amd64.Open
2018-02-15 20:22:51.7820000 Centos.73.Amd64.Open
2018-02-16 10:12:23.1500000 Ubuntu.1604.Amd64.Open
2018-02-20 06:22:51.5590000 Ubuntu.1604.Amd64.Open
2018-02-21 02:57:54.5360000 Debian.87.Amd64.Open
2018-02-21 06:11:29.5620000 OSX.1013.Amd64.Open
2018-02-21 06:21:19.3630000 OSX.1013.Amd64.Open
2018-02-21 06:22:25.1970000 Ubuntu.1404.Amd64.Open
2018-02-21 06:25:15.0180000 OSX.1012.Amd64.Open
2018-02-21 06:26:34.2860000 OSX.1013.Amd64.Open
2018-02-21 06:32:30.9720000 OSX.1013.Amd64.Open
2018-02-21 06:43:43.5080000 OSX.1012.Amd64.Open
2018-02-21 06:50:21.6910000 OSX.1012.Amd64.Open
2018-02-21 06:55:31.5970000 OSX.1012.Amd64.Open
2018-02-21 17:20:08.1200000 OSX.1012.Amd64.Open
2018-02-21 17:22:18.4330000 OSX.1013.Amd64.Open
2018-02-21 19:31:51.8080000 Ubuntu.1604.Amd64.Open
2018-02-21 19:32:35.5310000 Ubuntu.1404.Amd64.Open
2018-02-21 19:32:37.6770000 Ubuntu.1604.Amd64.Open
2018-02-21 19:33:15.0100000 Ubuntu.1404.Amd64.Open
2018-02-21 19:33:58.8690000 Ubuntu.1804.Amd64.Open
2018-02-21 19:34:28.4380000 Debian.87.Amd64.Open
2018-02-21 19:34:29.4850000 OpenSuse.423.Amd64.Open
2018-02-21 19:34:43.2060000 Ubuntu.1804.Amd64.Open
2018-02-21 19:35:13.2390000 Centos.74.Amd64.Open
2018-02-21 19:35:22.1310000 Debian.87.Amd64.Open
2018-02-21 19:35:51.3410000 Centos.74.Amd64.Open
2018-02-21 19:37:05.6610000 OpenSuse.423.Amd64.Open
2018-02-21 19:37:33.4700000 RedHat.74.Amd64.Open
2018-02-21 19:38:25.1180000 RedHat.74.Amd64.Open
2018-02-21 19:43:03.1380000 OSX.1013.Amd64.Open
2018-02-21 19:43:37.2190000 Fedora.26.Amd64.Open
2018-02-21 19:44:20.1740000 Fedora.26.Amd64.Open
2018-02-21 19:56:55.5770000 OSX.1013.Amd64.Open
2018-02-21 20:00:29.4390000 Debian.90.Amd64.Open
2018-02-21 20:02:07.7510000 SLES.12.Amd64.Open
2018-02-21 20:02:31.7560000 Ubuntu.1604.Amd64.Open
2018-02-21 20:03:22.1570000 Fedora.26.Amd64.Open
2018-02-21 20:03:31.7020000 Ubuntu.1404.Amd64.Open
2018-02-21 20:04:20.9590000 Fedora.27.Amd64.Open
2018-02-21 20:05:10.6790000 Ubuntu.1804.Amd64.Open
2018-02-21 20:05:23.4700000 OpenSuse.423.Amd64.Open
2018-02-21 20:05:39.5260000 Debian.87.Amd64.Open
2018-02-21 20:06:28.6250000 Centos.74.Amd64.Open
2018-02-21 20:09:35.4870000 RedHat.74.Amd64.Open
2018-02-21 20:13:36.7540000 OSX.1013.Amd64.Open
2018-02-21 20:22:49.6320000 OSX.1012.Amd64.Open
2018-02-21 20:43:28.5760000 OSX.1012.Amd64.Open
2018-02-21 20:51:07.5600000 OSX.1012.Amd64.Open
2018-02-21 23:36:18.5400000 Ubuntu.1804.Amd64.Open
2018-02-22 00:11:01.1030000 Ubuntu.1404.Amd64.Open
2018-02-23 01:29:34.6180000 Fedora.26.Amd64.Open
2018-02-23 01:31:45.4320000 RedHat.74.Amd64.Open
2018-02-23 06:09:27.5720000 Ubuntu.1804.Amd64.Open
2018-02-23 17:31:57.9830000 Ubuntu.1604.Amd64.Open
2018-02-24 00:23:04.8200000 OSX.1013.Amd64.Open
2018-02-24 00:25:34.7690000 OSX.1012.Amd64.Open
2018-02-24 01:24:00.9610000 Ubuntu.1404.Amd64.Open
2018-02-24 04:58:30.9490000 Ubuntu.1404.Amd64.Open
2018-02-24 16:59:51.7700000 Ubuntu.1604.Amd64.Open
2018-02-24 22:08:08.0970000 Ubuntu.1404.Amd64.Open
2018-02-26 22:59:55.6370000 RedHat.74.Amd64.Open
2018-02-27 12:24:48.0990000 Ubuntu.1404.Amd64.Open
2018-02-28 15:37:37.6050000 Ubuntu.1404.Amd64.Open
2018-02-28 22:44:05.8560000 OpenSuse.423.Amd64.Open
2018-03-01 13:52:51.5580000 OSX.1012.Amd64.Open
2018-03-01 13:56:14.3910000 OSX.1013.Amd64.Open
2018-03-02 17:50:07.4410000 Debian.90.Amd64.Open
2018-03-02 17:54:25.2820000 Ubuntu.1804.Amd64.Open
2018-03-04 03:50:31.2760000 RedHat.74.Amd64.Open
2018-03-04 16:03:37.4900000 Ubuntu.1804.Amd64.Open
2018-03-05 23:20:14.6140000 SLES.12.Amd64.Open
2018-03-06 03:23:09.4260000 OpenSuse.423.Amd64.Open

MattGal on 8 Mar 2018

@MattGal Thank you a lot!

krwq on 8 Mar 2018

Wasn't planning on merging my PR yet - please ping if it repros again.

krwq on 8 Mar 2018

Failed again here:
https://mc.dot.net/#/user/stephentoub/pr~2Fjenkins~2Fdotnet~2Fcorefx~2Fmaster~2F/test~2Ffunctional~2Fcli~2F/3e073e591ff556727475b38ee9436bfc9f9a46ce/workItem/System.Xml.XmlSchemaSet.Tests/wilogs
Unfortunately gdb isn't installed on the lab machine so Dan's buildtools updates didn't result in us getting a stack trace:

2018-03-09 17:12:23,814: INFO: proc(54): run_and_log_output: Output: Discovering: System.Xml.XmlSchemaSet.Tests
2018-03-09 17:12:24,016: INFO: proc(54): run_and_log_output: Output: Discovered:  System.Xml.XmlSchemaSet.Tests
2018-03-09 17:12:24,158: INFO: proc(54): run_and_log_output: Output: Starting:    System.Xml.XmlSchemaSet.Tests
2018-03-09 17:12:25,998: INFO: proc(54): run_and_log_output: Output: /home/helixbot/dotnetbuild/work/a627ecdb-b9c3-4bdd-b274-47806e3b54b8/Work/26afd538-9aa6-4dc7-b53a-35b6f0ab5c06/Unzip/RunTests.sh: line 87: 47430 Aborted                 (core dumped) $RUNTIME_PATH/dotnet xunit.console.netcore.exe System.Xml.XmlSchemaSet.Tests.dll -xml testResults.xml -notrait Benchmark=true -notrait category=nonnetcoreapptests -notrait category=nonlinuxtests -notrait category=OuterLoop -notrait category=failing
2018-03-09 17:12:26,033: INFO: proc(54): run_and_log_output: Output: Dumpling has not been installed yet. Please run "DumplingHelper.py install_dumpling" before collect_dumps.
2018-03-09 17:12:26,036: INFO: proc(54): run_and_log_output: Output: ~/dotnetbuild/work/a627ecdb-b9c3-4bdd-b274-47806e3b54b8/Work/26afd538-9aa6-4dc7-b53a-35b6f0ab5c06/Unzip
2018-03-09 17:12:26,037: INFO: proc(54): run_and_log_output: Output: ----- end 17:12:26 ----- exit code 134 ----------------------------------------------------------
2018-03-09 17:12:26,038: INFO: proc(54): run_and_log_output: Output: Looking around for any Linux dump...
2018-03-09 17:12:26,039: INFO: proc(54): run_and_log_output: Output: GDB was not found. Unable to print core file.
2018-03-09 17:12:26,041: INFO: proc(54): run_and_log_output: Output: Copying core file core to /tmp/coredumps/core.28805 in case you need it.
2018-03-09 17:12:26,200: INFO: proc(58): run_and_log_output: Exit Code: 134

stephentoub on 9 Mar 2018

Finally got a crash on a machine with gdb:
https://mc.dot.net/#/user/jswolf19/pr~2Fjenkins~2Fdotnet~2Fcorefx~2Fmaster~2F/test~2Ffunctional~2Fcli~2F/7f02d90fce8bc18f5668935c29fb9121f689870d/workItem/System.Xml.XmlSchemaSet.Tests/wilogs

Output: Discovering: System.Xml.XmlSchemaSet.Tests
Output: Discovered:  System.Xml.XmlSchemaSet.Tests
Output: Starting:    System.Xml.XmlSchemaSet.Tests
Output: /home/helixbot/dotnetbuild/work/4669dcde-7303-476e-aa92-88736f420e56/Work/cc997b9a-882e-490d-9b6a-2a3ad7f54b4a/Unzip/RunTests.sh: line 87: 34819 Aborted                 (core dumped) $RUNTIME_PATH/dotnet xunit.console.netcore.exe System.Xml.XmlSchemaSet.Tests.dll -xml testResults.xml -notrait Benchmark=true -notrait category=nonnetcoreapptests -notrait category=nonlinuxtests -notrait category=OuterLoop -notrait category=failing
Output: Dumpling has not been installed yet. Please run "DumplingHelper.py install_dumpling" before collect_dumps.
Output: ~/dotnetbuild/work/4669dcde-7303-476e-aa92-88736f420e56/Work/cc997b9a-882e-490d-9b6a-2a3ad7f54b4a/Unzip
Output: ----- end 06:02:21 ----- exit code 134 ----------------------------------------------------------
Output: Looking around for any Linux dump...
Output: Looking for files matching core.34819 ...
Output: Printing info from core file core.34819
Output: Missing separate debuginfo for /home/helixbot/dotnetbuild/work/4669dcde-7303-476e-aa92-88736f420e56/Payload/dotnet
Output: Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/9b/536afc1f88f5d52b1330876b83f8efc9f392b8.debug
Output: [New LWP 34828]
Output: [New LWP 34820]
Output: [New LWP 34819]
Output: [New LWP 34821]
Output: [New LWP 34822]
Output: [New LWP 34823]
Output: [New LWP 34824]
Output: [New LWP 34827]
Output: [New LWP 34829]
Output: [Thread debugging using libthread_db enabled]
Output: Using host libthread_db library "/lib64/libthread_db.so.1".
Output: Missing separate debuginfo for /home/helixbot/dotnetbuild/work/4669dcde-7303-476e-aa92-88736f420e56/Payload/host/fxr/9.9.9/libhostfxr.so
Output: Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/db/e98db32ffdd4bff56eea37de26bb516890c9bb.debug
Output: Missing separate debuginfo for /home/helixbot/dotnetbuild/work/4669dcde-7303-476e-aa92-88736f420e56/Payload/shared/Microsoft.NETCore.App/9.9.9/libhostpolicy.so
Output: Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/e7/93f574f205a700f5f8f7082a6a3a056ea00649.debug
Output: Missing separate debuginfo for /home/helixbot/dotnetbuild/work/4669dcde-7303-476e-aa92-88736f420e56/Payload/shared/Microsoft.NETCore.App/9.9.9/libclrjit.so
Output: Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/63/fd6de408f49e4556fd7617075183050836e703.debug
Output: Core was generated by `/home/helixbot/dotnetbuild/work/4669dcde-7303-476e-aa92-88736f420e56/Payload/do'.
Output: Program terminated with signal 6, Aborted.
Output: #0  0x00007fb4023531f7 in raise () from /lib64/libc.so.6
Output: 
Output: Thread 9 (Thread 0x7fb3fab0e700 (LWP 34829)):
Output: #0  0x00007fb3fe9d23c7 in Compiler::impImportBlockPending(BasicBlock*) () from /home/helixbot/dotnetbuild/work/4669dcde-7303-476e-aa92-88736f420e56/Payload/shared/Microsoft.NETCore.App/9.9.9/libclrjit.so
Output: No symbol table info available.
Output: dotnet/corefx#1  0x00007fb3fe972418 in Compiler::compCompile(void**, unsigned int*, JitFlags*) () from /home/helixbot/dotnetbuild/work/4669dcde-7303-476e-aa92-88736f420e56/Payload/shared/Microsoft.NETCore.App/9.9.9/libclrjit.so
Output: No symbol table info available.
Output: dotnet/corefx#2  0x00007fb3fe98be5c in Compiler::compCompileHelper(CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, CorInfoInstantiationVerification) () from /home/helixbot/dotnetbuild/work/4669dcde-7303-476e-aa92-88736f420e56/Payload/shared/Microsoft.NETCore.App/9.9.9/libclrjit.so
Output: No symbol table info available.
Output: dotnet/corefx#3  0x00007fb3fe98df5c in jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*) () from /home/helixbot/dotnetbuild/work/4669dcde-7303-476e-aa92-88736f420e56/Payload/shared/Microsoft.NETCore.App/9.9.9/libclrjit.so
Output: No symbol table info available.
Output: dotnet/corefx#4  0x00007fb3fe9902c3 in CILJit::compileMethod(ICorJitInfo*, CORINFO_METHOD_INFO*, unsigned int, unsigned char**, unsigned int*) () from /home/helixbot/dotnetbuild/work/4669dcde-7303-476e-aa92-88736f420e56/Payload/shared/Microsoft.NETCore.App/9.9.9/libclrjit.so
Output: No symbol table info available.
Output: dotnet/corefx#5  0x00007fb4016de082 in invokeCompileMethodHelper (jitMgr=0x1faa900, comp=0x7fb3fab0b328, info=<optimized out>, jitFlags=..., nativeEntry=<optimized out>, nativeSizeOfCode=<optimized out>) at /root/coreclr/src/vm/jitinterface.cpp:12181
Output:         ret = CORJIT_SKIPPED
Output: dotnet/corefx#6  0x00007fb4016de164 in invokeCompileMethod (jitMgr=0x1faa900, comp=0x7fb3fab0b328, info=0x7fb3fab0b3f8, jitFlags=..., nativeEntry=0x7fb3fab0b2f8, nativeSizeOfCode=0x7fb3fab0b2f4) at /root/coreclr/src/vm/jitinterface.cpp:12248
Output:         ret = <optimized out>
Output: dotnet/corefx#7  0x00007fb4016de275 in operator() (this=<optimized out>, pParam=<optimized out>) at /root/coreclr/src/vm/jitinterface.cpp:12300
Output: No locals.
Output: dotnet/corefx#8  CallCompileMethodWithSEHWrapper (jitMgr=0x7fb4033cc020, comp=0x7fb3fab0b328, info=0x7fb4033cdcf0, flags=..., nativeEntry=0x7fb3fab0b2f8, nativeSizeOfCode=0x2e, ftn=0x7fb4033d1e60) at /root/coreclr/src/vm/jitinterface.cpp:12330
Output: No locals.
Output: dotnet/corefx#9  0x00007fb4016deba6 in UnsafeJitFunction (ftn=<optimized out>, ILHeader=0x7fb3fab0b6a0, flags=..., pSizeOfCode=0x7fb3fab0b7cc) at /root/coreclr/src/vm/jitinterface.cpp:12795
Output:         pMethodForSecurity = <optimized out>
Output:         sizeOfCode = <optimized out>
Output:         ftnHnd = <optimized out>
Output:         jitMgr = <optimized out>
Output:         reserveForJumpStubs = 0
Output:         CURRENT_THREAD = <optimized out>
Output:         __pThread = <optimized out>
Output:         ret = 0
Output: dotnet/corefx#10 0x00007fb4018b9bde in MethodDesc::JitCompileCodeLocked (this=0x7fb388abaaf8, pConfig=0x7fb3fab0b8d0, pEntry=0x7fb34801f290, pSizeOfCode=0x7fb3fab0b7cc, pFlags=0x7fb3fab0b7c0) at /root/coreclr/src/vm/prestub.cpp:839
Output:         __exceptionHolder = <optimized out>
Output:         pOtherCode = 0
Output:         pilHeader = 0x7fb3fab0b6a0
Output: dotnet/corefx#11 0x00007fb4018b990e in MethodDesc::JitCompileCodeLockedEventWrapper (this=0x7fb388abaaf8, pConfig=0x7fb3fab0b8d0, pEntry=0x7fb34801f290) at /root/coreclr/src/vm/prestub.cpp:696
Output:         sizeOfCode = 1290634573
Output:         pCode = <optimized out>
Output:         flags = <error reading variable flags (Cannot access memory at address 0x1)>
Output: dotnet/corefx#12 0x00007fb4018b9130 in MethodDesc::JitCompileCode (this=0x7fb388abaaf8, pConfig=0x7fb3fab0b8d0) at /root/coreclr/src/vm/prestub.cpp:636
Output:         pCode = <optimized out>
Output: dotnet/corefx#13 0x00007fb4018b8eb7 in MethodDesc::PrepareILBasedCode (this=0x7fb388abaaf8, pConfig=0x7fb3fab0b8d0) at /root/coreclr/src/vm/prestub.cpp:310
Output:         pCode = <optimized out>
Output: dotnet/corefx#14 0x00007fb4018bb45a in PrepareCode (this=0x7fb388abaaf8, pConfig=0x7fb4033d1e60) at /root/coreclr/src/vm/prestub.cpp:292
Output: No locals.
Output: dotnet/corefx#15 PrepareInitialCode (this=0x7fb388abaaf8) at /root/coreclr/src/vm/prestub.cpp:248
Output:         config = {_vptr$PrepareCodeConfig = 0x0, m_pMethodDesc = 0x0, m_nativeCodeVersion = {m_storageKind = (unknown: 54337384), {m_pVersionNode = 0x7fb4033d1d58, m_synthetic = {m_pMethodDesc = 0x7fb4033d1d58}}}, m_needsMulticoreJitNotification = 0, m_mayUsePrecompiledCode = 0, m_ProfilerRejectedPrecompiledCode = 31, m_ReadyToRunRejectedPrecompiledCode = 0}
Output:         pCode = <optimized out>
Output: dotnet/corefx#16 MethodDesc::DoPrestub (this=0x7fb388abaaf8, pDispatchingMT=<optimized out>) at /root/coreclr/src/vm/prestub.cpp:1805
Output:         pCode = <optimized out>
Output:         pThread = <optimized out>
Output:         fCanBackpatchPrestub = <optimized out>
Output:         fWasPromotedToTier1 = <optimized out>
Output:         fEligibleForTieredCompilation = <optimized out>
Output:         pTieredCompilationManager = 0x0
Output:         fIsPointingToPrestub = <optimized out>
Output:         pStub = <optimized out>
Output: dotnet/runtime#13850 0x00007fb4018badc4 in PreStubWorker (pTransitionBlock=<optimized out>, pMD=0x7fb388abaaf8) at /root/coreclr/src/vm/prestub.cpp:1546
Output:         pDispatchingMT = <optimized out>
Output:         CURRENT_THREAD = 0x211f120
Output:         __fExceptionCatched = false
Output:         __pThread = 0x211f120
Output:         __pUnCEntryFrame = 0x7fb3fab0b970
Output:         CURRENT_THREAD = 0x211f120
Output:         pPFrame = <optimized out>
Output:         __pThread = 0x211f120
Output:         hasCaughtException = false
Output:         __dwLastError = 0
Output:         pbRetVal = 0
Output: dotnet/runtime#13851 0x00007fb40183d264 in ThePreStub () from /home/helixbot/dotnetbuild/work/4669dcde-7303-476e-aa92-88736f420e56/Payload/shared/Microsoft.NETCore.App/9.9.9/libcoreclr.so
Output: No symbol table info available.
Output: dotnet/corefx#19 0x00007fb3889f2aa2 in ?? ()
Output: No symbol table info available.
Output: dotnet/runtime#13852 0x00007fb360086008 in ?? ()
Output: No symbol table info available.
Output: dotnet/corefx#21 0x00007fb3600858a0 in ?? ()
Output: No symbol table info available.
Output: dotnet/runtime#13853 0x00007fb3fab0bb40 in ?? ()
Output: No symbol table info available.
Output: dotnet/corefx#23 0x00007fb3889eccda in ?? ()
Output: No symbol table info available.
Output: dotnet/corefx#24 0x00007fb3fab0bb00 in ?? ()
Output: No symbol table info available.
Output: dotnet/corefx#25 0x0000000000000000 in ?? ()
Output: No symbol table info available.
Output: 
Output: Thread 8 (Thread 0x7fb3fbbb9700 (LWP 34827)):
Output: #0  0x00007fb402f0c945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
Output: No symbol table info available.
Output: dotnet/corefx#1  0x00007fb401ab8fc2 in CorUnix::CPalSynchronizationManager::ThreadNativeWait (ptnwdNativeWaitData=<optimized out>, dwTimeout=<optimized out>, ptwrWakeupReason=<optimized out>, pdwSignaledObject=<optimized out>) at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:479
Output:         iWaitRet = <optimized out>
Output:         palErr = 1359
Output:         iRet = <optimized out>
Output:         tsAbsTmo = <optimized out>
Output: dotnet/corefx#2  0x00007fb401ab8ba4 in CorUnix::CPalSynchronizationManager::BlockThread (this=0x1f53dd0, pthrCurrent=0x20660d0, dwTimeout=4294967295, fAlertable=true, fIsSleep=<optimized out>, ptwrWakeupReason=0x7fb3fbbb8380, pdwSignaledObject=0xfffffffffffffe00) at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:302
Output:         twrWakeupReason = <error reading variable twrWakeupReason (Cannot access memory at address 0x17)>
Output:         palErr = 0
Output:         dwSigObjIdx = <error reading variable dwSigObjIdx (Cannot access memory at address 0xffffffffffffffff)>
Output:         dwWaitState = <optimized out>
Output:         fEarlyDeath = false
Output:         fRaceAlerted = true
Output:         pdwWaitState = <optimized out>
Output: dotnet/corefx#3  0x00007fb401abd574 in CorUnix::InternalWaitForMultipleObjectsEx (pThread=0x20660d0, nCount=<optimized out>, lpHandles=<optimized out>, bWaitAll=<optimized out>, dwMilliseconds=<optimized out>, bAlertable=<optimized out>, bPrioritize=<optimized out>) at /root/coreclr/src/pal/src/synchmgr/wait.cpp:636
Output:         twrWakeupReason = <error reading variable twrWakeupReason (Cannot access memory at address 0xb)>
Output:         pIPalObjStackArray = {0x2064d60, 0x0 <repeats 15 times>}
Output:         pISyncStackArray = {0x0 <repeats 16 times>}
Output:         iSignaledObjIndex = <optimized out>
Output:         palErr = <optimized out>
Output:         dwRet = 4294967295
Output:         fAbandoned = <optimized out>
Output:         fNeedToBlock = <optimized out>
Output:         fWAll = <optimized out>
Output:         ppISyncWaitCtrlrs = <optimized out>
Output:         ppIPalObjs = <optimized out>
Output:         wtWaitType = <optimized out>
Output:         iSignaledObjCount = 1
Output: dotnet/corefx#4  0x00007fb401abd8f1 in WaitForMultipleObjectsEx (nCount=1, lpHandles=0x7fb3fbbb8690, bWaitAll=1, dwMilliseconds=4294967295, bAlertable=1) at /root/coreclr/src/pal/src/synchmgr/wait.cpp:202
Output:         pThread = 0xfffffffffffffe00
Output:         dwRet = <optimized out>
Output: dotnet/corefx#5  0x00007fb401729ca1 in WaitForMultipleObjectsEx_SO_TOLERANT (nCount=<optimized out>, lpHandles=0x7fb3fbbb8690, bWaitAll=1, bAlertable=1, dwMilliseconds=<optimized out>) at /root/coreclr/src/vm/threads.cpp:3463
Output:         lastError = 0
Output: dotnet/corefx#6  DoAppropriateAptStateWait (numWaiters=1, timeout=<optimized out>, this=<optimized out>, pHandles=<optimized out>, bWaitAll=<optimized out>, mode=<optimized out>) at /root/coreclr/src/vm/threads.cpp:3497
Output:         alertable = 1
Output: dotnet/corefx#7  Thread::DoAppropriateWaitWorker (this=0x2065120, countHandles=<optimized out>, handles=<optimized out>, waitAll=1, millis=<optimized out>, mode=WaitMode_None) at /root/coreclr/src/vm/threads.cpp:3634
Output:         ret = 0
Output:         alertable = 1
Output:         dwEnd = <optimized out>
Output: dotnet/corefx#8  0x00007fb401724bf0 in operator() (this=<optimized out>, __pEEParam=<optimized out>) at /root/coreclr/src/vm/threads.cpp:3308
Output: No locals.
Output: dotnet/corefx#9  Thread::DoAppropriateWait (this=0x2066334, countHandles=1, handles=0x17, waitAll=-1, millis=33972992, mode=(WaitMode_Alertable | WaitMode_IgnoreSyncCtx | WaitMode_InDeadlock), syncState=<optimized out>) at /root/coreclr/src/vm/threads.cpp:3323
Output: No locals.
Output: dotnet/corefx#10 0x00007fb401775b20 in WaitHandleNative::CorWaitOneNative (safeWaitHandleUNSAFE=<optimized out>, timeout=<optimized out>, hasThreadAffinity=<optimized out>, exitContext=<optimized out>) at /root/coreclr/src/vm/comwaithandle.cpp:198
Output:         sPauseTime = 0
Output:         handles = {0x1ec}
Output:         pThread = 0x2065120
Output:         res = 4294967295
Output:         __fExceptionCatched = false
Output:         CURRENT_THREAD = 0x2065120
Output:         hasCaughtException = false
Output:         sh = <optimized out>
Output:         alwaysZero = 0
Output: dotnet/corefx#11 0x00007fb3886c6737 in ?? ()
Output: No symbol table info available.
Output: dotnet/corefx#12 0x00007fb360eadc20 in ?? ()
Output: No symbol table info available.
Output: dotnet/corefx#13 0x00007fb360028710 in ?? ()
Output: No symbol table info available.
Output: dotnet/corefx#14 0x00007fb3fbbb8810 in ?? ()
Output: No symbol table info available.
Output: dotnet/corefx#15 0x00007fb3886c5e74 in ?? ()
Output: No symbol table info available.
Output: dotnet/corefx#16 0x00007fb3fbbb87f0 in ?? ()
Output: No symbol table info available.
Output: dotnet/runtime#13850 0x00007fb38816d743 in ?? ()
Output: No symbol table info available.
Output: dotnet/runtime#13851 0x00007fb360075be0 in ?? ()
Output: No symbol table info available.
Output: dotnet/corefx#19 0x0000000000000000 in ?? ()
Output: No symbol table info available.
Output: 
Output: Thread 7 (Thread 0x7fb3fc7b2700 (LWP 34824)):
Output: #0  0x00007fb402f0f70d in read () from /lib64/libpthread.so.0
Output: No symbol table info available.
Output: dotnet/corefx#1  0x00007fb3fc7c4d8f in SignalHandlerLoop (arg=<optimized out>) at /mnt/j/workspace/dotnet_corefx/master/linux-TGroup_netcoreapp+CGroup_Release+AGroup_x64+TestOuter_false_prtest/src/Native/Unix/System.Native/pal_signal.c:100
Output:         pipeFd = 45
Output: dotnet/corefx#2  0x00007fb402f08e25 in start_thread () from /lib64/libpthread.so.0
Output: No symbol table info available.
Output: dotnet/corefx#3  0x00007fb40241634d in clone () from /lib64/libc.so.6
Output: No symbol table info available.
Output: 
Output: Thread 6 (Thread 0x7fb3ff3f6700 (LWP 34823)):
Output: #0  0x00007fb402f0ccf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
Output: No symbol table info available.
Output: dotnet/corefx#1  0x00007fb401ab8f85 in CorUnix::CPalSynchronizationManager::ThreadNativeWait (ptnwdNativeWaitData=<optimized out>, dwTimeout=<optimized out>, ptwrWakeupReason=<optimized out>, pdwSignaledObject=<optimized out>) at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:484
Output:         iWaitRet = 0
Output:         palErr = 1359
Output:         iRet = <optimized out>
Output:         tsAbsTmo = {tv_sec = 513, tv_nsec = 225937367}
Output: dotnet/corefx#2  0x00007fb401ab8ba4 in CorUnix::CPalSynchronizationManager::BlockThread (this=0x1f53dd0, pthrCurrent=0x201ab70, dwTimeout=2000, fAlertable=false, fIsSleep=<optimized out>, ptwrWakeupReason=0x7fb3ff3f5a50, pdwSignaledObject=0xfffffffffffffdfc) at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:302
Output:         twrWakeupReason = <error reading variable twrWakeupReason (Cannot access memory at address 0x3)>
Output:         palErr = 0
Output:         dwSigObjIdx = <error reading variable dwSigObjIdx (Cannot access memory at address 0xffffffffffffffff)>
Output:         dwWaitState = <optimized out>
Output:         fEarlyDeath = false
Output:         fRaceAlerted = true
Output:         pdwWaitState = <optimized out>
Output: dotnet/corefx#3  0x00007fb401abd574 in CorUnix::InternalWaitForMultipleObjectsEx (pThread=0x201ab70, nCount=<optimized out>, lpHandles=<optimized out>, bWaitAll=<optimized out>, dwMilliseconds=<optimized out>, bAlertable=<optimized out>, bPrioritize=<optimized out>) at /root/coreclr/src/pal/src/synchmgr/wait.cpp:636
Output:         twrWakeupReason = <error reading variable twrWakeupReason (Cannot access memory at address 0xffffffff)>
Output:         pIPalObjStackArray = {0x20196e0, 0x0 <repeats 15 times>}
Output:         pISyncStackArray = {0x0 <repeats 16 times>}
Output:         iSignaledObjIndex = <optimized out>
Output:         palErr = <optimized out>
Output:         dwRet = 4294967295
Output:         fAbandoned = <optimized out>
Output:         fNeedToBlock = <optimized out>
Output:         fWAll = <optimized out>
Output:         ppISyncWaitCtrlrs = <optimized out>
Output:         ppIPalObjs = <optimized out>
Output:         wtWaitType = <optimized out>
Output:         iSignaledObjCount = 1
Output: dotnet/corefx#4  0x00007fb401abd819 in WaitForSingleObjectEx (hHandle=<error reading variable: Cannot access memory at address 0x3>, dwMilliseconds=2000, bAlertable=0) at /root/coreclr/src/pal/src/synchmgr/wait.cpp:139
Output:         pThread = 0xfffffffffffffdfc
Output:         dwRet = <optimized out>
Output: dotnet/corefx#5  0x00007fb401801af1 in CLREventWaitHelper2 (handle=0x201add4, dwMilliseconds=<optimized out>, alertable=<optimized out>) at /root/coreclr/src/vm/synch.cpp:385
Output: No locals.
Output: dotnet/corefx#6  operator() (this=<optimized out>, pParam=<optimized out>) at /root/coreclr/src/vm/synch.cpp:411
Output: No locals.
Output: dotnet/corefx#7  CLREventWaitHelper (dwMilliseconds=2000, alertable=<optimized out>, handle=<optimized out>) at /root/coreclr/src/vm/synch.cpp:413
Output:         __exceptionHolder = <optimized out>
Output:         __param = 0x7fb3ff3f5c30
Output:         disposition = -1
Output: dotnet/corefx#8  CLREventBase::WaitEx (this=0x1f57240, dwMilliseconds=2000, mode=<optimized out>, syncState=0x0) at /root/coreclr/src/vm/synch.cpp:483
Output:         alertable = <optimized out>
Output:         pThread = <optimized out>
Output: dotnet/corefx#9  0x00007fb401791daf in FinalizerThread::WaitForFinalizerEvent (event=0x1f57240) at /root/coreclr/src/vm/finalizerthread.cpp:408
Output: No locals.
Output: dotnet/corefx#10 0x00007fb401791f55 in FinalizerThread::FinalizerThreadWorker (args=<optimized out>) at /root/coreclr/src/vm/finalizerthread.cpp:564
Output:         bPriorityBoosted = 0
Output: dotnet/corefx#11 0x00007fb40172f40d in ManagedThreadBase_DispatchInner (pCallState=0x7fb3ff3f5e20) at /root/coreclr/src/vm/threads.cpp:8851
Output:         pThread = 0x2019c80
Output: dotnet/corefx#12 ManagedThreadBase_DispatchMiddle (pCallState=0x7fb3ff3f5e20) at /root/coreclr/src/vm/threads.cpp:8902
Output: No locals.
Output: dotnet/corefx#13 operator() (this=<optimized out>, pParam=<optimized out>) at /root/coreclr/src/vm/threads.cpp:9140
Output: No locals.
Output: dotnet/corefx#14 operator() (this=<optimized out>, pArgs=<optimized out>) at /root/coreclr/src/vm/threads.cpp:9142
Output:         __exceptionHolder = <optimized out>
Output:         disposition = -1
Output:         __param = <optimized out>
Output: dotnet/corefx#15 ManagedThreadBase_DispatchOuter (pCallState=<optimized out>) at /root/coreclr/src/vm/threads.cpp:9179
Output:         pThread = 0x2019c80
Output:         param = <optimized out>
Output: dotnet/corefx#16 0x00007fb40172fbee in ManagedThreadBase_NoADTransition (pTarget=0x7fb401791ee0 <FinalizerThread::FinalizerThreadWorker(void*)>, filterType=FinalizerThread) at /root/coreclr/src/vm/threads.cpp:9224
Output:         CallState = {pAppDomainId = {m_dwId = 3}, pUnsafeAppDomain = 0x100000000, bDomainIsAsID = 0, pTarget = 0x201ada800000000, args = 0x300000000, filterType = 0, flags = MTCSF_NormalBase}
Output: dotnet/runtime#13850 ManagedThreadBase::FinalizerBase (pTarget=0x7fb401791ee0 <FinalizerThread::FinalizerThreadWorker(void*)>) at /root/coreclr/src/vm/threads.cpp:9250
Output: No locals.
Output: dotnet/runtime#13851 0x00007fb40179234c in FinalizerThread::FinalizerThreadStart (args=<optimized out>) at /root/coreclr/src/vm/finalizerthread.cpp:764
Output: No locals.
Output: dotnet/corefx#19 0x00007fb401ac3a42 in CorUnix::CPalThread::ThreadEntry (pvParam=0x201ab70) at /root/coreclr/src/pal/src/thread/thread.cpp:1684
Output:         palError = <optimized out>
Output:         pvPar = 0x201add4
Output:         retValue = <optimized out>
Output: dotnet/runtime#13852 0x00007fb402f08e25 in start_thread () from /lib64/libpthread.so.0
Output: No symbol table info available.
Output: dotnet/corefx#21 0x00007fb40241634d in clone () from /lib64/libc.so.6
Output: No symbol table info available.
Output: 
Output: Thread 5 (Thread 0x7fb3fff70700 (LWP 34822)):
Output: #0  0x00007fb402f0c945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
Output: No symbol table info available.
Output: dotnet/corefx#1  0x00007fb401ab8fc2 in CorUnix::CPalSynchronizationManager::ThreadNativeWait (ptnwdNativeWaitData=<optimized out>, dwTimeout=<optimized out>, ptwrWakeupReason=<optimized out>, pdwSignaledObject=<optimized out>) at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:479
Output:         iWaitRet = <optimized out>
Output:         palErr = 1359
Output:         iRet = <optimized out>
Output:         tsAbsTmo = <optimized out>
Output: dotnet/corefx#2  0x00007fb401ab8ba4 in CorUnix::CPalSynchronizationManager::BlockThread (this=0x1f53dd0, pthrCurrent=0x1ff4230, dwTimeout=4294967295, fAlertable=false, fIsSleep=<optimized out>, ptwrWakeupReason=0x7fb3fff6fcb0, pdwSignaledObject=0xfffffffffffffe00) at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:302
Output:         twrWakeupReason = <error reading variable twrWakeupReason (Cannot access memory at address 0x1)>
Output:         palErr = 0
Output:         dwSigObjIdx = <error reading variable dwSigObjIdx (Cannot access memory at address 0xffffffffffffffff)>
Output:         dwWaitState = <optimized out>
Output:         fEarlyDeath = false
Output:         fRaceAlerted = true
Output:         pdwWaitState = <optimized out>
Output: dotnet/corefx#3  0x00007fb401abd574 in CorUnix::InternalWaitForMultipleObjectsEx (pThread=0x1ff4230, nCount=<optimized out>, lpHandles=<optimized out>, bWaitAll=<optimized out>, dwMilliseconds=<optimized out>, bAlertable=<optimized out>, bPrioritize=<optimized out>) at /root/coreclr/src/pal/src/synchmgr/wait.cpp:636
Output:         twrWakeupReason = <error reading variable twrWakeupReason (Cannot access memory at address 0x0)>
Output:         pIPalObjStackArray = {0x1fe6a70, 0x1ff2e80, 0x1fe6890, 0x0 <repeats 13 times>}
Output:         pISyncStackArray = {0x0 <repeats 16 times>}
Output:         iSignaledObjIndex = <optimized out>
Output:         palErr = <optimized out>
Output:         dwRet = 4294967295
Output:         fAbandoned = <optimized out>
Output:         fNeedToBlock = <optimized out>
Output:         fWAll = <optimized out>
Output:         ppISyncWaitCtrlrs = <optimized out>
Output:         ppIPalObjs = <optimized out>
Output:         wtWaitType = <optimized out>
Output:         iSignaledObjCount = 1
Output: dotnet/corefx#4  0x00007fb401abd8f1 in WaitForMultipleObjectsEx (nCount=3, lpHandles=0x7fb3fff6fe50, bWaitAll=0, dwMilliseconds=4294967295, bAlertable=0) at /root/coreclr/src/pal/src/synchmgr/wait.cpp:202
Output:         pThread = 0xfffffffffffffe00
Output:         dwRet = <optimized out>
Output: dotnet/corefx#5  0x00007fb4019bdad8 in DebuggerRCThread::MainLoop (this=<optimized out>) at /root/coreclr/src/debug/ee/rcthread.cpp:1234
Output:         rghWaitSet = {0x44, 0x4c, 0x40, 0x1f51ab0}
Output: dotnet/corefx#6  0x00007fb4019bd96b in DebuggerRCThread::ThreadProc (this=0x1fe4650) at /root/coreclr/src/debug/ee/rcthread.cpp:1037
Output: No locals.
Output: dotnet/corefx#7  0x00007fb4019bd644 in DebuggerRCThread::ThreadProcStatic () at /root/coreclr/src/debug/ee/rcthread.cpp:1633
Output:         fSuccess = <optimized out>
Output:         t = 0x1ff4494
Output: dotnet/corefx#8  0x00007fb401ac3a42 in CorUnix::CPalThread::ThreadEntry (pvParam=0x1ff4230) at /root/coreclr/src/pal/src/thread/thread.cpp:1684
Output:         palError = <optimized out>
Output:         pvPar = 0x1ff4494
Output:         retValue = <optimized out>
Output: dotnet/corefx#9  0x00007fb402f08e25 in start_thread () from /lib64/libpthread.so.0
Output: No symbol table info available.
Output: dotnet/corefx#10 0x00007fb40241634d in clone () from /lib64/libc.so.6
Output: No symbol table info available.
Output: 
Output: Thread 4 (Thread 0x7fb400771700 (LWP 34821)):
Output: #0  0x00007fb402f0fead in open64 () from /lib64/libpthread.so.0
Output: No symbol table info available.
Output: dotnet/corefx#1  0x00007fb4019c793f in TwoWayPipe::WaitForConnection (this=0x1fe8f10) at /root/coreclr/src/debug/debug-pal/unix/twowaypipe.cpp:90
Output: No locals.
Output: dotnet/corefx#2  0x00007fb4019c0618 in DbgTransportSession::TransportWorker (this=0x1fe8e30) at /root/coreclr/src/debug/ee/../shared/dbgtransportsession.cpp:1320
Output:         pid = <optimized out>
Output: dotnet/corefx#3  0x00007fb4019bf4c9 in DbgTransportSession::TransportWorkerStatic (pvContext=0x1fe8f20) at /root/coreclr/src/debug/ee/../shared/dbgtransportsession.cpp:1236
Output: No locals.
Output: dotnet/corefx#4  0x00007fb401ac3a42 in CorUnix::CPalThread::ThreadEntry (pvParam=0x1ff3240) at /root/coreclr/src/pal/src/thread/thread.cpp:1684
Output:         palError = <optimized out>
Output:         pvPar = 0x1fe8f20
Output:         retValue = <optimized out>
Output: dotnet/corefx#5  0x00007fb402f08e25 in start_thread () from /lib64/libpthread.so.0
Output: No symbol table info available.
Output: dotnet/corefx#6  0x00007fb40241634d in clone () from /lib64/libc.so.6
Output: No symbol table info available.
Output: 
Output: Thread 3 (Thread 0x7fb403533740 (LWP 34819)):
Output: #0  0x00007fb402f0c945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
Output: No symbol table info available.
Output: dotnet/corefx#1  0x00007fb401ab8fc2 in CorUnix::CPalSynchronizationManager::ThreadNativeWait (ptnwdNativeWaitData=<optimized out>, dwTimeout=<optimized out>, ptwrWakeupReason=<optimized out>, pdwSignaledObject=<optimized out>) at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:479
Output:         iWaitRet = <optimized out>
Output:         palErr = 1359
Output:         iRet = <optimized out>
Output:         tsAbsTmo = <optimized out>
Output: dotnet/corefx#2  0x00007fb401ab8ba4 in CorUnix::CPalSynchronizationManager::BlockThread (this=0x1f53dd0, pthrCurrent=0x1f29da0, dwTimeout=4294967295, fAlertable=true, fIsSleep=<optimized out>, ptwrWakeupReason=0x7fff721f5470, pdwSignaledObject=0xfffffffffffffe00) at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:302
Output:         twrWakeupReason = <error reading variable twrWakeupReason (Cannot access memory at address 0x3)>
Output:         palErr = 0
Output:         dwSigObjIdx = <error reading variable dwSigObjIdx (Cannot access memory at address 0xffffffffffffffff)>
Output:         dwWaitState = <optimized out>
Output:         fEarlyDeath = false
Output:         fRaceAlerted = true
Output:         pdwWaitState = <optimized out>
Output: dotnet/corefx#3  0x00007fb401abd574 in CorUnix::InternalWaitForMultipleObjectsEx (pThread=0x1f29da0, nCount=<optimized out>, lpHandles=<optimized out>, bWaitAll=<optimized out>, dwMilliseconds=<optimized out>, bAlertable=<optimized out>, bPrioritize=<optimized out>) at /root/coreclr/src/pal/src/synchmgr/wait.cpp:636
Output:         twrWakeupReason = <error reading variable twrWakeupReason (Cannot access memory at address 0x1)>
Output:         pIPalObjStackArray = {0x2061850, 0x0 <repeats 15 times>}
Output:         pISyncStackArray = {0x0 <repeats 16 times>}
Output:         iSignaledObjIndex = <optimized out>
Output:         palErr = <optimized out>
Output:         dwRet = 4294967295
Output:         fAbandoned = <optimized out>
Output:         fNeedToBlock = <optimized out>
Output:         fWAll = <optimized out>
Output:         ppISyncWaitCtrlrs = <optimized out>
Output:         ppIPalObjs = <optimized out>
Output:         wtWaitType = <optimized out>
Output:         iSignaledObjCount = 1
Output: dotnet/corefx#4  0x00007fb401abd8f1 in WaitForMultipleObjectsEx (nCount=1, lpHandles=0x7fff721f5780, bWaitAll=1, dwMilliseconds=4294967295, bAlertable=1) at /root/coreclr/src/pal/src/synchmgr/wait.cpp:202
Output:         pThread = 0xfffffffffffffe00
Output:         dwRet = <optimized out>
Output: dotnet/corefx#5  0x00007fb401729ca1 in WaitForMultipleObjectsEx_SO_TOLERANT (nCount=<optimized out>, lpHandles=0x7fff721f5780, bWaitAll=1, bAlertable=1, dwMilliseconds=<optimized out>) at /root/coreclr/src/vm/threads.cpp:3463
Output:         lastError = 0
Output: dotnet/corefx#6  DoAppropriateAptStateWait (numWaiters=1, timeout=<optimized out>, this=<optimized out>, pHandles=<optimized out>, bWaitAll=<optimized out>, mode=<optimized out>) at /root/coreclr/src/vm/threads.cpp:3497
Output:         alertable = 1
Output: dotnet/corefx#7  Thread::DoAppropriateWaitWorker (this=0x1ff7050, countHandles=<optimized out>, handles=<optimized out>, waitAll=1, millis=<optimized out>, mode=WaitMode_None) at /root/coreclr/src/vm/threads.cpp:3634
Output:         ret = 0
Output:         alertable = 1
Output:         dwEnd = <optimized out>
Output: dotnet/corefx#8  0x00007fb401724bf0 in operator() (this=<optimized out>, __pEEParam=<optimized out>) at /root/coreclr/src/vm/threads.cpp:3308
Output: No locals.
Output: dotnet/corefx#9  Thread::DoAppropriateWait (this=0x1f2a004, countHandles=1, handles=0x3, waitAll=-1, millis=32677632, mode=WaitMode_Alertable, syncState=<optimized out>) at /root/coreclr/src/vm/threads.cpp:3323
Output: No locals.
Output: dotnet/corefx#10 0x00007fb401775b20 in WaitHandleNative::CorWaitOneNative (safeWaitHandleUNSAFE=<optimized out>, timeout=<optimized out>, hasThreadAffinity=<optimized out>, exitContext=<optimized out>) at /root/coreclr/src/vm/comwaithandle.cpp:198
Output:         sPauseTime = 0
Output:         handles = {0x1e8}
Output:         pThread = 0x1ff7050
Output:         res = 4294967295
Output:         __fExceptionCatched = false
Output:         CURRENT_THREAD = 0x1ff7050
Output:         hasCaughtException = false
Output:         sh = <optimized out>
Output:         alwaysZero = 0
Output: dotnet/corefx#11 0x00007fb3881f4bfc in ?? ()
Output: No symbol table info available.
Output: dotnet/corefx#12 0x0000000000000001 in ?? ()
Output: No symbol table info available.
Output: dotnet/corefx#13 0x00007fb360052528 in ?? ()
Output: No symbol table info available.
Output: dotnet/corefx#14 0x00007fff721f5a00 in ?? ()
Output: No symbol table info available.
Output: dotnet/corefx#15 0x00007fb3886b5545 in ?? ()
Output: No symbol table info available.
Output: dotnet/corefx#16 0x0000000000000000 in ?? ()
Output: No symbol table info available.
Output: 
Output: Thread 2 (Thread 0x7fb400f72700 (LWP 34820)):
Output: #0  0x00007fb40240ba3d in poll () from /lib64/libc.so.6
Output: No symbol table info available.
Output: dotnet/corefx#1  0x00007fb401ab92de in CorUnix::CPalSynchronizationManager::ThreadPrepareForShutdown () at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:3220
Output: No locals.
Output: dotnet/corefx#2  0x00007fb401abac70 in CorUnix::CPalSynchronizationManager::WorkerThread (pArg=<optimized out>) at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:1960
Output:         pthrWorker = 0x1f54320
Output:         palErr = 0
Output:         twrWakeUpReason = CorUnix::MutexAbondoned
Output: dotnet/corefx#3  0x00007fb401ac3a42 in CorUnix::CPalThread::ThreadEntry (pvParam=0x1f54320) at /root/coreclr/src/pal/src/thread/thread.cpp:1684
Output:         palError = <optimized out>
Output:         pvPar = 0x0
Output:         retValue = <optimized out>
Output: dotnet/corefx#4  0x00007fb402f08e25 in start_thread () from /lib64/libpthread.so.0
Output: No symbol table info available.
Output: dotnet/corefx#5  0x00007fb40241634d in clone () from /lib64/libc.so.6
Output: No symbol table info available.
Output: 
Output: Thread 1 (Thread 0x7fb3fb3b8700 (LWP 34828)):
Output: #0  0x00007fb4023531f7 in raise () from /lib64/libc.so.6
Output: No symbol table info available.
Output: dotnet/corefx#1  0x00007fb4023548e8 in abort () from /lib64/libc.so.6
Output: No symbol table info available.
Output: dotnet/corefx#2  0x00007fb401ac1d53 in PROCAbort () at /root/coreclr/src/pal/src/thread/process.cpp:3068
Output: No locals.
Output: dotnet/corefx#3  0x00007fb401ac0bbb in PROCEndProcess (hProcess=<optimized out>, uExitCode=3221225477, bTerminateUnconditionally=1) at /root/coreclr/src/pal/src/thread/process.cpp:1394
Output:         dwProcessId = <optimized out>
Output: dotnet/corefx#4  0x00007fb40177a79e in SafeExitProcess (exitCode=3221225477, fAbort=1, sca=SCA_ExitProcessWhenShutdownComplete) at /root/coreclr/src/vm/eepolicy.cpp:519
Output: No locals.
Output: dotnet/corefx#5  0x00007fb40177bc54 in EEPolicy::HandleFatalError (exitCode=<optimized out>, address=<optimized out>, pszMessage=0x0, pExceptionInfo=<optimized out>, errorSource=0x0) at /root/coreclr/src/vm/eepolicy.cpp:1532
Output:         context = <optimized out>
Output: dotnet/corefx#6  0x00007fb40182bc4b in ProcessCLRException (pExceptionRecord=<optimized out>, MemoryStackFp=<optimized out>, pContextRecord=0x7fb354017bf0, pDispatcherContext=<optimized out>) at /root/coreclr/src/vm/exceptionhandling.cpp:1029
Output:         bAsynchronousThreadStop = <optimized out>
Output:         STState = <optimized out>
Output:         pTracker = 0x1ff5050
Output:         status = <optimized out>
Output:         dwLastError = 0
Output:         dwExceptionFlags = 0
Output:         pThread = 0x211d3e0
Output: dotnet/corefx#7  0x00007fb40182fcd4 in UnwindManagedExceptionPass1 (ex=..., frameContext=<optimized out>) at /root/coreclr/src/vm/exceptionhandling.cpp:4630
Output:         codeInfo = <optimized out>
Output:         establisherFrame = 34828
Output:         controlPc = <optimized out>
Output:         disposition = 0
Output:         unwindStartContext = <optimized out>
Output: dotnet/corefx#8  0x00007fb40182ffd3 in DispatchManagedException (ex=..., isHardwareException=<optimized out>) at /root/coreclr/src/vm/exceptionhandling.cpp:4752
Output: No locals.
Output: dotnet/corefx#9  0x00007fb40182a874 in HandleHardwareException (ex=<optimized out>) at /root/coreclr/src/vm/exceptionhandling.cpp:5275
Output:         controlPc = <optimized out>
Output: dotnet/corefx#10 0x00007fb401a8a0c1 in SEHProcessException (exception=<optimized out>) at /root/coreclr/src/pal/src/exception/seh.cpp:286
Output:         exceptionRecord = <optimized out>
Output:         contextRecord = <optimized out>
Output: dotnet/corefx#11 0x00007fb401a8b90f in common_signal_handler (code=<optimized out>, siginfo=<optimized out>, sigcontext=<optimized out>, numParams=<optimized out>) at /root/coreclr/src/pal/src/exception/signal.cpp:897
Output:         params = {{gp_offset = 48, fp_offset = 48, overflow_arg_area = 0x7fb3fb3b5bd0, reg_save_area = 0x7fb3fb3b54c0}}
Output:         ucontext = <optimized out>
Output:         signalContextRecord = {P1Home = 140411120139856, P2Home = 140411093605856, P3Home = 21474836480, P4Home = 4294967427, P5Home = 140409069025536, P6Home = 140410990843880, ContextFlags = 135266379, MxCsr = 32691, SegCs = 51, SegDs = 0, SegEs = 0, SegFs = 0, SegGs = 3896, SegSs = 43, EFlags = 66050, Dr0 = 32705568, Dr1 = 32704704, Dr2 = 140411124494303, Dr3 = 0, Dr6 = 32706360, Dr7 = 103079215109, Rax = 16, Rcx = 140408387076792, Rdx = 21474836479, Rbx = 140410990845704, Rsp = 140410990845216, Rbp = 140410990845424, Rsi = 140410990845864, Rdi = 140408402215131, R8 = 42, R9 = 0, R10 = 16134982712070313, R11 = 10573920402283352473, R12 = 140408387030956, R13 = 140410990845704, R14 = 1, R15 = 4294967295, Rip = 140409068080350, {FltSave = {ControlWord = 895, StatusWord = 16384, TagWord = 0 '\000', Reserved1 = 0 '\000', ErrorOpcode = 0, ErrorOffset = 4265535571, ErrorSelector = 0, Reserved2 = 0, DataOffset = 1914652760, DataSelector = 0, Reserved3 = 0, MxCsr = 8096, MxCsr_Mask = 65535, FloatRegisters = {{Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 14339461213547659264, High = 16390}, {Low = 13835058055282163712, High = 16385}, {Low = 13835058055282163712, High = 16385}}, XmmRegisters = {{Low = 0, High = 0}, {Low = 19989382812925998, High = 144115188075856128}, {Low = 0, High = 0}, {Low = 18446744073709551615, High = -1}, {Low = 35747867511423103, High = 35747867511423103}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 18446744073709551615, High = -1}, {Low = 1082331758844, High = 1082331758844}, {Low = 30064771079, High = 30064771079}, {Low = 9223372039002259456, High = -9223372034707292160}, {Low = 9223372051887161347, High = -9223372021822390269}, {Low = 4294967297, High = 4294967297}, {Low = 35, High = 35}}, Reserved4 = '\000' <repeats 95 times>}, {Header = {{Low = 1073742719, High = 4265535571}, {Low = 1914652760, High = 281470681751456}}, Legacy = {{Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 14339461213547659264, High = 16390}, {Low = 13835058055282163712, High = 16385}, {Low = 13835058055282163712, High = 16385}}, Xmm0 = {Low = 0, High = 0}, Xmm1 = {Low = 19989382812925998, High = 144115188075856128}, Xmm2 = {Low = 0, High = 0}, Xmm3 = {Low = 18446744073709551615, High = -1}, Xmm4 = {Low = 35747867511423103, High = 35747867511423103}, Xmm5 = {Low = 0, High = 0}, Xmm6 = {Low = 0, High = 0}, Xmm7 = {Low = 0, High = 0}, Xmm8 = {Low = 0, High = 0}, Xmm9 = {Low = 18446744073709551615, High = -1}, Xmm10 = {Low = 1082331758844, High = 1082331758844}, Xmm11 = {Low = 30064771079, High = 30064771079}, Xmm12 = {Low = 9223372039002259456, High = -9223372034707292160}, Xmm13 = {Low = 9223372051887161347, High = -9223372021822390269}, Xmm14 = {Low = 4294967297, High = 4294967297}, Xmm15 = {Low = 35, High = 35}}}, VectorRegister = {{Low = 0, High = 0} <repeats 16 times>, {Low = 2, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}}, VectorControl = 0, DebugControl = 0, LastBranchToRip = 0, LastBranchFromRip = 0, LastExceptionToRip = 0, LastExceptionFromRip = 0}
Output:         signal_set = {__val = {1024, 0 <repeats 15 times>}}
Output:         exception = <error reading variable exception (Cannot access memory at address 0x8803)>
Output: dotnet/corefx#12 0x00007fb401a8b676 in signal_handler_worker (code=34819, siginfo=0x7fb3540090f0, context=0x6, returnPoint=0x7fb354008a30) at /root/coreclr/src/pal/src/exception/signal.cpp:436
Output:         signal_set = {__val = {8589934592, 0 <repeats 15 times>}}
Output: dotnet/corefx#13 0x00007fb401ac59f6 in CallSignalHandlerWrapper0 () from /home/helixbot/dotnetbuild/work/4669dcde-7303-476e-aa92-88736f420e56/Payload/shared/Microsoft.NETCore.App/9.9.9/libcoreclr.so
Output: No symbol table info available.
Output: dotnet/corefx#14 0x00007fb388a04cde in ?? ()
Output: No symbol table info available.
Output: dotnet/corefx#15 0x0047043cddd8002e in ?? ()
Output: No symbol table info available.
Output: dotnet/corefx#16 0x0000000000000000 in ?? ()
Output: No symbol table info available.
Output: Copying core file core.34819 to /tmp/coredumps/core.4087 in case you need it.
Exit Code: 134

stephentoub on 10 Mar 2018

Failed again here:
https://mc.dot.net/#/user/stephentoub/pr~2Fjenkins~2Fdotnet~2Fcorefx~2Fmaster~2F/test~2Ffunctional~2Fcli~2F/b89daaa1c196fadf8b4e6fcb3b3a30111eb4374a/workItem/System.Xml.XmlSchema.XmlSchemaValidatorApi.Tests/wilogs

2018-03-10 10:47:29,084: INFO: proc(54): run_and_log_output: Output: Discovering: System.Xml.XmlSchema.XmlSchemaValidatorApi.Tests
2018-03-10 10:47:29,281: INFO: proc(54): run_and_log_output: Output: Discovered:  System.Xml.XmlSchema.XmlSchemaValidatorApi.Tests
2018-03-10 10:47:29,394: INFO: proc(54): run_and_log_output: Output: Starting:    System.Xml.XmlSchema.XmlSchemaValidatorApi.Tests
2018-03-10 10:47:33,099: INFO: proc(54): run_and_log_output: Output: /home/helixbot/dotnetbuild/work/d61ec0d4-1ff2-490c-be87-d60d9eeee107/Work/acf9dc1a-3dd4-4e57-91c2-cfce35655b32/Unzip/RunTests.sh: line 87:  1504 Aborted                 (core dumped) $RUNTIME_PATH/dotnet xunit.console.netcore.exe System.Xml.XmlSchema.XmlSchemaValidatorApi.Tests.dll -xml testResults.xml -notrait Benchmark=true -notrait category=nonnetcoreapptests -notrait category=nonlinuxtests -notrait category=OuterLoop -notrait category=failing
2018-03-10 10:47:33,153: INFO: proc(54): run_and_log_output: Output: Dumpling has not been installed yet. Please run "DumplingHelper.py install_dumpling" before collect_dumps.
2018-03-10 10:47:33,158: INFO: proc(54): run_and_log_output: Output: ~/dotnetbuild/work/d61ec0d4-1ff2-490c-be87-d60d9eeee107/Work/acf9dc1a-3dd4-4e57-91c2-cfce35655b32/Unzip
2018-03-10 10:47:33,160: INFO: proc(54): run_and_log_output: Output: ----- end 10:47:33 ----- exit code 134 ----------------------------------------------------------
2018-03-10 10:47:33,161: INFO: proc(54): run_and_log_output: Output: Looking around for any Linux dump...
2018-03-10 10:47:33,163: INFO: proc(54): run_and_log_output: Output: Looking for files matching core.* ...
2018-03-10 10:47:33,163: INFO: proc(58): run_and_log_output: Exit Code: 134

stephentoub on 10 Mar 2018

Failed again on OpenSUSE:
https://mc.dot.net/#/user/stephentoub/pr~2Fjenkins~2Fdotnet~2Fcorefx~2Fmaster~2F/test~2Ffunctional~2Fcli~2F/db2a86e941db7153ed2214a30e28588c4bc85787/workItem/System.Xml.XmlSchema.XmlSchemaValidatorApi.Tests/wilogs

2018-03-14 05:23:57,447: INFO: proc(54): run_and_log_output: Output: Discovering: System.Xml.XmlSchema.XmlSchemaValidatorApi.Tests
2018-03-14 05:23:57,659: INFO: proc(54): run_and_log_output: Output: Discovered:  System.Xml.XmlSchema.XmlSchemaValidatorApi.Tests
2018-03-14 05:23:57,762: INFO: proc(54): run_and_log_output: Output: Starting:    System.Xml.XmlSchema.XmlSchemaValidatorApi.Tests
2018-03-14 05:23:59,157: INFO: proc(54): run_and_log_output: Output: /home/helixbot/dotnetbuild/work/76ed2635-a60d-4cf5-81d3-95e47d4244a4/Work/8e485f2a-f57b-4ba0-ba99-464974b83acf/Unzip/RunTests.sh: line 87: 19496 Aborted                 (core dumped) $RUNTIME_PATH/dotnet xunit.console.netcore.exe System.Xml.XmlSchema.XmlSchemaValidatorApi.Tests.dll -xml testResults.xml -notrait Benchmark=true -notrait category=nonnetcoreapptests -notrait category=nonlinuxtests -notrait category=OuterLoop -notrait category=failing
2018-03-14 05:23:59,211: INFO: proc(54): run_and_log_output: Output: Dumpling has not been installed yet. Please run "DumplingHelper.py install_dumpling" before collect_dumps.
2018-03-14 05:23:59,215: INFO: proc(54): run_and_log_output: Output: ~/dotnetbuild/work/76ed2635-a60d-4cf5-81d3-95e47d4244a4/Work/8e485f2a-f57b-4ba0-ba99-464974b83acf/Unzip
2018-03-14 05:23:59,217: INFO: proc(54): run_and_log_output: Output: ----- end 05:23:59 ----- exit code 134 ----------------------------------------------------------
2018-03-14 05:23:59,219: INFO: proc(54): run_and_log_output: Output: Looking around for any Linux dump...
2018-03-14 05:23:59,221: INFO: proc(54): run_and_log_output: Output: ... found no dump in /home/helixbot/dotnetbuild/work/76ed2635-a60d-4cf5-81d3-95e47d4244a4/Work/8e485f2a-f57b-4ba0-ba99-464974b83acf/Unzip
2018-03-14 05:23:59,221: INFO: proc(58): run_and_log_output: Exit Code: 134

stephentoub on 14 Mar 2018

@MattGal how can I get an updated list like you gave here. Thanks.

pjanotti on 27 Mar 2018

@pjanotti if you can get access to our Kusto instance (engsrvprod), it's this query:

WorkItems | where FriendlyName == "System.Xml.XmlSchemaSet.Tests" | where  ExitCode == 134 | join kind= inner (
   Jobs  | project  QueueName , JobId 
) on JobId   | project Finished, QueueName

SG is .NET Engineering Infrastructure Kusto Read

Recent-er results: Looking less crashy!?
Finished QueueName
2018-03-17 19:15:57.0850000 RedHat.74.Amd64.Open
2018-03-14 23:04:55.3280000 Ubuntu.1804.Amd64
2018-03-10 06:02:24.9300000 RedHat.74.Amd64.Open
2018-03-09 17:12:26.8350000 Ubuntu.1404.Amd64.Open

MattGal on 27 Mar 2018

@MattGal thanks! Yes, it looks less crashy...

pjanotti on 27 Mar 2018

10 days without hitting this issue while previously it was about once a day. Big speculation: the "fix" may have come from CoreCLR via commit 6cf7a0f2a9ee57977ccd9950153319856813b7ec since the issue _seems_ related to jitting and there were no more failures after that. Closing it...

pjanotti on 28 Mar 2018

Failed again (from https://github.com/dotnet/corefx/pull/28583):
RedHat.74.Amd64.Open-x64:Release
https://mc.dot.net/#/user/dotnet-bot/pr~2Fjenkins~2Fdotnet~2Fcorefx~2Fmaster~2F/test~2Ffunctional~2Fcli~2F/33e32ffc690ac54cfa1cb3571f8d9ee7cb7600be/workItem/System.Runtime.Tests/wilogs

2018-03-29 04:32:35,033: INFO: proc(54): run_and_log_output: Output: Discovering: System.Runtime.Tests
2018-03-29 04:32:35,977: INFO: proc(54): run_and_log_output: Output: Discovered:  System.Runtime.Tests
2018-03-29 04:32:36,840: INFO: proc(54): run_and_log_output: Output: Starting:    System.Runtime.Tests
2018-03-29 04:32:42,536: INFO: proc(54): run_and_log_output: Output: /home/helixbot/dotnetbuild/work/629aff01-20b2-4a00-90f6-6d8cba36b034/Work/9f8da0be-4e92-45b6-9df6-d943d54b9094/Unzip/RunTests.sh: line 89: 35277 Aborted                 (core dumped) $RUNTIME_PATH/dotnet xunit.console.netcore.exe System.Runtime.Tests.dll -xml testResults.xml -notrait Benchmark=true -notrait category=nonnetcoreapptests -notrait category=nonlinuxtests -notrait category=OuterLoop -notrait category=failing
2018-03-29 04:32:42,608: INFO: proc(54): run_and_log_output: Output: Dumpling has not been installed yet. Please run "DumplingHelper.py install_dumpling" before collect_dumps.
2018-03-29 04:32:42,615: INFO: proc(54): run_and_log_output: Output: ~/dotnetbuild/work/629aff01-20b2-4a00-90f6-6d8cba36b034/Work/9f8da0be-4e92-45b6-9df6-d943d54b9094/Unzip
2018-03-29 04:32:42,616: INFO: proc(54): run_and_log_output: Output: ----- end 04:32:42 ----- exit code 134 ----------------------------------------------------------
2018-03-29 04:32:42,618: INFO: proc(54): run_and_log_output: Output: Looking around for any Linux dump...
2018-03-29 04:32:42,619: INFO: proc(54): run_and_log_output: Output: Looking for files matching core.35277 ...
2018-03-29 04:32:42,620: INFO: proc(54): run_and_log_output: Output: Printing info from core file core.35277
2018-03-29 04:32:42,700: INFO: proc(54): run_and_log_output: Output: Missing separate debuginfo for /home/helixbot/dotnetbuild/work/629aff01-20b2-4a00-90f6-6d8cba36b034/Payload/dotnet
2018-03-29 04:32:42,700: INFO: proc(54): run_and_log_output: Output: Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/c4/51d0c1d735857ffa5017efa0f6151a70d7ab12.debug
2018-03-29 04:32:42,708: INFO: proc(54): run_and_log_output: Output: [New LWP 35287]
2018-03-29 04:32:42,708: INFO: proc(54): run_and_log_output: Output: [New LWP 35277]
2018-03-29 04:32:42,708: INFO: proc(54): run_and_log_output: Output: [New LWP 35278]
2018-03-29 04:32:42,708: INFO: proc(54): run_and_log_output: Output: [New LWP 35280]
2018-03-29 04:32:42,708: INFO: proc(54): run_and_log_output: Output: [New LWP 35281]
2018-03-29 04:32:42,709: INFO: proc(54): run_and_log_output: Output: [New LWP 35279]
2018-03-29 04:32:42,709: INFO: proc(54): run_and_log_output: Output: [New LWP 35282]
2018-03-29 04:32:42,709: INFO: proc(54): run_and_log_output: Output: [New LWP 35285]
2018-03-29 04:32:42,709: INFO: proc(54): run_and_log_output: Output: [New LWP 35286]
2018-03-29 04:32:42,737: INFO: proc(54): run_and_log_output: Output: [Thread debugging using libthread_db enabled]
2018-03-29 04:32:42,737: INFO: proc(54): run_and_log_output: Output: Using host libthread_db library "/lib64/libthread_db.so.1".
2018-03-29 04:32:42,789: INFO: proc(54): run_and_log_output: Output: Missing separate debuginfo for /home/helixbot/dotnetbuild/work/629aff01-20b2-4a00-90f6-6d8cba36b034/Payload/host/fxr/9.9.9/libhostfxr.so
2018-03-29 04:32:42,789: INFO: proc(54): run_and_log_output: Output: Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/7f/98e3ccdf06cdfa6228bf2fd856f1c82d731afc.debug
2018-03-29 04:32:42,790: INFO: proc(54): run_and_log_output: Output: Missing separate debuginfo for /home/helixbot/dotnetbuild/work/629aff01-20b2-4a00-90f6-6d8cba36b034/Payload/shared/Microsoft.NETCore.App/9.9.9/libhostpolicy.so
2018-03-29 04:32:42,790: INFO: proc(54): run_and_log_output: Output: Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/a9/bb170569e3cafcf56bb86c2532607fd7739976.debug
2018-03-29 04:32:44,458: INFO: proc(54): run_and_log_output: Output: Missing separate debuginfo for /home/helixbot/dotnetbuild/work/629aff01-20b2-4a00-90f6-6d8cba36b034/Payload/shared/Microsoft.NETCore.App/9.9.9/libclrjit.so
2018-03-29 04:32:44,458: INFO: proc(54): run_and_log_output: Output: Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/81/730b33125bf98ab752f2071615977a0e3b69bd.debug
2018-03-29 04:32:44,499: INFO: proc(54): run_and_log_output: Output: Core was generated by `/home/helixbot/dotnetbuild/work/629aff01-20b2-4a00-90f6-6d8cba36b034/Payload/do'.
2018-03-29 04:32:44,499: INFO: proc(54): run_and_log_output: Output: Program terminated with signal 6, Aborted.
2018-03-29 04:32:44,500: INFO: proc(54): run_and_log_output: Output: #0  0x00007f26d35e41f7 in raise () from /lib64/libc.so.6
2018-03-29 04:32:44,511: INFO: proc(54): run_and_log_output: Output: 
2018-03-29 04:32:44,511: INFO: proc(54): run_and_log_output: Output: Thread 9 (Thread 0x7f26cc8da700 (LWP 35286)):
2018-03-29 04:32:44,511: INFO: proc(54): run_and_log_output: Output: #0  0x00007f26d00dc294 in Compiler::fgMorphArgs(GenTreeCall*) () from /home/helixbot/dotnetbuild/work/629aff01-20b2-4a00-90f6-6d8cba36b034/Payload/shared/Microsoft.NETCore.App/9.9.9/libclrjit.so
2018-03-29 04:32:44,512: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:44,512: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#1  0x00007f26d00ef789 in Compiler::fgMorphCall(GenTreeCall*) () from /home/helixbot/dotnetbuild/work/629aff01-20b2-4a00-90f6-6d8cba36b034/Payload/shared/Microsoft.NETCore.App/9.9.9/libclrjit.so
2018-03-29 04:32:44,512: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:44,513: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#2  0x00007f26d00d98b0 in Compiler::fgMorphTree(GenTree*, Compiler::MorphAddrContext*) () from /home/helixbot/dotnetbuild/work/629aff01-20b2-4a00-90f6-6d8cba36b034/Payload/shared/Microsoft.NETCore.App/9.9.9/libclrjit.so
2018-03-29 04:32:44,514: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:44,514: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#3  0x00007f26d00fa68e in Compiler::fgMorph() () from /home/helixbot/dotnetbuild/work/629aff01-20b2-4a00-90f6-6d8cba36b034/Payload/shared/Microsoft.NETCore.App/9.9.9/libclrjit.so
2018-03-29 04:32:44,515: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:44,515: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#4  0x00007f26d0015fad in Compiler::compCompile(void**, unsigned int*, JitFlags*) () from /home/helixbot/dotnetbuild/work/629aff01-20b2-4a00-90f6-6d8cba36b034/Payload/shared/Microsoft.NETCore.App/9.9.9/libclrjit.so
2018-03-29 04:32:44,516: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:44,517: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#5  0x00007f26d002e882 in Compiler::compCompileHelper(CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, CorInfoInstantiationVerification) () from /home/helixbot/dotnetbuild/work/629aff01-20b2-4a00-90f6-6d8cba36b034/Payload/shared/Microsoft.NETCore.App/9.9.9/libclrjit.so
2018-03-29 04:32:44,517: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:44,518: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#6  0x00007f26d003089c in jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*) () from /home/helixbot/dotnetbuild/work/629aff01-20b2-4a00-90f6-6d8cba36b034/Payload/shared/Microsoft.NETCore.App/9.9.9/libclrjit.so
2018-03-29 04:32:44,518: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:44,519: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#7  0x00007f26d0032bd3 in CILJit::compileMethod(ICorJitInfo*, CORINFO_METHOD_INFO*, unsigned int, unsigned char**, unsigned int*) () from /home/helixbot/dotnetbuild/work/629aff01-20b2-4a00-90f6-6d8cba36b034/Payload/shared/Microsoft.NETCore.App/9.9.9/libclrjit.so
2018-03-29 04:32:44,666: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:44,666: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#8  0x00007f26d2958492 in invokeCompileMethodHelper (jitMgr=0x1b29700, comp=0x7f26cc8d7418, info=<optimized out>, jitFlags=..., nativeEntry=<optimized out>, nativeSizeOfCode=<optimized out>) at /root/coreclr/src/vm/jitinterface.cpp:12187
2018-03-29 04:32:44,667: INFO: proc(54): run_and_log_output: Output:         ret = CORJIT_SKIPPED
2018-03-29 04:32:44,667: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#9  0x00007f26d2958574 in invokeCompileMethod (jitMgr=0x1b29700, comp=0x7f26cc8d7418, info=0x7f26cc8d74e8, jitFlags=..., nativeEntry=0x7f26cc8d73e8, nativeSizeOfCode=0x7f26cc8d73e4) at /root/coreclr/src/vm/jitinterface.cpp:12254
2018-03-29 04:32:44,668: INFO: proc(54): run_and_log_output: Output:         ret = <optimized out>
2018-03-29 04:32:44,668: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#10 0x00007f26d2958685 in operator() (this=<optimized out>, pParam=<optimized out>) at /root/coreclr/src/vm/jitinterface.cpp:12306
2018-03-29 04:32:44,668: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-29 04:32:44,669: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#11 CallCompileMethodWithSEHWrapper (jitMgr=0x7f26d026e2f0 <PooledAllocator::s_pooledAllocator>, comp=0x7f26cc8d7418, info=0x7f26cbf96c68, flags=..., nativeEntry=0x7f26cc8d73e8, nativeSizeOfCode=0x7f26cbf96ac0, ftn=0x50) at /root/coreclr/src/vm/jitinterface.cpp:12336
2018-03-29 04:32:44,733: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-29 04:32:44,734: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#12 0x00007f26d2958fb6 in UnsafeJitFunction (ftn=<optimized out>, ILHeader=0x7f26cc8d7790, flags=..., pSizeOfCode=0x7f26cc8d78bc) at /root/coreclr/src/vm/jitinterface.cpp:12803
2018-03-29 04:32:44,734: INFO: proc(54): run_and_log_output: Output:         pMethodForSecurity = <optimized out>
2018-03-29 04:32:44,735: INFO: proc(54): run_and_log_output: Output:         sizeOfCode = <optimized out>
2018-03-29 04:32:44,735: INFO: proc(54): run_and_log_output: Output:         ftnHnd = <optimized out>
2018-03-29 04:32:44,735: INFO: proc(54): run_and_log_output: Output:         fAllowRel32 = 1
2018-03-29 04:32:44,735: INFO: proc(54): run_and_log_output: Output:         jitMgr = <optimized out>
2018-03-29 04:32:44,736: INFO: proc(54): run_and_log_output: Output:         reserveForJumpStubs = 0
2018-03-29 04:32:44,736: INFO: proc(54): run_and_log_output: Output:         CURRENT_THREAD = <optimized out>
2018-03-29 04:32:44,736: INFO: proc(54): run_and_log_output: Output:         __pThread = <optimized out>
2018-03-29 04:32:44,736: INFO: proc(54): run_and_log_output: Output:         ret = 0
2018-03-29 04:32:44,737: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#13 0x00007f26d2b348ce in MethodDesc::JitCompileCodeLocked (this=0x7f2659b07718, pConfig=0x7f26cc8d79c0, pEntry=0x7f262804b9c0, pSizeOfCode=0x7f26cc8d78bc, pFlags=0x7f26cc8d78b0) at /root/coreclr/src/vm/prestub.cpp:839
2018-03-29 04:32:44,737: INFO: proc(54): run_and_log_output: Output:         __exceptionHolder = <optimized out>
2018-03-29 04:32:44,737: INFO: proc(54): run_and_log_output: Output:         pOtherCode = 0
2018-03-29 04:32:44,737: INFO: proc(54): run_and_log_output: Output:         pilHeader = 0x7f26cc8d7790
2018-03-29 04:32:44,738: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#14 0x00007f26d2b345fe in MethodDesc::JitCompileCodeLockedEventWrapper (this=0x7f2659b07718, pConfig=0x7f26cc8d79c0, pEntry=0x7f262804b9c0) at /root/coreclr/src/vm/prestub.cpp:696
2018-03-29 04:32:44,738: INFO: proc(54): run_and_log_output: Output:         sizeOfCode = <error reading variable sizeOfCode (Cannot access memory at address 0x7)>
2018-03-29 04:32:44,738: INFO: proc(54): run_and_log_output: Output:         pCode = <optimized out>
2018-03-29 04:32:44,739: INFO: proc(54): run_and_log_output: Output:         flags = {corJitFlags = 350}
2018-03-29 04:32:44,739: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#15 0x00007f26d2b33e20 in MethodDesc::JitCompileCode (this=0x7f2659b07718, pConfig=0x7f26cc8d79c0) at /root/coreclr/src/vm/prestub.cpp:636
2018-03-29 04:32:44,739: INFO: proc(54): run_and_log_output: Output:         pCode = <optimized out>
2018-03-29 04:32:44,740: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#16 0x00007f26d2b33ba7 in MethodDesc::PrepareILBasedCode (this=0x7f2659b07718, pConfig=0x7f26cc8d79c0) at /root/coreclr/src/vm/prestub.cpp:310
2018-03-29 04:32:44,740: INFO: proc(54): run_and_log_output: Output:         pCode = <optimized out>
2018-03-29 04:32:44,740: INFO: proc(54): run_and_log_output: Output: dotnet/runtime#13850 0x00007f26d2b3614a in PrepareCode (this=0x7f2659b07718, pConfig=0x50) at /root/coreclr/src/vm/prestub.cpp:292
2018-03-29 04:32:44,740: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-29 04:32:44,741: INFO: proc(54): run_and_log_output: Output: dotnet/runtime#13851 PrepareInitialCode (this=0x7f2659b07718) at /root/coreclr/src/vm/prestub.cpp:248
2018-03-29 04:32:44,741: INFO: proc(54): run_and_log_output: Output:         config = <error reading variable config (Cannot access memory at address 0x50)>
2018-03-29 04:32:44,741: INFO: proc(54): run_and_log_output: Output:         pCode = <optimized out>
2018-03-29 04:32:44,741: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#19 MethodDesc::DoPrestub (this=0x7f2659b07718, pDispatchingMT=<optimized out>) at /root/coreclr/src/vm/prestub.cpp:1805
2018-03-29 04:32:44,742: INFO: proc(54): run_and_log_output: Output:         pCode = <optimized out>
2018-03-29 04:32:44,742: INFO: proc(54): run_and_log_output: Output:         pThread = <optimized out>
2018-03-29 04:32:44,742: INFO: proc(54): run_and_log_output: Output:         fCanBackpatchPrestub = <optimized out>
2018-03-29 04:32:44,742: INFO: proc(54): run_and_log_output: Output:         fWasPromotedToTier1 = <optimized out>
2018-03-29 04:32:44,743: INFO: proc(54): run_and_log_output: Output:         fEligibleForTieredCompilation = <optimized out>
2018-03-29 04:32:44,743: INFO: proc(54): run_and_log_output: Output:         pTieredCompilationManager = 0x0
2018-03-29 04:32:44,743: INFO: proc(54): run_and_log_output: Output:         fIsPointingToPrestub = <optimized out>
2018-03-29 04:32:44,743: INFO: proc(54): run_and_log_output: Output:         pStub = <optimized out>
2018-03-29 04:32:44,744: INFO: proc(54): run_and_log_output: Output: dotnet/runtime#13852 0x00007f26d2b35ab4 in PreStubWorker (pTransitionBlock=<optimized out>, pMD=0x7f2659b07718) at /root/coreclr/src/vm/prestub.cpp:1546
2018-03-29 04:32:44,827: INFO: proc(54): run_and_log_output: Output:         pDispatchingMT = <optimized out>
2018-03-29 04:32:44,827: INFO: proc(54): run_and_log_output: Output:         CURRENT_THREAD = 0x1c5d9f0
2018-03-29 04:32:44,827: INFO: proc(54): run_and_log_output: Output:         __fExceptionCatched = false
2018-03-29 04:32:44,827: INFO: proc(54): run_and_log_output: Output:         __pThread = 0x1c5d9f0
2018-03-29 04:32:44,828: INFO: proc(54): run_and_log_output: Output:         __pUnCEntryFrame = 0x7f26cc8d7a60
2018-03-29 04:32:44,828: INFO: proc(54): run_and_log_output: Output:         CURRENT_THREAD = 0x1c5d9f0
2018-03-29 04:32:44,828: INFO: proc(54): run_and_log_output: Output:         pPFrame = <optimized out>
2018-03-29 04:32:44,828: INFO: proc(54): run_and_log_output: Output:         __pThread = 0x1c5d9f0
2018-03-29 04:32:44,829: INFO: proc(54): run_and_log_output: Output:         hasCaughtException = false
2018-03-29 04:32:44,829: INFO: proc(54): run_and_log_output: Output:         __dwLastError = 0
2018-03-29 04:32:44,829: INFO: proc(54): run_and_log_output: Output:         pbRetVal = 0
2018-03-29 04:32:44,829: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#21 0x00007f26d2ab7e94 in ThePreStub () from /home/helixbot/dotnetbuild/work/629aff01-20b2-4a00-90f6-6d8cba36b034/Payload/shared/Microsoft.NETCore.App/9.9.9/libcoreclr.so
2018-03-29 04:32:44,830: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:44,830: INFO: proc(54): run_and_log_output: Output: dotnet/runtime#13853 0x00007f26d2ab71af in CallDescrWorkerInternal () from /home/helixbot/dotnetbuild/work/629aff01-20b2-4a00-90f6-6d8cba36b034/Payload/shared/Microsoft.NETCore.App/9.9.9/libcoreclr.so
2018-03-29 04:32:44,940: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:44,941: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#23 0x00007f26d29d723d in CallDescrWorkerWithHandler (pCallDescrData=0x7f26cc8d8060, fCriticalCall=<optimized out>) at /root/coreclr/src/vm/callhelpers.cpp:78
2018-03-29 04:32:44,943: INFO: proc(54): run_and_log_output: Output:         CURRENT_THREAD = <optimized out>
2018-03-29 04:32:44,943: INFO: proc(54): run_and_log_output: Output:         __pThread = <optimized out>
2018-03-29 04:32:44,944: INFO: proc(54): run_and_log_output: Output:         __exceptionHolder = {<NativeExceptionHolderBase> = {_vptr$NativeExceptionHolderBase = 0x7f26d2fc6bf8 <vtable for NativeExceptionHolderNoCatch+16>, m_head = 0x7f2628000900, m_next = 0x7f26cc8d7c38}, <No data fields>}
2018-03-29 04:32:44,944: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#24 0x00007f26d2a54e73 in operator() (this=<optimized out>, pParam=<optimized out>) at /root/coreclr/src/vm/reflectioninvocation.cpp:739
2018-03-29 04:32:44,944: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-29 04:32:44,945: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#25 CallDescrWorkerReflectionWrapper (pCallDescrData=<optimized out>, pFrame=<optimized out>) at /root/coreclr/src/vm/reflectioninvocation.cpp:741
2018-03-29 04:32:44,945: INFO: proc(54): run_and_log_output: Output:         __exceptionHolder = <optimized out>
2018-03-29 04:32:44,945: INFO: proc(54): run_and_log_output: Output:         __param = 0x7f26cbf97058
2018-03-29 04:32:44,947: INFO: proc(54): run_and_log_output: Output:         disposition = -1
2018-03-29 04:32:44,948: INFO: proc(54): run_and_log_output: Output: dotnet/runtime#13854 0x00007f26d2a55da2 in RuntimeMethodHandle::InvokeMethod (target=<optimized out>, objs=<optimized out>, pSigUNSAFE=<optimized out>, fConstructor=false, fWrapExceptions=<error reading variable: access outside bounds of object referenced via synthetic pointer>) at /root/coreclr/src/vm/reflectioninvocation.cpp:1355
2018-03-29 04:32:45,055: INFO: proc(54): run_and_log_output: Output:         nStackBytes = <optimized out>
2018-03-29 04:32:45,056: INFO: proc(54): run_and_log_output: Output:         pTransitionBlock = 0x7f26cc8d7d30 "`H\002\323&\177"
2018-03-29 04:32:45,057: INFO: proc(54): run_and_log_output: Output:         pValueClasses = 0x7f26cc8d7e50
2018-03-29 04:32:45,057: INFO: proc(54): run_and_log_output: Output:         pThread = 0x1c5d9f0
2018-03-29 04:32:45,057: INFO: proc(54): run_and_log_output: Output:         pAlloc = 0x7f26cc8d7cb0 "\220\202\215\314&\177"
2018-03-29 04:32:45,057: INFO: proc(54): run_and_log_output: Output:         fExceptionThrown = false
2018-03-29 04:32:45,058: INFO: proc(54): run_and_log_output: Output:         fHasRetBuffArg = <optimized out>
2018-03-29 04:32:45,058: INFO: proc(54): run_and_log_output: Output:         pRetBufStackCopy = <optimized out>
2018-03-29 04:32:45,058: INFO: proc(54): run_and_log_output: Output:         pAssem = <optimized out>
2018-03-29 04:32:45,058: INFO: proc(54): run_and_log_output: Output:         __fExceptionCatched = false
2018-03-29 04:32:45,059: INFO: proc(54): run_and_log_output: Output:         __pUnCException = <optimized out>
2018-03-29 04:32:45,059: INFO: proc(54): run_and_log_output: Output:         CURRENT_THREAD = 0x1c5d9f0
2018-03-29 04:32:45,059: INFO: proc(54): run_and_log_output: Output:         hasCaughtException = false
2018-03-29 04:32:45,059: INFO: proc(54): run_and_log_output: Output:         pMeth = 0x7f2659b07718
2018-03-29 04:32:45,059: INFO: proc(54): run_and_log_output: Output:         alwaysZero = 0
2018-03-29 04:32:45,060: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#27 0x00007f2659331357 in ?? ()
2018-03-29 04:32:45,060: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,060: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#28 0x0000000000000068 in ?? ()
2018-03-29 04:32:45,060: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,061: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#29 0x0000000000000000 in ?? ()
2018-03-29 04:32:45,075: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,075: INFO: proc(54): run_and_log_output: Output: 
2018-03-29 04:32:45,075: INFO: proc(54): run_and_log_output: Output: Thread 8 (Thread 0x7f26cd0db700 (LWP 35285)):
2018-03-29 04:32:45,076: INFO: proc(54): run_and_log_output: Output: #0  0x00007f26d419d945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
2018-03-29 04:32:45,077: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,077: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#1  0x00007f26d2d33502 in CorUnix::CPalSynchronizationManager::ThreadNativeWait (ptnwdNativeWaitData=<optimized out>, dwTimeout=<optimized out>, ptwrWakeupReason=<optimized out>, pdwSignaledObject=<optimized out>) at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:479
2018-03-29 04:32:45,077: INFO: proc(54): run_and_log_output: Output:         iWaitRet = <optimized out>
2018-03-29 04:32:45,082: INFO: proc(54): run_and_log_output: Output:         palErr = 1359
2018-03-29 04:32:45,083: INFO: proc(54): run_and_log_output: Output:         iRet = <optimized out>
2018-03-29 04:32:45,083: INFO: proc(54): run_and_log_output: Output:         tsAbsTmo = <optimized out>
2018-03-29 04:32:45,084: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#2  0x00007f26d2d330e4 in CorUnix::CPalSynchronizationManager::BlockThread (this=0x1a9af40, pthrCurrent=0x1ba6450, dwTimeout=4294967295, fAlertable=true, fIsSleep=<optimized out>, ptwrWakeupReason=0x7f26cd0da380, pdwSignaledObject=0xfffffffffffffe00) at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:302
2018-03-29 04:32:45,084: INFO: proc(54): run_and_log_output: Output:         twrWakeupReason = <error reading variable twrWakeupReason (Cannot access memory at address 0xef)>
2018-03-29 04:32:45,084: INFO: proc(54): run_and_log_output: Output:         palErr = 0
2018-03-29 04:32:45,085: INFO: proc(54): run_and_log_output: Output:         dwSigObjIdx = <error reading variable dwSigObjIdx (Cannot access memory at address 0xffffffffffffffff)>
2018-03-29 04:32:45,085: INFO: proc(54): run_and_log_output: Output:         dwWaitState = <optimized out>
2018-03-29 04:32:45,085: INFO: proc(54): run_and_log_output: Output:         fEarlyDeath = false
2018-03-29 04:32:45,085: INFO: proc(54): run_and_log_output: Output:         fRaceAlerted = true
2018-03-29 04:32:45,086: INFO: proc(54): run_and_log_output: Output:         pdwWaitState = <optimized out>
2018-03-29 04:32:45,086: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#3  0x00007f26d2d37ab4 in CorUnix::InternalWaitForMultipleObjectsEx (pThread=0x1ba6450, nCount=<optimized out>, lpHandles=<optimized out>, bWaitAll=<optimized out>, dwMilliseconds=<optimized out>, bAlertable=<optimized out>, bPrioritize=<optimized out>) at /root/coreclr/src/pal/src/synchmgr/wait.cpp:636
2018-03-29 04:32:45,087: INFO: proc(54): run_and_log_output: Output:         twrWakeupReason = <error reading variable twrWakeupReason (Cannot access memory at address 0x77)>
2018-03-29 04:32:45,088: INFO: proc(54): run_and_log_output: Output:         pIPalObjStackArray = {0x1af1b10, 0x0 <repeats 15 times>}
2018-03-29 04:32:45,088: INFO: proc(54): run_and_log_output: Output:         pISyncStackArray = {0x0 <repeats 16 times>}
2018-03-29 04:32:45,089: INFO: proc(54): run_and_log_output: Output:         iSignaledObjIndex = <optimized out>
2018-03-29 04:32:45,089: INFO: proc(54): run_and_log_output: Output:         palErr = <optimized out>
2018-03-29 04:32:45,089: INFO: proc(54): run_and_log_output: Output:         dwRet = 4294967295
2018-03-29 04:32:45,089: INFO: proc(54): run_and_log_output: Output:         fAbandoned = <optimized out>
2018-03-29 04:32:45,090: INFO: proc(54): run_and_log_output: Output:         fNeedToBlock = <optimized out>
2018-03-29 04:32:45,090: INFO: proc(54): run_and_log_output: Output:         fWAll = <optimized out>
2018-03-29 04:32:45,090: INFO: proc(54): run_and_log_output: Output:         ppISyncWaitCtrlrs = <optimized out>
2018-03-29 04:32:45,090: INFO: proc(54): run_and_log_output: Output:         ppIPalObjs = <optimized out>
2018-03-29 04:32:45,090: INFO: proc(54): run_and_log_output: Output:         wtWaitType = <optimized out>
2018-03-29 04:32:45,091: INFO: proc(54): run_and_log_output: Output:         iSignaledObjCount = 1
2018-03-29 04:32:45,091: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#4  0x00007f26d2d37e31 in WaitForMultipleObjectsEx (nCount=1, lpHandles=0x7f26cd0da690, bWaitAll=1, dwMilliseconds=4294967295, bAlertable=1) at /root/coreclr/src/pal/src/synchmgr/wait.cpp:202
2018-03-29 04:32:45,091: INFO: proc(54): run_and_log_output: Output:         pThread = 0xfffffffffffffe00
2018-03-29 04:32:45,092: INFO: proc(54): run_and_log_output: Output:         dwRet = <optimized out>
2018-03-29 04:32:45,092: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#5  0x00007f26d29a3e41 in WaitForMultipleObjectsEx_SO_TOLERANT (nCount=<optimized out>, lpHandles=0x7f26cd0da690, bWaitAll=1, bAlertable=1, dwMilliseconds=<optimized out>) at /root/coreclr/src/vm/threads.cpp:3462
2018-03-29 04:32:45,092: INFO: proc(54): run_and_log_output: Output:         lastError = 0
2018-03-29 04:32:45,093: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#6  DoAppropriateAptStateWait (numWaiters=1, timeout=<optimized out>, this=<optimized out>, pHandles=<optimized out>, bWaitAll=<optimized out>, mode=<optimized out>) at /root/coreclr/src/vm/threads.cpp:3496
2018-03-29 04:32:45,093: INFO: proc(54): run_and_log_output: Output:         alertable = 1
2018-03-29 04:32:45,093: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#7  Thread::DoAppropriateWaitWorker (this=0x1bb1690, countHandles=<optimized out>, handles=<optimized out>, waitAll=1, millis=<optimized out>, mode=WaitMode_None) at /root/coreclr/src/vm/threads.cpp:3633
2018-03-29 04:32:45,093: INFO: proc(54): run_and_log_output: Output:         ret = 0
2018-03-29 04:32:45,093: INFO: proc(54): run_and_log_output: Output:         alertable = 1
2018-03-29 04:32:45,094: INFO: proc(54): run_and_log_output: Output:         dwEnd = <optimized out>
2018-03-29 04:32:45,094: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#8  0x00007f26d299ed90 in operator() (this=<optimized out>, __pEEParam=<optimized out>) at /root/coreclr/src/vm/threads.cpp:3307
2018-03-29 04:32:45,124: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-29 04:32:45,125: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#9  Thread::DoAppropriateWait (this=0x1ba66b4, countHandles=1, handles=0xef, waitAll=-1, millis=28993024, mode=(WaitMode_Alertable | WaitMode_IgnoreSyncCtx | WaitMode_ADUnload | unknown: 112), syncState=<optimized out>) at /root/coreclr/src/vm/threads.cpp:3322
2018-03-29 04:32:45,125: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-29 04:32:45,125: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#10 0x00007f26d29ef830 in WaitHandleNative::CorWaitOneNative (safeWaitHandleUNSAFE=<optimized out>, timeout=<optimized out>, hasThreadAffinity=<optimized out>, exitContext=<optimized out>) at /root/coreclr/src/vm/comwaithandle.cpp:198
2018-03-29 04:32:45,126: INFO: proc(54): run_and_log_output: Output:         sPauseTime = 0
2018-03-29 04:32:45,126: INFO: proc(54): run_and_log_output: Output:         handles = {0x204}
2018-03-29 04:32:45,126: INFO: proc(54): run_and_log_output: Output:         pThread = 0x1bb1690
2018-03-29 04:32:45,126: INFO: proc(54): run_and_log_output: Output:         res = 4294967295
2018-03-29 04:32:45,127: INFO: proc(54): run_and_log_output: Output:         __fExceptionCatched = false
2018-03-29 04:32:45,127: INFO: proc(54): run_and_log_output: Output:         CURRENT_THREAD = 0x1bb1690
2018-03-29 04:32:45,127: INFO: proc(54): run_and_log_output: Output:         hasCaughtException = false
2018-03-29 04:32:45,127: INFO: proc(54): run_and_log_output: Output:         sh = <optimized out>
2018-03-29 04:32:45,127: INFO: proc(54): run_and_log_output: Output:         alwaysZero = 0
2018-03-29 04:32:45,128: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#11 0x00007f26598ee0e7 in ?? ()
2018-03-29 04:32:45,128: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,128: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#12 0x00007f2634304790 in ?? ()
2018-03-29 04:32:45,129: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,129: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#13 0x00007f26340144b8 in ?? ()
2018-03-29 04:32:45,130: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,130: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#14 0x00007f26cd0da810 in ?? ()
2018-03-29 04:32:45,131: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,131: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#15 0x00007f26598ed454 in ?? ()
2018-03-29 04:32:45,132: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,132: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#16 0x00007f26cd0da7f0 in ?? ()
2018-03-29 04:32:45,133: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,133: INFO: proc(54): run_and_log_output: Output: dotnet/runtime#13850 0x00007f265939d283 in ?? ()
2018-03-29 04:32:45,134: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,134: INFO: proc(54): run_and_log_output: Output: dotnet/runtime#13851 0x00007f2634247d70 in ?? ()
2018-03-29 04:32:45,135: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,135: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#19 0x0000000000000000 in ?? ()
2018-03-29 04:32:45,137: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,137: INFO: proc(54): run_and_log_output: Output: 
2018-03-29 04:32:45,138: INFO: proc(54): run_and_log_output: Output: Thread 7 (Thread 0x7f26cde56700 (LWP 35282)):
2018-03-29 04:32:45,138: INFO: proc(54): run_and_log_output: Output: #0  0x00007f26d41a070d in read () from /lib64/libpthread.so.0
2018-03-29 04:32:45,138: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,139: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#1  0x00007f26cde68d8f in SignalHandlerLoop (arg=<optimized out>) at /mnt/j/workspace/dotnet_corefx/master/linux-TGroup_netcoreapp+CGroup_Release+AGroup_x64+TestOuter_false_prtest/src/Native/Unix/System.Native/pal_signal.c:100
2018-03-29 04:32:45,139: INFO: proc(54): run_and_log_output: Output:         pipeFd = 46
2018-03-29 04:32:45,139: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#2  0x00007f26d4199e25 in start_thread () from /lib64/libpthread.so.0
2018-03-29 04:32:45,140: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,140: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#3  0x00007f26d36a734d in clone () from /lib64/libc.so.6
2018-03-29 04:32:45,142: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,142: INFO: proc(54): run_and_log_output: Output: 
2018-03-29 04:32:45,142: INFO: proc(54): run_and_log_output: Output: Thread 6 (Thread 0x7f26d1e25700 (LWP 35279)):
2018-03-29 04:32:45,142: INFO: proc(54): run_and_log_output: Output: #0  0x00007f26d41a0ead in open64 () from /lib64/libpthread.so.0
2018-03-29 04:32:45,176: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,176: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#1  0x00007f26d2c41eef in TwoWayPipe::WaitForConnection (this=0x1b30690) at /root/coreclr/src/debug/debug-pal/unix/twowaypipe.cpp:90
2018-03-29 04:32:45,176: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-29 04:32:45,177: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#2  0x00007f26d2c3abc8 in DbgTransportSession::TransportWorker (this=0x1b305b0) at /root/coreclr/src/debug/ee/../shared/dbgtransportsession.cpp:1320
2018-03-29 04:32:45,178: INFO: proc(54): run_and_log_output: Output:         pid = <optimized out>
2018-03-29 04:32:45,178: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#3  0x00007f26d2c39a79 in DbgTransportSession::TransportWorkerStatic (pvContext=0x1b306a0) at /root/coreclr/src/debug/ee/../shared/dbgtransportsession.cpp:1236
2018-03-29 04:32:45,178: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-29 04:32:45,179: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#4  0x00007f26d2d3df82 in CorUnix::CPalThread::ThreadEntry (pvParam=0x1b3a9c0) at /root/coreclr/src/pal/src/thread/thread.cpp:1682
2018-03-29 04:32:45,179: INFO: proc(54): run_and_log_output: Output:         palError = <optimized out>
2018-03-29 04:32:45,179: INFO: proc(54): run_and_log_output: Output:         pvPar = 0x1b306a0
2018-03-29 04:32:45,179: INFO: proc(54): run_and_log_output: Output:         retValue = <optimized out>
2018-03-29 04:32:45,180: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#5  0x00007f26d4199e25 in start_thread () from /lib64/libpthread.so.0
2018-03-29 04:32:45,180: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,181: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#6  0x00007f26d36a734d in clone () from /lib64/libc.so.6
2018-03-29 04:32:45,182: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,182: INFO: proc(54): run_and_log_output: Output: 
2018-03-29 04:32:45,183: INFO: proc(54): run_and_log_output: Output: Thread 5 (Thread 0x7f26d0aaa700 (LWP 35281)):
2018-03-29 04:32:45,183: INFO: proc(54): run_and_log_output: Output: #0  0x00007f26d419dcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
2018-03-29 04:32:45,183: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,184: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#1  0x00007f26d2d334c5 in CorUnix::CPalSynchronizationManager::ThreadNativeWait (ptnwdNativeWaitData=<optimized out>, dwTimeout=<optimized out>, ptwrWakeupReason=<optimized out>, pdwSignaledObject=<optimized out>) at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:484
2018-03-29 04:32:45,184: INFO: proc(54): run_and_log_output: Output:         iWaitRet = 0
2018-03-29 04:32:45,184: INFO: proc(54): run_and_log_output: Output:         palErr = 1359
2018-03-29 04:32:45,185: INFO: proc(54): run_and_log_output: Output:         iRet = <optimized out>
2018-03-29 04:32:45,185: INFO: proc(54): run_and_log_output: Output:         tsAbsTmo = {tv_sec = 702, tv_nsec = 543085289}
2018-03-29 04:32:45,186: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#2  0x00007f26d2d330e4 in CorUnix::CPalSynchronizationManager::BlockThread (this=0x1a9af40, pthrCurrent=0x1b620d0, dwTimeout=2000, fAlertable=false, fIsSleep=<optimized out>, ptwrWakeupReason=0x7f26d0aa9a50, pdwSignaledObject=0xfffffffffffffdfc) at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:302
2018-03-29 04:32:45,186: INFO: proc(54): run_and_log_output: Output:         twrWakeupReason = <error reading variable twrWakeupReason (Cannot access memory at address 0x9)>
2018-03-29 04:32:45,186: INFO: proc(54): run_and_log_output: Output:         palErr = 0
2018-03-29 04:32:45,186: INFO: proc(54): run_and_log_output: Output:         dwSigObjIdx = <error reading variable dwSigObjIdx (Cannot access memory at address 0xffffffffffffffff)>
2018-03-29 04:32:45,187: INFO: proc(54): run_and_log_output: Output:         dwWaitState = <optimized out>
2018-03-29 04:32:45,187: INFO: proc(54): run_and_log_output: Output:         fEarlyDeath = false
2018-03-29 04:32:45,187: INFO: proc(54): run_and_log_output: Output:         fRaceAlerted = true
2018-03-29 04:32:45,187: INFO: proc(54): run_and_log_output: Output:         pdwWaitState = <optimized out>
2018-03-29 04:32:45,188: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#3  0x00007f26d2d37ab4 in CorUnix::InternalWaitForMultipleObjectsEx (pThread=0x1b620d0, nCount=<optimized out>, lpHandles=<optimized out>, bWaitAll=<optimized out>, dwMilliseconds=<optimized out>, bAlertable=<optimized out>, bPrioritize=<optimized out>) at /root/coreclr/src/pal/src/synchmgr/wait.cpp:636
2018-03-29 04:32:45,188: INFO: proc(54): run_and_log_output: Output:         twrWakeupReason = <error reading variable twrWakeupReason (Cannot access memory at address 0xffffffff)>
2018-03-29 04:32:45,188: INFO: proc(54): run_and_log_output: Output:         pIPalObjStackArray = {0x1b60c40, 0x0 <repeats 15 times>}
2018-03-29 04:32:45,189: INFO: proc(54): run_and_log_output: Output:         pISyncStackArray = {0x0 <repeats 16 times>}
2018-03-29 04:32:45,189: INFO: proc(54): run_and_log_output: Output:         iSignaledObjIndex = <optimized out>
2018-03-29 04:32:45,189: INFO: proc(54): run_and_log_output: Output:         palErr = <optimized out>
2018-03-29 04:32:45,189: INFO: proc(54): run_and_log_output: Output:         dwRet = 4294967295
2018-03-29 04:32:45,189: INFO: proc(54): run_and_log_output: Output:         fAbandoned = <optimized out>
2018-03-29 04:32:45,190: INFO: proc(54): run_and_log_output: Output:         fNeedToBlock = <optimized out>
2018-03-29 04:32:45,190: INFO: proc(54): run_and_log_output: Output:         fWAll = <optimized out>
2018-03-29 04:32:45,190: INFO: proc(54): run_and_log_output: Output:         ppISyncWaitCtrlrs = <optimized out>
2018-03-29 04:32:45,190: INFO: proc(54): run_and_log_output: Output:         ppIPalObjs = <optimized out>
2018-03-29 04:32:45,190: INFO: proc(54): run_and_log_output: Output:         wtWaitType = <optimized out>
2018-03-29 04:32:45,197: INFO: proc(54): run_and_log_output: Output:         iSignaledObjCount = 1
2018-03-29 04:32:45,198: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#4  0x00007f26d2d37d59 in WaitForSingleObjectEx (hHandle=<error reading variable: Cannot access memory at address 0x9>, dwMilliseconds=2000, bAlertable=0) at /root/coreclr/src/pal/src/synchmgr/wait.cpp:139
2018-03-29 04:32:45,198: INFO: proc(54): run_and_log_output: Output:         pThread = 0xfffffffffffffdfc
2018-03-29 04:32:45,198: INFO: proc(54): run_and_log_output: Output:         dwRet = <optimized out>
2018-03-29 04:32:45,199: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#5  0x00007f26d2a7c1f1 in CLREventWaitHelper2 (handle=0x1b62334, dwMilliseconds=<optimized out>, alertable=<optimized out>) at /root/coreclr/src/vm/synch.cpp:385
2018-03-29 04:32:45,199: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-29 04:32:45,199: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#6  operator() (this=<optimized out>, pParam=<optimized out>) at /root/coreclr/src/vm/synch.cpp:411
2018-03-29 04:32:45,199: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-29 04:32:45,200: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#7  CLREventWaitHelper (dwMilliseconds=2000, alertable=<optimized out>, handle=<optimized out>) at /root/coreclr/src/vm/synch.cpp:413
2018-03-29 04:32:45,200: INFO: proc(54): run_and_log_output: Output:         __exceptionHolder = <optimized out>
2018-03-29 04:32:45,200: INFO: proc(54): run_and_log_output: Output:         __param = 0x7f26d0aa9c30
2018-03-29 04:32:45,240: INFO: proc(54): run_and_log_output: Output:         disposition = -1
2018-03-29 04:32:45,240: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#8  CLREventBase::WaitEx (this=0x1a816d0, dwMilliseconds=2000, mode=<optimized out>, syncState=0x0) at /root/coreclr/src/vm/synch.cpp:483
2018-03-29 04:32:45,240: INFO: proc(54): run_and_log_output: Output:         alertable = <optimized out>
2018-03-29 04:32:45,241: INFO: proc(54): run_and_log_output: Output:         pThread = <optimized out>
2018-03-29 04:32:45,241: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#9  0x00007f26d2a0c31f in FinalizerThread::WaitForFinalizerEvent (event=0x1a816d0) at /root/coreclr/src/vm/finalizerthread.cpp:408
2018-03-29 04:32:45,241: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-29 04:32:45,242: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#10 0x00007f26d2a0c4c5 in FinalizerThread::FinalizerThreadWorker (args=<optimized out>) at /root/coreclr/src/vm/finalizerthread.cpp:564
2018-03-29 04:32:45,242: INFO: proc(54): run_and_log_output: Output:         bPriorityBoosted = 0
2018-03-29 04:32:45,243: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#11 0x00007f26d29a95ad in ManagedThreadBase_DispatchInner (pCallState=0x7f26d0aa9e20) at /root/coreclr/src/vm/threads.cpp:8850
2018-03-29 04:32:45,243: INFO: proc(54): run_and_log_output: Output:         pThread = 0x1b611e0
2018-03-29 04:32:45,243: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#12 ManagedThreadBase_DispatchMiddle (pCallState=0x7f26d0aa9e20) at /root/coreclr/src/vm/threads.cpp:8901
2018-03-29 04:32:45,243: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-29 04:32:45,244: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#13 operator() (this=<optimized out>, pParam=<optimized out>) at /root/coreclr/src/vm/threads.cpp:9139
2018-03-29 04:32:45,244: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-29 04:32:45,244: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#14 operator() (this=<optimized out>, pArgs=<optimized out>) at /root/coreclr/src/vm/threads.cpp:9141
2018-03-29 04:32:45,244: INFO: proc(54): run_and_log_output: Output:         __exceptionHolder = <optimized out>
2018-03-29 04:32:45,245: INFO: proc(54): run_and_log_output: Output:         disposition = -1
2018-03-29 04:32:45,245: INFO: proc(54): run_and_log_output: Output:         __param = <optimized out>
2018-03-29 04:32:45,246: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#15 ManagedThreadBase_DispatchOuter (pCallState=<optimized out>) at /root/coreclr/src/vm/threads.cpp:9178
2018-03-29 04:32:45,246: INFO: proc(54): run_and_log_output: Output:         pThread = 0x1b611e0
2018-03-29 04:32:45,247: INFO: proc(54): run_and_log_output: Output:         param = <optimized out>
2018-03-29 04:32:45,247: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#16 0x00007f26d29a9d8e in ManagedThreadBase_NoADTransition (pTarget=0x7f26d2a0c450 <FinalizerThread::FinalizerThreadWorker(void*)>, filterType=FinalizerThread) at /root/coreclr/src/vm/threads.cpp:9223
2018-03-29 04:32:45,370: INFO: proc(54): run_and_log_output: Output:         CallState = {pAppDomainId = {m_dwId = 9}, pUnsafeAppDomain = 0x400000000, bDomainIsAsID = 0, pTarget = 0x1b6230800000000, args = 0x300000000, filterType = 0, flags = MTCSF_NormalBase}
2018-03-29 04:32:45,370: INFO: proc(54): run_and_log_output: Output: dotnet/runtime#13850 ManagedThreadBase::FinalizerBase (pTarget=0x7f26d2a0c450 <FinalizerThread::FinalizerThreadWorker(void*)>) at /root/coreclr/src/vm/threads.cpp:9249
2018-03-29 04:32:45,370: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-29 04:32:45,371: INFO: proc(54): run_and_log_output: Output: dotnet/runtime#13851 0x00007f26d2a0c8bc in FinalizerThread::FinalizerThreadStart (args=<optimized out>) at /root/coreclr/src/vm/finalizerthread.cpp:764
2018-03-29 04:32:45,371: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-29 04:32:45,372: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#19 0x00007f26d2d3df82 in CorUnix::CPalThread::ThreadEntry (pvParam=0x1b620d0) at /root/coreclr/src/pal/src/thread/thread.cpp:1682
2018-03-29 04:32:45,372: INFO: proc(54): run_and_log_output: Output:         palError = <optimized out>
2018-03-29 04:32:45,373: INFO: proc(54): run_and_log_output: Output:         pvPar = 0x1b62334
2018-03-29 04:32:45,373: INFO: proc(54): run_and_log_output: Output:         retValue = <optimized out>
2018-03-29 04:32:45,373: INFO: proc(54): run_and_log_output: Output: dotnet/runtime#13852 0x00007f26d4199e25 in start_thread () from /lib64/libpthread.so.0
2018-03-29 04:32:45,374: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,374: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#21 0x00007f26d36a734d in clone () from /lib64/libc.so.6
2018-03-29 04:32:45,375: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,376: INFO: proc(54): run_and_log_output: Output: 
2018-03-29 04:32:45,376: INFO: proc(54): run_and_log_output: Output: Thread 4 (Thread 0x7f26d1624700 (LWP 35280)):
2018-03-29 04:32:45,376: INFO: proc(54): run_and_log_output: Output: #0  0x00007f26d419d945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
2018-03-29 04:32:45,376: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,377: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#1  0x00007f26d2d33502 in CorUnix::CPalSynchronizationManager::ThreadNativeWait (ptnwdNativeWaitData=<optimized out>, dwTimeout=<optimized out>, ptwrWakeupReason=<optimized out>, pdwSignaledObject=<optimized out>) at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:479
2018-03-29 04:32:45,377: INFO: proc(54): run_and_log_output: Output:         iWaitRet = <optimized out>
2018-03-29 04:32:45,378: INFO: proc(54): run_and_log_output: Output:         palErr = 1359
2018-03-29 04:32:45,378: INFO: proc(54): run_and_log_output: Output:         iRet = <optimized out>
2018-03-29 04:32:45,378: INFO: proc(54): run_and_log_output: Output:         tsAbsTmo = <optimized out>
2018-03-29 04:32:45,379: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#2  0x00007f26d2d330e4 in CorUnix::CPalSynchronizationManager::BlockThread (this=0x1a9af40, pthrCurrent=0x1b3b7b0, dwTimeout=4294967295, fAlertable=false, fIsSleep=<optimized out>, ptwrWakeupReason=0x7f26d1623cb0, pdwSignaledObject=0xfffffffffffffe00) at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:302
2018-03-29 04:32:45,379: INFO: proc(54): run_and_log_output: Output:         twrWakeupReason = <error reading variable twrWakeupReason (Cannot access memory at address 0x1)>
2018-03-29 04:32:45,379: INFO: proc(54): run_and_log_output: Output:         palErr = 0
2018-03-29 04:32:45,380: INFO: proc(54): run_and_log_output: Output:         dwSigObjIdx = <error reading variable dwSigObjIdx (Cannot access memory at address 0xffffffffffffffff)>
2018-03-29 04:32:45,380: INFO: proc(54): run_and_log_output: Output:         dwWaitState = <optimized out>
2018-03-29 04:32:45,380: INFO: proc(54): run_and_log_output: Output:         fEarlyDeath = false
2018-03-29 04:32:45,380: INFO: proc(54): run_and_log_output: Output:         fRaceAlerted = true
2018-03-29 04:32:45,381: INFO: proc(54): run_and_log_output: Output:         pdwWaitState = <optimized out>
2018-03-29 04:32:45,381: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#3  0x00007f26d2d37ab4 in CorUnix::InternalWaitForMultipleObjectsEx (pThread=0x1b3b7b0, nCount=<optimized out>, lpHandles=<optimized out>, bWaitAll=<optimized out>, dwMilliseconds=<optimized out>, bAlertable=<optimized out>, bPrioritize=<optimized out>) at /root/coreclr/src/pal/src/synchmgr/wait.cpp:636
2018-03-29 04:32:45,381: INFO: proc(54): run_and_log_output: Output:         twrWakeupReason = <error reading variable twrWakeupReason (Cannot access memory at address 0x0)>
2018-03-29 04:32:45,382: INFO: proc(54): run_and_log_output: Output:         pIPalObjStackArray = {0x1b2e1f0, 0x1b3a600, 0x1b2e010, 0x0 <repeats 13 times>}
2018-03-29 04:32:45,382: INFO: proc(54): run_and_log_output: Output:         pISyncStackArray = {0x0 <repeats 16 times>}
2018-03-29 04:32:45,382: INFO: proc(54): run_and_log_output: Output:         iSignaledObjIndex = <optimized out>
2018-03-29 04:32:45,382: INFO: proc(54): run_and_log_output: Output:         palErr = <optimized out>
2018-03-29 04:32:45,383: INFO: proc(54): run_and_log_output: Output:         dwRet = 4294967295
2018-03-29 04:32:45,383: INFO: proc(54): run_and_log_output: Output:         fAbandoned = <optimized out>
2018-03-29 04:32:45,383: INFO: proc(54): run_and_log_output: Output:         fNeedToBlock = <optimized out>
2018-03-29 04:32:45,383: INFO: proc(54): run_and_log_output: Output:         fWAll = <optimized out>
2018-03-29 04:32:45,383: INFO: proc(54): run_and_log_output: Output:         ppISyncWaitCtrlrs = <optimized out>
2018-03-29 04:32:45,384: INFO: proc(54): run_and_log_output: Output:         ppIPalObjs = <optimized out>
2018-03-29 04:32:45,384: INFO: proc(54): run_and_log_output: Output:         wtWaitType = <optimized out>
2018-03-29 04:32:45,417: INFO: proc(54): run_and_log_output: Output:         iSignaledObjCount = 1
2018-03-29 04:32:45,417: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#4  0x00007f26d2d37e31 in WaitForMultipleObjectsEx (nCount=3, lpHandles=0x7f26d1623e50, bWaitAll=0, dwMilliseconds=4294967295, bAlertable=0) at /root/coreclr/src/pal/src/synchmgr/wait.cpp:202
2018-03-29 04:32:45,418: INFO: proc(54): run_and_log_output: Output:         pThread = 0xfffffffffffffe00
2018-03-29 04:32:45,418: INFO: proc(54): run_and_log_output: Output:         dwRet = <optimized out>
2018-03-29 04:32:45,418: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#5  0x00007f26d2c38078 in DebuggerRCThread::MainLoop (this=<optimized out>) at /root/coreclr/src/debug/ee/rcthread.cpp:1234
2018-03-29 04:32:45,419: INFO: proc(54): run_and_log_output: Output:         rghWaitSet = {0x48, 0x50, 0x44, 0x1aa7d20}
2018-03-29 04:32:45,419: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#6  0x00007f26d2c37f0b in DebuggerRCThread::ThreadProc (this=0x1b2bdd0) at /root/coreclr/src/debug/ee/rcthread.cpp:1037
2018-03-29 04:32:45,419: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-29 04:32:45,420: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#7  0x00007f26d2c37be4 in DebuggerRCThread::ThreadProcStatic () at /root/coreclr/src/debug/ee/rcthread.cpp:1633
2018-03-29 04:32:45,420: INFO: proc(54): run_and_log_output: Output:         fSuccess = <optimized out>
2018-03-29 04:32:45,420: INFO: proc(54): run_and_log_output: Output:         t = 0x1b3ba14
2018-03-29 04:32:45,421: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#8  0x00007f26d2d3df82 in CorUnix::CPalThread::ThreadEntry (pvParam=0x1b3b7b0) at /root/coreclr/src/pal/src/thread/thread.cpp:1682
2018-03-29 04:32:45,421: INFO: proc(54): run_and_log_output: Output:         palError = <optimized out>
2018-03-29 04:32:45,421: INFO: proc(54): run_and_log_output: Output:         pvPar = 0x1b3ba14
2018-03-29 04:32:45,421: INFO: proc(54): run_and_log_output: Output:         retValue = <optimized out>
2018-03-29 04:32:45,422: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#9  0x00007f26d4199e25 in start_thread () from /lib64/libpthread.so.0
2018-03-29 04:32:45,422: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,422: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#10 0x00007f26d36a734d in clone () from /lib64/libc.so.6
2018-03-29 04:32:45,424: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,424: INFO: proc(54): run_and_log_output: Output: 
2018-03-29 04:32:45,424: INFO: proc(54): run_and_log_output: Output: Thread 3 (Thread 0x7f26d2626700 (LWP 35278)):
2018-03-29 04:32:45,425: INFO: proc(54): run_and_log_output: Output: #0  0x00007f26d369ca3d in poll () from /lib64/libc.so.6
2018-03-29 04:32:45,425: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,425: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#1  0x00007f26d2d3381e in CorUnix::CPalSynchronizationManager::ThreadPrepareForShutdown () at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:3220
2018-03-29 04:32:45,425: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-29 04:32:45,426: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#2  0x00007f26d2d351b0 in CorUnix::CPalSynchronizationManager::WorkerThread (pArg=<optimized out>) at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:1960
2018-03-29 04:32:45,426: INFO: proc(54): run_and_log_output: Output:         pthrWorker = 0x1a70230
2018-03-29 04:32:45,426: INFO: proc(54): run_and_log_output: Output:         palErr = 0
2018-03-29 04:32:45,427: INFO: proc(54): run_and_log_output: Output:         twrWakeUpReason = CorUnix::MutexAbondoned
2018-03-29 04:32:45,427: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#3  0x00007f26d2d3df82 in CorUnix::CPalThread::ThreadEntry (pvParam=0x1a70230) at /root/coreclr/src/pal/src/thread/thread.cpp:1682
2018-03-29 04:32:45,427: INFO: proc(54): run_and_log_output: Output:         palError = <optimized out>
2018-03-29 04:32:45,427: INFO: proc(54): run_and_log_output: Output:         pvPar = 0x0
2018-03-29 04:32:45,428: INFO: proc(54): run_and_log_output: Output:         retValue = <optimized out>
2018-03-29 04:32:45,428: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#4  0x00007f26d4199e25 in start_thread () from /lib64/libpthread.so.0
2018-03-29 04:32:45,429: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,429: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#5  0x00007f26d36a734d in clone () from /lib64/libc.so.6
2018-03-29 04:32:45,430: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,431: INFO: proc(54): run_and_log_output: Output: 
2018-03-29 04:32:45,431: INFO: proc(54): run_and_log_output: Output: Thread 2 (Thread 0x7f26d47c4740 (LWP 35277)):
2018-03-29 04:32:45,431: INFO: proc(54): run_and_log_output: Output: #0  0x00007f26d419d945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
2018-03-29 04:32:45,431: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,432: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#1  0x00007f26d2d33502 in CorUnix::CPalSynchronizationManager::ThreadNativeWait (ptnwdNativeWaitData=<optimized out>, dwTimeout=<optimized out>, ptwrWakeupReason=<optimized out>, pdwSignaledObject=<optimized out>) at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:479
2018-03-29 04:32:45,432: INFO: proc(54): run_and_log_output: Output:         iWaitRet = <optimized out>
2018-03-29 04:32:45,433: INFO: proc(54): run_and_log_output: Output:         palErr = 1359
2018-03-29 04:32:45,433: INFO: proc(54): run_and_log_output: Output:         iRet = <optimized out>
2018-03-29 04:32:45,433: INFO: proc(54): run_and_log_output: Output:         tsAbsTmo = <optimized out>
2018-03-29 04:32:45,434: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#2  0x00007f26d2d330e4 in CorUnix::CPalSynchronizationManager::BlockThread (this=0x1a9af40, pthrCurrent=0x1a9a590, dwTimeout=4294967295, fAlertable=true, fIsSleep=<optimized out>, ptwrWakeupReason=0x7ffd93e23450, pdwSignaledObject=0xfffffffffffffe00) at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:302
2018-03-29 04:32:45,434: INFO: proc(54): run_and_log_output: Output:         twrWakeupReason = <error reading variable twrWakeupReason (Cannot access memory at address 0x3)>
2018-03-29 04:32:45,434: INFO: proc(54): run_and_log_output: Output:         palErr = 0
2018-03-29 04:32:45,435: INFO: proc(54): run_and_log_output: Output:         dwSigObjIdx = <error reading variable dwSigObjIdx (Cannot access memory at address 0xffffffffffffffff)>
2018-03-29 04:32:45,435: INFO: proc(54): run_and_log_output: Output:         dwWaitState = <optimized out>
2018-03-29 04:32:45,435: INFO: proc(54): run_and_log_output: Output:         fEarlyDeath = false
2018-03-29 04:32:45,435: INFO: proc(54): run_and_log_output: Output:         fRaceAlerted = true
2018-03-29 04:32:45,436: INFO: proc(54): run_and_log_output: Output:         pdwWaitState = <optimized out>
2018-03-29 04:32:45,436: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#3  0x00007f26d2d37ab4 in CorUnix::InternalWaitForMultipleObjectsEx (pThread=0x1a9a590, nCount=<optimized out>, lpHandles=<optimized out>, bWaitAll=<optimized out>, dwMilliseconds=<optimized out>, bAlertable=<optimized out>, bPrioritize=<optimized out>) at /root/coreclr/src/pal/src/synchmgr/wait.cpp:636
2018-03-29 04:32:45,437: INFO: proc(54): run_and_log_output: Output:         twrWakeupReason = <error reading variable twrWakeupReason (Cannot access memory at address 0x1)>
2018-03-29 04:32:45,437: INFO: proc(54): run_and_log_output: Output:         pIPalObjStackArray = {0x1a66500, 0x0 <repeats 15 times>}
2018-03-29 04:32:45,437: INFO: proc(54): run_and_log_output: Output:         pISyncStackArray = {0x0 <repeats 16 times>}
2018-03-29 04:32:45,437: INFO: proc(54): run_and_log_output: Output:         iSignaledObjIndex = <optimized out>
2018-03-29 04:32:45,438: INFO: proc(54): run_and_log_output: Output:         palErr = <optimized out>
2018-03-29 04:32:45,438: INFO: proc(54): run_and_log_output: Output:         dwRet = 4294967295
2018-03-29 04:32:45,438: INFO: proc(54): run_and_log_output: Output:         fAbandoned = <optimized out>
2018-03-29 04:32:45,438: INFO: proc(54): run_and_log_output: Output:         fNeedToBlock = <optimized out>
2018-03-29 04:32:45,439: INFO: proc(54): run_and_log_output: Output:         fWAll = <optimized out>
2018-03-29 04:32:45,439: INFO: proc(54): run_and_log_output: Output:         ppISyncWaitCtrlrs = <optimized out>
2018-03-29 04:32:45,439: INFO: proc(54): run_and_log_output: Output:         ppIPalObjs = <optimized out>
2018-03-29 04:32:45,439: INFO: proc(54): run_and_log_output: Output:         wtWaitType = <optimized out>
2018-03-29 04:32:45,440: INFO: proc(54): run_and_log_output: Output:         iSignaledObjCount = 1
2018-03-29 04:32:45,440: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#4  0x00007f26d2d37e31 in WaitForMultipleObjectsEx (nCount=1, lpHandles=0x7ffd93e23760, bWaitAll=1, dwMilliseconds=4294967295, bAlertable=1) at /root/coreclr/src/pal/src/synchmgr/wait.cpp:202
2018-03-29 04:32:45,440: INFO: proc(54): run_and_log_output: Output:         pThread = 0xfffffffffffffe00
2018-03-29 04:32:45,440: INFO: proc(54): run_and_log_output: Output:         dwRet = <optimized out>
2018-03-29 04:32:45,441: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#5  0x00007f26d29a3e41 in WaitForMultipleObjectsEx_SO_TOLERANT (nCount=<optimized out>, lpHandles=0x7ffd93e23760, bWaitAll=1, bAlertable=1, dwMilliseconds=<optimized out>) at /root/coreclr/src/vm/threads.cpp:3462
2018-03-29 04:32:45,441: INFO: proc(54): run_and_log_output: Output:         lastError = 0
2018-03-29 04:32:45,442: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#6  DoAppropriateAptStateWait (numWaiters=1, timeout=<optimized out>, this=<optimized out>, pHandles=<optimized out>, bWaitAll=<optimized out>, mode=<optimized out>) at /root/coreclr/src/vm/threads.cpp:3496
2018-03-29 04:32:45,442: INFO: proc(54): run_and_log_output: Output:         alertable = 1
2018-03-29 04:32:45,442: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#7  Thread::DoAppropriateWaitWorker (this=0x1b3e5b0, countHandles=<optimized out>, handles=<optimized out>, waitAll=1, millis=<optimized out>, mode=WaitMode_None) at /root/coreclr/src/vm/threads.cpp:3633
2018-03-29 04:32:45,443: INFO: proc(54): run_and_log_output: Output:         ret = 0
2018-03-29 04:32:45,443: INFO: proc(54): run_and_log_output: Output:         alertable = 1
2018-03-29 04:32:45,443: INFO: proc(54): run_and_log_output: Output:         dwEnd = <optimized out>
2018-03-29 04:32:45,443: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#8  0x00007f26d299ed90 in operator() (this=<optimized out>, __pEEParam=<optimized out>) at /root/coreclr/src/vm/threads.cpp:3307
2018-03-29 04:32:45,444: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-29 04:32:45,444: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#9  Thread::DoAppropriateWait (this=0x1a9a7f4, countHandles=1, handles=0x3, waitAll=-1, millis=27895552, mode=WaitMode_Alertable, syncState=<optimized out>) at /root/coreclr/src/vm/threads.cpp:3322
2018-03-29 04:32:45,444: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-29 04:32:45,445: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#10 0x00007f26d29ef830 in WaitHandleNative::CorWaitOneNative (safeWaitHandleUNSAFE=<optimized out>, timeout=<optimized out>, hasThreadAffinity=<optimized out>, exitContext=<optimized out>) at /root/coreclr/src/vm/comwaithandle.cpp:198
2018-03-29 04:32:45,445: INFO: proc(54): run_and_log_output: Output:         sPauseTime = 0
2018-03-29 04:32:45,445: INFO: proc(54): run_and_log_output: Output:         handles = {0x200}
2018-03-29 04:32:45,446: INFO: proc(54): run_and_log_output: Output:         pThread = 0x1b3e5b0
2018-03-29 04:32:45,446: INFO: proc(54): run_and_log_output: Output:         res = 4294967295
2018-03-29 04:32:45,446: INFO: proc(54): run_and_log_output: Output:         __fExceptionCatched = false
2018-03-29 04:32:45,446: INFO: proc(54): run_and_log_output: Output:         CURRENT_THREAD = 0x1b3e5b0
2018-03-29 04:32:45,447: INFO: proc(54): run_and_log_output: Output:         hasCaughtException = false
2018-03-29 04:32:45,447: INFO: proc(54): run_and_log_output: Output:         sh = <optimized out>
2018-03-29 04:32:45,447: INFO: proc(54): run_and_log_output: Output:         alwaysZero = 0
2018-03-29 04:32:45,447: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#11 0x00007f2659424d3c in ?? ()
2018-03-29 04:32:45,447: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,448: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#12 0x0000000000000001 in ?? ()
2018-03-29 04:32:45,448: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,448: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#13 0x00007f26340217a0 in ?? ()
2018-03-29 04:32:45,448: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,449: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#14 0x00007ffd93e239e0 in ?? ()
2018-03-29 04:32:45,449: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,449: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#15 0x00007f26598d56f5 in ?? ()
2018-03-29 04:32:45,449: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,449: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#16 0x0000000000000000 in ?? ()
2018-03-29 04:32:45,450: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,450: INFO: proc(54): run_and_log_output: Output: 
2018-03-29 04:32:45,450: INFO: proc(54): run_and_log_output: Output: Thread 1 (Thread 0x7f26cbe9d700 (LWP 35287)):
2018-03-29 04:32:45,451: INFO: proc(54): run_and_log_output: Output: #0  0x00007f26d35e41f7 in raise () from /lib64/libc.so.6
2018-03-29 04:32:45,451: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,451: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#1  0x00007f26d35e58e8 in abort () from /lib64/libc.so.6
2018-03-29 04:32:45,456: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,456: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#2  0x00007f26d2d3c293 in PROCAbort () at /root/coreclr/src/pal/src/thread/process.cpp:3068
2018-03-29 04:32:45,496: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-29 04:32:45,497: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#3  0x00007f26d2d3b0fb in PROCEndProcess (hProcess=<optimized out>, uExitCode=3221225477, bTerminateUnconditionally=1) at /root/coreclr/src/pal/src/thread/process.cpp:1394
2018-03-29 04:32:45,497: INFO: proc(54): run_and_log_output: Output:         dwProcessId = <optimized out>
2018-03-29 04:32:45,498: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#4  0x00007f26d29f44be in SafeExitProcess (exitCode=3221225477, fAbort=1, sca=SCA_ExitProcessWhenShutdownComplete) at /root/coreclr/src/vm/eepolicy.cpp:519
2018-03-29 04:32:45,550: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-29 04:32:45,551: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#5  0x00007f26d29f5988 in EEPolicy::HandleFatalError (exitCode=<optimized out>, address=<optimized out>, pszMessage=0x0, pExceptionInfo=<optimized out>, errorSource=0x0, argExceptionString=0x0) at /root/coreclr/src/vm/eepolicy.cpp:1545
2018-03-29 04:32:45,551: INFO: proc(54): run_and_log_output: Output:         context = <optimized out>
2018-03-29 04:32:45,551: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#6  0x00007f26d2aa675e in ProcessCLRException (pExceptionRecord=<optimized out>, MemoryStackFp=<optimized out>, pContextRecord=0x7f261c018c30, pDispatcherContext=<optimized out>) at /root/coreclr/src/vm/exceptionhandling.cpp:1029
2018-03-29 04:32:45,552: INFO: proc(54): run_and_log_output: Output:         bAsynchronousThreadStop = <optimized out>
2018-03-29 04:32:45,552: INFO: proc(54): run_and_log_output: Output:         STState = <optimized out>
2018-03-29 04:32:45,552: INFO: proc(54): run_and_log_output: Output:         pTracker = 0x1b3c5b0
2018-03-29 04:32:45,552: INFO: proc(54): run_and_log_output: Output:         status = <optimized out>
2018-03-29 04:32:45,553: INFO: proc(54): run_and_log_output: Output:         dwLastError = 0
2018-03-29 04:32:45,553: INFO: proc(54): run_and_log_output: Output:         dwExceptionFlags = 0
2018-03-29 04:32:45,553: INFO: proc(54): run_and_log_output: Output:         pThread = 0x1c5f850
2018-03-29 04:32:45,553: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#7  0x00007f26d2aaa7e4 in UnwindManagedExceptionPass1 (ex=..., frameContext=<optimized out>) at /root/coreclr/src/vm/exceptionhandling.cpp:4630
2018-03-29 04:32:45,553: INFO: proc(54): run_and_log_output: Output:         codeInfo = <optimized out>
2018-03-29 04:32:45,554: INFO: proc(54): run_and_log_output: Output:         establisherFrame = 35287
2018-03-29 04:32:45,554: INFO: proc(54): run_and_log_output: Output:         controlPc = <optimized out>
2018-03-29 04:32:45,554: INFO: proc(54): run_and_log_output: Output:         disposition = 0
2018-03-29 04:32:45,554: INFO: proc(54): run_and_log_output: Output:         unwindStartContext = <optimized out>
2018-03-29 04:32:45,555: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#8  0x00007f26d2aaaae3 in DispatchManagedException (ex=..., isHardwareException=<optimized out>) at /root/coreclr/src/vm/exceptionhandling.cpp:4752
2018-03-29 04:32:45,555: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-29 04:32:45,555: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#9  0x00007f26d2aa5384 in HandleHardwareException (ex=<optimized out>) at /root/coreclr/src/vm/exceptionhandling.cpp:5275
2018-03-29 04:32:45,556: INFO: proc(54): run_and_log_output: Output:         controlPc = <optimized out>
2018-03-29 04:32:45,556: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#10 0x00007f26d2d045d1 in SEHProcessException (exception=<optimized out>) at /root/coreclr/src/pal/src/exception/seh.cpp:286
2018-03-29 04:32:45,556: INFO: proc(54): run_and_log_output: Output:         exceptionRecord = <optimized out>
2018-03-29 04:32:45,556: INFO: proc(54): run_and_log_output: Output:         contextRecord = <optimized out>
2018-03-29 04:32:45,557: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#11 0x00007f26d2d05e1f in common_signal_handler (code=<optimized out>, siginfo=<optimized out>, sigcontext=<optimized out>, numParams=<optimized out>) at /root/coreclr/src/pal/src/exception/signal.cpp:897
2018-03-29 04:32:45,557: INFO: proc(54): run_and_log_output: Output:         params = {{gp_offset = 48, fp_offset = 48, overflow_arg_area = 0x7f26cbe9ab40, reg_save_area = 0x7f26cbe9a430}}
2018-03-29 04:32:45,558: INFO: proc(54): run_and_log_output: Output:         ucontext = <optimized out>
2018-03-29 04:32:45,564: INFO: proc(54): run_and_log_output: Output:         signalContextRecord = {P1Home = 139802693833728, P2Home = 29751376, P3Home = 17864864936673296, P4Home = 139802677250808, P5Home = 139802676920168, P6Home = 1, ContextFlags = 135266379, MxCsr = 0, SegCs = 51, SegDs = 23008, SegEs = 32550, SegFs = 0, SegGs = 43280, SegSs = 43, EFlags = 66067, Dr0 = 139802693009440, Dr1 = 85899345940, Dr2 = 139802693352144, Dr3 = 4294967297, Dr6 = 1, Dr7 = 0, Rax = 0, Rcx = 0, Rdx = 255, Rbx = 139802069549123, Rsp = 139804606573712, Rbp = 139804606573712, Rsi = 0, Rdi = 139802069549123, R8 = 21, R9 = 4716521588802453544, R10 = 10729268994940096007, R11 = 8049312439267323969, R12 = 8, R13 = 139804606574344, R14 = 139802061342872, R15 = 0, Rip = 139802687858041, {FltSave = {ControlWord = 895, StatusWord = 16640, TagWord = 0 '\000', Reserved1 = 0 '\000', ErrorOpcode = 0, ErrorOffset = 3483969619, ErrorSelector = 0, Reserved2 = 0, DataOffset = 2481072200, DataSelector = 0, Reserved3 = 0, MxCsr = 8096, MxCsr_Mask = 65535, FloatRegisters = {{Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 14339461213547659264, High = 16390}, {Low = 9223372036854775808, High = 16385}, {Low = 9223372036854775808, High = 16385}}, XmmRegisters = {{Low = 0, High = 0}, {Low = 71776119077928960, High = -281474959998976}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 28429470871519353, High = 29555198975148141}, {Low = 23362560509739057, High = 28429470871519353}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 144680345676153346, High = 144680345676153346}, {Low = 18446744073709551615, High = -1}, {Low = 1082331758844, High = 1082331758844}, {Low = 30064771079, High = 30064771079}, {Low = 9223372039002259456, High = -9223372034707292160}, {Low = 9223372051887161347, High = -9223372021822390269}, {Low = 4294967297, High = 4294967297}, {Low = 18446744073709551609, High = -8}}, Reserved4 = "\000&\301ߠ\327\307=@\253\351\313&\177\000\000\b\251\351\313&\177\000\000`9\340Y&\177\000\000\260\250\351\313&\177\000\000DM\263\322&\177\000\000\000&\301ߠ\327\307= x\347Y&\177\000\000\000&\301ߠ\327\307=`9\340Y&\177\000\000\020\251\351\313&\177\000\000<\252\351\313&\177\000"}, {Header = {{Low = 1090519935, High = 3483969619}, {Low = 2481072200, High = 281470681751456}}, Legacy = {{Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 14339461213547659264, High = 16390}, {Low = 9223372036854775808, High = 16385}, {Low = 9223372036854775808, High = 16385}}, Xmm0 = {Low = 0, High = 0}, Xmm1 = {Low = 71776119077928960, High = -281474959998976}, Xmm2 = {Low = 0, High = 0}, Xmm3 = {Low = 0, High = 0}, Xmm4 = {Low = 28429470871519353, High = 29555198975148141}, Xmm5 = {Low = 23362560509739057, High = 28429470871519353}, Xmm6 = {Low = 0, High = 0}, Xmm7 = {Low = 0, High = 0}, Xmm8 = {Low = 144680345676153346, High = 144680345676153346}, Xmm9 = {Low = 18446744073709551615, High = -1}, Xmm10 = {Low = 1082331758844, High = 1082331758844}, Xmm11 = {Low = 30064771079, High = 30064771079}, Xmm12 = {Low = 9223372039002259456, High = -9223372034707292160}, Xmm13 = {Low = 9223372051887161347, High = -9223372021822390269}, Xmm14 = {Low = 4294967297, High = 4294967297}, Xmm15 = {Low = 18446744073709551609, High = -8}}}, VectorRegister = {{Low = 0, High = 0} <repeats 16 times>, {Low = 139801655300384, High = 139804725626296}, {Low = 139804725626252, High = 139804725626256}, {Low = 240, High = 139804725626208}, {Low = 1, High = 0}, {Low = 29751376, High = 1}, {Low = 139804606573120, High = 1}, {Low = 139804606573376, High = 139801655300384}, {Low = 28475200, High = 139804722521640}, {Low = 28475200, High = 139801655300384}, {Low = 139804606573168, High = 139804718560920}}, VectorControl = 0, DebugControl = 0, LastBranchToRip = 139802693826592, LastBranchFromRip = 139802693826608, LastExceptionToRip = 139802693351776, LastExceptionFromRip = 0}
2018-03-29 04:32:45,565: INFO: proc(54): run_and_log_output: Output:         signal_set = {__val = {1024, 0 <repeats 15 times>}}
2018-03-29 04:32:45,565: INFO: proc(54): run_and_log_output: Output:         exception = <error reading variable exception (Cannot access memory at address 0x89cd)>
2018-03-29 04:32:45,566: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#12 0x00007f26d2d05b86 in signal_handler_worker (code=35277, siginfo=0x7f261c0040f0, context=0x6, returnPoint=0x7f261c003a30) at /root/coreclr/src/pal/src/exception/signal.cpp:436
2018-03-29 04:32:45,566: INFO: proc(54): run_and_log_output: Output:         signal_set = {__val = {8589934592, 0 <repeats 15 times>}}
2018-03-29 04:32:45,566: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#13 0x00007f26d2d3ff36 in CallSignalHandlerWrapper0 () from /home/helixbot/dotnetbuild/work/629aff01-20b2-4a00-90f6-6d8cba36b034/Payload/shared/Microsoft.NETCore.App/9.9.9/libcoreclr.so
2018-03-29 04:32:45,567: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,567: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#14 0x00007f26598c6579 in ?? ()
2018-03-29 04:32:45,568: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,568: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#15 0x00007f26cbe9ad40 in ?? ()
2018-03-29 04:32:45,569: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,569: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#16 0x00007f2659e039dc in ?? ()
2018-03-29 04:32:45,570: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,571: INFO: proc(54): run_and_log_output: Output: dotnet/runtime#13850 0x0065007400730079 in ?? ()
2018-03-29 04:32:45,572: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,572: INFO: proc(54): run_and_log_output: Output: dotnet/runtime#13851 0x0000000000000000 in ?? ()
2018-03-29 04:32:45,574: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-29 04:32:45,587: INFO: proc(54): run_and_log_output: Output: Copying core file core.35277 to /tmp/coredumps/core.29500 in case you need it.
2018-03-29 04:32:45,719: INFO: proc(58): run_and_log_output: Exit Code: 134
2018-03-29 04:32:45,720: ERROR: scriptrunner(91): _main: Error: No exception thrown, but XUnit results not created
2018-03-29 04:32:45,720: ERROR: helix_test_execution(83): report_error: Error running xunit None

OSX.1013.Amd64.Open-x64:Debug
https://mc.dot.net/#/user/dotnet-bot/pr~2Fjenkins~2Fdotnet~2Fcorefx~2Fmaster~2F/test~2Ffunctional~2Fcli~2F/33e32ffc690ac54cfa1cb3571f8d9ee7cb7600be/workItem/System.IO.FileSystem.Tests/wilogs

2018-03-28 21:47:50,965: INFO: proc(54): run_and_log_output: Output: /Users/dotnet-bot/dotnetbuild/work/b6a7f00e-5641-4edb-8f93-54f23875a5f8/Work/726cc166-ffd3-4b9d-abfa-4a6c43b23ec5/Unzip/RunTests.sh: line 89: 74196 Killed: 9               $RUNTIME_PATH/dotnet xunit.console.netcore.exe System.IO.FileSystem.Tests.dll -xml testResults.xml -notrait Benchmark=true -notrait category=nonnetcoreapptests -notrait category=nonosxtests -notrait category=OuterLoop -notrait category=failing
2018-03-28 21:47:51,114: INFO: proc(54): run_and_log_output: Output: Trying to find crash dumps for project: System.IO.FileSystem.Tests
2018-03-28 21:47:51,114: INFO: proc(54): run_and_log_output: Output: No new dump file was found in /Users/dotnet-bot/dotnetbuild/work/b6a7f00e-5641-4edb-8f93-54f23875a5f8/Work/726cc166-ffd3-4b9d-abfa-4a6c43b23ec5/Unzip
2018-03-28 21:47:51,117: INFO: proc(54): run_and_log_output: Output: ~/dotnetbuild/work/b6a7f00e-5641-4edb-8f93-54f23875a5f8/Work/726cc166-ffd3-4b9d-abfa-4a6c43b23ec5/Unzip
2018-03-28 21:47:51,121: INFO: proc(54): run_and_log_output: Output: ----- end 21:47:51 ----- exit code 137 ----------------------------------------------------------
2018-03-28 21:47:51,125: INFO: proc(58): run_and_log_output: Exit Code: 137
2018-03-28 21:47:51,127: ERROR: scriptrunner(91): _main: Error: No exception thrown, but XUnit results not created
2018-03-28 21:47:51,128: ERROR: helix_test_execution(83): report_error: Error running xunit None

ahsonkhan on 29 Mar 2018

@dotnet/jit-contrib this test is failing frequently on the CI and seeing JIT couple of times on the stack.
I currently can't see anything obviously wrong with XML code which could lead to such stack.

Any chance, someone saw something similar in the past or could take a quick look and give us some pointers on what could be wrong? Any advice might be useful here.

krwq on 29 Mar 2018

Looks like maybe an AV in Compiler::fgMorphArgs(GenTreeCall*)?

AndyAyersMS on 29 Mar 2018

Thanks @AndyAyersMS, any pointers on someone who might be familiar with this code and could help?

krwq on 29 Mar 2018

Likely anyone on the jit team can investigate -- @RussKeldorph FYI

Things will go a lot faster if there's a crashdump available. I can't tell from the above if that's the case. Otherwise we'll have to try recreating the problem. Speaking of which is there a repro environment?

AndyAyersMS on 29 Mar 2018

Seems like the repro is gone after restarting the run in the PR @dotnet/dnceng - is it possible to somehow get repro environment/dump for the issue @ahsonkhan has submitted (or similar failure)? (or some kind of instructions in how to get that repro/dump)

Here is info @MattGal has previously gave which might be useful in finding it:

Kusto query

WorkItems | where FriendlyName == "System.Xml.XmlSchemaSet.Tests" | where  ExitCode == 134 | join kind= inner (
   Jobs  | project  QueueName , JobId 
) on JobId   | project Finished, QueueName

SG is .NET Engineering Infrastructure Kusto Read

krwq on 29 Mar 2018

Is it possible to somehow get repro environment/dump for the issue @ahsonkhan has submitted (or similar failure)

If you have the jobName and the workItemName you could run the Repro Tool passing those parameters.
i.e. helix-repro.exe -j=<jobName> -w=<workItemName> + the access tokens

Ping me if you need help with that

maririos on 29 Mar 2018

Thanks @maririos, is jobid one of those on the path? If not how do we get it?
/home/helixbot/dotnetbuild/work/629aff01-20b2-4a00-90f6-6d8cba36b034/Work/9f8da0be-4e92-45b6-9df6-d943d54b9094/Unzip/RunTests.sh
or in this link somewher? https://mc.dot.net/#/user/dotnet-bot/pr~2Fjenkins~2Fdotnet~2Fcorefx~2Fmaster~2F/test~2Ffunctional~2Fcli~2F/33e32ffc690ac54cfa1cb3571f8d9ee7cb7600be/workItem/System.Runtime.Tests/wilogs

This seems to have been restarted in the PR and now is showing that it passes...

@AndyAyersMS full invocation will be something like this:
"\bvtsrv2\Tools\ReproEnvironmentTool\helix-repro.exe" -j= -w=System.Xml.XmlSchemaSet.Tests --helix-access-key= --repro-tool-access-key=

a627ecdb-b9c3-4bdd-b274-47806e3b54b8 is a job-id
Helix API key: https://helix.dot.net/
Repro API key: https://repro-tool.westus2.cloudapp.azure.com/

krwq on 29 Mar 2018

It is not in the url. If you are in MC is:

If you have the logs, like /home/helixbot/dotnetbuild/work/629aff01-20b2-4a00-90f6-6d8cba36b034/Work/9f8da0be-4e92-45b6-9df6-d943d54b9094/Unzip/RunTests.sh it is the string after the word Work, so in this case: 629aff01-20b2-4a00-90f6-6d8cba36b034

maririos on 29 Mar 2018

👍1

Please note that when the workitem fails, you automatically get a link that says Get Repro Environment and that will give you all the data you need

maririos on 29 Mar 2018

@erozenfeld Could you see if you can get the repro and/or a crashdump and investigate?

RussKeldorph on 30 Mar 2018

@maririos the last PR hitting it reported Copying core file core.35277 to /tmp/coredumps/core.29500 in case you need it is it preserved somewhere?

pjanotti on 30 Mar 2018

Idk about the dumps. maybe @danmosemsft or @MattGal ?

maririos on 30 Mar 2018

Clarification: the original crash regarding System.Xml.XmlSchemaSet.Tests with exit code 134 had its last repro on 2018-03-17T19:15:57.085. It still hits System.Xml.XmlSchema.XmlSchemaValidatorApi.Tests though. Some of the other crashes listed above w/ exit code 134 are for:

System.Runtime.Tests
System.Text.Encoding.Tests
System.IO.FileSystem.Tests

/cc @karelz @danmosemsft

pjanotti on 30 Mar 2018

Unless a crash is close to 100% likely to repro every time, using the repro tool won't give you the dump from logs. Automated dump uploading was handled by Dumpling previously and despite suggestions noone wanted to use the built-in upload_result script to put dumps in the results container for runs. Not much I can do unless someone wants to invest in fixing this. (@DrewScoggins is dumpling going back online any time soon?)

MattGal on 30 Mar 2018

@anipik is signed up to do the work to bypass dumpling and upload dumps - he has just been fixing ZBB issues. If the callstack isn't sufficient, I would grab a repro box and loop the test under a debugger.

danmosemsft on 30 Mar 2018

The real fix as you say is dumpling.

danmosemsft on 30 Mar 2018

I see an email offline from @DrewScoggins that the dumpling work should be wrapped up this week.

danmosemsft on 30 Mar 2018

@danmosemsft this week as end of today or next week?

krwq on 30 Mar 2018

Today, but @DrewScoggins needs to give an update.

danmosemsft on 30 Mar 2018

Today

DrewScoggins on 30 Mar 2018

👍1

The uploading piece I just turned back on. The viewing piece should be on later today, I am just waiting to get my CNAME so that I can get SSL on.

DrewScoggins on 30 Mar 2018

@danmosemsft I'm only trying to get the repro for the JIT team here but we can leave it assigned to me until then

krwq on 30 Mar 2018

Yeah that's fine...if it's not actionable today then it's not.

danmosemsft on 30 Mar 2018

@RussKeldorph @erozenfeld I was able to get some core dumps in my Ubuntu 16.04 VM for System.Xml.XmlSchemaValidatorApi.Tests let me know how to better make them available to you.

pjanotti on 30 Mar 2018

Reopened recently, marking as Post-ZBB

karelz on 31 Mar 2018

Talked with @erozenfeld offline, he will investigate it today/Monday

krwq on 31 Mar 2018

Failed again from https://github.com/dotnet/corefx/pull/28670
https://mc.dot.net/#/user/dotnet-maestro-bot/pr~2Fjenkins~2Fdotnet~2Fcorefx~2Fmaster~2F/test~2Ffunctional~2Fcli~2F/54951919147a4cd2fc641263b447179aa32b5f27/workItem/System.Xml.XmlSchema.XmlSchemaValidatorApi.Tests/wilogs
Centos.74.Amd64.Open-x64:Release

2018-03-31 02:32:39,264: INFO: proc(54): run_and_log_output: Output: Starting:    System.Xml.XmlSchema.XmlSchemaValidatorApi.Tests
2018-03-31 02:32:40,473: INFO: proc(54): run_and_log_output: Output: /home/helixbot/dotnetbuild/work/2f848e59-c80a-44b9-a520-66c4d51997b6/Work/3104ffbf-7ea7-4baa-93e7-2bb5f1ee7424/Unzip/RunTests.sh: line 131: 25897 Aborted                 (core dumped) $RUNTIME_PATH/dotnet xunit.console.netcore.exe System.Xml.XmlSchema.XmlSchemaValidatorApi.Tests.dll -xml testResults.xml -notrait Benchmark=true -notrait category=nonnetcoreapptests -notrait category=nonlinuxtests -notrait category=OuterLoop -notrait category=failing
2018-03-31 02:32:40,519: INFO: proc(54): run_and_log_output: Output: Dumpling has not been installed yet. Please run "DumplingHelper.py install_dumpling" before collect_dumps.
2018-03-31 02:32:40,522: INFO: proc(54): run_and_log_output: Output: ~/dotnetbuild/work/2f848e59-c80a-44b9-a520-66c4d51997b6/Work/3104ffbf-7ea7-4baa-93e7-2bb5f1ee7424/Unzip
2018-03-31 02:32:40,523: INFO: proc(54): run_and_log_output: Output: ----- end 02:32:40 ----- exit code 134 ----------------------------------------------------------
2018-03-31 02:32:40,524: INFO: proc(54): run_and_log_output: Output: exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.
2018-03-31 02:32:40,525: INFO: proc(54): run_and_log_output: Output: Looking around for any Linux dump...
2018-03-31 02:32:40,527: INFO: proc(54): run_and_log_output: Output: Looking for files matching core.25897 ...
2018-03-31 02:32:40,527: INFO: proc(54): run_and_log_output: Output: Printing info from core file core.25897
2018-03-31 02:32:40,527: INFO: proc(54): run_and_log_output: Output: lldb was not found. Unable to print core file.
2018-03-31 02:32:40,528: INFO: proc(54): run_and_log_output: Output: ----- start =============== GDB Output =====================================================
2018-03-31 02:32:40,528: INFO: proc(54): run_and_log_output: Output: printing native stack.
2018-03-31 02:32:40,685: INFO: proc(54): run_and_log_output: Output: Missing separate debuginfo for /home/helixbot/dotnetbuild/work/2f848e59-c80a-44b9-a520-66c4d51997b6/Payload/dotnet
2018-03-31 02:32:40,686: INFO: proc(54): run_and_log_output: Output: Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/f7/ffe392dbfa89201700ff51fb4d2aa6a32a1410.debug
2018-03-31 02:32:40,705: INFO: proc(54): run_and_log_output: Output: [New LWP 25907]
2018-03-31 02:32:40,706: INFO: proc(54): run_and_log_output: Output: [New LWP 25898]
2018-03-31 02:32:40,706: INFO: proc(54): run_and_log_output: Output: [New LWP 25906]
2018-03-31 02:32:40,706: INFO: proc(54): run_and_log_output: Output: [New LWP 25897]
2018-03-31 02:32:40,706: INFO: proc(54): run_and_log_output: Output: [New LWP 25905]
2018-03-31 02:32:40,706: INFO: proc(54): run_and_log_output: Output: [New LWP 25899]
2018-03-31 02:32:40,706: INFO: proc(54): run_and_log_output: Output: [New LWP 25900]
2018-03-31 02:32:40,706: INFO: proc(54): run_and_log_output: Output: [New LWP 25901]
2018-03-31 02:32:40,707: INFO: proc(54): run_and_log_output: Output: [New LWP 25902]
2018-03-31 02:32:40,728: INFO: proc(54): run_and_log_output: Output: [Thread debugging using libthread_db enabled]
2018-03-31 02:32:40,729: INFO: proc(54): run_and_log_output: Output: Using host libthread_db library "/lib64/libthread_db.so.1".
2018-03-31 02:32:40,763: INFO: proc(54): run_and_log_output: Output: Missing separate debuginfo for /home/helixbot/dotnetbuild/work/2f848e59-c80a-44b9-a520-66c4d51997b6/Payload/host/fxr/9.9.9/libhostfxr.so
2018-03-31 02:32:40,763: INFO: proc(54): run_and_log_output: Output: Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/bd/279a5cbf7148bc4e3d080e1398a86ecd14dd9f.debug
2018-03-31 02:32:40,769: INFO: proc(54): run_and_log_output: Output: Missing separate debuginfo for /home/helixbot/dotnetbuild/work/2f848e59-c80a-44b9-a520-66c4d51997b6/Payload/shared/Microsoft.NETCore.App/9.9.9/libhostpolicy.so
2018-03-31 02:32:40,770: INFO: proc(54): run_and_log_output: Output: Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/e8/c36b56ad0f7fce98676fa4b7b4cb545b573467.debug
2018-03-31 02:32:42,345: INFO: proc(54): run_and_log_output: Output: Missing separate debuginfo for /home/helixbot/dotnetbuild/work/2f848e59-c80a-44b9-a520-66c4d51997b6/Payload/shared/Microsoft.NETCore.App/9.9.9/libclrjit.so
2018-03-31 02:32:42,346: INFO: proc(54): run_and_log_output: Output: Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/3c/6003d47554fb78e5297c602247283a1c1afa78.debug
2018-03-31 02:32:42,447: INFO: proc(54): run_and_log_output: Output: Core was generated by `/home/helixbot/dotnetbuild/work/2f848e59-c80a-44b9-a520-66c4d51997b6/Payload/do'.
2018-03-31 02:32:42,447: INFO: proc(54): run_and_log_output: Output: Program terminated with signal 6, Aborted.
2018-03-31 02:32:42,447: INFO: proc(54): run_and_log_output: Output: #0  0x00007f93de47c1f7 in raise () from /lib64/libc.so.6
2018-03-31 02:32:42,462: INFO: proc(54): run_and_log_output: Output: 
2018-03-31 02:32:42,462: INFO: proc(54): run_and_log_output: Output: Thread 9 (Thread 0x7f93d8ced700 (LWP 25902)):
2018-03-31 02:32:42,462: INFO: proc(54): run_and_log_output: Output: #0  0x00007f93df03870d in read () from /lib64/libpthread.so.0
2018-03-31 02:32:42,463: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,463: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#1  0x00007f93d8cffd8f in SignalHandlerLoop (arg=<optimized out>) at /mnt/j/workspace/dotnet_corefx/master/linux-TGroup_netcoreapp+CGroup_Release+AGroup_x64+TestOuter_false_prtest/src/Native/Unix/System.Native/pal_signal.c:100
2018-03-31 02:32:42,464: INFO: proc(54): run_and_log_output: Output:         pipeFd = 46
2018-03-31 02:32:42,464: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#2  0x00007f93df031e25 in start_thread () from /lib64/libpthread.so.0
2018-03-31 02:32:42,464: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,465: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#3  0x00007f93de53f34d in clone () from /lib64/libc.so.6
2018-03-31 02:32:42,479: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,479: INFO: proc(54): run_and_log_output: Output: 
2018-03-31 02:32:42,480: INFO: proc(54): run_and_log_output: Output: Thread 8 (Thread 0x7f93db942700 (LWP 25901)):
2018-03-31 02:32:42,480: INFO: proc(54): run_and_log_output: Output: #0  0x00007f93df035cf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
2018-03-31 02:32:42,480: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,481: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#1  0x00007f93ddbcb625 in CorUnix::CPalSynchronizationManager::ThreadNativeWait (ptnwdNativeWaitData=<optimized out>, dwTimeout=<optimized out>, ptwrWakeupReason=<optimized out>, pdwSignaledObject=<optimized out>) at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:484
2018-03-31 02:32:42,481: INFO: proc(54): run_and_log_output: Output:         iWaitRet = 0
2018-03-31 02:32:42,481: INFO: proc(54): run_and_log_output: Output:         palErr = 1359
2018-03-31 02:32:42,481: INFO: proc(54): run_and_log_output: Output:         iRet = <optimized out>
2018-03-31 02:32:42,486: INFO: proc(54): run_and_log_output: Output:         tsAbsTmo = {tv_sec = 453, tv_nsec = 498258621}
2018-03-31 02:32:42,487: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#2  0x00007f93ddbcb244 in CorUnix::CPalSynchronizationManager::BlockThread (this=0x14406f0, pthrCurrent=0x152fdd0, dwTimeout=2000, fAlertable=false, fIsSleep=<optimized out>, ptwrWakeupReason=0x7f93db941a50, pdwSignaledObject=0xfffffffffffffdfc) at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:302
2018-03-31 02:32:42,487: INFO: proc(54): run_and_log_output: Output:         twrWakeupReason = <error reading variable twrWakeupReason (Cannot access memory at address 0x5)>
2018-03-31 02:32:42,488: INFO: proc(54): run_and_log_output: Output:         palErr = 0
2018-03-31 02:32:42,488: INFO: proc(54): run_and_log_output: Output:         dwSigObjIdx = <error reading variable dwSigObjIdx (Cannot access memory at address 0xffffffffffffffff)>
2018-03-31 02:32:42,488: INFO: proc(54): run_and_log_output: Output:         dwWaitState = <optimized out>
2018-03-31 02:32:42,488: INFO: proc(54): run_and_log_output: Output:         fEarlyDeath = false
2018-03-31 02:32:42,489: INFO: proc(54): run_and_log_output: Output:         fRaceAlerted = true
2018-03-31 02:32:42,489: INFO: proc(54): run_and_log_output: Output:         pdwWaitState = <optimized out>
2018-03-31 02:32:42,489: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#3  0x00007f93ddbcfc14 in CorUnix::InternalWaitForMultipleObjectsEx (pThread=0x152fdd0, nCount=<optimized out>, lpHandles=<optimized out>, bWaitAll=<optimized out>, dwMilliseconds=<optimized out>, bAlertable=<optimized out>, bPrioritize=<optimized out>) at /root/coreclr/src/pal/src/synchmgr/wait.cpp:636
2018-03-31 02:32:42,491: INFO: proc(54): run_and_log_output: Output:         twrWakeupReason = <error reading variable twrWakeupReason (Cannot access memory at address 0xffffffff)>
2018-03-31 02:32:42,491: INFO: proc(54): run_and_log_output: Output:         pIPalObjStackArray = {0x152e940, 0x0 <repeats 15 times>}
2018-03-31 02:32:42,492: INFO: proc(54): run_and_log_output: Output:         pISyncStackArray = {0x0 <repeats 16 times>}
2018-03-31 02:32:42,492: INFO: proc(54): run_and_log_output: Output:         iSignaledObjIndex = <optimized out>
2018-03-31 02:32:42,492: INFO: proc(54): run_and_log_output: Output:         palErr = <optimized out>
2018-03-31 02:32:42,492: INFO: proc(54): run_and_log_output: Output:         dwRet = 4294967295
2018-03-31 02:32:42,492: INFO: proc(54): run_and_log_output: Output:         fAbandoned = <optimized out>
2018-03-31 02:32:42,493: INFO: proc(54): run_and_log_output: Output:         fNeedToBlock = <optimized out>
2018-03-31 02:32:42,493: INFO: proc(54): run_and_log_output: Output:         fWAll = <optimized out>
2018-03-31 02:32:42,493: INFO: proc(54): run_and_log_output: Output:         ppISyncWaitCtrlrs = <optimized out>
2018-03-31 02:32:42,493: INFO: proc(54): run_and_log_output: Output:         ppIPalObjs = <optimized out>
2018-03-31 02:32:42,493: INFO: proc(54): run_and_log_output: Output:         wtWaitType = <optimized out>
2018-03-31 02:32:42,501: INFO: proc(54): run_and_log_output: Output:         iSignaledObjCount = 1
2018-03-31 02:32:42,502: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#4  0x00007f93ddbcfeb9 in WaitForSingleObjectEx (hHandle=<error reading variable: Cannot access memory at address 0x5>, dwMilliseconds=2000, bAlertable=0) at /root/coreclr/src/pal/src/synchmgr/wait.cpp:139
2018-03-31 02:32:42,502: INFO: proc(54): run_and_log_output: Output:         pThread = 0xfffffffffffffdfc
2018-03-31 02:32:42,503: INFO: proc(54): run_and_log_output: Output:         dwRet = <optimized out>
2018-03-31 02:32:42,503: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#5  0x00007f93dd9141f1 in CLREventWaitHelper2 (handle=0x1530034, dwMilliseconds=<optimized out>, alertable=<optimized out>) at /root/coreclr/src/vm/synch.cpp:385
2018-03-31 02:32:42,503: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-31 02:32:42,503: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#6  operator() (this=<optimized out>, pParam=<optimized out>) at /root/coreclr/src/vm/synch.cpp:411
2018-03-31 02:32:42,504: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-31 02:32:42,504: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#7  CLREventWaitHelper (dwMilliseconds=2000, alertable=<optimized out>, handle=<optimized out>) at /root/coreclr/src/vm/synch.cpp:413
2018-03-31 02:32:42,504: INFO: proc(54): run_and_log_output: Output:         __exceptionHolder = <optimized out>
2018-03-31 02:32:42,504: INFO: proc(54): run_and_log_output: Output:         __param = 0x7f93db941c30
2018-03-31 02:32:42,580: INFO: proc(54): run_and_log_output: Output:         disposition = -1
2018-03-31 02:32:42,581: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#8  CLREventBase::WaitEx (this=0x145ba20, dwMilliseconds=2000, mode=<optimized out>, syncState=0x0) at /root/coreclr/src/vm/synch.cpp:483
2018-03-31 02:32:42,581: INFO: proc(54): run_and_log_output: Output:         alertable = <optimized out>
2018-03-31 02:32:42,581: INFO: proc(54): run_and_log_output: Output:         pThread = <optimized out>
2018-03-31 02:32:42,582: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#9  0x00007f93dd8a431f in FinalizerThread::WaitForFinalizerEvent (event=0x145ba20) at /root/coreclr/src/vm/finalizerthread.cpp:408
2018-03-31 02:32:42,659: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-31 02:32:42,660: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#10 0x00007f93dd8a44c5 in FinalizerThread::FinalizerThreadWorker (args=<optimized out>) at /root/coreclr/src/vm/finalizerthread.cpp:564
2018-03-31 02:32:42,660: INFO: proc(54): run_and_log_output: Output:         bPriorityBoosted = 0
2018-03-31 02:32:42,660: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#11 0x00007f93dd8415ad in ManagedThreadBase_DispatchInner (pCallState=0x7f93db941e20) at /root/coreclr/src/vm/threads.cpp:8850
2018-03-31 02:32:42,661: INFO: proc(54): run_and_log_output: Output:         pThread = 0x152eee0
2018-03-31 02:32:42,661: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#12 ManagedThreadBase_DispatchMiddle (pCallState=0x7f93db941e20) at /root/coreclr/src/vm/threads.cpp:8901
2018-03-31 02:32:42,661: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-31 02:32:42,662: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#13 operator() (this=<optimized out>, pParam=<optimized out>) at /root/coreclr/src/vm/threads.cpp:9139
2018-03-31 02:32:42,662: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-31 02:32:42,662: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#14 operator() (this=<optimized out>, pArgs=<optimized out>) at /root/coreclr/src/vm/threads.cpp:9141
2018-03-31 02:32:42,663: INFO: proc(54): run_and_log_output: Output:         __exceptionHolder = <optimized out>
2018-03-31 02:32:42,663: INFO: proc(54): run_and_log_output: Output:         disposition = -1
2018-03-31 02:32:42,663: INFO: proc(54): run_and_log_output: Output:         __param = <optimized out>
2018-03-31 02:32:42,663: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#15 ManagedThreadBase_DispatchOuter (pCallState=<optimized out>) at /root/coreclr/src/vm/threads.cpp:9178
2018-03-31 02:32:42,663: INFO: proc(54): run_and_log_output: Output:         pThread = 0x152eee0
2018-03-31 02:32:42,664: INFO: proc(54): run_and_log_output: Output:         param = <optimized out>
2018-03-31 02:32:42,664: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#16 0x00007f93dd841d8e in ManagedThreadBase_NoADTransition (pTarget=0x7f93dd8a4450 <FinalizerThread::FinalizerThreadWorker(void*)>, filterType=FinalizerThread) at /root/coreclr/src/vm/threads.cpp:9223
2018-03-31 02:32:42,787: INFO: proc(54): run_and_log_output: Output:         CallState = {pAppDomainId = {m_dwId = 5}, pUnsafeAppDomain = 0x200000000, bDomainIsAsID = 0, pTarget = 0x153000800000000, args = 0x300000000, filterType = 0, flags = MTCSF_NormalBase}
2018-03-31 02:32:42,788: INFO: proc(54): run_and_log_output: Output: dotnet/runtime#13850 ManagedThreadBase::FinalizerBase (pTarget=0x7f93dd8a4450 <FinalizerThread::FinalizerThreadWorker(void*)>) at /root/coreclr/src/vm/threads.cpp:9249
2018-03-31 02:32:42,788: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-31 02:32:42,789: INFO: proc(54): run_and_log_output: Output: dotnet/runtime#13851 0x00007f93dd8a48bc in FinalizerThread::FinalizerThreadStart (args=<optimized out>) at /root/coreclr/src/vm/finalizerthread.cpp:764
2018-03-31 02:32:42,789: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-31 02:32:42,789: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#19 0x00007f93ddbd60e2 in CorUnix::CPalThread::ThreadEntry (pvParam=0x152fdd0) at /root/coreclr/src/pal/src/thread/thread.cpp:1682
2018-03-31 02:32:42,789: INFO: proc(54): run_and_log_output: Output:         palError = <optimized out>
2018-03-31 02:32:42,790: INFO: proc(54): run_and_log_output: Output:         pvPar = 0x1530034
2018-03-31 02:32:42,790: INFO: proc(54): run_and_log_output: Output:         retValue = <optimized out>
2018-03-31 02:32:42,790: INFO: proc(54): run_and_log_output: Output: dotnet/runtime#13852 0x00007f93df031e25 in start_thread () from /lib64/libpthread.so.0
2018-03-31 02:32:42,791: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,791: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#21 0x00007f93de53f34d in clone () from /lib64/libc.so.6
2018-03-31 02:32:42,793: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,793: INFO: proc(54): run_and_log_output: Output: 
2018-03-31 02:32:42,793: INFO: proc(54): run_and_log_output: Output: Thread 7 (Thread 0x7f93dc4bc700 (LWP 25900)):
2018-03-31 02:32:42,793: INFO: proc(54): run_and_log_output: Output: #0  0x00007f93df035945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
2018-03-31 02:32:42,794: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,794: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#1  0x00007f93ddbcb662 in CorUnix::CPalSynchronizationManager::ThreadNativeWait (ptnwdNativeWaitData=<optimized out>, dwTimeout=<optimized out>, ptwrWakeupReason=<optimized out>, pdwSignaledObject=<optimized out>) at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:479
2018-03-31 02:32:42,795: INFO: proc(54): run_and_log_output: Output:         iWaitRet = <optimized out>
2018-03-31 02:32:42,795: INFO: proc(54): run_and_log_output: Output:         palErr = 1359
2018-03-31 02:32:42,795: INFO: proc(54): run_and_log_output: Output:         iRet = <optimized out>
2018-03-31 02:32:42,796: INFO: proc(54): run_and_log_output: Output:         tsAbsTmo = <optimized out>
2018-03-31 02:32:42,796: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#2  0x00007f93ddbcb244 in CorUnix::CPalSynchronizationManager::BlockThread (this=0x14406f0, pthrCurrent=0x15094b0, dwTimeout=4294967295, fAlertable=false, fIsSleep=<optimized out>, ptwrWakeupReason=0x7f93dc4bbcb0, pdwSignaledObject=0xfffffffffffffe00) at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:302
2018-03-31 02:32:42,796: INFO: proc(54): run_and_log_output: Output:         twrWakeupReason = <error reading variable twrWakeupReason (Cannot access memory at address 0x1)>
2018-03-31 02:32:42,796: INFO: proc(54): run_and_log_output: Output:         palErr = 0
2018-03-31 02:32:42,797: INFO: proc(54): run_and_log_output: Output:         dwSigObjIdx = <error reading variable dwSigObjIdx (Cannot access memory at address 0xffffffffffffffff)>
2018-03-31 02:32:42,797: INFO: proc(54): run_and_log_output: Output:         dwWaitState = <optimized out>
2018-03-31 02:32:42,797: INFO: proc(54): run_and_log_output: Output:         fEarlyDeath = false
2018-03-31 02:32:42,798: INFO: proc(54): run_and_log_output: Output:         fRaceAlerted = true
2018-03-31 02:32:42,798: INFO: proc(54): run_and_log_output: Output:         pdwWaitState = <optimized out>
2018-03-31 02:32:42,799: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#3  0x00007f93ddbcfc14 in CorUnix::InternalWaitForMultipleObjectsEx (pThread=0x15094b0, nCount=<optimized out>, lpHandles=<optimized out>, bWaitAll=<optimized out>, dwMilliseconds=<optimized out>, bAlertable=<optimized out>, bPrioritize=<optimized out>) at /root/coreclr/src/pal/src/synchmgr/wait.cpp:636
2018-03-31 02:32:42,799: INFO: proc(54): run_and_log_output: Output:         twrWakeupReason = <error reading variable twrWakeupReason (Cannot access memory at address 0x0)>
2018-03-31 02:32:42,799: INFO: proc(54): run_and_log_output: Output:         pIPalObjStackArray = {0x14fbef0, 0x1508300, 0x14fbd10, 0x0 <repeats 13 times>}
2018-03-31 02:32:42,800: INFO: proc(54): run_and_log_output: Output:         pISyncStackArray = {0x0 <repeats 16 times>}
2018-03-31 02:32:42,800: INFO: proc(54): run_and_log_output: Output:         iSignaledObjIndex = <optimized out>
2018-03-31 02:32:42,800: INFO: proc(54): run_and_log_output: Output:         palErr = <optimized out>
2018-03-31 02:32:42,800: INFO: proc(54): run_and_log_output: Output:         dwRet = 4294967295
2018-03-31 02:32:42,800: INFO: proc(54): run_and_log_output: Output:         fAbandoned = <optimized out>
2018-03-31 02:32:42,801: INFO: proc(54): run_and_log_output: Output:         fNeedToBlock = <optimized out>
2018-03-31 02:32:42,801: INFO: proc(54): run_and_log_output: Output:         fWAll = <optimized out>
2018-03-31 02:32:42,801: INFO: proc(54): run_and_log_output: Output:         ppISyncWaitCtrlrs = <optimized out>
2018-03-31 02:32:42,801: INFO: proc(54): run_and_log_output: Output:         ppIPalObjs = <optimized out>
2018-03-31 02:32:42,801: INFO: proc(54): run_and_log_output: Output:         wtWaitType = <optimized out>
2018-03-31 02:32:42,839: INFO: proc(54): run_and_log_output: Output:         iSignaledObjCount = 1
2018-03-31 02:32:42,839: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#4  0x00007f93ddbcff91 in WaitForMultipleObjectsEx (nCount=3, lpHandles=0x7f93dc4bbe50, bWaitAll=0, dwMilliseconds=4294967295, bAlertable=0) at /root/coreclr/src/pal/src/synchmgr/wait.cpp:202
2018-03-31 02:32:42,840: INFO: proc(54): run_and_log_output: Output:         pThread = 0xfffffffffffffe00
2018-03-31 02:32:42,840: INFO: proc(54): run_and_log_output: Output:         dwRet = <optimized out>
2018-03-31 02:32:42,840: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#5  0x00007f93ddad0078 in DebuggerRCThread::MainLoop (this=<optimized out>) at /root/coreclr/src/debug/ee/rcthread.cpp:1234
2018-03-31 02:32:42,841: INFO: proc(54): run_and_log_output: Output:         rghWaitSet = {0x48, 0x50, 0x44, 0x1476d20}
2018-03-31 02:32:42,841: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#6  0x00007f93ddacff0b in DebuggerRCThread::ThreadProc (this=0x14f9ad0) at /root/coreclr/src/debug/ee/rcthread.cpp:1037
2018-03-31 02:32:42,841: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-31 02:32:42,842: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#7  0x00007f93ddacfbe4 in DebuggerRCThread::ThreadProcStatic () at /root/coreclr/src/debug/ee/rcthread.cpp:1633
2018-03-31 02:32:42,842: INFO: proc(54): run_and_log_output: Output:         fSuccess = <optimized out>
2018-03-31 02:32:42,843: INFO: proc(54): run_and_log_output: Output:         t = 0x1509714
2018-03-31 02:32:42,843: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#8  0x00007f93ddbd60e2 in CorUnix::CPalThread::ThreadEntry (pvParam=0x15094b0) at /root/coreclr/src/pal/src/thread/thread.cpp:1682
2018-03-31 02:32:42,843: INFO: proc(54): run_and_log_output: Output:         palError = <optimized out>
2018-03-31 02:32:42,843: INFO: proc(54): run_and_log_output: Output:         pvPar = 0x1509714
2018-03-31 02:32:42,844: INFO: proc(54): run_and_log_output: Output:         retValue = <optimized out>
2018-03-31 02:32:42,844: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#9  0x00007f93df031e25 in start_thread () from /lib64/libpthread.so.0
2018-03-31 02:32:42,844: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,845: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#10 0x00007f93de53f34d in clone () from /lib64/libc.so.6
2018-03-31 02:32:42,846: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,847: INFO: proc(54): run_and_log_output: Output: 
2018-03-31 02:32:42,847: INFO: proc(54): run_and_log_output: Output: Thread 6 (Thread 0x7f93dccbd700 (LWP 25899)):
2018-03-31 02:32:42,848: INFO: proc(54): run_and_log_output: Output: #0  0x00007f93df038ead in open64 () from /lib64/libpthread.so.0
2018-03-31 02:32:42,883: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,884: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#1  0x00007f93ddad9eef in TwoWayPipe::WaitForConnection (this=0x14fe390) at /root/coreclr/src/debug/debug-pal/unix/twowaypipe.cpp:90
2018-03-31 02:32:42,884: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-31 02:32:42,884: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#2  0x00007f93ddad2bc8 in DbgTransportSession::TransportWorker (this=0x14fe2b0) at /root/coreclr/src/debug/ee/../shared/dbgtransportsession.cpp:1320
2018-03-31 02:32:42,885: INFO: proc(54): run_and_log_output: Output:         pid = <optimized out>
2018-03-31 02:32:42,885: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#3  0x00007f93ddad1a79 in DbgTransportSession::TransportWorkerStatic (pvContext=0x14fe3a0) at /root/coreclr/src/debug/ee/../shared/dbgtransportsession.cpp:1236
2018-03-31 02:32:42,885: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-31 02:32:42,886: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#4  0x00007f93ddbd60e2 in CorUnix::CPalThread::ThreadEntry (pvParam=0x15086c0) at /root/coreclr/src/pal/src/thread/thread.cpp:1682
2018-03-31 02:32:42,886: INFO: proc(54): run_and_log_output: Output:         palError = <optimized out>
2018-03-31 02:32:42,887: INFO: proc(54): run_and_log_output: Output:         pvPar = 0x14fe3a0
2018-03-31 02:32:42,887: INFO: proc(54): run_and_log_output: Output:         retValue = <optimized out>
2018-03-31 02:32:42,887: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#5  0x00007f93df031e25 in start_thread () from /lib64/libpthread.so.0
2018-03-31 02:32:42,888: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,888: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#6  0x00007f93de53f34d in clone () from /lib64/libc.so.6
2018-03-31 02:32:42,889: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,890: INFO: proc(54): run_and_log_output: Output: 
2018-03-31 02:32:42,890: INFO: proc(54): run_and_log_output: Output: Thread 5 (Thread 0x7f93d8109700 (LWP 25905)):
2018-03-31 02:32:42,890: INFO: proc(54): run_and_log_output: Output: #0  0x00007f93df035945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
2018-03-31 02:32:42,891: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,891: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#1  0x00007f93ddbcb662 in CorUnix::CPalSynchronizationManager::ThreadNativeWait (ptnwdNativeWaitData=<optimized out>, dwTimeout=<optimized out>, ptwrWakeupReason=<optimized out>, pdwSignaledObject=<optimized out>) at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:479
2018-03-31 02:32:42,891: INFO: proc(54): run_and_log_output: Output:         iWaitRet = <optimized out>
2018-03-31 02:32:42,892: INFO: proc(54): run_and_log_output: Output:         palErr = 1359
2018-03-31 02:32:42,892: INFO: proc(54): run_and_log_output: Output:         iRet = <optimized out>
2018-03-31 02:32:42,892: INFO: proc(54): run_and_log_output: Output:         tsAbsTmo = <optimized out>
2018-03-31 02:32:42,893: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#2  0x00007f93ddbcb244 in CorUnix::CPalSynchronizationManager::BlockThread (this=0x14406f0, pthrCurrent=0x157c950, dwTimeout=4294967295, fAlertable=true, fIsSleep=<optimized out>, ptwrWakeupReason=0x7f93d8108380, pdwSignaledObject=0xfffffffffffffe00) at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:302
2018-03-31 02:32:42,893: INFO: proc(54): run_and_log_output: Output:         twrWakeupReason = <error reading variable twrWakeupReason (Cannot access memory at address 0x2f)>
2018-03-31 02:32:42,894: INFO: proc(54): run_and_log_output: Output:         palErr = 0
2018-03-31 02:32:42,894: INFO: proc(54): run_and_log_output: Output:         dwSigObjIdx = <error reading variable dwSigObjIdx (Cannot access memory at address 0xffffffffffffffff)>
2018-03-31 02:32:42,894: INFO: proc(54): run_and_log_output: Output:         dwWaitState = <optimized out>
2018-03-31 02:32:42,895: INFO: proc(54): run_and_log_output: Output:         fEarlyDeath = false
2018-03-31 02:32:42,895: INFO: proc(54): run_and_log_output: Output:         fRaceAlerted = true
2018-03-31 02:32:42,895: INFO: proc(54): run_and_log_output: Output:         pdwWaitState = <optimized out>
2018-03-31 02:32:42,896: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#3  0x00007f93ddbcfc14 in CorUnix::InternalWaitForMultipleObjectsEx (pThread=0x157c950, nCount=<optimized out>, lpHandles=<optimized out>, bWaitAll=<optimized out>, dwMilliseconds=<optimized out>, bAlertable=<optimized out>, bPrioritize=<optimized out>) at /root/coreclr/src/pal/src/synchmgr/wait.cpp:636
2018-03-31 02:32:42,896: INFO: proc(54): run_and_log_output: Output:         twrWakeupReason = <error reading variable twrWakeupReason (Cannot access memory at address 0x17)>
2018-03-31 02:32:42,897: INFO: proc(54): run_and_log_output: Output:         pIPalObjStackArray = {0x1578ad0, 0x0 <repeats 15 times>}
2018-03-31 02:32:42,897: INFO: proc(54): run_and_log_output: Output:         pISyncStackArray = {0x0 <repeats 16 times>}
2018-03-31 02:32:42,897: INFO: proc(54): run_and_log_output: Output:         iSignaledObjIndex = <optimized out>
2018-03-31 02:32:42,897: INFO: proc(54): run_and_log_output: Output:         palErr = <optimized out>
2018-03-31 02:32:42,898: INFO: proc(54): run_and_log_output: Output:         dwRet = 4294967295
2018-03-31 02:32:42,898: INFO: proc(54): run_and_log_output: Output:         fAbandoned = <optimized out>
2018-03-31 02:32:42,898: INFO: proc(54): run_and_log_output: Output:         fNeedToBlock = <optimized out>
2018-03-31 02:32:42,898: INFO: proc(54): run_and_log_output: Output:         fWAll = <optimized out>
2018-03-31 02:32:42,899: INFO: proc(54): run_and_log_output: Output:         ppISyncWaitCtrlrs = <optimized out>
2018-03-31 02:32:42,899: INFO: proc(54): run_and_log_output: Output:         ppIPalObjs = <optimized out>
2018-03-31 02:32:42,899: INFO: proc(54): run_and_log_output: Output:         wtWaitType = <optimized out>
2018-03-31 02:32:42,899: INFO: proc(54): run_and_log_output: Output:         iSignaledObjCount = 1
2018-03-31 02:32:42,900: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#4  0x00007f93ddbcff91 in WaitForMultipleObjectsEx (nCount=1, lpHandles=0x7f93d8108690, bWaitAll=1, dwMilliseconds=4294967295, bAlertable=1) at /root/coreclr/src/pal/src/synchmgr/wait.cpp:202
2018-03-31 02:32:42,900: INFO: proc(54): run_and_log_output: Output:         pThread = 0xfffffffffffffe00
2018-03-31 02:32:42,900: INFO: proc(54): run_and_log_output: Output:         dwRet = <optimized out>
2018-03-31 02:32:42,900: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#5  0x00007f93dd83be41 in WaitForMultipleObjectsEx_SO_TOLERANT (nCount=<optimized out>, lpHandles=0x7f93d8108690, bWaitAll=1, bAlertable=1, dwMilliseconds=<optimized out>) at /root/coreclr/src/vm/threads.cpp:3462
2018-03-31 02:32:42,901: INFO: proc(54): run_and_log_output: Output:         lastError = 0
2018-03-31 02:32:42,901: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#6  DoAppropriateAptStateWait (numWaiters=1, timeout=<optimized out>, this=<optimized out>, pHandles=<optimized out>, bWaitAll=<optimized out>, mode=<optimized out>) at /root/coreclr/src/vm/threads.cpp:3496
2018-03-31 02:32:42,901: INFO: proc(54): run_and_log_output: Output:         alertable = 1
2018-03-31 02:32:42,902: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#7  Thread::DoAppropriateWaitWorker (this=0x157ba00, countHandles=<optimized out>, handles=<optimized out>, waitAll=1, millis=<optimized out>, mode=WaitMode_None) at /root/coreclr/src/vm/threads.cpp:3633
2018-03-31 02:32:42,902: INFO: proc(54): run_and_log_output: Output:         ret = 0
2018-03-31 02:32:42,902: INFO: proc(54): run_and_log_output: Output:         alertable = 1
2018-03-31 02:32:42,902: INFO: proc(54): run_and_log_output: Output:         dwEnd = <optimized out>
2018-03-31 02:32:42,903: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#8  0x00007f93dd836d90 in operator() (this=<optimized out>, __pEEParam=<optimized out>) at /root/coreclr/src/vm/threads.cpp:3307
2018-03-31 02:32:42,933: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-31 02:32:42,934: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#9  Thread::DoAppropriateWait (this=0x157cbb4, countHandles=1, handles=0x2f, waitAll=-1, millis=22530816, mode=(WaitMode_Alertable | WaitMode_IgnoreSyncCtx | WaitMode_ADUnload | unknown: 16), syncState=<optimized out>) at /root/coreclr/src/vm/threads.cpp:3322
2018-03-31 02:32:42,936: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-31 02:32:42,937: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#10 0x00007f93dd887830 in WaitHandleNative::CorWaitOneNative (safeWaitHandleUNSAFE=<optimized out>, timeout=<optimized out>, hasThreadAffinity=<optimized out>, exitContext=<optimized out>) at /root/coreclr/src/vm/comwaithandle.cpp:198
2018-03-31 02:32:42,937: INFO: proc(54): run_and_log_output: Output:         sPauseTime = 0
2018-03-31 02:32:42,937: INFO: proc(54): run_and_log_output: Output:         handles = {0x1e4}
2018-03-31 02:32:42,938: INFO: proc(54): run_and_log_output: Output:         pThread = 0x157ba00
2018-03-31 02:32:42,938: INFO: proc(54): run_and_log_output: Output:         res = 4294967295
2018-03-31 02:32:42,938: INFO: proc(54): run_and_log_output: Output:         __fExceptionCatched = false
2018-03-31 02:32:42,938: INFO: proc(54): run_and_log_output: Output:         CURRENT_THREAD = 0x157ba00
2018-03-31 02:32:42,939: INFO: proc(54): run_and_log_output: Output:         hasCaughtException = false
2018-03-31 02:32:42,939: INFO: proc(54): run_and_log_output: Output:         sh = <optimized out>
2018-03-31 02:32:42,939: INFO: proc(54): run_and_log_output: Output:         alwaysZero = 0
2018-03-31 02:32:42,939: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#11 0x00007f9364786457 in ?? ()
2018-03-31 02:32:42,940: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,940: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#12 0x00007f933ca761b0 in ?? ()
2018-03-31 02:32:42,940: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,941: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#13 0x00007f933c013eb8 in ?? ()
2018-03-31 02:32:42,941: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,942: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#14 0x00007f93d8108810 in ?? ()
2018-03-31 02:32:42,943: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,943: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#15 0x00007f9364785b94 in ?? ()
2018-03-31 02:32:42,944: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,944: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#16 0x00007f93d81087f0 in ?? ()
2018-03-31 02:32:42,945: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,945: INFO: proc(54): run_and_log_output: Output: dotnet/runtime#13850 0x00007f936423d5b3 in ?? ()
2018-03-31 02:32:42,946: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,946: INFO: proc(54): run_and_log_output: Output: dotnet/runtime#13851 0x00007f933c06cf58 in ?? ()
2018-03-31 02:32:42,947: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,948: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#19 0x0000000000000000 in ?? ()
2018-03-31 02:32:42,949: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,949: INFO: proc(54): run_and_log_output: Output: 
2018-03-31 02:32:42,950: INFO: proc(54): run_and_log_output: Output: Thread 4 (Thread 0x7f93df65b740 (LWP 25897)):
2018-03-31 02:32:42,950: INFO: proc(54): run_and_log_output: Output: #0  0x00007f93df035945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
2018-03-31 02:32:42,950: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,951: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#1  0x00007f93ddbcb662 in CorUnix::CPalSynchronizationManager::ThreadNativeWait (ptnwdNativeWaitData=<optimized out>, dwTimeout=<optimized out>, ptwrWakeupReason=<optimized out>, pdwSignaledObject=<optimized out>) at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:479
2018-03-31 02:32:42,951: INFO: proc(54): run_and_log_output: Output:         iWaitRet = <optimized out>
2018-03-31 02:32:42,952: INFO: proc(54): run_and_log_output: Output:         palErr = 1359
2018-03-31 02:32:42,952: INFO: proc(54): run_and_log_output: Output:         iRet = <optimized out>
2018-03-31 02:32:42,952: INFO: proc(54): run_and_log_output: Output:         tsAbsTmo = <optimized out>
2018-03-31 02:32:42,953: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#2  0x00007f93ddbcb244 in CorUnix::CPalSynchronizationManager::BlockThread (this=0x14406f0, pthrCurrent=0x143fd40, dwTimeout=4294967295, fAlertable=true, fIsSleep=<optimized out>, ptwrWakeupReason=0x7fff14f1e800, pdwSignaledObject=0xfffffffffffffe00) at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:302
2018-03-31 02:32:42,953: INFO: proc(54): run_and_log_output: Output:         twrWakeupReason = <error reading variable twrWakeupReason (Cannot access memory at address 0x3)>
2018-03-31 02:32:42,953: INFO: proc(54): run_and_log_output: Output:         palErr = 0
2018-03-31 02:32:42,954: INFO: proc(54): run_and_log_output: Output:         dwSigObjIdx = <error reading variable dwSigObjIdx (Cannot access memory at address 0xffffffffffffffff)>
2018-03-31 02:32:42,954: INFO: proc(54): run_and_log_output: Output:         dwWaitState = <optimized out>
2018-03-31 02:32:42,954: INFO: proc(54): run_and_log_output: Output:         fEarlyDeath = false
2018-03-31 02:32:42,954: INFO: proc(54): run_and_log_output: Output:         fRaceAlerted = true
2018-03-31 02:32:42,955: INFO: proc(54): run_and_log_output: Output:         pdwWaitState = <optimized out>
2018-03-31 02:32:42,955: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#3  0x00007f93ddbcfc14 in CorUnix::InternalWaitForMultipleObjectsEx (pThread=0x143fd40, nCount=<optimized out>, lpHandles=<optimized out>, bWaitAll=<optimized out>, dwMilliseconds=<optimized out>, bAlertable=<optimized out>, bPrioritize=<optimized out>) at /root/coreclr/src/pal/src/synchmgr/wait.cpp:636
2018-03-31 02:32:42,956: INFO: proc(54): run_and_log_output: Output:         twrWakeupReason = <error reading variable twrWakeupReason (Cannot access memory at address 0x1)>
2018-03-31 02:32:42,956: INFO: proc(54): run_and_log_output: Output:         pIPalObjStackArray = {0x1578340, 0x0 <repeats 15 times>}
2018-03-31 02:32:42,956: INFO: proc(54): run_and_log_output: Output:         pISyncStackArray = {0x0 <repeats 16 times>}
2018-03-31 02:32:42,957: INFO: proc(54): run_and_log_output: Output:         iSignaledObjIndex = <optimized out>
2018-03-31 02:32:42,957: INFO: proc(54): run_and_log_output: Output:         palErr = <optimized out>
2018-03-31 02:32:42,957: INFO: proc(54): run_and_log_output: Output:         dwRet = 4294967295
2018-03-31 02:32:42,957: INFO: proc(54): run_and_log_output: Output:         fAbandoned = <optimized out>
2018-03-31 02:32:42,958: INFO: proc(54): run_and_log_output: Output:         fNeedToBlock = <optimized out>
2018-03-31 02:32:42,958: INFO: proc(54): run_and_log_output: Output:         fWAll = <optimized out>
2018-03-31 02:32:42,958: INFO: proc(54): run_and_log_output: Output:         ppISyncWaitCtrlrs = <optimized out>
2018-03-31 02:32:42,958: INFO: proc(54): run_and_log_output: Output:         ppIPalObjs = <optimized out>
2018-03-31 02:32:42,959: INFO: proc(54): run_and_log_output: Output:         wtWaitType = <optimized out>
2018-03-31 02:32:42,959: INFO: proc(54): run_and_log_output: Output:         iSignaledObjCount = 1
2018-03-31 02:32:42,959: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#4  0x00007f93ddbcff91 in WaitForMultipleObjectsEx (nCount=1, lpHandles=0x7fff14f1eb10, bWaitAll=1, dwMilliseconds=4294967295, bAlertable=1) at /root/coreclr/src/pal/src/synchmgr/wait.cpp:202
2018-03-31 02:32:42,960: INFO: proc(54): run_and_log_output: Output:         pThread = 0xfffffffffffffe00
2018-03-31 02:32:42,960: INFO: proc(54): run_and_log_output: Output:         dwRet = <optimized out>
2018-03-31 02:32:42,961: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#5  0x00007f93dd83be41 in WaitForMultipleObjectsEx_SO_TOLERANT (nCount=<optimized out>, lpHandles=0x7fff14f1eb10, bWaitAll=1, bAlertable=1, dwMilliseconds=<optimized out>) at /root/coreclr/src/vm/threads.cpp:3462
2018-03-31 02:32:42,961: INFO: proc(54): run_and_log_output: Output:         lastError = 0
2018-03-31 02:32:42,961: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#6  DoAppropriateAptStateWait (numWaiters=1, timeout=<optimized out>, this=<optimized out>, pHandles=<optimized out>, bWaitAll=<optimized out>, mode=<optimized out>) at /root/coreclr/src/vm/threads.cpp:3496
2018-03-31 02:32:42,962: INFO: proc(54): run_and_log_output: Output:         alertable = 1
2018-03-31 02:32:42,962: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#7  Thread::DoAppropriateWaitWorker (this=0x150c2b0, countHandles=<optimized out>, handles=<optimized out>, waitAll=1, millis=<optimized out>, mode=WaitMode_None) at /root/coreclr/src/vm/threads.cpp:3633
2018-03-31 02:32:42,962: INFO: proc(54): run_and_log_output: Output:         ret = 0
2018-03-31 02:32:42,962: INFO: proc(54): run_and_log_output: Output:         alertable = 1
2018-03-31 02:32:42,963: INFO: proc(54): run_and_log_output: Output:         dwEnd = <optimized out>
2018-03-31 02:32:42,963: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#8  0x00007f93dd836d90 in operator() (this=<optimized out>, __pEEParam=<optimized out>) at /root/coreclr/src/vm/threads.cpp:3307
2018-03-31 02:32:42,963: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-31 02:32:42,964: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#9  Thread::DoAppropriateWait (this=0x143ffa4, countHandles=1, handles=0x3, waitAll=-1, millis=21233408, mode=WaitMode_Alertable, syncState=<optimized out>) at /root/coreclr/src/vm/threads.cpp:3322
2018-03-31 02:32:42,964: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-31 02:32:42,965: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#10 0x00007f93dd887830 in WaitHandleNative::CorWaitOneNative (safeWaitHandleUNSAFE=<optimized out>, timeout=<optimized out>, hasThreadAffinity=<optimized out>, exitContext=<optimized out>) at /root/coreclr/src/vm/comwaithandle.cpp:198
2018-03-31 02:32:42,965: INFO: proc(54): run_and_log_output: Output:         sPauseTime = 0
2018-03-31 02:32:42,966: INFO: proc(54): run_and_log_output: Output:         handles = {0x1e0}
2018-03-31 02:32:42,966: INFO: proc(54): run_and_log_output: Output:         pThread = 0x150c2b0
2018-03-31 02:32:42,966: INFO: proc(54): run_and_log_output: Output:         res = 4294967295
2018-03-31 02:32:42,966: INFO: proc(54): run_and_log_output: Output:         __fExceptionCatched = false
2018-03-31 02:32:42,967: INFO: proc(54): run_and_log_output: Output:         CURRENT_THREAD = 0x150c2b0
2018-03-31 02:32:42,967: INFO: proc(54): run_and_log_output: Output:         hasCaughtException = false
2018-03-31 02:32:42,967: INFO: proc(54): run_and_log_output: Output:         sh = <optimized out>
2018-03-31 02:32:42,967: INFO: proc(54): run_and_log_output: Output:         alwaysZero = 0
2018-03-31 02:32:42,967: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#11 0x00007f93642c514c in ?? ()
2018-03-31 02:32:42,968: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,968: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#12 0x0000000000000001 in ?? ()
2018-03-31 02:32:42,968: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,968: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#13 0x00007f933c0524f0 in ?? ()
2018-03-31 02:32:42,969: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,969: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#14 0x00007fff14f1ed90 in ?? ()
2018-03-31 02:32:42,969: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,969: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#15 0x00007f9364775615 in ?? ()
2018-03-31 02:32:42,970: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,970: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#16 0x0000000000000000 in ?? ()
2018-03-31 02:32:42,970: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,970: INFO: proc(54): run_and_log_output: Output: 
2018-03-31 02:32:42,970: INFO: proc(54): run_and_log_output: Output: Thread 3 (Thread 0x7f93d7908700 (LWP 25906)):
2018-03-31 02:32:42,971: INFO: proc(54): run_and_log_output: Output: #0  0x00007f93642adf82 in ?? ()
2018-03-31 02:32:42,971: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,971: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#1  0x00007f93645b8138 in ?? ()
2018-03-31 02:32:42,972: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,972: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#2  0x00007f933c0b0e74 in ?? ()
2018-03-31 02:32:42,973: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,973: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#3  0x00007f93d7905e30 in ?? ()
2018-03-31 02:32:42,974: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,974: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#4  0x00007f93642770ed in ?? ()
2018-03-31 02:32:42,976: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,976: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#5  0x00007f93d7905de0 in ?? ()
2018-03-31 02:32:42,977: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,977: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#6  0x0000063a64277640 in ?? ()
2018-03-31 02:32:42,979: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,979: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#7  0x0000000000000000 in ?? ()
2018-03-31 02:32:42,981: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,981: INFO: proc(54): run_and_log_output: Output: 
2018-03-31 02:32:42,981: INFO: proc(54): run_and_log_output: Output: Thread 2 (Thread 0x7f93dd4be700 (LWP 25898)):
2018-03-31 02:32:42,982: INFO: proc(54): run_and_log_output: Output: #0  0x00007f93de534a3d in poll () from /lib64/libc.so.6
2018-03-31 02:32:42,982: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,982: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#1  0x00007f93ddbcb97e in CorUnix::CPalSynchronizationManager::ThreadPrepareForShutdown () at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:3220
2018-03-31 02:32:42,983: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-31 02:32:42,983: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#2  0x00007f93ddbcd310 in CorUnix::CPalSynchronizationManager::WorkerThread (pArg=<optimized out>) at /root/coreclr/src/pal/src/synchmgr/synchmanager.cpp:1960
2018-03-31 02:32:42,983: INFO: proc(54): run_and_log_output: Output:         pthrWorker = 0x1469ed0
2018-03-31 02:32:42,983: INFO: proc(54): run_and_log_output: Output:         palErr = 0
2018-03-31 02:32:42,984: INFO: proc(54): run_and_log_output: Output:         twrWakeUpReason = CorUnix::MutexAbondoned
2018-03-31 02:32:42,984: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#3  0x00007f93ddbd60e2 in CorUnix::CPalThread::ThreadEntry (pvParam=0x1469ed0) at /root/coreclr/src/pal/src/thread/thread.cpp:1682
2018-03-31 02:32:42,984: INFO: proc(54): run_and_log_output: Output:         palError = <optimized out>
2018-03-31 02:32:42,985: INFO: proc(54): run_and_log_output: Output:         pvPar = 0x0
2018-03-31 02:32:42,985: INFO: proc(54): run_and_log_output: Output:         retValue = <optimized out>
2018-03-31 02:32:42,985: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#4  0x00007f93df031e25 in start_thread () from /lib64/libpthread.so.0
2018-03-31 02:32:42,986: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,986: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#5  0x00007f93de53f34d in clone () from /lib64/libc.so.6
2018-03-31 02:32:42,988: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,988: INFO: proc(54): run_and_log_output: Output: 
2018-03-31 02:32:42,988: INFO: proc(54): run_and_log_output: Output: Thread 1 (Thread 0x7f93d6fc6700 (LWP 25907)):
2018-03-31 02:32:42,988: INFO: proc(54): run_and_log_output: Output: #0  0x00007f93de47c1f7 in raise () from /lib64/libc.so.6
2018-03-31 02:32:42,989: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,989: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#1  0x00007f93de47d8e8 in abort () from /lib64/libc.so.6
2018-03-31 02:32:42,994: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:42,994: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#2  0x00007f93ddbd43f3 in PROCAbort () at /root/coreclr/src/pal/src/thread/process.cpp:3068
2018-03-31 02:32:43,036: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-31 02:32:43,037: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#3  0x00007f93ddbd325b in PROCEndProcess (hProcess=<optimized out>, uExitCode=3221225477, bTerminateUnconditionally=1) at /root/coreclr/src/pal/src/thread/process.cpp:1394
2018-03-31 02:32:43,037: INFO: proc(54): run_and_log_output: Output:         dwProcessId = <optimized out>
2018-03-31 02:32:43,038: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#4  0x00007f93dd88c4be in SafeExitProcess (exitCode=3221225477, fAbort=1, sca=SCA_ExitProcessWhenShutdownComplete) at /root/coreclr/src/vm/eepolicy.cpp:519
2018-03-31 02:32:43,099: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-31 02:32:43,099: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#5  0x00007f93dd88d988 in EEPolicy::HandleFatalError (exitCode=<optimized out>, address=<optimized out>, pszMessage=0x0, pExceptionInfo=<optimized out>, errorSource=0x0, argExceptionString=0x0) at /root/coreclr/src/vm/eepolicy.cpp:1545
2018-03-31 02:32:43,100: INFO: proc(54): run_and_log_output: Output:         context = <optimized out>
2018-03-31 02:32:43,100: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#6  0x00007f93dd93e75e in ProcessCLRException (pExceptionRecord=<optimized out>, MemoryStackFp=<optimized out>, pContextRecord=0x7f9324018e70, pDispatcherContext=<optimized out>) at /root/coreclr/src/vm/exceptionhandling.cpp:1029
2018-03-31 02:32:43,101: INFO: proc(54): run_and_log_output: Output:         bAsynchronousThreadStop = <optimized out>
2018-03-31 02:32:43,101: INFO: proc(54): run_and_log_output: Output:         STState = <optimized out>
2018-03-31 02:32:43,101: INFO: proc(54): run_and_log_output: Output:         pTracker = 0x150a2b0
2018-03-31 02:32:43,101: INFO: proc(54): run_and_log_output: Output:         status = <optimized out>
2018-03-31 02:32:43,102: INFO: proc(54): run_and_log_output: Output:         dwLastError = 0
2018-03-31 02:32:43,102: INFO: proc(54): run_and_log_output: Output:         dwExceptionFlags = 0
2018-03-31 02:32:43,102: INFO: proc(54): run_and_log_output: Output:         pThread = 0x1633100
2018-03-31 02:32:43,102: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#7  0x00007f93dd9427e4 in UnwindManagedExceptionPass1 (ex=..., frameContext=<optimized out>) at /root/coreclr/src/vm/exceptionhandling.cpp:4630
2018-03-31 02:32:43,103: INFO: proc(54): run_and_log_output: Output:         codeInfo = <optimized out>
2018-03-31 02:32:43,103: INFO: proc(54): run_and_log_output: Output:         establisherFrame = 25907
2018-03-31 02:32:43,103: INFO: proc(54): run_and_log_output: Output:         controlPc = <optimized out>
2018-03-31 02:32:43,103: INFO: proc(54): run_and_log_output: Output:         disposition = 0
2018-03-31 02:32:43,103: INFO: proc(54): run_and_log_output: Output:         unwindStartContext = <optimized out>
2018-03-31 02:32:43,104: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#8  0x00007f93dd942ae3 in DispatchManagedException (ex=..., isHardwareException=<optimized out>) at /root/coreclr/src/vm/exceptionhandling.cpp:4752
2018-03-31 02:32:43,104: INFO: proc(54): run_and_log_output: Output: No locals.
2018-03-31 02:32:43,104: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#9  0x00007f93dd93d384 in HandleHardwareException (ex=<optimized out>) at /root/coreclr/src/vm/exceptionhandling.cpp:5275
2018-03-31 02:32:43,105: INFO: proc(54): run_and_log_output: Output:         controlPc = <optimized out>
2018-03-31 02:32:43,106: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#10 0x00007f93ddb9c5d1 in SEHProcessException (exception=<optimized out>) at /root/coreclr/src/pal/src/exception/seh.cpp:286
2018-03-31 02:32:43,106: INFO: proc(54): run_and_log_output: Output:         exceptionRecord = <optimized out>
2018-03-31 02:32:43,106: INFO: proc(54): run_and_log_output: Output:         contextRecord = <optimized out>
2018-03-31 02:32:43,107: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#11 0x00007f93ddb9de1f in common_signal_handler (code=<optimized out>, siginfo=<optimized out>, sigcontext=<optimized out>, numParams=<optimized out>) at /root/coreclr/src/pal/src/exception/signal.cpp:897
2018-03-31 02:32:43,107: INFO: proc(54): run_and_log_output: Output:         params = {{gp_offset = 48, fp_offset = 48, overflow_arg_area = 0x7f93d6fc33f0, reg_save_area = 0x7f93d6fc2ce0}}
2018-03-31 02:32:43,107: INFO: proc(54): run_and_log_output: Output:         ucontext = <optimized out>
2018-03-31 02:32:43,162: INFO: proc(54): run_and_log_output: Output:         signalContextRecord = {P1Home = 0, P2Home = 21462616, P3Home = 140273084967712, P4Home = 140273051803300, P5Home = 140273078423120, P6Home = 140273051795128, ContextFlags = 135266379, MxCsr = 5, SegCs = 51, SegDs = 0, SegEs = 1, SegFs = 0, SegGs = 44864, SegSs = 43, EFlags = 66050, Dr0 = 140272943771688, Dr1 = 140272943771648, Dr2 = 1, Dr3 = 21462616, Dr6 = 21261776, Dr7 = 21260912, Rax = 16, Rcx = 140270344265288, Rdx = 21474836479, Rbx = 140272943773480, Rsp = 140272943772992, Rbp = 140272943773200, Rsi = 140272943773640, Rdi = 140270358489331, R8 = 17, R9 = 140269940961240, R10 = 0, R11 = 643, R12 = 140270344186124, R13 = 140272943773480, R14 = 1, R15 = 4294967295, Rip = 140271027401054, {FltSave = {ControlWord = 895, StatusWord = 0, TagWord = 0 '\000', Reserved1 = 0 '\000', ErrorOpcode = 0, ErrorOffset = 3739051624, ErrorSelector = 0, Reserved2 = 0, DataOffset = 351400680, DataSelector = 0, Reserved3 = 0, MxCsr = 8096, MxCsr_Mask = 65535, FloatRegisters = {{Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 14339461213547659264, High = 16390}, {Low = 14339461213547659264, High = 16390}, {Low = 12682136550675316736, High = 16386}}, XmmRegisters = {{Low = 0, High = 0}, {Low = 7882821499128804723, High = 110447100310113}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 68719476752, High = 68719476752}, {Low = 481036337168, High = 42949672976}, {Low = 68719476752, High = 68719476752}, {Low = 68719476752, High = 68719476752}, {Low = 0, High = 0}, {Low = 18446744073709551615, High = -1}, {Low = 1082331758844, High = 1082331758844}, {Low = 30064771079, High = 30064771079}, {Low = 9223372039002259456, High = -9223372034707292160}, {Low = 9223372051887161347, High = -9223372021822390269}, {Low = 4294967297, High = 4294967297}, {Low = 35, High = 35}}, Reserved4 = "@1\374֓\177\000\000\024\b\200ݓ\177\000\000\000\000\000\000\000\000\000\000\244m\221ݓ\177", '\000' <repeats 65 times>}, {Header = {{Low = 895, High = 3739051624}, {Low = 351400680, High = 281470681751456}}, Legacy = {{Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 14339461213547659264, High = 16390}, {Low = 14339461213547659264, High = 16390}, {Low = 12682136550675316736, High = 16386}}, Xmm0 = {Low = 0, High = 0}, Xmm1 = {Low = 7882821499128804723, High = 110447100310113}, Xmm2 = {Low = 0, High = 0}, Xmm3 = {Low = 0, High = 0}, Xmm4 = {Low = 68719476752, High = 68719476752}, Xmm5 = {Low = 481036337168, High = 42949672976}, Xmm6 = {Low = 68719476752, High = 68719476752}, Xmm7 = {Low = 68719476752, High = 68719476752}, Xmm8 = {Low = 0, High = 0}, Xmm9 = {Low = 18446744073709551615, High = -1}, Xmm10 = {Low = 1082331758844, High = 1082331758844}, Xmm11 = {Low = 30064771079, High = 30064771079}, Xmm12 = {Low = 9223372039002259456, High = -9223372034707292160}, Xmm13 = {Low = 9223372051887161347, High = -9223372021822390269}, Xmm14 = {Low = 4294967297, High = 4294967297}, Xmm15 = {Low = 35, High = 35}}}, VectorRegister = {{Low = 0, High = 0} <repeats 16 times>, {Low = 3, High = 140273060137720}, {Low = 140272943772480, High = 140273054959383}, {Low = 2, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}}, VectorControl = 0, DebugControl = 0, LastBranchToRip = 0, LastBranchFromRip = 0, LastExceptionToRip = 0, LastExceptionFromRip = 0}
2018-03-31 02:32:43,164: INFO: proc(54): run_and_log_output: Output:         signal_set = {__val = {1024, 0 <repeats 15 times>}}
2018-03-31 02:32:43,164: INFO: proc(54): run_and_log_output: Output:         exception = <error reading variable exception (Cannot access memory at address 0x6529)>
2018-03-31 02:32:43,165: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#12 0x00007f93ddb9db86 in signal_handler_worker (code=25897, siginfo=0x7f93240040f0, context=0x6, returnPoint=0x7f9324003a30) at /root/coreclr/src/pal/src/exception/signal.cpp:436
2018-03-31 02:32:43,165: INFO: proc(54): run_and_log_output: Output:         signal_set = {__val = {8589934592, 0 <repeats 15 times>}}
2018-03-31 02:32:43,165: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#13 0x00007f93ddbd8096 in CallSignalHandlerWrapper0 () from /home/helixbot/dotnetbuild/work/2f848e59-c80a-44b9-a520-66c4d51997b6/Payload/shared/Microsoft.NETCore.App/9.9.9/libcoreclr.so
2018-03-31 02:32:43,166: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:43,166: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#14 0x00007f9364c2b15e in ?? ()
2018-03-31 02:32:43,167: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:43,167: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#15 0x6d65686373627573 in ?? ()
2018-03-31 02:32:43,168: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:43,168: INFO: proc(54): run_and_log_output: Output: dotnet/corefx#16 0x0000000000000000 in ?? ()
2018-03-31 02:32:43,172: INFO: proc(54): run_and_log_output: Output: No symbol table info available.
2018-03-31 02:32:43,182: INFO: proc(54): run_and_log_output: Output: ----- end =============== GDB Output =======================================================
2018-03-31 02:32:43,184: INFO: proc(54): run_and_log_output: Output: Copying core file core.25897 to /tmp/coredumps/core.16617 in case you need it.
2018-03-31 02:32:43,721: INFO: proc(58): run_and_log_output: Exit Code: 134
2018-03-31 02:32:43,729: ERROR: scriptrunner(91): _main: Error: No exception thrown, but XUnit results not created
2018-03-31 02:32:43,729: ERROR: helix_test_execution(83): report_error: Error running xunit None

2018-03-31 02:32:43,730: INFO: azure_storage(205): _upload: Uploading single blob: 'error.log'
2018-03-31 02:32:43,777: INFO: saferequests(90): request_with_retry: Response complete with status code '201'

ahsonkhan on 31 Mar 2018

Crashed again here:
https://mc.dot.net/#/user/stephentoub/pr~2Fjenkins~2Fdotnet~2Fcorefx~2Fmaster~2F/test~2Ffunctional~2Fcli~2F/155361ca207ff26ad91a6bc7f03765367b4a2cba/workItem/System.Xml.XmlSchema.XmlSchemaValidatorApi.Tests/wilogs

2018-03-31 04:23:29,654: INFO: proc(54): run_and_log_output: Output: Discovering: System.Xml.XmlSchema.XmlSchemaValidatorApi.Tests
2018-03-31 04:23:29,868: INFO: proc(54): run_and_log_output: Output: Discovered:  System.Xml.XmlSchema.XmlSchemaValidatorApi.Tests
2018-03-31 04:23:29,975: INFO: proc(54): run_and_log_output: Output: Starting:    System.Xml.XmlSchema.XmlSchemaValidatorApi.Tests
2018-03-31 04:23:32,200: INFO: proc(54): run_and_log_output: Output: /home/helixbot/dotnetbuild/work/9b03aeee-b5d4-4201-83fa-87f06e311769/Work/c04ba7d4-f004-483b-960f-7931846055dd/Unzip/RunTests.sh: line 137: 45621 Aborted                 (core dumped) $RUNTIME_PATH/dotnet xunit.console.netcore.exe System.Xml.XmlSchema.XmlSchemaValidatorApi.Tests.dll -xml testResults.xml -notrait category=nonnetcoreapptests -notrait category=nonlinuxtests -notrait category=failing
2018-03-31 04:23:52,175: INFO: proc(54): run_and_log_output: Output: processing dump file /home/helixbot/dotnetbuild/work/9b03aeee-b5d4-4201-83fa-87f06e311769/Work/c04ba7d4-f004-483b-960f-7931846055dd/Unzip/core
2018-03-31 04:23:52,176: INFO: proc(54): run_and_log_output: Output: creating dumpling dump 39c7e1b0e9f2a82b5fb3e0468b3127bcdfa0dd64
2018-03-31 04:23:52,176: INFO: proc(54): run_and_log_output: Output: Traceback (most recent call last):
2018-03-31 04:23:52,176: INFO: proc(54): run_and_log_output: Output:   File "/home/helixbot/.dumpling/dumpling.py", line 1180, in <module>
2018-03-31 04:23:52,176: INFO: proc(54): run_and_log_output: Output:     main(sys.argv)
2018-03-31 04:23:52,177: INFO: proc(54): run_and_log_output: Output:   File "/home/helixbot/.dumpling/dumpling.py", line 1175, in main
2018-03-31 04:23:52,177: INFO: proc(54): run_and_log_output: Output:     cmdProc.Process(config)
2018-03-31 04:23:52,177: INFO: proc(54): run_and_log_output: Output:   File "/home/helixbot/.dumpling/dumpling.py", line 530, in Process
2018-03-31 04:23:52,177: INFO: proc(54): run_and_log_output: Output:     self.Upload(config)
2018-03-31 04:23:52,177: INFO: proc(54): run_and_log_output: Output:   File "/home/helixbot/.dumpling/dumpling.py", line 626, in Upload
2018-03-31 04:23:52,178: INFO: proc(54): run_and_log_output: Output:     self.UploadDump(config)
2018-03-31 04:23:52,178: INFO: proc(54): run_and_log_output: Output:   File "/home/helixbot/.dumpling/dumpling.py", line 637, in UploadDump
2018-03-31 04:23:52,178: INFO: proc(54): run_and_log_output: Output:     dumpdata = self._filequeue.UploadDump(config.dumppath, config.incpaths, config.user, config.displayname)
2018-03-31 04:23:52,179: INFO: proc(54): run_and_log_output: Output:   File "/home/helixbot/.dumpling/dumpling.py", line 519, in UploadDump
2018-03-31 04:23:52,179: INFO: proc(54): run_and_log_output: Output:     dumpData = self._dumpSvc.UploadDump(dumppath, hash, origin, displayname, fUpld)   
2018-03-31 04:23:52,179: INFO: proc(54): run_and_log_output: Output:   File "/home/helixbot/.dumpling/dumpling.py", line 294, in UploadDump
2018-03-31 04:23:52,179: INFO: proc(54): run_and_log_output: Output:     dumplingid = self.CreateDump(hash, origin, displayname)
2018-03-31 04:23:52,180: INFO: proc(54): run_and_log_output: Output:   File "/home/helixbot/.dumpling/dumpling.py", line 321, in CreateDump
2018-03-31 04:23:52,180: INFO: proc(54): run_and_log_output: Output:     response = requests.get(url)
2018-03-31 04:23:52,180: INFO: proc(54): run_and_log_output: Output:   File "/home/helixbot/.local/lib/python2.7/site-packages/requests/api.py", line 69, in get
2018-03-31 04:23:52,180: INFO: proc(54): run_and_log_output: Output:     return request('get', url, params=params, **kwargs)
2018-03-31 04:23:52,181: INFO: proc(54): run_and_log_output: Output:   File "/home/helixbot/.local/lib/python2.7/site-packages/requests/api.py", line 50, in request
2018-03-31 04:23:52,181: INFO: proc(54): run_and_log_output: Output:     response = session.request(method=method, url=url, **kwargs)
2018-03-31 04:23:52,181: INFO: proc(54): run_and_log_output: Output:   File "/home/helixbot/.local/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
2018-03-31 04:23:52,181: INFO: proc(54): run_and_log_output: Output:     resp = self.send(prep, **send_kwargs)
2018-03-31 04:23:52,182: INFO: proc(54): run_and_log_output: Output:   File "/home/helixbot/.local/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
2018-03-31 04:23:52,182: INFO: proc(54): run_and_log_output: Output:     r = adapter.send(request, **kwargs)
2018-03-31 04:23:52,182: INFO: proc(54): run_and_log_output: Output:   File "/home/helixbot/.local/lib/python2.7/site-packages/requests/adapters.py", line 431, in send
2018-03-31 04:23:52,182: INFO: proc(54): run_and_log_output: Output:     raise SSLError(e, request=request)
2018-03-31 04:23:52,183: INFO: proc(54): run_and_log_output: Output: requests.exceptions.SSLError: hostname 'dumpling.azurewebsites.net' doesn't match 'dumpling.int-dot.net'
2018-03-31 04:23:52,193: INFO: proc(54): run_and_log_output: Output: Trying to find crash dumps for project: System.Xml.XmlSchema.XmlSchemaValidatorApi.Tests
2018-03-31 04:23:52,194: INFO: proc(54): run_and_log_output: Output: Uploading dump file: /home/helixbot/dotnetbuild/work/9b03aeee-b5d4-4201-83fa-87f06e311769/Work/c04ba7d4-f004-483b-960f-7931846055dd/Unzip/core
2018-03-31 04:23:52,200: INFO: proc(54): run_and_log_output: Output: ~/dotnetbuild/work/9b03aeee-b5d4-4201-83fa-87f06e311769/Work/c04ba7d4-f004-483b-960f-7931846055dd/Unzip
2018-03-31 04:23:52,201: INFO: proc(54): run_and_log_output: Output: ----- end 04:23:52 ----- exit code 134 ----------------------------------------------------------
2018-03-31 04:23:52,201: INFO: proc(54): run_and_log_output: Output: exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.
2018-03-31 04:23:52,203: INFO: proc(54): run_and_log_output: Output: Looking around for any Linux dump...
2018-03-31 04:23:52,204: INFO: proc(54): run_and_log_output: Output: found a dump named core in /home/helixbot/dotnetbuild/work/9b03aeee-b5d4-4201-83fa-87f06e311769/Work/c04ba7d4-f004-483b-960f-7931846055dd/Unzip !
2018-03-31 04:23:52,204: INFO: proc(54): run_and_log_output: Output: Printing info from core file core
2018-03-31 04:23:52,204: INFO: proc(54): run_and_log_output: Output: lldb was not found. Unable to print core file.
2018-03-31 04:23:52,205: INFO: proc(54): run_and_log_output: Output: GDB was not found. Unable to print core file.
2018-03-31 04:23:52,206: INFO: proc(54): run_and_log_output: Output: Copying core file core to /tmp/coredumps/core.18664 in case you need it.

stephentoub on 31 Mar 2018

This is the second time that I use lldb so take this with a grain of salt :smile: but fortunately it seems similar enough to windbg+sos for me to be able to see some stuff...

According to core dumps there are two crashes happening in XmlSchemaValidatorApi tests:

https://github.com/dotnet/coreclr/blob/9496bc0546b73fdc4fdc8a176b2abd66bacf54db/src/mscorlib/shared/System/SpanHelpers.Byte.cs#L125

/root/coreclr/src/mscorlib/shared/System/SpanHelpers.Byte.cs @ 123:
00007fc837b26575 4983c0f8             add     r8, -0x8
>>> 00007fc837b26579 440fb60c0f           movzx   r9d, byte ptr [rdi + rcx]
00007fc837b2657e 443bce               cmp     r9d, esi
00007fc837b26581 0f84f4010000         je      0x7fc837b2677b

the name of the temp file being used seems fine, e.g.: /tmp/kpurfkrv.ia0.

The other crash happens at:

https://github.com/dotnet/corefx/blob/83c87a02da70b05ea0cd76a39430dd399bd6bbdc/src/System.IO.FileSystem/src/System/IO/Enumeration/FileSystemEntry.Unix.cs#L62

00007febd42fed5a 488b7dd0             mov     rdi, qword ptr [rbp - 0x30]
>>> 00007febd42fed5e 803f2e               cmp     byte ptr [rdi], 0x2e
00007febd42fed61 7503                 jne     0x7febd42fed66
00007febd42fed63 83c802               or      eax, 0x2
00007febd42fed66 85c0                 test    eax, eax

These two issues are hit by the tests being disabled by PR dotnet/corefx#28679 they have in common their usage of TempDirectory, so in principle any other test using this should be subject to the same error. This doesn't happen on every run.

pjanotti on 31 Mar 2018

Ok, these latest crashes are understood and I will be testing the fix.

pjanotti on 1 Apr 2018

A quick recap to make sense of this issue:

Initially a couple of call stacks for this seemed to indicate something related to jit
However, that initial test crash (XmlSchemaSet.Tests) did not repro since 3/17
Other SIGABRT crashes in tests were added to this bug, for the ones that I got repro, it was always the issue fixed by PR dotnet/corefx#28692
This issue can potentially affect any code using the FileSystemEnumerator. This includes TempDirectory and FileCleanUpTestBase so it may account for many "random" test crashes in Unix.

@RussKeldorph @erozenfeld at this point I don't see anything related to clr/jit to be investigated, thanks for jumping in.

pjanotti on 1 Apr 2018

👍1

@pjanotti there are 3 test cases related to this:

corefx/27740: System.Private.Xml/tests/XmlSchema/XmlSchemaSet/TC_SchemaSet_Compile.cs:88:        [ActiveIssue(27740, TestPlatforms.AnyUnix)]
corefx/27740: System.Private.Xml/tests/XmlSchema/XmlSchemaValidatorApi/ValidateMisc.cs:896:        [ActiveIssue(27740, TestPlatforms.AnyUnix)]
corefx/27740: System.Private.Xml/tests/XmlSchema/XmlSchemaValidatorApi/ValidateMisc.cs:923:        [ActiveIssue(27740, TestPlatforms.AnyUnix)]

should we reenable?

krwq on 4 Apr 2018

Ops, forgot to re-enable them...

pjanotti on 4 Apr 2018

Runtime: System.Xml.XmlSchemaSet.Tests crashed on Linux in CI

All 56 comments

Related issues