runtime 🚀 - [ARM/Linux] Unit Test Regression - JIT/Regression/CLR-x86-JIT/V1.2-M01/b07369

@wateret I also reproduced this regression with coreclr(361edab2925026970715546bdc7ccca0b6c058cd)(release built with -O1) and unix tests (x64, release) on arm-softfp.

hqueue on 14 Jul 2016

Looks like there's something wrong during compilation, specifically range analysis.

(gdb) bt
#0  0xb21d3242 in ExpandArrayStack<ValueNumStore::Chunk*>::GetNoExpand(unsigned int) () from /opt/home/owner/0714r/libclrjit.so [expandarray.h:165]
dotnet/coreclr#1  0xb2242498 in ValueNumStore::GetVNFunc(unsigned int, VNFuncApp*) () from /opt/home/owner/0714r/libclrjit.so  [valuenum.cpp:3240]
dotnet/coreclr#2  0xb2244a80 in ValueNumStore::GetConstantBoundInfo(unsigned int, ValueNumStore::ConstantBoundInfo*) () from /opt/home/owner/0714r/libclrjit.so [valuenum.cpp:2871]
dotnet/coreclr#3  0xb2232492 in RangeCheck::MergeEdgeAssertions(GenTree*, unsigned int*, Range*) () from /opt/home/owner/0714r/libclrjit.so [rangecheck.cpp:588]
dotnet/coreclr#4  0xb2232f52 in RangeCheck::ComputeRange(BasicBlock*, GenTree*, GenTree*, SimplerHashTable<GenTree*, PtrKeyFuncs<GenTree>, BasicBlock*, JitSimplerHashBehavior>*, bool) () from /opt/home/owner/0714r/libclrjit.so
dotnet/coreclr#5  0xb2231d9a in RangeCheck::GetRange(BasicBlock*, GenTree*, GenTree*, SimplerHashTable<GenTree*, PtrKeyFuncs<GenTree>, BasicBlock*, JitSimplerHashBehavior>*, bool) () from /opt/home/owner/0714r/libclrjit.so
dotnet/coreclr#6  0xb223306a in RangeCheck::ComputeRange(BasicBlock*, GenTree*, GenTree*, SimplerHashTable<GenTree*, PtrKeyFuncs<GenTree>, BasicBlock*, JitSimplerHashBehavior>*, bool) () from /opt/home/owner/0714r/libclrjit.so
dotnet/coreclr#7  0xb2231d9a in RangeCheck::GetRange(BasicBlock*, GenTree*, GenTree*, SimplerHashTable<GenTree*, PtrKeyFuncs<GenTree>, BasicBlock*, JitSimplerHashBehavior>*, bool) () from /opt/home/owner/0714r/libclrjit.so
dotnet/coreclr#8  0xb223293e in RangeCheck::ComputeRangeForLocalDef(BasicBlock*, GenTree*, GenTree*, SimplerHashTable<GenTree*, PtrKeyFuncs<GenTree>, BasicBlock*, JitSimplerHashBehavior>*, bool) () from /opt/home/owner/0714r/libclrjit.so
dotnet/coreclr#9  0xb2232f38 in RangeCheck::ComputeRange(BasicBlock*, GenTree*, GenTree*, SimplerHashTable<GenTree*, PtrKeyFuncs<GenTree>, BasicBlock*, JitSimplerHashBehavior>*, bool) () from /opt/home/owner/0714r/libclrjit.so
dotnet/coreclr#10 0xb2231d9a in RangeCheck::GetRange(BasicBlock*, GenTree*, GenTree*, SimplerHashTable<GenTree*, PtrKeyFuncs<GenTree>, BasicBlock*, JitSimplerHashBehavior>*, bool) () from /opt/home/owner/0714r/libclrjit.so
dotnet/coreclr#11 0xb2231c98 in RangeCheck::OptimizeRangeCheck(BasicBlock*, GenTree*, GenTree*) () from /opt/home/owner/0714r/libclrjit.so
dotnet/coreclr#12 0xb223326e in RangeCheck::OptimizeRangeChecks() () from /opt/home/owner/0714r/libclrjit.so
dotnet/coreclr#13 0xb21dc2aa in Compiler::compCompile(void**, unsigned int*, CORJIT_FLAGS*) () from /opt/home/owner/0714r/libclrjit.so
...

wateret on 14 Jul 2016

This happens after 36efe1f7527ed207b25283832b035a0db0fee2c8. And this commit is not related to range anlaysis at all.
The backtrace shows the error is from range analysis phase but I don't think it is actual problematic place. What do you think? @papaslavik

% This problem only occurs for coreclr-release and TC-release

wateret on 15 Jul 2016

@wateret so does it only happen after the commit, and does it happen only on softfp? I should have a look, although it's weird it's a release version...

papaslavik on 15 Jul 2016

cannot get running release system yet...

papaslavik on 15 Jul 2016

@mkborg builds everything for release, but corerun crashes very early at any input.
It crashes in code related to tls.

papaslavik on 15 Jul 2016

now we were able to do that

papaslavik on 15 Jul 2016

Though both TC versions release and debug are crashing to me.

papaslavik on 15 Jul 2016

@wateret do you have a generally stable release version in which the test fails just in case both of the TC and coreclr are release? And that it happens only after the commit 36efe1f ?

We can build the release version only with O1 right now, and we have crashes for both release and debug versions of the test.
And the crash occurs even for the commit 4ec1a4908 preceding 36efe1f

papaslavik on 18 Jul 2016

The crashes are also different from what you showed:

#0  0xb6771444 in ?? () from /usr/lib/arm-linux-gnueabi/libunwind-arm.so.8
dotnet/coreclr#1  0xb6771b00 in _Uarm_is_signal_frame () from /usr/lib/arm-linux-gnueabi/libunwind-arm.so.8
dotnet/coreclr#2  0xb6a76d34 in PAL_VirtualUnwind () from /home/krivtsov/v.nikolaev/coreoverlay/libcoreclr.so
dotnet/coreclr#3  0xb6a76f7c in RtlpRaiseException(_EXCEPTION_RECORD*) () from /home/krivtsov/v.nikolaev/coreoverlay/libcoreclr.so
dotnet/coreclr#4  0xb6a76f48 in RaiseException () from /home/krivtsov/v.nikolaev/coreoverlay/libcoreclr.so
dotnet/coreclr#5  0xb6876108 in RaiseTheExceptionInternalOnly(Object*, int, int)::$_1::operator()(RaiseTheExceptionInternalOnly(Object*, int, int)::Param*) const () from /home/krivtsov/v.nikolaev/coreoverlay/libcoreclr.so
dotnet/coreclr#6  0xb6875f84 in RaiseTheExceptionInternalOnly(Object*, int, int) () from /home/krivtsov/v.nikolaev/coreoverlay/libcoreclr.so
dotnet/coreclr#7  0xb6876688 in UnwindAndContinueRethrowHelperAfterCatch(Frame*, Exception*) () from /home/krivtsov/v.nikolaev/coreoverlay/libcoreclr.so
dotnet/coreclr#8  0xb68f7e44 in JIT_RngChkFail() () from /home/krivtsov/v.nikolaev/coreoverlay/libcoreclr.so
dotnet/coreclr#9  0xb20ed532 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

papaslavik on 18 Jul 2016

@Dmitri-Botcharnikov

papaslavik on 18 Jul 2016

So, to make sure I tried the clang3.6 provided here
https://github.com/dotnet/coreclr/blob/master/Documentation/building/linux-instructions.md#how-to-enable--o3-optimization-level-for-armlinux

and built on the newest commit 4ec1a4908 in release mode (with -O3) and didn't encounter any issues with the test.

I also tried it on the commit 36efe1f and it didn't have issues as well.

As for clang3.8 the issue occured in release version with -O1 for both debug and release versions of the b07369.exe and also it was present even before the commit 36efe1f

papaslavik on 18 Jul 2016

Perhaps I downloaded a different version of the exe... but it was claimed to be the newest release at the moment

papaslavik on 18 Jul 2016

@papaslavik
Here's how I build coreclr-release.

clang3.6 vanilla
optimization level changed - O3 to O1
test were built from coreclr source that is 2-3 weeks older than now. IMHO, this does not matter.

So I'll try clang3.6 patch you mentioned. However I don't think 4ec1a49 precedes 36efe1f. It is merged yesterday, isn't it?

wateret on 19 Jul 2016

/cc @parjong

wateret on 19 Jul 2016

@papaslavik In my side, b07369 still crashes as follows:

FAILED   - [ 8115 :    1 sec ] JIT/Regression/CLR-x86-JIT/V1.2-M01/b07369/b07369/b07369.sh
               BEGIN EXECUTION
               /opt/dotnet/Linux.arm-softfp.Release/corerun b07369.exe
               ./b07369.sh: line 117:  8987 Segmentation fault      $_DebuggerFullPath "$CORE_ROOT/corerun" b07369.exe $CLRTestExecutionArguments
               Expected: 100
               Actual: 139
               END EXECUTION - FAILED

I used Release TC from 387d9fc0a, and Release CoreCLR from 6b4ad33e5bf.

When I built CoreCLR, I used clang 3.6 with patch provided by @leemgs, and changed optimization level from O3 into O1. In addition, I manually reverted aab8856ce03.

parjong on 19 Jul 2016

However I don't think 4ec1a49 precedes 36efe1f.

I copied not the correct hash to the message, but I actually took 429139c44

Anyway, did you try the preceding commit?

papaslavik on 19 Jul 2016

@parjong why did you revert the commit aab8856 ?

papaslavik on 19 Jul 2016

So the optimization level should be changed to O1...

It's pity there're so many options. We normally build with clang3.8 while there's also clang 3.5 and clang 3.6 etc...

Anyway, I think I tried 3.6 with O1, though I can check it once again.

papaslavik on 19 Jul 2016

@mkborg told me there's information about the release build with just 28-36 crashes... so it's stable.

Mikhail, I think we should maintain the same procedure, or agree with the procedure with the colleagues.

papaslavik on 19 Jul 2016

@papaslavik Please refer to dotnet/coreclr#6298 for details.

parjong on 19 Jul 2016

👍1

The test I downloaded from the daily builds site works well on the procedure you described (I mean if I copy just the correct libcoreclr.so/corerun).

I will try now the test which @mkborg built locally.

@parjong perhaps you can somehow upload the test or send via email, just to be sure I'm not working on a wrong test version?

papaslavik on 19 Jul 2016

@papaslavik Here is the testcase that I used. (I just renamed b07369.exe as b07369.txt)

b07369.txt

parjong on 19 Jul 2016

👍1

OK. thank you. I was able to reproduce the problem appearing after the commit 36efe1f and not appearing before.

papaslavik on 19 Jul 2016

Also, perhaps today I have done some wrong library copies when testing before I could reproduce... :(

papaslavik on 19 Jul 2016

And also the debug version of TC has the crash issue.

papaslavik on 19 Jul 2016

@papaslavik

Anyway, did you try the preceding commit?

Yes, it passes with the immediate previous commit. I also tried 429139c which you mentioned, it passes for me.

wateret on 19 Jul 2016

Yeah, after all I was also able to reproduce the behavior

papaslavik on 19 Jul 2016

By the way, another note: with the native System.Private.CoreLib.ni.dll the test also works.

That sounds also quite strange.

papaslavik on 19 Jul 2016

@papaslavik How did you get crosse-gened System.Private.CorLib.dll (System.Private.CorLib.dll)? As far as I know, crossgen is not yet supported for ARM/Linux. If you are using cross-build, then it might come from AMD64/Linux. If so, it might indicate that there is some inconsistency between RyuJIT and LegacyJIT since LegacyJIT is used for ARM/Linux, but RyuJIT is used for AMD64/Linux.

parjong on 19 Jul 2016

We fixed that in dotnet/runtime#6252

papaslavik on 19 Jul 2016

We even tried running tests with that, and it makes the whole test run time 3 times faster.

But as we see now, some tests crash after jitting the dll...

papaslavik on 19 Jul 2016

We tried debug tests, not the release ones, to make sure.

papaslavik on 19 Jul 2016

I generate System.Private.CoreLib.ni.dll in chroot on ARM board

mkborg on 19 Jul 2016

@papaslavik @mkborg Wow!! As crossgen-related unittests (such as readytorun/mainv1) failed, I didn't expect that crossgen works for System.Private.CorLib.dll. Have you ever tried running full unittests with that native image?

parjong on 19 Jul 2016

Yes, @mkborg made the debug run and it was much faster...

papaslavik on 19 Jul 2016

=======================
     Test Results
=======================
# CoreCLR Bin Dir  : coreclr-testing/20160718.debug/odroid-2.arm-softfp/odroid-2.coreclr_4ec1a49.corefx_c3b7411.arm-softfp/CORECLR_NATIVE/coreclr/bin/Product/Linux.arm-softfp.Debug
# Tests Discovered : 9872
# Passed           : 9421
# Failed           : 98
# Skipped          : 353
=======================
486 minutes and 1 seconds taken to run CoreCLR tests.

=======================
     Test Results
=======================
# CoreCLR Bin Dir  : coreclr-testing/20160718.release/odroid-1.arm-softfp.release/odroid-1.coreclr_edc0bf7.corefx_c3b7411.arm-softfp/CORECLR_NATIVE/coreclr/bin/Product/Linux.arm-softfp.Release
# Tests Discovered : 9872
# Passed           : 6781
# Failed           : 2738
# Skipped          : 353
=======================
133 minutes and 41 seconds taken to run CoreCLR tests.

coreclr edc0bf7 is the same as coreclr 4ec1a49 but with -O1 option for release build.
In both cases tests we launched with System.Private.CoreLib.ni.dll

mkborg on 19 Jul 2016

Regarding the current issue it seems to me so far that the crash is somehow related to an empty frame or too small a frame or something. When there're too many pre-spiils the issue is hidden.

So I tried making soft fp false on the preceding commit and I got it crashing the same way. Just to be sure I made everything right, I will show the diffs I used:

git diff 
diff --git a/src/jit/compiler.h b/src/jit/compiler.h
index 251b03e..4b10d7c 100644
--- a/src/jit/compiler.h
+++ b/src/jit/compiler.h
@@ -7751,7 +7751,7 @@ public :
 #endif

 #ifdef ARM_SOFTFP
-        static const bool compUseSoftFP = true;
+        static const bool compUseSoftFP = false;
 #else // !ARM_SOFTFP
         static const bool compUseSoftFP = false;
 #endif
v.nikolaev@rpcc:~/CoreClr/coreclr$ 
v.nikolaev@rpcc:~/CoreClr/coreclr$ 
v.nikolaev@rpcc:~/CoreClr/coreclr$ git show 
commit 42eee84ab4a36d2dd7ad03cdce8fce8e1c065b2a
Author: Mikhail I. Krivtsov <[email protected]>
Date:   Fri Jul 15 18:40:52 2016 +0300

    Build 'release' with -O1

diff --git a/src/pal/tools/clang-compiler-override.txt b/src/pal/tools/clang-compiler-override.txt
index ee1e63b..1414ca2 100644
--- a/src/pal/tools/clang-compiler-override.txt
+++ b/src/pal/tools/clang-compiler-override.txt
@@ -9,7 +9,7 @@ SET (CMAKE_C_FLAGS_RELWITHDEBINFO_INIT "-g -O2")
 SET (CMAKE_CXX_FLAGS_INIT                "-Wall -Wno-null-conversion -std=c++11")
 SET (CMAKE_CXX_FLAGS_DEBUG_INIT          "-g -O0")
 SET (CLR_CXX_FLAGS_CHECKED_INIT          "-g -O2")
-SET (CMAKE_CXX_FLAGS_RELEASE_INIT        "-g -O3")
+SET (CMAKE_CXX_FLAGS_RELEASE_INIT        "-g -O1")
 SET (CMAKE_CXX_FLAGS_RELWITHDEBINFO_INIT "-g -O2")

 SET (CLR_DEFINES_DEBUG_INIT              DEBUG _DEBUG _DBG URTBLDENV_FRIENDLY=Checked BUILDENV_CHECKED=1)
v.nikolaev@rpcc:~/CoreClr/coreclr$ 
v.nikolaev@rpcc:~/CoreClr/coreclr$ git show HEAD^ 
commit 429139c4474ec944c271243c595d2b252aea52c7
Merge: b33bd67 cdb8a21
Author: Bruce Forstall <[email protected]>
Date:   Tue Jul 12 11:14:56 2016 -0700

    Merge pull request dotnet/coreclr#5966 from mikedn/x86-neg-long

    Implement GT_NEG decomposition

And the crash stack is the same as we had:

Program received signal SIGSEGV, Segmentation fault.
0xb20d1ace in ExpandArrayStack<ValueNumStore::Chunk*>::GetNoExpand(unsigned int) () from /home/krivtsov/v.nikolaev/coreoverlay/libclrjit.so
(gdb) bt
#0  0xb20d1ace in ExpandArrayStack<ValueNumStore::Chunk*>::GetNoExpand(unsigned int) () from /home/krivtsov/v.nikolaev/coreoverlay/libclrjit.so
dotnet/coreclr#1  0xb2141b58 in ValueNumStore::GetVNFunc(unsigned int, VNFuncApp*) () from /home/krivtsov/v.nikolaev/coreoverlay/libclrjit.so
dotnet/coreclr#2  0xb21441ac in ValueNumStore::GetConstantBoundInfo(unsigned int, ValueNumStore::ConstantBoundInfo*) () from /home/krivtsov/v.nikolaev/coreoverlay/libclrjit.so
dotnet/coreclr#3  0xb2131604 in RangeCheck::MergeEdgeAssertions(GenTree*, unsigned int*, Range*) () from /home/krivtsov/v.nikolaev/coreoverlay/libclrjit.so
dotnet/coreclr#4  0xb21320a6 in RangeCheck::ComputeRange(BasicBlock*, GenTree*, GenTree*, SimplerHashTable<GenTree*, PtrKeyFuncs<GenTree>, BasicBlock*, JitSimplerHashBehavior>*, bool) () from /home/krivtsov/v.nikolaev/coreoverlay/libclrjit.so
dotnet/coreclr#5  0xb2130f0e in RangeCheck::GetRange(BasicBlock*, GenTree*, GenTree*, SimplerHashTable<GenTree*, PtrKeyFuncs<GenTree>, BasicBlock*, JitSimplerHashBehavior>*, bool) () from /home/krivtsov/v.nikolaev/coreoverlay/libclrjit.so
dotnet/coreclr#6  0xb21321be in RangeCheck::ComputeRange(BasicBlock*, GenTree*, GenTree*, SimplerHashTable<GenTree*, PtrKeyFuncs<GenTree>, BasicBlock*, JitSimplerHashBehavior>*, bool) () from /home/krivtsov/v.nikolaev/coreoverlay/libclrjit.so
dotnet/coreclr#7  0xb2130f0e in RangeCheck::GetRange(BasicBlock*, GenTree*, GenTree*, SimplerHashTable<GenTree*, PtrKeyFuncs<GenTree>, BasicBlock*, JitSimplerHashBehavior>*, bool) () from /home/krivtsov/v.nikolaev/coreoverlay/libclrjit.so
dotnet/coreclr#8  0xb2131a94 in RangeCheck::ComputeRangeForLocalDef(BasicBlock*, GenTree*, GenTree*, SimplerHashTable<GenTree*, PtrKeyFuncs<GenTree>, BasicBlock*, JitSimplerHashBehavior>*, bool) () from /home/krivtsov/v.nikolaev/coreoverlay/libclrjit.so
dotnet/coreclr#9  0xb213208c in RangeCheck::ComputeRange(BasicBlock*, GenTree*, GenTree*, SimplerHashTable<GenTree*, PtrKeyFuncs<GenTree>, BasicBlock*, JitSimplerHashBehavior>*, bool) () from /home/krivtsov/v.nikolaev/coreoverlay/libclrjit.so
dotnet/coreclr#10 0xb2130f0e in RangeCheck::GetRange(BasicBlock*, GenTree*, GenTree*, SimplerHashTable<GenTree*, PtrKeyFuncs<GenTree>, BasicBlock*, JitSimplerHashBehavior>*, bool) () from /home/krivtsov/v.nikolaev/coreoverlay/libclrjit.so
dotnet/coreclr#11 0xb2130e0c in RangeCheck::OptimizeRangeCheck(BasicBlock*, GenTree*, GenTree*) () from /home/krivtsov/v.nikolaev/coreoverlay/libclrjit.so
dotnet/coreclr#12 0xb21323c4 in RangeCheck::OptimizeRangeChecks() () from /home/krivtsov/v.nikolaev/coreoverlay/libclrjit.so
dotnet/coreclr#13 0xb20daaee in Compiler::compCompile(void**, unsigned int*, CORJIT_FLAGS*) () from /home/krivtsov/v.nikolaev/coreoverlay/libclrjit.so
dotnet/coreclr#14 0xb20db182 in Compiler::compCompileHelper(CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, CORJIT_FLAGS*, CorInfoInstantiationVerification) () from /home/krivtsov/v.nikolaev/coreoverlay/libclrjit.so
dotnet/coreclr#15 0xb20dae8c in Compiler::compCompile(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, CORJIT_FLAGS*)::$_1::operator()(Compiler::compCompile(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, CORJIT_FLAGS*)::__JITParam*) const () from /home/krivtsov/v.nikolaev/coreoverlay/libclrjit.so
dotnet/coreclr#16 0xb20dade0 in Compiler::compCompile(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, CORJIT_FLAGS*) () from /home/krivtsov/v.nikolaev/coreoverlay/libclrjit.so
dotnet/coreclr#17 0xb20db844 in jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, CORJIT_FLAGS*, void*)::$_3::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, CORJIT_FLAGS*, void*)::__JITParam*) const::{lambda(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, CORJIT_FLAGS*, void*)::$_3::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, CORJIT_FLAGS*, void*)::__JITParam*) const::__JITParam*)#1}::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, CORJIT_FLAGS*, void*)::$_3::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, CORJIT_FLAGS*, void*)::__JITParam*) const::__JITParam*) const ()
   from /home/krivtsov/v.nikolaev/coreoverlay/libclrjit.so
dotnet/coreclr#18 0xb20db70e in jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, CORJIT_FLAGS*, void*)::$_3::operator()(jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, CORJIT_FLAGS*, void*)::__JITParam*) const () from /home/krivtsov/v.nikolaev/coreoverlay/libclrjit.so
dotnet/coreclr#19 0xb20db600 in jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, CORJIT_FLAGS*, void*) () from /home/krivtsov/v.nikolaev/coreoverlay/libclrjit.so
dotnet/coreclr#20 0xb20de086 in CILJit::compileMethod(ICorJitInfo*, CORINFO_METHOD_INFO*, unsigned int, unsigned char**, unsigned int*) () from /home/krivtsov/v.nikolaev/coreoverlay/libclrjit.so
dotnet/coreclr#21 0xb68ec29e in invokeCompileMethodHelper (jitMgr=<optimized out>, comp=0xbeffe87c, info=0xbeffe8e4, flags=<optimized out>, flags2=0, nativeEntry=0xbeffe83c, nativeSizeOfCode=0xbeffe838) at /home/v.nikolaev/CoreClr/coreclr/src/vm/jitinterface.cpp:11774
dotnet/coreclr#22 0xb68ec32a in invokeCompileMethod (jitMgr=0xb4d48f6c, comp=0xb4d4183c, info=0xb4d42ea4, flags=3033804452, flags2=0, nativeEntry=0xbeffe83c, nativeSizeOfCode=0xbeffe838) at /home/v.nikolaev/CoreClr/coreclr/src/vm/jitinterface.cpp:11829
dotnet/coreclr#23 0xb68ec428 in CallCompileMethodWithSEHWrapper(EEJitManager*, CEEInfo*, CORINFO_METHOD_INFO*, unsigned int, unsigned int, unsigned char**, unsigned int*, MethodDesc*)::$_4::operator()(CallCompileMethodWithSEHWrapper(EEJitManager*, CEEInfo*, CORINFO_METHOD_INFO*, unsigned int, unsigned int, unsigned char**, unsigned int*, MethodDesc*)::Param*) const (this=<optimized out>, pParam=<optimized out>) at /home/v.nikolaev/CoreClr/coreclr/src/vm/jitinterface.cpp:11891
dotnet/coreclr#24 0xb68ec3c0 in CallCompileMethodWithSEHWrapper (jitMgr=<optimized out>, comp=<optimized out>, info=0xb4d42ea4, flags=<optimized out>, flags2=0, nativeEntry=0x0, nativeSizeOfCode=0xbeffe838, ftn=<optimized out>)
    at /home/v.nikolaev/CoreClr/coreclr/src/vm/jitinterface.cpp:11934
dotnet/runtime#3858 0xb68ec842 in UnsafeJitFunction (ftn=0xb5f35638, ILHeader=<optimized out>, flags=1048592, flags2=0, pSizeOfCode=0xb4d48f6c) at /home/v.nikolaev/CoreClr/coreclr/src/vm/jitinterface.cpp:12575
dotnet/runtime#3859 0xb69ede0c in MethodDesc::MakeJitWorker (this=0xb5f35638, ILHeader=<optimized out>, flags=0, flags2=0) at /home/v.nikolaev/CoreClr/coreclr/src/vm/prestub.cpp:462
dotnet/coreclr#27 0xb69ee720 in MethodDesc::DoPrestub (this=0xb5f35638, pDispatchingMT=0x0) at /home/v.nikolaev/CoreClr/coreclr/src/vm/prestub.cpp:1454
dotnet/runtime#3860 0xb69ee326 in PreStubWorker (pTransitionBlock=<optimized out>, pMD=0xb5f35638) at /home/v.nikolaev/CoreClr/coreclr/src/vm/prestub.cpp:1041
dotnet/runtime#3861 0xb69a4fea in ThePreStub () at /home/v.nikolaev/CoreClr/coreclr/src/vm/typehandle.h:194
dotnet/runtime#3862 0xb23a9452 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

papaslavik on 19 Jul 2016

I wonder if it crashes the same way for hard firware

papaslavik on 19 Jul 2016

How did you get crosse-gened System.Private.CorLib.dll (System.Private.CorLib.dll)? As far as I know, crossgen is not yet supported for ARM/Linux.

@parjong , FYI. You can run CoreCLR run-time after creating mscorlib.ni.dll with crossgen command on Linux/ARM emulator as following. However, Crossgen for Linux/ARM is not compatible with all managed files.

root@arm-emul#> ./crossgen mscorlib.dll
root@arm-emul#> mv mscorlib.dll mscorlib.dll.orig
root@arm-emul#> mv mscorlib.ni.dll mscorlib.dll
root@arm-emul#> ./corerun /dotent/example/hello.exe

leemgs on 19 Jul 2016

Some time ago mscorlib.dll was renamed to System.Private.CoreLib.dll.

mkborg on 19 Jul 2016

Some time ago mscorlib.dll was renamed to System.Private.CoreLib.dll.

Right. The mscorlib.dll was splitted into two files (e.g., msocrlib.dll and System.Private.CoreLib.dll)

leemgs on 19 Jul 2016

On hard the test seems to be passing though.

papaslavik on 19 Jul 2016

And the latest version of arm softfp crashes as well.

papaslavik on 19 Jul 2016

@leemgs @mkborg FYI, dotnet/coreclr#6333 explains the relationship between mscorlib.dll and System.Private.CorLib.dll

parjong on 20 Jul 2016

@papaslavik @mkborg @leemgs @wateret I got the same result as @papaslavik.

Here is the result from Release CoreCLR without native image:

=======================
     Test Results
=======================
# CoreCLR Bin Dir  :
# Tests Discovered : 9870
# Passed           : 9505
# Failed           : 27
# Skipped          : 338
=======================
208 minutes and 12 seconds taken to run CoreCLR tests.

Here is the result from the same CoreCLR but with native image:

=======================
     Test Results
=======================
# CoreCLR Bin Dir  :
# Tests Discovered : 9870
# Passed           : 9504
# Failed           : 28
# Skipped          : 338
=======================
121 minutes and 51 seconds taken to run CoreCLR tests.

The difference in failure mostly comes from some non-determistic behavior related with string_cs_?? testcases.

If we take into account that non-deterministic behavior, then the only difference between two results is that the unittest of interest does not fail if we use native image. (Please refers to the attached logs for details)

[b1b8d2b8509][TC 387d9fc0a Release] [mainline-ni].txt
[b1b8d2b8509][TC 387d9fc0a Release] [mainline].txt

parjong on 20 Jul 2016

It's also weird that with debug-tc, it fails in __tls_get_addr ()

coreclr was built with clang3.6 vanilla, Release, -O1.

Program received signal SIGSEGV, Segmentation fault.
0xb6fe2538 in __tls_get_addr () from /lib/ld-linux.so.3
(gdb) bt
#0  0xb6fe2538 in __tls_get_addr () from /lib/ld-linux.so.3
dotnet/coreclr#1  0xb6a29372 in ClassLoader::LoadTypeHandleForTypeKey_Body(TypeKey*, TypeHandle, ClassLoadLevel) () from /opt/home/owner/0714r/libcoreclr.so
dotnet/coreclr#2  0xb6a264ba in ClassLoader::LoadTypeHandleForTypeKey(TypeKey*, TypeHandle, ClassLoadLevel, InstantiationContext const*) ()
   from /opt/home/owner/0714r/libcoreclr.so
dotnet/coreclr#3  0xb6a273cc in ClassLoader::LoadTypeDefThrowing(Module*, unsigned int, ClassLoader::NotFoundAction, ClassLoader::PermitUninstantiatedFlag, unsigned int, ClassLoadLevel, Instantiation*) () from /opt/home/owner/0714r/libcoreclr.so
dotnet/coreclr#4  0xb6a24ebc in ClassLoader::LoadTypeHandleThrowing(NameHandle*, ClassLoadLevel, Module*) () from /opt/home/owner/0714r/libcoreclr.so
dotnet/coreclr#5  0xb6a24c6c in ClassLoader::LoadTypeByNameThrowing(Assembly*, char const*, char const*, ClassLoader::NotFoundAction, ClassLoader::LoadTypesFlag, ClassLoadLevel) () from /opt/home/owner/0714r/libcoreclr.so
dotnet/coreclr#6  0xb6a11a38 in MscorlibBinder::LookupClass(BinderClassID) ()
   from /opt/home/owner/0714r/libcoreclr.so
dotnet/coreclr#7  0xb69f8b06 in SystemDomain::LoadBaseSystemClasses() ()
   from /opt/home/owner/0714r/libcoreclr.so
dotnet/coreclr#8  0xb69f88e8 in SystemDomain::Init() ()
   from /opt/home/owner/0714r/libcoreclr.so
dotnet/coreclr#9  0xb696af2a in EEStartupHelper(tagCOINITEE) ()
   from /opt/home/owner/0714r/libcoreclr.so
dotnet/coreclr#10 0xb696a3fa in EEStartup(tagCOINITEE) ()
---Type <return> to continue, or q <return> to quit---
   from /opt/home/owner/0714r/libcoreclr.so
dotnet/coreclr#11 0xb696a2da in EnsureEEStarted(tagCOINITEE) ()
   from /opt/home/owner/0714r/libcoreclr.so
dotnet/coreclr#12 0xb68e2e70 in CorHost2::Start() () from /opt/home/owner/0714r/libcoreclr.so
dotnet/coreclr#13 0xb68c85fa in coreclr_initialize ()
   from /opt/home/owner/0714r/libcoreclr.so
dotnet/coreclr#14 0x0000be4e in ExecuteManagedAssembly (
    currentExeAbsolutePath=0x17014 "/opt/home/owner/0714r/corerun", 
    clrFilesAbsolutePath=0x17074 "/opt/home/owner/0714r", 
    managedAssemblyAbsolutePath=0x1709c "/opt/media/SDCardA1/dotnet/Windows_NT.x64.Debug/JIT/Regression/CLR-x86-JIT/V1.2-M01/b07369/b07369/b07369.exe", 
    managedAssemblyArgc=0, managedAssemblyArgv=0x0)
    at /home/hanjoung/ws/dxl/dotnet/coreclr/src/coreclr/hosts/unixcoreruncommon/coreruncommon.cpp:387
dotnet/coreclr#15 0x0000b4c2 in corerun (argc=2, argv=0xbefffb54)
    at /home/hanjoung/ws/dxl/dotnet/coreclr/src/coreclr/hosts/unixcorerun/corerun.cpp:149
dotnet/coreclr#16 0x0000b570 in main (argc=2, argv=0xbefffb54)
    at /home/hanjoung/ws/dxl/dotnet/coreclr/src/coreclr/hosts/unixcorerun/corerun.cpp:161

wateret on 20 Jul 2016

t's also weird that with debug-tc, it fails in __tls_get_addr ()
coreclr was built with clang3.6 vanilla, Release, -O1.

@wateret , If you always face the issue in debug build mode, Please install the patched clang3.6 as following:

First of all, remove the existing clang package with apt-get command.
Then, execute the below commands to avoid _tls_get_addr issue.
lgs@ubuntu sudo add-apt-repository ppa:leemgs/dotnet
lgs@ubuntu sudo apt-get update
lgs@ubuntu sudo apt-get install clang-3.6 llvm-3.6 lldb-3.6

For more details - https://github.com/dotnet/coreclr/blob/master/Documentation/building/linux-instructions.md#how-to-enable--o3-optimization-level-for-armlinux

leemgs on 20 Jul 2016

@leemgs After applying your instruction, debug-tc passes. Thanks!

wateret on 20 Jul 2016

The issue is most probably related to the fact that there's no constructor for AssertionDsc

papaslavik on 20 Jul 2016

I mean, no non-compiler generated one, of course.

papaslavik on 20 Jul 2016

And not everyone calls Compiler::optCreateAssertion it seems.

papaslavik on 20 Jul 2016

👍2

@leemgs After applying your instruction, debug-tc passes. Thanks!

@wateret , Could you share me again after getting enough evaluation test & aging test? If the __tls_get_addr issue is always generated in debug build mode as well as in release build mode, I will update the below instruction.

https://github.com/dotnet/coreclr/blob/master/Documentation/building/linux-instructions.md#how-to-enable--o3-optimization-level-for-armlinux

leemgs on 21 Jul 2016

You can look in the dotnet/coreclr#6353 PR discussion that the problem is related to an array overrun.

Now I have to switch to another urgent task...

papaslavik on 21 Jul 2016

This is eventually fixed by dotnet/coreclr#6484

papaslavik on 28 Jul 2016

Runtime: [ARM/Linux] Unit Test Regression - JIT/Regression/CLR-x86-JIT/V1.2-M01/b07369

Most helpful comment

All 55 comments

Related issues