Hi,
I am pretty in new to .net core. I have a .net console application. I converted my code to a .net core project, and published it using this command for Linux arm (raspberry pi): dotnet publish -r linux-arm
The application works fine for a couple of hours and some time for one or two days. But it crashes with this message: "Aborted"
Unisg the command "ulimit -c unlimited", I got a large core dump file after the crash. I analyzed the file using gdb command, and it is the result:
pi@Unit10:/ga/linux-arm/publish $ sudo gdb ./FieldUnitDotNetCore -c core
GNU gdb (Raspbian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./FieldUnitDotNetCore...(no debugging symbols found)...done.
warning: core file may not match specified executable file.
[New LWP 16774]
[New LWP 16768]
[New LWP 16764]
[New LWP 16767]
[New LWP 17167]
[New LWP 16776]
[New LWP 17160]
[New LWP 16772]
[New LWP 17162]
[New LWP 16766]
[New LWP 16771]
[New LWP 16777]
[New LWP 16775]
[New LWP 16765]
[New LWP 17163]
[New LWP 17195]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
Core was generated by `./FieldUnitDotNetCore'.
Program terminated with signal SIGABRT, Aborted.
51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x6d9ff450 (LWP 16774))]
(gdb) where
dotnet/coreclr#1 0x76c36824 in __GI_abort () at abort.c:89
dotnet/coreclr#2 0x768c2342 in PROCAbort () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#3 0x768c1560 in PROCEndProcess(void, unsigned int, int) () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#4 0x766155c0 in StackFrameIterator::ProcessCurrentFrame() () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#5 0x76615186 in StackFrameIterator::Init(Thread, Frame, REGDISPLAY, unsigned int) () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#6 0x76614fc0 in Thread::StackWalkFramesEx(REGDISPLAY, StackWalkAction ()(CrawlFrame, void), void, unsigned int, Frame) () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#7 0x766b8112 in HandleGCSuspensionForInterruptedThread(_CONTEXT) () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#8 0x7689c70c in inject_activation_handler(int, siginfo_t, void) () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#9
dotnet/coreclr#10 0x71986860 in ?? ()
dotnet/coreclr#11 0x7198279e in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) frame 10
dotnet/coreclr#10 0x71986860 in ?? ()
(gdb) list +
46 in ../sysdeps/unix/sysv/linux/raise.c
(gdb) frame 11
dotnet/coreclr#11 0x7198279e in ?? ()
(gdb) list +
46 in ../sysdeps/unix/sysv/linux/raise.c
(gdb) frame 9
dotnet/coreclr#9
(gdb) list +
75 ../sysdeps/unix/sysv/linux/arm/sigrestorer.S: No such file or directory.
(gdb) frame 8
dotnet/coreclr#8 0x7689c70c in inject_activation_handler(int, siginfo_t
(gdb) list +
75 in ../sysdeps/unix/sysv/linux/arm/sigrestorer.S
Can someone help me to fix it.
Thanks.
@sinisaz can you please try to run it couple of times and see if the call stack in GDB is always the same or similar? It might be caused by dotnet/runtime#10523.
@janvorli Thanks for the answer. I ran the project on three more raspbian overnight. It seems to me all have the same issue. these are the results:
pi@Unit9:/ga/linux-arm/publish $ sudo gdb ./FieldUnitDotNetCore -c core
GNU gdb (Raspbian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./FieldUnitDotNetCore...(no debugging symbols found)...done.
warning: core file may not match specified executable file.
[New LWP 16158]
[New LWP 16149]
[New LWP 16148]
[New LWP 16151]
[New LWP 16152]
[New LWP 21964]
[New LWP 16160]
[New LWP 16156]
[New LWP 16150]
[New LWP 16159]
[New LWP 21980]
[New LWP 21937]
[New LWP 16161]
[New LWP 21923]
[New LWP 21988]
[New LWP 16155]
[New LWP 21990]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
Core was generated by `./FieldUnitDotNetCore'.
Program terminated with signal SIGABRT, Aborted.
51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
pi@Pi08:/ga/linux-arm/publish $ sudo gdb ./FieldUnitDotNetCore -c core
GNU gdb (Raspbian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./FieldUnitDotNetCore...(no debugging symbols found)...done.
warning: core file may not match specified executable file.
[New LWP 27290]
[New LWP 27281]
[New LWP 27285]
[New LWP 27282]
[New LWP 27284]
[New LWP 6322]
[New LWP 6223]
[New LWP 6283]
[New LWP 6030]
[New LWP 27283]
[New LWP 6312]
[New LWP 27294]
[New LWP 27288]
[New LWP 27293]
[New LWP 27292]
[New LWP 27289]
[New LWP 6313]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
Core was generated by `./FieldUnitDotNetCore'.
Program terminated with signal SIGABRT, Aborted.
51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
pi@Unit10:/ga/linux-arm/publish $ sudo gdb ./FieldUnitDotNetCore -c core
GNU gdb (Raspbian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./FieldUnitDotNetCore...(no debugging symbols found)...done.
warning: core file may not match specified executable file.
[New LWP 17675]
[New LWP 17673]
[New LWP 17670]
[New LWP 17671]
[New LWP 21019]
[New LWP 17678]
[New LWP 17683]
[New LWP 20995]
[New LWP 20986]
[New LWP 17672]
[New LWP 17681]
[New LWP 17674]
[New LWP 20950]
[New LWP 21021]
[New LWP 17682]
[New LWP 20940]
[New LWP 21020]
[New LWP 17677]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
Core was generated by `./FieldUnitDotNetCore'.
Program terminated with signal SIGABRT, Aborted.
51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x6f723450 (LWP 17675))]
BTW, I'm using the latest version of dotnet core : 2.2.100-preview1-009349
@sinisaz I just got a report of the same issue from someone else, including a very simple repro. I've verified that it happens even with the latest version of coreclr from the master branch. I am looking into it.
Thanks, looking forward to hearing from you.
Thank @janvorli ,
Hi @vivmishra, when can I expect this fix to be released?
i have similar issue that caused by .NET CORE internally check thread stack gs cookie and then abort my app randomly on linux arm32, can't wait for the fix for such a critical bug!
Hi @janvorli , using the nightly build which includes dotnet/coreclr#20116 and dotnet/coreclr#20117, I run my project, but still get the same error:
pi@Unit4:/ga/linux-arm/publish $ sudo gdb ./FieldUnitDotNetCore -c core
GNU gdb (Raspbian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./FieldUnitDotNetCore...(no debugging symbols found)...done.
warning: core file may not match specified executable file.
[New LWP 2122]
[New LWP 2113]
[New LWP 2112]
[New LWP 2128]
[New LWP 2115]
[New LWP 2124]
[New LWP 2125]
[New LWP 2120]
[New LWP 2117]
[New LWP 10348]
[New LWP 10369]
[New LWP 2119]
[New LWP 2114]
[New LWP 10388]
[New LWP 2123]
[New LWP 2116]
[New LWP 3405]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
Core was generated by `./FieldUnitDotNetCore'.
Program terminated with signal SIGABRT, Aborted.
51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x6d8ff450 (LWP 2122))]
wh(gdb) where
dotnet/coreclr#1 0x76ba3824 in __GI_abort () at abort.c:89
dotnet/coreclr#2 0x7682f342 in PROCAbort () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#3 0x7682e560 in PROCEndProcess(void, unsigned int, int) () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#4 0x765825c0 in StackFrameIterator::ProcessCurrentFrame() () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#5 0x76582186 in StackFrameIterator::Init(Thread, Frame, REGDISPLAY, unsigned int) () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#6 0x76581fc0 in Thread::StackWalkFramesEx(REGDISPLAY, StackWalkAction ()(CrawlFrame, void), void, unsigned int, Frame) () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#7 0x76625112 in HandleGCSuspensionForInterruptedThread(_CONTEXT) () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#8 0x7680970c in inject_activation_handler(int, siginfo_t, void) () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#9
dotnet/coreclr#10 0x7198685c in ?? ()
dotnet/coreclr#11 0x051777ca in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) frame 10
dotnet/coreclr#10 0x7198685c in ?? ()
(gdb) list +
46 in ../sysdeps/unix/sysv/linux/raise.c
(gdb) frame 11
dotnet/coreclr#11 0x051777ca in ?? ()
(gdb) list +
46 in ../sysdeps/unix/sysv/linux/raise.c
(gdb) list +
46 in ../sysdeps/unix/sysv/linux/raise.c
(gdb) frame 9
dotnet/coreclr#9
(gdb) list +
75 ../sysdeps/unix/sysv/linux/arm/sigrestorer.S: No such file or directory.
(gdb) frame 8
dotnet/coreclr#8 0x7680970c in inject_activation_handler(int, siginfo_t
(gdb) list +
\75 in ../sysdeps/unix/sysv/linux/arm/sigrestorer.S
Could you please have look on this. Thanks
BTW, my dotnet version is: 3.0.100-alpha1-009630
This is also the output of running under gdb debugger:
Thread 6 "FieldUnitDotNet" received signal SIG34, Real-time event 34.
[Switching to Thread 0x6f9c9450 (LWP 11056)]
0x766b1e94 in JIT_MonExit_Portable () from /ga/linux-arm/publish/libcoreclr.so
(gdb) where
dotnet/coreclr#1 0x71ba192e in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) frame 1
dotnet/coreclr#1 0x71ba192e in ?? ()
(gdb) list +
1 ../sysdeps/unix/sysv/linux/arm/dl-procinfo.c: No such file or directory.
(gdb) frame 0
(gdb) list +
1 in ../sysdeps/unix/sysv/linux/arm/dl-procinfo.c
(gdb)
(gdb) continue
Continuing.
Thread 10 "FieldUnitDotNet" received signal SIG34, Real-time event 34.
[Switching to Thread 0x6e1ff450 (LWP 11060)]
0x76f7a4b8 in __pthread_mutex_unlock_usercnt (mutex=
55 pthread_mutex_unlock.c: No such file or directory.
(gdb) where
dotnet/coreclr#1 0x768f39e0 in CorUnix::CThreadSynchronizationInfo::RunDeferredThreadConditionSignalings() () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#2 0x768ef13c in CorUnix::CSynchControllerBase::Release() () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#3 0x768ed814 in SetEvent () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#4 0x766b2060 in JIT_MonExit_Signal(Object*) () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#5 0x766b1e76 in JIT_MonExit_Portable () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#6 0x71ba192e in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Thread 11 "FieldUnitDotNet" received signal SIG34, Real -time event 34.
[Switching to Thread 0x6d9ff450 (LWP 23295)]
0x76f7c94c in __pthread_cond_wait (cond=0xff690, mutex= 0xff678) at pthread_cond_wait.c:186
186 pthread_cond_wait.c: No such file or directory.
(gdb) where
dotnet/coreclr#1 0x768f1312 in CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData, unsigned int, CorUnix::ThreadWakeupReason, unsigned int) ()
from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#2 0x768f0fb2 in CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread, unsigned int, bool, bool, CorUnix::ThreadWakeupReason, unsigned int) () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#3 0x768f429c in CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread, unsigned int, void const, int, unsigned int, int, int) () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#4 0x768f442e in WaitForSingleObjectEx () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#5 0x766e3fa0 in CLREventBase::WaitEx(unsigned int, WaitMode, PendingSync) () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#6 0x766e9b26 in Thread::RareDisablePreemptiveGC() () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#7 0x766b14f4 in JIT_MonEnter_Helper(Object, unsigned char, void*) () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#8 0x766b1878 in JIT_MonReliableEnter_Portable () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#9 0x71ba0740 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
@sinisaz are you sure you are using the latest nightly build of the runtime and not just the latest nightly build of the SDK? These things can be different - the latest SDK can still be targeting and using older runtime.
To verify that, could you please run
strings /ga/linux-arm/publish/libcoreclr.so | grep "@(#)"
and copy and paste the result here?
@janvorli thanks for the quick answer. This is the result:
@(#)Version 4.6.26628.05 @BuiltBy: root-220e191eb7a3 @Branch: release/2.1-MSRC @SrcCode: https://github.com/dotnet/coreclr/tree/c6e7fbc5516842945b9b375f7724438dece10627
Thank you, that means it is as I've suspected. The runtime version is release/2.1-MSRC with last commit from June 28, so it cannot have the fix. You need to update your .csproj to target the latest runtime by changing the TargetFramework:
<TargetFramework>netcoreapp3.0</TargetFramework>
You will also need to create a NuGet.Config file e.g. next to your .csproj containing the following stuff since the 3.0 packages are available on myget only.
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<packageSources>
<add key="myget" value="https://dotnet.myget.org/F/dotnet-core/api/v3/index.json" />
</packageSources>
</configuration>
Thanks @janvorli , now I have this resault:
pi@Unit4:/ $ strings /ga/linux-arm/publish/libcoreclr.so | grep "@(#)"
@(#)Version 4.6.26927.03 @BuiltBy: root-9f97e44efa12 @Branch: master @SrcCode: https://github.com/dotnet/coreclr/tree/0bc8c62cfb7514b91b4ebd5f073835391e0502a3
I still get an error when run the project using the gdb debugger. not sure it is the same as previous or not!
Thread 11 "FieldUnitDotNet" received signal SIG34, Real -time event 34.
[Switching to Thread 0x6da76450 (LWP 23674)]
0x76f7c94c in __pthread_cond_wait (cond=0x171320, mutex =0x171308) at pthread_cond_wait.c:186
186 pthread_cond_wait.c: No such file or directory.
(gdb) where
dotnet/coreclr#1 0x7691c482 in CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData, unsigned int, CorUnix::ThreadWakeupReason, unsigned int) ()
from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#2 0x7691c122 in CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread, unsigned int, bool, bool, CorUnix::ThreadWakeupReason, unsigned int) () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#3 0x7691f40c in CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread, unsigned int, void const, int, unsigned int, int, int) () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#4 0x7691f59e in WaitForSingleObjectEx () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#5 0x76714f6e in CLREventBase::WaitEx(unsigned int, WaitMode, PendingSync) () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#6 0x7671968e in Thread::RareDisablePreemptiveGC() () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#7 0x766e2948 in JIT_MonEnter_Helper(Object, unsigned char, void*) () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#8 0x766e2ccc in JIT_MonReliableEnter_Portable () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#9 0x6b1d4c60 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
another run:
Thread 7 "FieldUnitDotNet" received signal SIG34, Real-time event 34.
[Switching to Thread 0x71464450 (LWP 23709)]
0x6eea9876 in ?? ()
(gdb) where
dotnet/coreclr#1 0x76923620 in SwitchToThread () from /ga/linux-arm/publish/libcoreclr.so
dotnet/coreclr#2 0x72a141ac in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
The break on SIG34 is expected. We use this signal for internal purposes. You need to disable this signal in GDB:
handle SIG34 nostop noprint
@janvorli Thank you so much.
Hi,
we are also experiencing this issue when running our application on .NET Core 2.1.5 on Raspbian (published for linux-arm). While the fix for dotnet/runtime#10523 (which also caused a crash on Raspbian) has been included in 2.1.5, I can see that the fix for this issue is scheduled for 2.1.6.
As this issue still blocks us from moving from .NET Core 2.0 to 2.1, can you give us a hint when 2.1.6 will be released (since 2.0 is now already end-of-life)? Or is there a way to deploy a nightly version of the 2.1.x runtime with dotnet publish -r linux-arm?
Edit: .NET Core 2.1.6 was released on Nov 13, 2018.
Thank you!