Runtime: Debugging Linux Apps - Dump Analysis Clarification

[abbotware]

I have hanging app on linux and was attempting to get take a memory dump... upon reading the docs for memory dump analysis I found nothing talking about mention hung, deadlocks or memory leaks: they are mostly geared to crash analysis.

the way i see it there are 2 types of dumps:

1. Crash (created by operating system/debugger event) for crash analysis
2. Live / On Demand (created by user) for memory, lock, hang analysis

I think this link needs to be updated to at least mention the second category of memory dump analysis is also possible (and any deviation in steps required)

https://github.com/dotnet/coreclr/blob/master/Documentation/building/debugging-instructions.md

Example text from above link that should be modified:

It is also possible to debug .NET Core crash dumps (what about hung processes?)
(NOTE: this is not an exhaust list of text that needs to be updated)

What is CreateDump?

On Linux, there is an utility called **createdump** (see doc) that can be setup to generate core dumps when a managed app throws an unhandled exception or faults. ?

The current wording seems to imply a 'separate utility' - but I can't find it in the (linux/arm) sdk

According to the following links it seems that it might be a seprate util, or part of sos?

(Command Line Options)?
https://github.com/dotnet/coreclr/blob/master/Documentation/botr/xplat-minidump-generation.md

(SOS Plugin Command?)
https://github.com/dotnet/coreclr/blob/master/Documentation/building/debugging-instructions.md

Please Clarify the following:

1. Are dumps only able to be analyzed if they are created via lldb + sos ?
2. What is 'create dump' utility and where is it? (contradictory docs)
3. Update the above linked docs since they seem have no mention that memory dumps from 'live' processes are supported.
4. a quick how to guide for creating a dump of a live process would be nice if the steps vastly are different from crash dump analysis.

regarding item 1 - I attempted to create memory dumps using 'gcore' and load them in lldb but this just caused lldb to crash so i posted a bug with lldb and sought guidance on coreclr glitter for more input

@karelz - also sent me the links above but I if you read my above notes: I think more clarity is needed to deal with the second class of dumps that I want to analyze on linux/arm hardware.

[karelz]

@janvorli @wfurt can you provide advice here?
cc @jkotas

[jkotas]

@mikem8361

[karelz]

cc @tmds

[wfurt]

lldb & sosplugin should work. With clrstack and eestack you should see what threads are doing.
(assuming you have matching pdb installed)

You can also do remote debug from visual studio via ssh.
That may be most convenient as VS also helps unwinding tasks and Async operations (if you use them)

[tmds]

The createdump program comes with the runtime. For 2.1.1 for example it is at ./shared/Microsoft.NETCore.App/2.1.1/createdump.
It seems pretty straight forward how to use it:

./shared/Microsoft.NETCore.App/2.1.1/createdump --help
createdump [options] pid
-f, --name - dump path and file name. The pid can be placed in the name with %d. The default is '/tmp/coredump.%d'
-n, --normal - create minidump.
-h, --withheap - create minidump with heap (default).
-t, --triage - create triage minidump.
-u, --full - create full core dump.
-d, --diag - enable diagnostic messages.

I assume those dump files can be opened up in VS.

[tmds]

@jkotas @wfurt @mikem8361 do you have some thoughts on building a command line tool that is capable of showing the managed stacks of a running .NET process? I think it would be useful to help devs figure out where there program is hanging without having to learn and setup lldb+sos.

[wfurt]

good questions @tmds. I think one can go long way by scripting lldb with sos or writing Python extension. However remote debug from VS still provides superior option IMHO.

As far as the createdump, it is not in arm version as far as I can tell. That perhaps deserves separate issue.

[tmds]

I think one can go long way by scripting lldb with sos or writing Python extension.

It can be quite a ceremony to install a matching version of lldb on your production environment. Especially if that is a container where you are not running as root. I'm thinking: something that works out of the box and can help users get an insight into what a dotnet program is doing.

[jkotas]

thoughts on building a command line tool that is capable of showing the managed stacks of a running .NET process?
something that works out of the box and can help users get an insight into what a dotnet program is doing

EventPipe should enable this eventually. Discussion on this should be separate issue. cc @brianrob

[abbotware]

@wfurt As far as the createdump, it is not in arm version as far as I can tell.

1. So how do I get this?
2. Is lldb + sos able to take an equivalent memory dump?

@tmds I assume those dump files can be opened up in VS.
can this be confirmed? - that would certainly make dump analysis a lot more pleasant. (I assume matching binaries + pdbs are required)

[karelz]

I would first focus on the key thing - how to get a dump on Linux (any distro). Either for crashes or on demand (e.g. when there's memory leak, etc.)
Having nice tools to print call stacks, etc. may be the next steps, but let's cover the basics first ...

[mikem8361]

1) Linux (ELF) core dumps can NOT be opened by VS.
2) "createdump" is both an separate command line utility and an SOS command, but the SOS command can be very slow to create dumps if the heaps are large.
3) There should be an arm Linux version of the createdump utility built but it probably hasn't been tested and I personally haven't done anything on the arm platforms.
4) Currently lldb + SOS is the only way to analysis core dumps. We are currently investigating ways to improve this.
5) We do need to improve the documentation on hang, deadlocks and memory leaks. I'm current working on improving the documentation on getting/building/setting up lldb and SOS in this new repo "diagnostics". Currently it is still a work in progress.
6) You can usually load core dumps from other machines/Linux distros on another machine/distro if you have the matching .NET Core binaries especially the program ("dotnet") when you load the dump. The probably is getting the right version of them and we have been working on a symbol/module downloader tool that will make this easier. It is intended to be a dotnet (2.1) global extension but I still working on getting it published.

/cc: @noahfalk @leculver @tommcdon

[tmds]

Is there something specific about the dumps generated with createdump? Or are these 'generic' elf dumps?

[wfurt]

They seems to be "normal" core files.

furt     26328  0.5  0.0 2662932 31824 pts/6   SLl+ 11:14   0:00 dotnet exec /tmp/aa/bin/Debug/netcoreapp2.1/aa.dll

~/dotnet-2.1/shared/Microsoft.NETCore.App/2.1.0/createdump 263281
riting minidump with heap to file /tmp/coredump.26328
Written 70725632 bytes (17267 pages) to core file
furt@net-dale:~$ file /tmp/coredump.26328
/tmp/coredump.26328: ELF 64-bit LSB core file x86-64, version 1 (GNU/Linux), SVR4-style, from 'dotnet', real uid: 1001, effective uid: 1001, real gid: 1001, effective gid: 1001, execfn: '/home/furt/dotnet-2.1/dotnet', platform: 'x86_64'

You can use 'gcore ' from gdb package:

armory:~# gcore 16563
[New LWP 16565]
[New LWP 16566]
[New LWP 16567]
[New LWP 16568]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
__libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
46  ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S: No such file or directory.
warning: target file /proc/16563/cmdline contained unexpected null characters
Saved corefile core.16563

armory:~# file core.16563
core.16563: ELF 32-bit LSB core file ARM, version 1, SVR4-style, from 'dotnet'

I was able to load it in gdb:

(gdb) bt
#0  __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
dotnet/coreclr#1  0xb6f7c03a in __pthread_cond_timedwait (cond=<optimized out>, mutex=0x1057390, abstime=0xbed24bec) at pthread_cond_timedwait.c:198
dotnet/coreclr#2  0xb698c2fa in CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /mnt/dotnet-2.1.301/shared/Microsoft.NETCore.App/2.1.1/libcoreclr.so
dotnet/coreclr#3  0xb698bfae in CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) ()
   from /mnt/dotnet-2.1.301/shared/Microsoft.NETCore.App/2.1.1/libcoreclr.so
dotnet/coreclr#4  0xb698f834 in SleepEx () from /mnt/dotnet-2.1.301/shared/Microsoft.NETCore.App/2.1.1/libcoreclr.so
dotnet/coreclr#5  0xb66f2386 in Thread::UserSleep(int) () from /mnt/dotnet-2.1.301/shared/Microsoft.NETCore.App/2.1.1/libcoreclr.so
dotnet/coreclr#6  0xb6722790 in ThreadNative::Sleep(int) () from /mnt/dotnet-2.1.301/shared/Microsoft.NETCore.App/2.1.1/libcoreclr.so
dotnet/coreclr#7  0xb1a3cefe in ?? ()

```Backtrace stopped: previous frame identical to this frame (corrupt stack?)

lldb seems to have issues with same file. That looks more like a lldb bug to me.
(so as lldb crash) I'll do more testing with ARM specifically and

[mikem8361]

They are ELF core dumps but createdump uses the same DAC api to trim the memory chunks down to what is needed for the managed state like a Windows minidump. There are options to the utility and configuration environment variables to coreclr that control how much memory to write the dump. A “full” dump should be like gcore dumps with the right “coredump_filter” settings. See https://github.com/dotnet/coreclr/blob/master/Documentation/botr/xplat-minidump-generation.md for more details.

[abbotware]

@wfurt
lldb seems to have issues with same file. That looks more like a lldb bug to me.
Is that the same as this problem that I was having? or different? https://bugs.llvm.org/show_bug.cgi?id=37986

[wfurt]

I have different issues on different Linux versions. However, you can try the VS path:

I verified just now that you can use VS to debug Linux/ARM.
Run this on target board:

curl -sSL https://aka.ms/getvsdbgsh | bash /dev/stdin -v latest -l ~/vsdbg

In VS (2017):

 Tools -> Options -> Cross Platform -> Connection manager -> Add (add ssh to your system) 
 Debug -> Attach to process. Select Connection type: ssh and pick your target. 

You should get list of running processes. Select your app.

Now , you can do "Stop Al"l and inspect threads and their stack traces.
You should have usual capabilities available to inspect data structures.

[abbotware]

@wfurt so i tried this on my build agent (arm hardware) (with dotnet sdk installed) and dev hardware (no sdk installed, but has the application with a self contained runtime) and both failed:

root@dev-build:~# curl -sSL https://aka.ms/getvsdbgsh | bash /dev/stdin -v latest -l ~/vsdbg
Using arguments
    Version                    : 'latest'
    Location                   : '/root/vsdbg'
    SkipDownloads              : 'false'
    LaunchVsDbgAfter           : 'false'
    RemoveExistingOnUpgrade    : 'false'
Info: Using vsdbg version '15.7.20425.2'
Info: Previous installation at /root/vsdbg not found
Info: Using Runtime ID ''
Downloading https://vsdebugger.azureedge.net/vsdbg-15-7-20425-2/vsdbg-.zip

ERROR: Could not download https://vsdebugger.azureedge.net/vsdbg-15-7-20425-2/vsdbg-.zip

[abbotware]

FYI - for 'installing' the sdk, i mimic'd these steps: https://stackoverflow.com/questions/38146877/net-core-on-raspbian-arm (I am not using a pi, but similar hardware) I replaced sdk version with 2.1.301 though since i knew 2.1.1 was out when i read it. Not sure if there is an official install guide, but you can bet a lot of people are following that right now! ;-)

[wfurt]

What base OS do you use? It does not seems to be recognized properly.
You can download the install script and then run it as
getvsdbgsh -r linux-arm -v latest -l ~/vsdbg

In my case, it resolves to https://vsdebugger.azureedge.net/vsdbg-15-7-20425-2/vsdbg-linux-arm.zip. (also you will need unzip utility installed)

[abbotware]

its an Ubuntu 16.04 derivative: I know it supposedly 'unsupported', but I have been using .net core 2.0 on it for over 8 months... I am working on an 18.04 base image for my hardware in parallel

[wfurt]

That is ok. When you look at the script it uses uname -m to determine arch. I'm wondering if your's is different. Forcing rid via command line should hopefully fix you up.

[abbotware]

uname=aarch64
(64bit arm hardware/OS ... running 32bit arm net core)
there are a lot of arm architectures out there.. armhf, aarch64, armel...

[abbotware]

FYI - specifying -r linux-arm seemed to install! i setup a connection in the cross platform section and that is where it fails... but so close I think... visual studio is looking in the wrong location?

    Version                    : 'vs2017u5'
    Location                   : '/root/.vs-debugger/vs2017u5'
    SkipDownloads              : 'false'
    LaunchVsDbgAfter           : 'false'
    RemoveExistingOnUpgrade    : 'false'

1. The location seems to be off - is there a setting in visual studio for that? (I just reinstalled the debugger to -l .vs-debugger for now

2. it can't find version vs2017u5 - my current deployed code is running net core 2.0 - If i have to upgrade to 2.1.1 I can do that (I rolled back due to the random hangs)

[wfurt]

Use the original flags: -v latest -l ~/vsdbg
I don't know if that can be configured in vs. You may need to experiment with links or copy.

[abbotware]

    elif [[ "$(uname -m)" == "armv7l" ]]; then
        __RuntimeID=linux-arm
    fi

that is probably very incomplete :-)

[abbotware]

I think my pi returns armv61 (running 32bit raspbian os)... and a sopine64 (target hardware) returns aarch64 - to get .net core to run on aarch64 - I installed all the required .net core shared libaries (ie: libc6, libunwind, etc) apt-get targeting armhf specfically... (was a bit of work to figure out.. but.. it works! - same binaries run fine on both hardware and os)

[abbotware]

so i copied the files into /root/.vs-debugger/vs2017u5 - but that did not seem to do much... visual studio doesnt even display an error now :-/

FYI remote debugging natively via gdb worked! so that is pretty cool!! 👍

If we can get the remote debugger to work for managed code... then should really be added to documentation! :-) This route is so much nicer than lldb + SOS

[wfurt]

I assume ~/vsdbg/vsdbg runs? Did the connection manager let you save the configuration?
There may be level of trickery as your OS arch is not app arch. But at least the process list should work. That may also cause the lldb issues.

Also just for the record: original pi (armv6) is not supported by runtime.

also note that 2.1 has early support for aarch64 so you may be able to run it without much trickery.
In either way, you are already on highly experimental path.

[abbotware]

I assume ~/vsdbg/vsdbg runs?

Nope!

./vsdbg: No such file or directory
That error usually means a 32bit static lib is missing (when running on 64bit os)

[abbotware]

ok! installed the missing dependencies for vsdbg... (readelf -d) and now it vsdbg runs... but still nothing in visual studio... are there logs?

(in case your wonder how I am even running 32bit .net core on the aarch64 in the first place... its all running in an ubuntu snap)

[wfurt]

I would probably look at sshd logs first. You can also configure sshd to log all executed commands.
You can possibly wrap vsdbg with some shell script.
What exactly do you see on VS? Getting more logs may be possible but I don't know how.

[abbotware]

VS - nothing - its as if the attach to process dialog disappears after I select 'managed net core for unix' the show output form 'debug' is empy too... before it at least showed the ssh output

sshd logging all commands - have you actually done that before? It sounds like a good idea, but i can't seem to figure that one out - the only way I have access to the hardware is via ssh, so i have to run a 2nd sshd on a different port (/usr/sbin/sshd -dD -p 23) and all i see is the debut output from ssh:

debug1: server_init_dispatch_20
debug1: server_input_channel_open: ctype session rchan 256 win 16384 max 16384
debug1: input_session_request
debug1: channel 0: new [server-session]
debug1: session_new: session 0
debug1: session_open: channel 0
debug1: session_open: session 0: link with channel 0
debug1: server_input_channel_open: confirm session
debug1: server_input_channel_req: channel 0 request pty-req reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req pty-req
debug1: Allocating pty.
debug1: session_pty_req: session 0 alloc /dev/pts/1
debug1: Ignoring unsupported tty mode opcode 42 (0x2a)
debug1: SELinux support disabled
debug1: server_input_channel_req: channel 0 request shell reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req shell
Starting session: shell on pts/1 for root from 192.168.2.100 port 58564 id 0
debug1: Setting controlling tty using TIOCSCTTY.

[tmds]

@wfurt @mikem8361 if you attach VS to a .NET Core process running on Linux. Can it create minidumps that VS can understand and diff to debug memory leaks (https://msdn.microsoft.com/en-us/library/dn342825.aspx)? Or does this also not work due to the ELF format?

[wfurt]

sos seems broken or arm. I opened dotnet/coreclr#18889 to track it.

[mikem8361]

VS can not create minidumps or core dumps when attached to a remote Linux process. We are investigate ways to improve the memory leaks and dump analysis for Linux. We don’t have a time frame yet.

[wfurt]

opened dotnet/coreclr#18893 for createdump. It seems to function ok on arm so you can build it your self @abbotware

Is there anything left on this issue?

[abbotware]

@wfurt - Well, if i build createdump (found the build steps here: https://github.com/dotnet/coreclr/blob/master/Documentation/building/linux-instructions.md) myself and manage to create a dump, how would I analyze it if SOS is broken on arm (#18889) ?

[wfurt]

If you cross-compiled coreclr, could you try libsosplugin.so from that build? I use tree synced to ff5d7adfc8. (and cross rootfs created via ./cross/build-rootfs.sh arm xenial lldb4.0)

pi@raspberrypi:~ $ lldb-4.0 ./dotnet-2.1.302/dotnet -c /tmp/coredump.16979
(lldb) plugin load ~/libsosplugin.so
(lldb) clrstack
OS Thread Id: 0x4253 (1)
Child SP       IP Call Site
7EE8AD34 00000000 [HelperMethodFrame: 7ee8ad34] Unknown
(lldb) clrthreads
ThreadCount:      2
UnstartedThread:  0
BackgroundThread: 1
PendingThread:    0
DeadThread:       0
Hosted Runtime:   no
                                                                         Lock
       ID OSID ThreadOBJ    State GC Mode     GC Alloc Context  Domain   Count Apt Exception
   1    1 4253 00DDE738   2020020 Preemptive  00000000:729160A0 00DD0BA0 0     Ukn
   5    2 4258 00DEE808     21220 Preemptive  00000000:00000000 00DD0BA0 0     Ukn (Finalizer)

[wfurt]

Actually, you need to follow https://github.com/dotnet/coreclr/blob/master/Documentation/building/cross-building.md
Running the script will fetch most the tools and packages you need.

I can also post my binary if you want to give it try.

[abbotware]

sure binary is easy for me to test! - i never did a compile of the coreclr - already hitting some issues I have to dig into

[wfurt]

libsosplugin.so.gz

This may be some problem with build. I'm trying to look into it.

[wfurt]

as you see, I used dotnet-2.1.302 for my testing. I think it should be ok if you get close enough.

[abbotware]

im using 2.1.301, lldb didnt seem to like the -c option, so i took it off:

root@pine64so:/tmp# lldb-4.0 /usr/local/bin/dotnet
(lldb) target create "/usr/local/bin/dotnet"
Current executable set to '/usr/local/bin/dotnet' (arm).
(lldb) plugin load libsosplugin.so
(lldb) clrstack
The libcoreclr.so module is not loaded yet in the target process

[wfurt]

That matched x64. Without loaded core, there are no clr internals. You can try to attach to real process or figure out how to load the core. The core I use is the one created with createdump. I don't know if that matters. what core file are you trying to load?

[abbotware]

shouldnt lldb be able to load the SOS by itself?

root@pine64so:/tmp# lldb-4.0
(lldb) plugin load libsosplugin.so
(lldb) sos
The libcoreclr.so module is not loaded yet in the target process