Diagnostics: LLDB hangs in Linux when opening a dotnet core 2.1 dump

[vtortola]

Hi there,

I want to get a process dump of an application that is consuming more memory than expected and do a simple !dupmheap to try to figure out the problem.

Please check this self-contained example. It will create an image with all the tools already installed, then create a dotnet core 2.1 application that runs in a continuous loop printing messages with a 1s sleep between messages, and use it as entry point:

FROM microsoft/dotnet:2.1-sdk
WORKDIR /app
RUN apt-get update && apt-get install curl -y && \ 
    apt-get install gpg -y && \
    apt-get install apt-transport-https -y && \
    curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.gpg && \
    mv microsoft.gpg /etc/apt/trusted.gpg.d/microsoft.gpg && \
    sh -c 'echo "deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-ubuntu-xenial-prod xenial main" > /etc/apt/sources.list.d/microsoft.list' && \
    apt-get update && \
    apt-get install procdump -y && \
    apt-get install lldb-3.9 -y
RUN dotnet tool install -g dotnet-symbol && \
    export PATH="$PATH:/root/.dotnet/tools" && \
    dotnet symbol coredump && \
    dotnet new console && \
    echo 'using System; namespace app { class Program { static void Main(string[] args) { while(true) { Console.WriteLine($"{DateTime.Now:HH:mm:ss} Hello!"); System.Threading.Thread.Sleep(1000);} } } } '  > Program.cs && \ 
    dotnet build -c Release
WORKDIR /app/bin/Release/netcoreapp2.1
ENTRYPOINT [ "dotnet", "app.dll" ]

Then you need to run it with ptrace support and connect to it:

$ docker build --no-cache --rm -t lldb-poc .
$ docker run --cap-add=SYS_PTRACE --security-opt seccomp:unconfined -d --name lldb-poc-c lldb-poc 
$ docker exec -it --privileged lldb-poc-c /bin/bash

Once in, you will see that the application has the PID 1 (as usual when it is the entry point).
You can take the dump using procdump that is also installed:

# procdump --pid 1

ProcDump v1.0.1 - Sysinternals process dump utility
Copyright (C) 2017 Microsoft Corporation. All rights reserved. Licensed under the MIT license.
Mark Russinovich, Mario Hewardt, John Salem, Javid Habibi
Monitors a process and writes a dump file when the process exceeds the
specified criteria.

Process:                dotnet (1)
CPU Threshold:          n/a
Commit Threshold:       n/a
Threshold Seconds:      10
Number of Dumps:        1

Press Ctrl-C to end monitoring without terminating the process.

[07:31:51 - INFO]: Timed:
[07:31:54 - INFO]: Core dump 1 generated: dotnet_time_2018-10-22_07:31:51.1

Now that the dump is take, I try to open it in LLDB...

# lldb-3.9 `which dotnet` -c dotnet_time_2018-10-22_07\:31\:51.1
(lldb) target create "/usr/bin/dotnet" --core "dotnet_time_2018-10-22_07:31:51.1"

But it hangs forever... the prompt never gets to (lldb) again. I tried several variants, like loading the SOS plugin first, and then do the target create after, but the outcome is always the same.

I tried also to attach to the running process, but no luck:

# lldb-3.9 `which dotnet`
(lldb) target create "/usr/bin/dotnet"
Current executable set to '/usr/bin/dotnet' (x86_64).
(lldb) plugin load /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/libsosplugin.so
(lldb) process attach -p 1
error: attach failed: unable to attach

Could you please give me some light on how to explore the process' memory with LLDB in linux?

[amartinkyan]

Hi,
According to this link there is a bug in lldb until ver. 4.0. libsospling.so (up to ver. 2.1.6, haven't check 2.2 previews) is linked to lldb v. 3.9, so it's need to manually rebuild using provided instruction for your environment. You just need to install appropriate versions of lldb and liblldb before.
After you need to load plugin from <path to cloned repo>/bin/Product/Linux.x64.Debug/libsosplugin.so
P.S. I haven't search for lldb >= v4.0 for alpine, I've just built my container with dotnet:2.1.6-runtime-bionic

[mikem8361]

Actually the libsosplugin.so built for lldb 3.9 will work with lldb 4.0 and greater.

[mikem8361]

I would recommend trying lldb 4.0 on the Ubuntu or Debian distros (which I think this issue was originally reported). As far as Alpine, I haven't found a version lldb that works. I've even built 4.0 from source and it still hangs on Alpine.

[mikem8361]

I loaded the image, installed lldb-3.9 and procdump, created a testapp using your source, generated the core dump with procdump and ran into the same problem loading it in lldb. But when I used the runtime's "createdump" program to generate the dump, lldb-3.9 can load it just fine.

/usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.6/createdump -?
createdump [options] pid
-f, --name - dump path and file name. The pid can be placed in the name with %d. The default is '/tmp/coredump.%d'
-n, --normal - create minidump.
-h, --withheap - create minidump with heap (default).
-t, --triage - create triage minidump.
-u, --full - create full core dump.
-d, --diag - enable diagnostic messages.
root@3ec22aad5879:~/testapp# /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.6/createdump 1173
Writing minidump with heap to file /tmp/coredump.1173
Written 70950912 bytes (17322 pages) to core file

lldb-3.9 /usr/bin/dotnet -c /tmp/coredump.1173
(lldb) target create "/usr/bin/dotnet" --core "/tmp/coredump.1173"
Core file '/tmp/coredump.1173' (x86_64) was loaded.

There may be something wrong with the core dumps that procdump generates.

[mikem8361]

The problem seems to be between procdump and lldb and has nothing to do with SOS or .NET Core.

[vtortola]

Thanks a million @mikem8361 ! I can confirm I can do a sos DumpHeap :)