Diagnostics: Crash reporting and .NET dumps

[bruno-garcia]

With dotnet dump, I was hoping to get a better understanding of:

1. What is the use-case for it in a usual JIT environment, where we can capture managed exceptions and when portable pdbs are available, so are the managed stack traces? Is the goal to improve troubleshooting native code in the process like the GC? And other code called by P/Invoke?

2. Is there a native (unmanaged) hook we can use to capture these dumps and upload to a crash service like sentry.io? @davidfowl mentioned we could inspect the file system for dumps but I was hoping to run the code in the same process as the crashing app, ideally before the crash (before and/or after generating the dump) or perhaps after restart once the process is "in a good state".

3. How does this relate to the new AOT scenarios coming to .NET? How can we symbolicate the memory dumps back to the original managed code. On Mono, the mvid and aotid are embbedded in the stack trace, but only once ToString is called and there's no API that I could find to get these values other than parsing that string. That's how the mono-symbolcate tool works.

4. How does symbolication work for crash dumps? Are the portable pdb and native symbols both needed? Is the code doing that in this repo, that could help clarify.

In other words I'm just a bit confused how we'll be able to capture crashes, upload them as they happen, and provide nice stack traces to users without their need to do anything other than upload debug symbols to the service, and integrate the crash handling SDK in their app.

[josalem]

dotnet-dump is mainly for on-demand dump generation and not reactionary dump generation. The mechanism it uses is asynchronous, so you won't be able to hook dotnet-dump to, say, take a dump on a first chance exception in managed code while that code is in the catch block. You could use it for collecting dumps based on counter thresholds, e.g., when the # of first chance exceptions per minute goes over some value take a dump. If you used the diagnostics SDK directly, you could use the underlying IPC protocol to issue the dump command on-demand, though.

I think what you are trying to ask would be better suited for the dotnet/runtime repo, but the same people who would be answering your questions there will see it here. There is code in the runtime for collecting a dump of itself--it's the same code that dotnet-dump invokes from outside. There is also an environment variable pair that can be set that will cause the runtime to take a dump of itself on fatal crashes, COMPlus_DbgEnableMiniDump and COMPlus_DbgMiniDumpName (see: https://github.com/dotnet/runtime/blob/master/docs/design/coreclr/botr/xplat-minidump-generation.md for more details).

CC @mikem8361 @noahfalk

[noahfalk]

Everything @josalem above is accurate, let me add to it because I don't think all the questions got answered.

What is the use-case for it in a usual JIT environment

Today I expect the most useful scenarios would be:
a) Capture a dump of a hung app to analyze all the thread stacks and locks
b) Capture a dump of an app that is using too much memory to analyze what objects are on the GC heap

when portable pdbs are available, so are the managed stack traces?

Yes, if you analyze the dump with a tool that understands managed stack traces. Visual Studio, dotnet dump analyze, and sos loaded into windbg/lldb can all do it.

Is the goal to improve troubleshooting native code in the process like the GC? And other code called by P/Invoke?

It depends what behavior you are trying to troubleshoot. The dump will include native thread stacks and might be very useful if a p/invoke call was deadlocked. On the other hand if your native code was supposed to compute the result "Apple" and instead it keeps returning "Orange", a native debugger or some logging is probably better suited.

Is there a native (unmanaged) hook we can use to capture these dumps and upload to a crash service like sentry.io?

@josalem's info about COMPlus_DbgEnableMiniDump is probably what you want here

How does this relate to the new AOT scenarios coming to .NET?

At the moment, it doesn't : ) dotnet-dump was originally designed to work for coreclr. I am interested to see if it can be expanded to work with mono's runtime as well but so far this isn't an area of the tooling that converged.

How does symbolication work for crash dumps? Are the portable pdb and native symbols both needed?

If you want to see ___ in your dump viewer then you need _____

1. Managed method names -> Managed assembly metadata. (In heap or full dumps this metadata is captured in the dump, otherwise it is available in the application's assemblies)
2. Managed source file names and line numbers -> Portable PDBs
3. Native method names and source info -> native symbols

There is some related info about the underlying APIs in the thread here

Is the code doing [symbolication] in this repo, that could help clarify.

Symbolication is done typically as part of viewing the dump. Visual Studio has an implementation, windbg has an implementation, lldb has an implementation and so on. Symbolication is also a fairly broad set of steps so we may need to clarify what part you want to learn about to find the right resources. Roughly the implementation involves:
unwinding a stack, parsing binary file formats (PE, ELF, Metadata), searching for files that are external to the dump (dlls, symbols), downloading and caching files from symbol servers, parsing symbolic formats (DWARF, CodeView, Portable PDB), doing the symbolic lookups, and formatting the final results.

In repos that dotnet owns there are two dump viewing scenarios one in "dotnet dump analyze" and another by loading SOS into lldb or windbg. Both of those scenarios share most of the underlying code to print the stack trace
This is also the dotnet-symbol tool which can download binaries/symbols offline.

[mikem8361]

@bruno-garcia did you get the answer's you need?

[bruno-garcia]

There is also an environment variable pair that can be set that will cause the runtime to take a dump of itself on fatal crashes, COMPlus_DbgEnableMiniDump and COMPlus_DbgMiniDumpName (see: https://github.com/dotnet/runtime/blob/master/docs/design/coreclr/botr/xplat-minidump-generation.md for more details).

Thanks @josalem for the pointer, the link is very helpful.

Today I expect the most useful scenarios would be:
a) Capture a dump of a hung app to analyze all the thread stacks and locks
b) Capture a dump of an app that is using too much memory to analyze what objects are on the GC heap

That clarifies things, thanks.

That's clearly very useful. I wish I had that many times in the past.

I'm looking at it mostly from the crash reporting perspective. Sentry already supports (through symbolic) most debug file formats like breakpad, ELF, DWARF, Windows PDBs but there's no support for Portable PDBs. Besides the .NET PDBs, with the AOT scenarios, I originally looked at it to support Mono AOT, which is used by Xamarin on iOS and I expect to be .NET for iOS on .NET 6.

did you get the answer's you need?

@mikem8361 yes!

Thanks everyone for the clarifications.