Diagnostics: Use diagnostics global tools functionality at runtime for problem reporting

[bitbonk]

Is there a way I could use the diagnostics global tools (or its functionality) at runtime in a deployed application for self-diagnosing or to generate a problem report? Is there some guidance on how to do this?

For example:

1. In case an application crashes due to an unhandled excpetion I would like to generate a dump (global tool dotnet dump) right inside the AppDomain.UnhandledException of the application itself.
2. In case the used memory of the current process exceeds a certain value (global tool dotnet counter) the process itself should generate a gcdump (global tool dotnet gcdump).
3. In case the used CPU utilisation of the current process exceeds a current value (global tool dotnet counter) the process itself should collect performance data (global tool dotnet trace).
4. In a servicing scenario a customer should be able to manually trigger the creation dumps and performance collections for the application.

It would be OK to do all of the above out of process. But the installation of a .NET Core SDK on the customer's machine is often not possible, so I suppose there is no way to install or use the global tools directly?

If this is not easily possible, consider this to be a feature request. 😁

[josalem]

We're actively working on a mechanism for installing the tools without requiring the SDK, but your scenario sounds like it would be better suited for the client library directly. @sywhang has a few samples where he does essentially what you're asking for in a couple lines of code.

[lg2de]

Hey @josalem, sound great but wondering why Windows is not supported!?

System.PlatformNotSupportedException: Unsupported operating system: Microsoft Windows 10.0.18362
   at Microsoft.Diagnostics.NETCore.Client.DiagnosticsClient.WriteDump(DumpType dumpType, String dumpPath, Boolean logDumpGeneration)
   at XXX.Program.Main(String[] args) in ...\Program.cs:line 51
   at XXX.Program.<Main>(String[] args)

https://github.com/dotnet/diagnostics/blob/d792a1b38f5cd0283ff55f2dd4143d3a09ea50e0/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsClient/DiagnosticsClient.cs#L50-L51

This is the first feature I found in dotnet core world that FIRST supports Linux...

[bitbonk]

Linux only? That’s a bummer. The weird thing is, dotnet dump does support Windows: https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-dump#description

[josalem]

Ah, you’ve stumbled across one of the few things that is different between Linux and Windows. Dumping is supported, just not via the same mechanism. For dumping we just use the Windows APIs directly rather than using the in-proc mechanism we use on Linux (see: https://github.com/dotnet/diagnostics/blob/d792a1b38f5cd0283ff55f2dd4143d3a09ea50e0/src/Tools/dotnet-dump/Dumper.cs#L60). The client library doesn’t hide this from users and throws that exception if you attempt to use the Diagnostics Server for dumping. You can do what dotnet-dump does in this case.

cc @sywhang @mikem8361

[lg2de]

You can do what dotnet-dump does in this case.

Can we add this to the client API?
Would you accept such PR?

[sywhang]

@lg2de IMO I don't think this belongs to the client API since the client library is concerned with handling diagnostics IPC protocol with the runtime, and triggering a dump on Windows already has a native API designed for it that can be wrapped like dotnet-dump has. The client library is not intended to be a tool itself, but rather a library for tool authors to leverage when dealing with the diagnostics IPC protocol with the tools they write.

I am open to discussions on this though.

cc @dotnet/dotnet-diag

[noahfalk]

One gotcha with the MiniDumpWriteDump API is that it may not work if you invoke it from within the process you are trying to dump. Implementing a call to MiniDumpWriteDump in the client API would create a leaky abstraction. If @bitbonk tried to use the hypothetical updated client library API support in his in-proc scenario it would probably work on Linux but fail in unexpected ways on Windows.

One possibility that could bring Windows/Linux more in sync here is to:

1. Implement CreateDump for Windows (_much_ simpler than the Linux implementation because it can call MiniDumpWriteDump to do all the work)

1. Change the ifdefs to enable Windows runtime support for the IPC event.

For anyone who wanted to have a process capture a dump of itself right now on Windows my recommendation would be to launch a 2nd process that will call MiniDumpWriteDump on your behalf. dotnet-dump could be that 2nd process, or you could make your own helper process using the implementation of dotnet-dump as an example.

[lg2de]

@sywhang: I don't think this belongs to the client API since the client library is concerned with handling diagnostics IPC protocol with the runtime

Mmh, Microsoft.Diagnostics.NETCore.Client sounds like a generic API.

@noahfalk: One gotcha with the MiniDumpWriteDump API is that it may not work if you invoke it from within the process you are trying to dump.

It _may_ not work, but we are using this API already for several years in native and .net framework applications without problems.

@noahfalk: Implementing a call to MiniDumpWriteDump in the client API would create a leaky abstraction. If @bitbonk tried to use the hypothetical updated client library API support in his in-proc scenario it would probably work on Linux but fail in unexpected ways on Windows.

I think, that current library is a leaky abstraction. It looks like a generic solution but it is limited to Linux.

[bitbonk]

I agree with @lg2de here.

Looking at the name Microsoft.Diagnostics.NETCore.Client and the description on Nuget , it appears to be a generic package that helps with diagnosing .NET Core processes. I would have expected that it works on all platforms that .NET Core runs on. The fact that it uses the IPC protocol for some of its features is (should be) merely an implementation detail.
From that perspective, the fact that it only works on Linux looks like a much bigger leaky abstraction to me than the fact that writing a dump sometimes may not work on Windows in the case that it was called for the same process.

So if Microsoft.Diagnostics.NETCore.Client is not and will not be a generic collection of APIs to diagnose any .NET Core process, I think there should be one. It should offer APIs that are equivalent to all the things you can do with the dotnet CLI diagnostic tools.

[noahfalk]

Mmh, Microsoft.Diagnostics.NETCore.Client sounds like a generic API

Fair criticism. The design intent and current implementation of the library are aligned around solely being a protocol client for the diagnostic server. We'll try to do a better job messaging that.

So if Microsoft.Diagnostics.NETCore.Client is not and will not be a generic collection of APIs to diagnose any .NET Core process, I think there should be one

I have no objetion to letting DiagnosticClient.WriteDump() API work for Windows, only to one particular way of implementing it. To the broader idea that we might want a library that has higher level abstractions or more generic mechanisms that are not solely tied to being an IPC client - Agreed 👍. Recently we've been exploring the idea of a monitoring agent and inside the agent there could be a library that is also hostable from in-proc. So far it is very early investigation and I can't promise where (or when) it will go exactly. I don't want to discourage you or anyone in the community from scratching this itch if you have more specific goals and timeline in mind.

we are using this API already for several years ... without problems

Unfortunately I also have to consider the guidance from the team at Microsoft which supports MiniDumpWriteDump and I have heard stories from other teams like Watson and Visual Studio whose results were not as uniformly successful. As a maintainer I can't be confident if I rely only on examples of success while disregarding evidence of problems.

the fact that it only works on Linux looks like a much bigger leaky abstraction to me

I think of the current implementation as being incomplete and agreed, that too can be classified a leaky abstraction. The distinction in my mind is that an incomplete implementation can be completed whereas if we implement it to have different failure modes then it isn't easy to change away from that.

I do appreciate the feedback and I don't relish being the guy who stands in the way of a quick and pragmatic implementation that probably would work in many cases. I hope one or more of these options to move forward is a reasonable alternative though:
1) Use the client library as-is and implement a call to MiniDumpWriteDump to work around the incomplete Windows support. Code from dotnet-dump can provide an example.
2) Microsoft and/or community completes the implementation of the IPC + CreateDump mechanism for dumps on Windows to bring the client API to parity with Linux
3) Microsoft and/or community puts together a library with higher level abstractions that is free to leverage mechanisms other than the IPC channel.

Hope this helps!
-Noah

[lg2de]

I tried creating dump similar to implementation of the "dotnet dump" tool.

The dump creation always fails with ERROR_PARTIAL_COPY.
https://github.com/dotnet/diagnostics/blob/0d368aa2edfad485d9eff694bda43792cdd4c3c8/src/Tools/dotnet-dump/Dumper.Windows.cs#L47

Do you have any suggestion?

[noahfalk]

@mikem8361 authored our dotnet dump code so he might have a better recollection of that error and why he put a retry loop on it. I've heard of a couple reasons it might occur, though I can't tell if either applies to your case:

1. cross bitness issues
2. Bad exception pointer

[mikem8361]

I put the retry loop for ERROR_PARTIAL_COPY because of occasional failures that would go away after one or two retries. I'm not sure why it is always returning this error on retry other than what Noah has suggested.