Diagnostics: [Feature suggestion] Need a simple tool to get stacktrace of all threads, like jstack in java world

[wsq003]

It's painful (even impossible) to get callstack of all threads, which is very useful for diagnosing performance issue.

We need a simple tool equivalent to jstack.

[benaadams]

On WIndows you can right click in Task manager on the process and choose "Create Dump File"; then open in VS or WinDebug

For 3.0 on Linux you can use dotnet-dump

Might be other ways also

[josalem]

With 3.0, you can use the dotnet-trace tool to do CPU sampling with stack traces. See here for a tutorial on diagnosing high cpu. dotnet-trace can also output a speedscope formatted file if you're trying to open the file on Linux or Mac.

@dotnet/dotnet-diag

[noahfalk]

The thought for a jstack like tool has come up a few times and I am happy to have an issue tracking it.

I'd echo that both @benaadams and @josalem's options should approximate the jstack scenario and they are available in 3.0 previews. As far as I can tell both of these solutions offer a super-set of the data you would get with jstack, but the small downside is that each requires multiple steps in the UI to complete. I'm hoping the 'painful (event impossible)' description is because these options aren't well documented and that if we improve awareness and documentation we'll be at "easily doable, but still a little more overhead than jStack"

If we wanted to build a UI tool that approximated jStack more closely my suggestion would be to start from the code of dotnet-trace, hard-code the stack sampling provider as the config, wait to observe one set of stack samples, then stop the session and use TraceEvent to format the stacks on the console. If anyone wanted a fun diagnostics project, this should be fully doable as an OOB tool with runtime support that is already in 3.0.

[GSPP]

Some java server apps such as Solr have the ability to display stacks in the browser. This is very useful.

Whenever a production application is having performance issues, sampling a few stacks is a very effective way to debug this. Being able to do this programmatically and in a way that is suitable for production use is valuable.

[acolombi]

small downside is that each requires multiple steps in the UI to complete

One of my main use cases for jstack is as proactive collection, rather than reactive. In the past, anywhere I've run a production Java service, I've also run a periodic (usually like once a minute) jstack and appended its output to a log file. Basically you're always collecting jstacks, not just when you think you might have a problem. This helps diagnose issues that are so severe they destabilize the runtime itself, and things like connecting a profiler stop working. For this use case I think any interactive UI at all is probably a deal breaker. You want to be able to run this from a simple command line and append the output to a log that you aggregate somewhere.

dotnet-trace might work. It's definitely more information than what you get from jstack. It might be a suitable alternative, if the collected data doesn't grow too rapidly, and its suitable to run in production basically at all times.

[noahfalk]

Thanks for the info @acolombi !

dotnet-trace might work. It's definitely more information than what you get from jstack. It might be a suitable alternative, if the collected data doesn't grow too rapidly, and its suitable to run in production basically at all times.

Dotnet-trace has some production use cases, but the cpu sampling in specific can be expensive. The exact overhead is going to depend on the app and the amount of load already present on the service. For someone wanting to use it proactively, using it only intermittently might mitigate that impact. For example run it for a few seconds every 10 minutes and then you accept that during those few seconds that node of your service may handle fewer requests than normal. You could measure the overhead to decide if it was acceptable.

One thing to be aware of for the dotnet-trace style workaround is that dotnet-trace does something called 'rundown' at the end of the trace. Rundown captures information about every method loaded in the process so that the IPs collected in stack traces can be resolved. It is probably the limiting factor for creating a small trace file right now. In a high quality implementation of jstack we'd probably want to disable rundown and replace it with something that only logs information about the IPs that were observed in collected callstacks to further shrink the data collection overhead.

[josalem]

This issue has been open for a while, but I was going back through some old side projects I had thrown together. I wrote a sample app that does roughly what jstack does for dotnet apps: https://github.com/josalem/dotstack. I just went through and bumped it up to the newest versions of its dependencies. I also added corert for easily getting an AOT'd, single file binary.

I'll take a look at adding similar functionality to our diagnostic tools in dotnet/diagnostics. For now, I'll move this issue over there and use it for tracking this feature.

[tmds]

This is super useful!

@josalem any idea when dotnet-stack may show up on nuget.org? Is it .NET 6, or will it be earlier?
I have the same question for dotnet-monitor.

[sywhang]

@tmds Our tools ship monthly, and we recently shipped a release this week. So you can expect to get a publicly available version of this some time next month :)