Spring-boot: Add support for plain text output to the threaddump endpoint

Created on 12 Jan 2015 · 24Comments · Source: spring-projects/spring-boot

The typical use of threaddumps is in troubleshooting JVM hangs or other performance issues where you want an indication of what the threads in the JVM are doing. The /dump endpoint enabled by the actuator module is a step in the right direction; however the stack traces of all the threads are returned in JSON which cannot be parsed by ANY of the standard threaddump parsing tools. I tried all of them - jconsole, jstack, Thread Dump Utility, etc.. The volume of information in the dump is such that visually identifying the culprit thread is not possible. In some circumstances you are typically looking at 3 or more threaddumps to understand if the the threads are moving. Therefore in my view /dump is not serving the purpose it was designed for. I propose the following -

Provide a separate endpoint or modify the existing /dump, to provide a threaddump in a format (perhaps *.tdump) that can be parsed by a thread dump utility or tool. This could be as simple as providing a StringHttpMessageConverter for the response.
Leverage all the runtime information from the ManagementFactory.getThreadMXBean() including memory, OS information and lock information.

Threaddumps are poor man's profilers and enabling this function will go a long way to make the spring boot a production framework without a need of custom APM solutions to diagnose performance issues.

enhancement

Source

kelapure

👍14

Most helpful comment

Sorry for this. I would have prefered the format jstack outputs. The current json format can be kept (maybe someone find it useful or already have a parser for it). But we for sure should have the option o get a more standard format such as jstack one.

renannprado on 8 Jun 2017

👍6

All 24 comments

References:

kelapure on 12 Jan 2015

👍1

Great idea. I'll target this to 1.3.

philwebb on 12 Jan 2015

I thought the best way to do this was to use content negotiation to extract a text/plain result from the existing endpoint. But there is no HttpMessageConverter registered by default that can handle it (we need one that just does a toString() on all elements in a Collection and concatenates the result).

As an aside: I didn't find anything in jvisualvm that parsed the .tdump files into anything anyway (so it is quite happy to read JSON). Is there an actual specific parser plugin that you can recommend?

dsyer on 27 Aug 2015

@kelapure Did you see Dave's question?

As an aside: I didn't find anything in jvisualvm that parsed the .tdump files into anything anyway (so it is quite happy to read JSON). Is there an actual specific parser plugin that you can recommend?

wilkinsona on 30 Oct 2015

visualvm plugin tda can parse tdump file. and dump it to text like this https://github.com/cqyijifu/watcher/blob/master/watcher-core%2Fsrc%2Fmain%2Fjava%2Fcom%2Fyiji%2Fframework%2Fwatcher%2Fmetrics%2FJstackMetrics.java#L38

bohrqiu on 3 Nov 2015

we can use Sigar to expose os info like https://github.com/cqyijifu/watcher/tree/master/watcher-core/src/main/java/com/yiji/framework/watcher/metrics/os

bohrqiu on 3 Nov 2015

Do you have a link to the plugin somewhere (the consumer of the dumps)?

dsyer on 3 Nov 2015

https://java.net/projects/tda/

bohrqiu on 3 Nov 2015

Interesting. It looks like it hasn't been touched in 4 years though, and I didn't see any prebuilt binaries. How do you install it in jvisualvm?

dsyer on 4 Nov 2015

you can download it https://java.net/projects/tda/downloads/directory/visualvm

bohrqiu on 4 Nov 2015

I installed it into jvisualvm but it's not active and is greyed out in the "plugins" dialogue (no way to activate it that I can see). I can't imagine it's very widely used if it's this difficult to install.

dsyer on 5 Nov 2015

you need install three plugins in that page,this plugins aren't very useful. but thread dump can output into text format is very useful.

like watcher do https://github.com/cqyijifu/watcher/blob/master/watcher-core%2Fsrc%2Fmain%2Fjava%2Fcom%2Fyiji%2Fframework%2Fwatcher%2Fmetrics%2FJstackMetrics.java#L38

bohrqiu on 5 Nov 2015

The tool (works across JREs) I use for parsing thread dumps is https://www.ibm.com/developerworks/community/groups/service/html/communityview?communityUuid=2245aa39-fa5c-4475-b891-14c205f7333c
Download link: ftp://public.dhe.ibm.com/software/websphere/appserv/support/tools/jca/jca457.jar

kelapure on 10 Nov 2015

I like @bohrqiu idea of adding OS level metrics to the thread dump https://github.com/cqyijifu/watcher/tree/master/watcher-core/src/main/java/com/yiji/framework/watcher/metrics/os

For inspiration we can look what the IBM JRE puts in its threaddumps
http://www-01.ibm.com/support/knowledgecenter/SSYKE2_8.0.0/com.ibm.java.lnx.80.doc/diag/tools/javadump_interpret.html

Javadump tags
The Javadump file contains sections separated by eyecatcher title areas to aid readability of the 

Javadump.
TITLE, GPINFO, and ENVINFO sections
At the start of a Javadump, the first three sections are the TITLE, GPINFO, and ENVINFO sections. They provide useful information about the cause of the dump.

Native memory (NATIVEMEMINFO)
The NATIVEMEMINFO section of a Javadump provides information about the native memory allocated by the Java Runtime Environment (JRE).

Storage Management (MEMINFO)
The MEMINFO section provides information about the Memory Manager.

Locks, monitors, and deadlocks (LOCKS)
An example of the LOCKS component part of a Javadump taken during a deadlock.

Threads and stack trace (THREADS)
For the application programmer, one of the most useful pieces of a Java dump is the THREADS section. This section shows a list of Java threads, native threads, and stack traces.

Shared Classes (SHARED CLASSES)
An example of the shared classes section that includes summary information about the shared data cache.
Class loaders and Classes (CLASSES)
An example of the classloader (CLASSES) section that includes Classloader summaries and Classloader loaded classes. Classloader summaries are the defined class loaders and the relationship between them. Classloader loaded classes are the classes that are loaded by each class loader.

kelapure on 10 Nov 2015

An alternative format, preferably something that looks like what the JVM dumps to stdout when you send a kill -QUIT, would also be nice since it is easier on the eyes. The json formatting for the current endpoint useful for programmatic parsing but is mostly noise for us humans.

b-hoyt on 12 Jan 2017

@renannprado In an attempt to turn your comment into something constructive, what format would you have preferred?

wilkinsona on 8 Jun 2017

renannprado on 8 Jun 2017

👍6

@wilkinsona Also as containers are getting popular and taking thread dumps from a container is tricky, having this issue fixed is very important to help investigation of performance issues.

renannprado on 8 Jun 2017

I'd also like to see jstack style outputs. Are there any third party scripts available that take the JSON and format like jstack?