Async-profiler: Profiling distributed application using async-profiler

[qqibrow]

I used async-profiler to profile flink application. Flink is a distributed application. It runs with one master and multiple workers. In order to profile, I go to one worker, run async-profiler, get the flamegraph and analyze that. However, since this is a distributed application, I need to repeat this steps on multiple workers to confirm my observation.

Is there a way to aggregate all perf events data collect from multiple workers and get a single flamegraph? thanks!

[krzysztofslusarski]

You can do it like this:

• run async-profiler with output set to collapsed stack on every worker
• do simple text join of output files
• convert joined file with https://github.com/brendangregg/FlameGraph (./flamegraph.pl --color java file > flame.svg)

[apangin]

Right, concatenating output files in collapsed format is probably the easiest way to get an aggregated profile.

[JigarJoshi]

I am not familiar with JFR file format, is there a way to merge multiple JFR captures?

[krzysztofslusarski]

I wrote a tool, that parses multiple wall-clock JFR output from async-profiler, and merges them into 2 collapsed stack files (one contains CPU profile, other with wall-clock profile). It is not what you exactly wants, but maybe it will solve your problem.

https://github.com/krzysztofslusarski/collapse-jfr

[JigarJoshi]

I think from this it looks like it is just plain contamination of the files for JFR.