Async-profiler: Marry async-profiler and JMH

[leventov]

It would be nice to filter async-profiler's output when profiling JMH runs to include only traces from the benchmarked methods, excluding e. g. @Setup and @Teardown.

[apangin]

async-profiler produces output in a simple text format that you can easily filter with grep etc.

[retronym]

I think recently integrated async-profiler and JMH (with flamegraphs).
Details:

http://malaw.ski/2017/12/10/automatic-flamegraph-generation-from-jmh-benchmarks-using-sbt-jmh-extras-plain-java-too/

[nitsanw]

This is, IMO, a bad idea for any benchmark involving multiple threads.You are essentially saying you are not interested in any profile data from: 

• Other java threads
   * Native library threads
   * GC threads
   * Compiler threads

If your cpu cycles are spent in your benchmark thread, great, you got the profile you want anyhow, with some marginal data. If this is not the result you already got, why are you trying to ignore all this data about where the cycles are actually spent?
Also, grep.

[retronym]

My integration does gather samples from all threads, but only starts
profiling after the warm up iterations conclude.

[leventov]

@nitsanw in my case @Setup contains some very heavy stuff, in fact heavier than the execution itself, and it contaminates the profile. It doesn't make the analysis impossible, but harder.

You can say the same thing about JMH's -gc option - why would you not include GC in benchmarking, it's a part of a real application. But sometimes this option is useful.

[leventov]

@apangin yes, it could probably be something as simple as after-the-fact filtering (although I think not as primitive as grep). I just think it's worth to maintain such utility in the async-profiler repository.

[apangin]

Why grep is not enough? What else do you expect from the feature?

[nitsanw]

@leventov OK, reading the conversation more closely I think I understand your requirement, still not sure async-profiler is the place for it, but lets see.
I believe what you are looking for is a jmh-profiler integration for async-profiler:

• profile only includes the measurement period
• profiler is started with some command line param, e.g: `-prof async-profiler:e=cpu,...

I think I misunderstood what you meant by including only the benchmark methods, I believe what you meant is include only the measurement period, which is the standard way JMH profilers work. That makes sense?
From my recollections of looking at profiler integration for JMH this is very doable, but it's not clear to me that such a project needs to be part of the async-profiler repo. It would make sense IMO to have it in it's own repo, and require users to include it in their "project to be profiled" and expect them to install async-profiler separately. This is consistent with the perf profiler experience(except that it is internal and therefore requires no extra dep).
Makes sense?

[apangin]

1. Async-profiler has both external API and now internal Java API that is enough for integration with other tools.
2. There is already AsyncProfiler plugin for JMH available here.
3. JMH profilers may depend on external tools (for example, perf or dtrace), but async-profiler hardly depends on JMH.
   So, I believe async-profiler requires no further actions regarding this issue.

[retronym]

I'm going to contribute an async-profiler integration to JMH. Here's my work so far: https://github.com/retronym/jmh/pull/2

[retronym]

JMH 1.24 includes -prof async.