Async-profiler: How to properly exclude garbage collector threads?

[nobeh]

I've been trying to use -X option to exclude specifically threads that are running GC on the target JVM. I've had no luck with this option:

$ profiler.sh -e cpu|wall -i 1ms -d 10 \
      -X '*GC*' \
      -X '*GarbageCollector*' \
      -X '*Generation*' \
      -X '*.gc*'
      $JAVA_PID

In the current situation, it is known that JVM spends a lot of time on GC. Using the above options still shows the GC threads which are G1 Young and Old generation threads. It is also quite the same with -e cpu.

What am I missing to use this feature correctly? Thanks!

[apangin]

Can you post a reproducible example? I don't see a problem with the above command line - GC related stack traces are correctly filtered from the output.

Is it possible that an older version of the agent library has been already loaded into the target process? In this case the newer version of async-profiler won't take effect until you restart the JVM.

[nobeh]

We're not using the agent mode. The profiler is executed on every invocation. Version 1.6 is the one that is used.

With the above example command, this is an example screen that I get from SpeedScope. There's large stack frame labeled with [GC_Active] although it's being excluded via the command option. Creating an isolated reproducible example would be a task on its own.

[apangin]

We're not using the agent mode

No matter if you launch profiler with -agentpath option or attach it in runtime using profiler.sh script, the profiler library gets loaded into the target JVM and cannot be updated until JVM restarts.

Version 1.6 is the one that is used.

Method filters appeared only in a recent version from master (1.7-ea3).
profiler.sh -v <pid> should print Async-profiler 1.7-ea3 ...

There's large stack frame labeled with [GC_Active]

Normally, [GC_active] should never appear in the profile, unless

• the JVM is not HotSpot (e.g. Zing), or
• libjvm.so binary misses debug symbols. In this case profiler prints WARNING: Install JVM debug symbols to improve profile accuracy in the JVM console. See README for the instructions on installing debug symbols.

What output format do you use? Note that methods filters do not work with JFR format for the reasons explained here.

[nobeh]

Thanks for the comments @apangin.

• We're still on 1.6 which does not have the new filtering switches.
• We're using OpenJDK which does not have the debug symbols by default. Not sure it'd be easy to install them on production.
• We're using collapsed format and not JFR.

[apangin]

collapsed format can be easily filtered with grep.
Otherwise upgrade to async-profiler 1.7 to use -I / -X switches.