Async-profiler: [Feature request] Add support for DWARF stack unwinding to show inlined native frames

[franz1981]

On recent(-ish) versions of Fedora/RHEL/CentOS perf uses the elfutils/libdw unwinder instead of
libunwind since it is faster and more reliable on all supported
architectures: supporting stack unwinding using DWARF would be great to allow a better observability of the native space, given that inlined native frame are not available while using commong FP stack walking.

[apangin]

I looked at libdw, but did not see how it could be easily integrated into async-profiler.
The main problem is that the profiler can currently use only async signal safe API for stack walking.

However, I'll leave this issue open in case I decide to revise how async-profiler process stack traces (i.e. switch to offline processing).

[franz1981]

I can write again to Jiri Olsa, dev of libdw, to ask for some help about it...maybe I will get some good hints on how to integrate it and will send a PR, but many thanks to have taken a look!!!

[apangin]

I spent some more time researching the topic, and came up with two findings: a good one and a bad one.

The bad one is that it doesn't look feasible to do DWARF-based unwinding on the fly. DWARF unwinding is really expensive both in computation and in memory used. While it's not usually a problem to get a stack trace during debugging, interpreting DWARF information during profiling will cause too much overhead. Furthermore, in async-profiler's case, stack traces are collected inside a signal handler, that incurs strict limitations on what can be done: e.g. no memory allocation, no locks etc.

There is a recent study [1] on improving DWARF unwinding performance by an order of magnitude using precompiled unwind tables. Sounds cool, except that implementing this approach in async-profiler would mean creating a specialized JIT compiler for unwind tables. This task alone is bigger than the rest of the profiler :) That's going to be too much effort for a Java profiler.

But I also have a good news for you.

Modern Intel CPUs can profile branch instructions, including calls and rets, and store their source and destination addresses (Last Branch Records) in hardware registers. Starting from Haswell, CPU can match these addresses to form a branch stack. This branch stack will be effectively a call chain automatically collected by the hardware.

What makes LBR call chain especially useful is that it works even without frame pointers. perf has supported LBR since Linux 4.1, and now I've added LBR call chain support to async-profiler.

The new option is --cstack lbr. It works only with hardware events like -e cycles (instructions, cache-misses etc.)

The maximum call chain depth is 32 (hardware limit). LBR stacks are not always complete or accurate, but they still appear much more helpful comparing to fp-based stack walking, when a native library is compiled with omitted frame pointers.

Here is an example of a traditionally collected profile:

and the same application profiled with --cstack lbr:

[apangin]

Taking the above into account, there are no plans to support DWARF in async-profiler.
LBR stack unwinding is a viable alternative.