Bpftrace: ustack not symbolicated if traced process exits first

[ahupp]

I have process that allocates once and then sleeps:

#include <unistd.h>
#include <stdlib.h>

int main() {
  void* a = malloc(2);
  sleep(1000);
  return 0;
}

And I trace it with this command:

sudo bpftrace -e 'uprobe:/lib64/libc.so.6:malloc /comm=="test_malloc"/ { @[ustack] = count(); }'

If I exit test_malloc first before exiting bpftrace, I get unsymbolicated output:

@[
0x7fc578a18800
0x7fc5789b5445
]: 2

But if I exit bpftrace first it is symbolicated:

@[
__libc_malloc+0
__libc_start_main+245
]: 2

So it seems like the symbolication step depends on having the process running. It would be convenient for the things I'm doing to have it symbolicate regardless of whether the process has exited.

[tyroguru]

I've just taken a quick look at this and we don't appear to attempt to resolve the stacks until bpftrace exits in this case (at least that's how it appears under a debugger). If I'm interpreting the code correctly then this would be the wrong thing to do. We should try and resolve symbols as soon as we are handed the array of PCs.

Maybe someone who understands this code better will chip in but I'll take a look next week if nobody does in the meantime.

[mmarchini]

We should try and resolve symbols as soon as we are handed the array of PCs.

As @tjfontaine said in https://github.com/iovisor/bpftrace/issues/286, if we try to resolve symbols too soon we'll introduce extra overhead into the traced application. I guess we could try to resolve symbols in a separate bpftrace thread, but even then there might be some unforeseen overhead.

What we can do is to keep the memory mapped information of traced processes cached in bpftrace, and use this cached information if the process has exited when we're resolving symbols. This might be more laborious than it sounds though, since symbol resolution is implemented in bcc.

[aktau]

What we can do is to keep the memory mapped information of traced processes cached in bpftrace, and use this cached information if the process has exited when we're resolving symbols. This might be more laborious than it sounds though, since symbol resolution is implemented in bcc.

Could symbol resolution be extracted into a separate library? It seems like a self-contained thing. (I came here because I noticed lots of Error looking up stack id 0 (pid 0) in offwake.bt output.

[danobi]

I'm considering an offline symbol resolution approach. If we store base addr and exe path, we should be able to resolve symbols even after the process exits. However, it'll only work for .text addrs. The advantage over real time symbolizing is this avoids races (the exception being the binary is updated).

I spoke w/ @yonghong-song and he's open to putting this functionality in bcc.