Bpftrace: BEGIN probe not working

[danobi]

$ sudo ./src/bpftrace -e 'BEGIN { printf("hello\n"); }'
Attaching 1 probe...
Could not resolve symbol: /proc/self/exe:BEGIN_trigger

Built from master at time of writing.

[danobi]

So that was on my fedora box. It works ok on my arch linux box.

[caringi]

On my fedora it's working...

$ cat /etc/fedora-release 
Fedora release 30 (Thirty)
$ uname -r
5.2.9-200.fc30.x86_64+debug
$ git rev-parse HEAD
2e73e04e27c439cf87d3687be24c77cce2cf65ae
$ sudo ./src/bpftrace -e 'BEGIN { printf("hello\n"); }'
Attaching 1 probe...
hello
^C

[danobi]

More info on my box:

$ cat /etc/fedora-release
Fedora release 30 (Thirty)
$ uname -r
5.2.16-200.fc30.x86_64

[brendangregg]

Here's a clue:

/usr/local/bin/bpftrace doesn't work:

(root) /mnt/src/bpftrace/build # bpftrace -e 'BEGIN { @++ }'
Attaching 1 probe...
Could not resolve symbol: /proc/self/exe:BEGIN_trigger
(root) /mnt/src/bpftrace/build # which bpftrace
/usr/local/bin/bpftrace

But build/src/bpftrace does:

(root) /mnt/src/bpftrace/build # ./src/bpftrace -e 'BEGIN { @++ }'
Attaching 1 probe...
^C

@: 1

They are the same version(!):

(root) /mnt/src/bpftrace/build # bpftrace --version
bpftrace v0.9.2-154-gd95a

(root) /mnt/src/bpftrace/build # ./src/bpftrace --version
bpftrace v0.9.2-154-gd95a

but they are not quite the same binary:

(root) /mnt/src/bpftrace/build # file /usr/local/bin/bpftrace src/bpftrace 
/usr/local/bin/bpftrace: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/l, for GNU/Linux 3.2.0, BuildID[sha1]=9a58fc11648c14151a3c49d8a7cc592bf21bd16b, with debug_info, not stripped
src/bpftrace:            ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/l, for GNU/Linux 3.2.0, BuildID[sha1]=091d31d33877b70c104dd42fbce8508850fc5376, not stripped

(root) /mnt/src/bpftrace/build # ls -lh /usr/local/bin/bpftrace src/bpftrace 
-rwxr-xr-x 1 root nac  5.0M Dec 17 19:51 src/bpftrace
-rwxr-xr-x 1 root root  18M Dec 17 19:33 /usr/local/bin/bpftrace

make install is turning build/src/bpftrace into a bpftrace version that includes debug info, and also doesn't work.

[brendangregg]

It's also not just BEGIN, it's any uprobe:

(root) /mnt/src/bpftrace/build # bpftrace -e 'uprobe:/bin/bash:readline { @++ }'
Attaching 1 probe...
Could not resolve symbol: /bin/bash:readline

(root) /mnt/src/bpftrace/build # ./src/bpftrace -e 'uprobe:/bin/bash:readline { @++ }'
Attaching 1 probe...
^C

[dalehamel]

What bcc y'all linking against? I also have this issue, latest master of bpftrace, bcc 0.12.0. Linux kernel 5.3.0

I wonder if maybe your system bpftrace could be linking against a different bcc than the one in your build directory @brendangregg ?

Super weird that one would work and not the other if they are on the same revision and linking the same libs...

[brendangregg]

I've resolved my issue: When building a fresh bcc & bpftrace, I switched the install path to /usr/bin, which has a working bpftrace. /usr/local/bin contains an older broken bpftrace. Even though they are the same version (v0.9.2-154-gd95a) and point to the same libraries (ldd)! The /usr/local/bin version was built with a previous bcc version, and contains code that doesn't work on the newer one.

For other people hitting this: does the bpftrace in the build/src directory work?

[danobi]

For other people hitting this: does the bpftrace in the build/src directory work?

It works for me (same fedora box as I originally opened this issue with). Just did a fresh install of bcc master and a clean build of bpftrace.

Edit: misread the question

[dalehamel]

The cause of this was discovered by @fbs.

edit: it may have also just been an overall uprobe issue

The problem is that these executables have been stripped, and they need to leave the symbol BEGIN_trigger. I've added this to the post-processing of the release artifacts of #1041 with the --keep-symbol flag to strip. To verify, just run nm on the bpftrace binary. For it to work you must see:

nm src/bpftrace
0000000000b5a750 T BEGIN_trigger

I think this can be closed as solved - don't strip this symbol as bpftrace's BEGIN probe works by probing itself, which I think is the intended behavior, but poses a quirk for released binaries.

It's also not just BEGIN, it's any uprobe:

That doesn't explain the uprobe error though I've just realized, so perhaps this is backwards - because the BEGIN probe is trying to probe itself, it wouldn't work because uprobes were broken.

[mmarchini]

My general advice is: don't strip binaries. It will mess with tracing tools and profilers, since they need symbols to properly translate addresses. I've seen many incorrect profiles with perf because the binaries were stripped.

But the issue reported by @brendangregg was with a unstripped binary, so I suggest keeping this open until someone confirms it has been fixed.

[fbs]

Closing this as no one seems to have this issue anymore

[ebergen]

I'm just dropping this comment for anyone else who hits the same issue. I hit this on my bone stock Fedora 31 install and it was resolved by installing the debug package.

dnf debuginfo-install bpftrace

[twhitehead]

Seems to be broken on my NixOS system.

# bpftrace -e 'uprobe:/bin/sh:readline { @++ }'
Attaching 1 probe...
Could not resolve symbol: /bin/sh:readline

I believe the symbol is in the binary though (or at least the readline library it links against)

# nm /bin/sh | fgrep readline
...
                 U readline
...

# ldd /bin/sh
...
    libreadline.so.7 => /nix/store/ms1ris36xzyx9rzyss4h7pir759adc2d-readline-7.0p5/lib/libreadline.so.7 (0x00007f972bdf1000)
...

The bpftrace binary is not stripped and it has the BEGIN_trigger symbol defined

# file $(which bpftrace)
/nix/store/jnms9sxc6gvbnsrrivvciah83d9rbdqi-bpftrace-0.9.4/bin/bpftrace: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /nix/store/9hy6c2hv8lcwc6clnc1p2jf09cs5q9dp-glibc-2.30/lib/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, not stripped

# nm $(which bpftrace) | fgrep BEGIN_trigger
0000000000477b70 T BEGIN_trigger

[danobi]

@twhitehead can you try:

# bpftrace -e 'uprobe:/nix/store/ms1ris36xzyx9rzyss4h7pir759adc2d-readline-7.0p5/lib/libreadline.so.7:readline { @++ }'

and see if that works?

My guess is the readline symbol is defined in that library. On my system it seems like readline is not a dynamic symbol.

[twhitehead]

You are correct. If I put the trace on the readline library it works, and then I can easily use a comm filter to select a particular program if I want.

Thanks so much for clarifying! I have a better mental picture now of what is happening. :+1:

[WGH-]

The functionality is also broken when you have the debug info, but it's stored in a separate file.

The filename bpftrace passes to bcc is "/proc/self/exe", and bcc ends up searching for the wrong filename:

$ sudo strace -e file bpftrace -e 'BEGIN { print("hi\n") }'
<skip>
openat(AT_FDCWD, "/proc/self/exe", O_RDONLY) = 29
stat("/proc/self/bpftrace.debug", 0x7ffff6d02060) = -1 ENOENT (No such file or directory)
access("/proc/self/bpftrace.debug", F_OK) = -1 ENOENT (No such file or directory)
access("/proc/self/.debug/bpftrace.debug", F_OK) = -1 ENOENT (No such file or directory)
access("/usr/lib/debug/proc/self/bpftrace.debug", F_OK) = -1 ENOENT (No such file or directory)
Could not resolve symbol: /proc/self/exe:BEGIN_trigger
+++ exited with 255 +++