bpftrace can't execute when shell contains 'BEGIN' or 'END' for Fedora32 rpm 0.11.0.1

[hermithuang]

bpftrace can't execute when shell contains 'BEGIN' or 'END' for Fedora32 rpm 0.11.0.1

#bpftrace --info
System
  OS: Linux 5.7.9-200.fc32.x86_64+debug #1 SMP Fri Jul 17 16:09:14 UTC 2020
  Arch: x86_64

Build
  version: v0.11.0
  LLVM: 10
  foreach_sym: yes
  unsafe uprobe: no
  bfd: yes
  bpf_attach_kfunc: yes
  bcc_usdt_addsem: yes
  libbpf: yes
  libbpf btf dump: yes
  libbpf btf dump type decl: yes

Kernel helpers
  probe_read: yes
  probe_read_str: yes
  probe_read_user: yes
  probe_read_user_str: yes
  probe_read_kernel: yes
  probe_read_kernel_str: yes
  get_current_cgroup_id: yes
  send_signal: yes
  override_return: yes

Kernel features
  Instruction limit: 1000000
  Loop support: yes
  btf: yes

Map types
  hash: yes
  percpu hash: yes
  array: yes
  percpu array: yes
  stack_trace: yes
  perf_event_array: yes

Probe types
  kprobe: yes
  tracepoint: yes
  perf_event: yes
  kfunc: yes

#sudo bpftrace -e 'BEGIN { printf("Hello eBPF!\n"); }'
Attaching 1 probe...
Could not resolve symbol: /proc/self/exe:BEGIN_trigger



md5-4d5ecf79746089b18c7e3d217ee21ef6



#nm /usr/bin/bpftrace | grep -i begin
nm: /usr/bin/bpftrace: no symbols



md5-4d5ecf79746089b18c7e3d217ee21ef6



#rpm -qi bpftrace
Name        : bpftrace
Version     : 0.11.0
Release     : 1.fc32
Architecture: x86_64
Install Date: Sat 25 Jul 2020 09:53:09 AM CST
Group       : Unspecified
Size        : 4166025
License     : ASL 2.0
Signature   : RSA/SHA256, Fri 17 Jul 2020 03:55:30 AM CST, Key ID 6c13026d12c944d0
Source RPM  : bpftrace-0.11.0-1.fc32.src.rpm
Build Date  : Fri 17 Jul 2020 03:37:32 AM CST
Build Host  : buildvm-x86-08.iad2.fedoraproject.org
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : https://github.com/iovisor/bpftrace
Bug URL     : https://bugz.fedoraproject.org/bpftrace
Summary     : High-level tracing language for Linux eBPF
Description :
BPFtrace is a high-level tracing language for Linux enhanced Berkeley Packet
Filter (eBPF) available in recent Linux kernels (4.x). BPFtrace uses LLVM as a
backend to compile scripts to BPF-bytecode and makes use of BCC for
interacting with the Linux BPF system, as well as existing Linux tracing
capabilities: kernel dynamic tracing (kprobes), user-level dynamic tracing
(uprobes), and tracepoints. The BPFtrace language is inspired by awk and C,
and predecessor tracers such as DTrace and SystemTap



md5-22ae271432855fbdaf61830fe0ceb882



#sudo dnf downgrade bpftrace
Last metadata expiration check: 0:11:20 ago on Tue 28 Jul 2020 04:48:30 PM CST.
Dependencies resolved.
==============================================================================================================================================================================================
 Package                                      Architecture                               Version                                             Repository                                  Size
==============================================================================================================================================================================================
Downgrading:
 bpftrace                                     x86_64                                     0.9.4-1.fc32                                        fedora                                     1.0 M

Transaction Summary
==============================================================================================================================================================================================
Downgrade  1 Package

Total download size: 1.0 M
Is this ok [y/N]: y
Downloading Packages:
bpftrace-0.9.4-1.fc32.x86_64.rpm                                                                                                                              258 kB/s | 1.0 MB     00:04    
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                          50 kB/s | 1.0 MB     00:21     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                      1/1 
  Downgrading      : bpftrace-0.9.4-1.fc32.x86_64                                                                                                                                         1/2 
  Cleanup          : bpftrace-0.11.0-1.fc32.x86_64                                                                                                                                        2/2 
  Running scriptlet: bpftrace-0.11.0-1.fc32.x86_64                                                                                                                                        2/2 
  Verifying        : bpftrace-0.9.4-1.fc32.x86_64                                                                                                                                         1/2 
  Verifying        : bpftrace-0.11.0-1.fc32.x86_64                                                                                                                                        2/2 

Downgraded:
  bpftrace-0.9.4-1.fc32.x86_64                                                                                                                                                                

Complete!



md5-4d5ecf79746089b18c7e3d217ee21ef6



#sudo bpftrace -e 'BEGIN { printf("Hello eBPF!\n"); }'
Attaching 1 probe...
Hello eBPF!
^C

munmap_chunk(): invalid pointer
Aborted



md5-3b03168820e321f345bfaaea35199a1e



#sudo ./offcputime.bt
libclang: crash detected during parsing: {
  'source_filename' : 'definitions.h'
  'command_line_args' : ['clang', '-isystem', '/usr/local/include', '-isystem', '/bpftrace/include', '-isystem', '/usr/include', '-nostdinc', '-isystem', '/virtual/lib/clang/include', '-I/lib/modules/5.7.9-200.fc32.x86_64+debug/source/arch/x86/include', '-I/lib/modules/5.7.9-200.fc32.x86_64+debug/build/arch/x86/include/generated', '-I/lib/modules/5.7.9-200.fc32.x86_64+debug/source/include', '-I/lib/modules/5.7.9-200.fc32.x86_64+debug/build/include', '-I/lib/modules/5.7.9-200.fc32.x86_64+debug/source/arch/x86/include/uapi', '-I/lib/modules/5.7.9-200.fc32.x86_64+debug/build/arch/x86/include/generated/uapi', '-I/lib/modules/5.7.9-200.fc32.x86_64+debug/source/include/uapi', '-I/lib/modules/5.7.9-200.fc32.x86_64+debug/build/include/generated/uapi', '-include', '/lib/modules/5.7.9-200.fc32.x86_64+debug/source/include/linux/kconfig.h', '-D__KERNEL__', '-D__BPF_TRACING__', '-D__HAVE_BUILTIN_BSWAP16__', '-D__HAVE_BUILTIN_BSWAP32__', '-D__HAVE_BUILTIN_BSWAP64__', '-DKBUILD_MODNAME='"bpftrace"'', '-include', 'clang_workarounds.h'],
  'unsaved_files' : [('definitions.h', '...', 25), ('/bpftrace/include/__stddef_max_align_t.h', '...', 1771), ('/bpftrace/include/float.h', '...', 5192), ('/bpftrace/include/limits.h', '...', 3735), ('/bpftrace/include/stdarg.h', '...', 2025), ('/bpftrace/include/stddef.h', '...', 4499), ('/bpftrace/include/stdint.h', '...', 23390), ('/bpftrace/include/clang_workarounds.h', '...', 921)],
  'options' : 1,
}
free(): double free detected in tcache 2
Aborted

forgot about the older version's error, hope the latest version 0.11.0-1.fc32 could change something to fix this bug.
Thanks!

[caringi]

Possibly the explanation (and the fix) is here:

https://github.com/iovisor/bpftrace/pull/1438

I'm going to do some tests and if that's the case, update the Fedora package (to include that patch)...

[danobi]

@caringi I'm reverting #1438 in #1441 b/c it's breaking the embedded builds. I'll try to reproduce this issue later today.

[caringi]

BTW, I can confirm that applying that patch (from #1438) fixes the issue for Fedora...

[hermithuang]

thank you all for the replies. I know the issue and expect for the update ...

[danobi]

There's a kernel patchset in development that allows userspace to directly trigger bpf progs. Long term we should look into using that feature (if the patchset lands).

This isn't the first time BEGIN/END probes have broken so it'd be great to have a more reliable way to trigger BEGIN/END probes.

[caringi]

@danobi I'm considering applying the patch from #1438 in order to fix the issue in Fedora bpftrace packages... Do you see any possible problem?

[danobi]

@caringi In theory there shouldn't be any issues but we were seeing bpftrace failing to print any output in the CI embedded builds. I haven't had time to isolate it yet, though. If things look like they work in fedora then I'd go ahead and apply it.

[danobi]

Having trouble reproducing. I built in release mode on fedora 32 and I can still see symbols:

[vagrant@localhost src]$ nm bpftrace | grep BEGIN
000000000045c9a0 T BEGIN_trigger
000000000042a310 t _GLOBAL__sub_I_BEGIN_trigger

[vagrant@localhost src]$ cat /etc/fedora-release
Fedora release 32 (Thirty Two)

@caringi can you point me to the fedora build config? Curious if something else is going on.

[caringi]

So it might be related to some compiler/linker flags set by rpm macros...

You can see a f32 bpftrace package build log here:

https://kojipkgs.fedoraproject.org//work/tasks/8773/48228773/build.log

[fbs]

Yes it's an RPM thing. It moves the symbol to a debug info package. With my centos 7 bpftrace package I ended up hitting the same issues, ended up doing this

# The post hooks strip the binary which removes
# the BEGIN_trigger and END_trigger functions
# which are needed for the BEGIN and END probes
%global __os_install_post %{nil}
%global _find_debuginfo_opts -g

[caringi]

@fbs thanks for the info.

Maybe I'll utilize the same idea for Fedora...

@hermithuang for now there is a workround for this problem:

debuginfo-install bpftrace

[hermithuang]

@caringi , yes i test the new version and it's perfect. Thank you all for fix the bug