bpftrace and bcc-tools seg fault on Centos 7.6 with updated 5.1 kernel

[gbansal2]

Hi,
I am new to bpftrace, so any help will be appreciated.

I am trying to install bpftrace on Centos 7.6. Updated the kernel to 5.1.12 (latest ml) as the default kernel is much older. Installed clang-8.0.0 and llvm-8.0.0

bpftrace commands segfault instantly.
./src/bpftrace -e 'kprobe:do_nanosleep { printf("sleep by %s\n", comm); }'
Error creating perf event map (-1)
Attaching 1 probe...
Segmentation fault (core dumped)

Attaching to gdb gives the following:
Program received signal SIGSEGV, Segmentation fault.
0x00007fffef730901 in __strlen_sse2_pminub () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install elfutils-libelf-0.172-2.el7.x86_64 glibc-2.17-260.el7.x86_64 zlib-1.2.7-18.el7.x86_64
(gdb) bt

0 0x00007fffef730901 in __strlen_sse2_pminub () from /lib64/libc.so.6

1 0x00007ffff3fbf28a in __bpf_object__open () from /usr/local/lib64/libbcc.so.0

2 0x00007ffff3fc3e02 in bpf_prog_load_xattr () from /usr/local/lib64/libbcc.so.0

3 0x00007ffff3fc3d7d in bpf_prog_load () from /usr/local/lib64/libbcc.so.0

4 0x0000000000696559 in bpftrace::AttachedProbe::load_prog() ()

5 0x0000000000697022 in bpftrace::AttachedProbe::AttachedProbe(bpftrace::Probe&, std::tuple) ()

6 0x00000000006b4904 in bpftrace::BPFtrace::attach_probe(bpftrace::Probe&, bpftrace::BpfOrc const&) ()

7 0x00000000006b5d44 in bpftrace::BPFtrace::run(std::unique_ptr >) ()

8 0x000000000064462b in main ()

Further, all bcc tools also segfault. I am guessing these are related.

Thanks,
Gaurav.

[danobi]

Hmm, that's interesting. Do you mind filing an issue with bcc? They're probably better equipped to debug this.

[bbn-bert]

Recently built bpftrace on centos7 with elrepo 5.1.9 kernel. No issues here. I built llvm/clang from source using the https://github.com/llvm/llvm-project repo. This required a newer gcc and cmake as well to build bpftrace.

~$ bpftrace --version
bpftrace v0.9-232-g60e6
~$ ls -la /usr/local/lib/libLLVM-9
libLLVM-9.0.0svn.so  libLLVM-9svn.so      
~$ uname -r
5.1.9-1.el7.elrepo.x86_64
~$ gcc --version
gcc (GCC) 9.1.0

[gbansal2]

Thanks @bbn-zbertils and @danobi for your replies.

I followed the same steps as @bbn-zbertils
@bbn-zbertils - Can you please share the cmake configure command line you used for building llvm/clang from source? Perhaps I am missing something. I am also using gcc 9.1.0. My kernel is a little newer though - 5.1.12, but I don't think that's the issue.

I will also file the issue on bcc repo.

[bbn-bert]

llvm/clang with specific commit I used

git clone https://github.com/llvm/llvm-project.git
git checkout 591c793b48e487382c30e352c0c3b7ae60b45a87
mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGET_ARCH=X86 -DLLVM_TARGETS_TO_BUILD='X86;BPF' -DLLVM_BUILD_LLVM_DYLIB=ON -DLLVM_ENABLE_RTTI=ON -DLLVM_ENABLE_PROJECTS=clang -G "Unix Makefiles" ../llvm
make && make clang
sudo make install

And gcc needs the newer libstdc++ libraries installed too, this is the install-target-libstdc++-v3.

wget https://ftp.gnu.org/gnu/gcc/gcc-9.1.0/gcc-9.1.0.tar.gz
tar -xf gcc-9.1.0.tar.gz
cd gcc-9.1.0
./configure --with-system-zlib --disable-multilib --enable-languages=c,c++
make -j4
sudo make install
sudo make install-target-libstdc++-v3

You may get library path errors running bpftrace since the llvm/clang libraries get installed to /usr/local/lib instead of /usr/lib, either symlink the libraries to /usr/lib or setup your sudo/root LD_LIBRARY_PATH variable to point to /usr/local/lib.

You should also make sure the kernel was built with BPF, kprobe, tracing, etc. options on

[gbansal2]

Thanks much @bbn-zbertils. Appreciate your help. These instructions work for me!