Bpftrace: Sporadic crashes during header parsing (static bpftrace, alpine linux linking musl)

[dalehamel]

There seems to be a bug in header parsing.

Here is the debug output during the crash:

#include <linux/tcp.h>
#include <net/sock.h>

Program
 BEGIN
  call: printf
   string: Tracing udp connections. Hit Ctrl-C to end.\n
  call: printf
   string: %-8s %-16s %-8s %-20s %-20s\n
   string: TIME
   string: COMM
   string: PID
   string: SRC:PORT
   string: DEST:PORT
 kprobe:tcp_connect
  =
   variable: $sk
   (sock*)
    builtin: arg0
  call: printf
   string: %llx\n
   .
    .
     dereference
      variable: $sk
     __sk_common
    skc_daddr

libclang: crash detected during parsing: {
  'source_filename' : 'definitions.h'
  'command_line_args' : ['clang', '-I', '/bpftrace/include', '-nostdinc', '-isystem', '/virtual/lib/clang/include', '-I/lib/modules/4.14.56+/build/arch/x86/include', '-I/lib/modules/4.14.56+/build/arch/x86/include/generated/uapi', '-I/lib/modules/4.14.56+/build/arch/x86/include/generated', '-I/lib/modules/4.14.56+/build/include', '-I/lib/modules/4.14.56+/build/./arch/x86/include/uapi', '-I/lib/modules/4.14.56+/build/arch/x86/include/generated/uapi', '-I/lib/modules/4.14.56+/build/include/uapi', '-I/lib/modules/4.14.56+/build/include/generated', '-I/lib/modules/4.14.56+/build/include/generated/uapi', '-I./arch/x86/include', '-Iarch/x86/include/generated/uapi', '-Iarch/x86/include/generated', '-Iinclude', '-I./arch/x86/include/uapi', '-Iarch/x86/include/generated/uapi', '-I./include/uapi', '-Iinclude/generated/uapi', '-include', './include/linux/kconfig.h', '-D__KERNEL__', '-D__HAVE_BUILTIN_BSWAP16__', '-D__HAVE_BUILTIN_BSWAP32__', '-D__HAVE_BUILTIN_BSWAP64__', '-Wno-unused-value', '-Wno-pointer-sign', '-fno-stack-protector'],
  'unsaved_files' : [('definitions.h', '...', 45), ('/bpftrace/include/__stddef_max_align_t.h', '...', 1771), ('/bpftrace/include/float.h', '...', 5192), ('/bpftrace/include/limits.h', '...', 3735), ('/bpftrace/include/stdarg.h', '...', 2025), ('/bpftrace/include/stddef.h', '...', 4499), ('/bpftrace/include/stdint.h', '...', 23388)],
  'options' : 0,
}
Clang error while parsing C definitions: 2
Input (45): #include <linux/tcp.h>
#include <net/sock.h>

Unknown struct/union: 'sock'

Note that the code continues, to execute, but it lacks the definition of the struct sock since it never seems to be parsed (so the program doesn't crash per-se, it catches the error and continues only to fail execution later).

Here is the backtrace from gdb:

#0  0x00007ffff585337b in clang::CIndexer::getClangResourcesPath[abi:cxx11]() ()
#1  0x00007ffff5850fcd in clang_parseTranslationUnit_Impl(void*, char const*, char const* const*, int, llvm::ArrayRef<CXUnsavedFile>, unsigned int, CXTranslationUnitImpl**) ()
#2  0x00007ffff58513b9 in void llvm::function_ref<void ()>::callback_fn<clang_parseTranslationUnit2FullArgv::{lambda()#1}>(long) ()
#3  0x00007ffff6c12e5c in llvm::CrashRecoveryContext::RunSafely(llvm::function_ref<void ()>) ()
#4  0x00007ffff6c12e7b in RunSafelyOnThread_Dispatch(void*) ()
#5  0x00007ffff6c4ed9a in ExecuteOnThread_Dispatch(void*) ()
#6  0x00007ffff6cc24f5 in start (p=0x7ffff5256ae8) at src/thread/pthread_create.c:150
#7  0x00007ffff6cc3709 in __clone () at src/thread/x86_64/clone.s:21

And here is the script that crashes it:

#include <linux/tcp.h>
#include <net/sock.h>

BEGIN
{
  printf("Tracing udp connections. Hit Ctrl-C to end.\n");
  printf("%-8s %-16s %-8s %-20s %-20s\n", "TIME", "COMM", "PID", "SRC:PORT", "DEST:PORT" )
}

kprobe:tcp_connect
{
  $sk = ((sock *) arg0);

  printf("%llx\n", $sk->__sk_common.skc_daddr);
}

When I run this script, I always receive a crash.

However, if I make minor, random changes to the script (such as inserting a printf, a useless variable assignment), it is possible to get the script working again. For instance, modifying it to add a printf("foo\n"); is enough to get this script working:

#include <linux/tcp.h>
#include <net/sock.h>

BEGIN
{
  printf("Tracing udp connections. Hit Ctrl-C to end.\n");
  printf("%-8s %-16s %-8s %-20s %-20s\n", "TIME", "COMM", "PID", "SRC:PORT", "DEST:PORT" )
}

kprobe:tcp_connect
{
  $sk = ((sock *) arg0);

  printf("foo\n");
  printf("%llx\n", $sk->__sk_common.skc_daddr);
}

This is the most minimal script I've been able to produce in a controlled way, but in earlier debugging attempts I've been able to get it to crash with pretty much just the #include statements at the top. If I can get a more minimal script to crash, I'll add that.

I've had this happen quite a lot while writing various scripts, and I've always been able to resolve it with similar no-op changes such as above.

The crash seems to be coming from here: https://github.com/iovisor/bpftrace/blob/master/src/clang_parser.cpp#L228-L234

Breakpoint 1, bpftrace::ClangParser::parse (this=0x7fffffffdbfa,
    program=0x7ffff80369e0, structs=...) at /app/src/clang_parser.cpp:228
228     in /app/src/clang_parser.cpp
(gdb) print unsaved_files
$1 = {{Filename = 0x7ffff6ccce28 "definitions.h",
    Contents = 0x7ffff8035840 "#include <linux/tcp.h>\n#include <net/sock.h>\n",
    Length = 45}, {
    Filename = 0x7ffff6ccce38 "/bpftrace/include/__stddef_max_align_t.h",
    Contents = 0x7ffff6cd7820 <__stddef_max_align_t_h> "/*===---- __stddef_max_align_t.h - Definition of max_align_t for modules ---===\n *\n * Copyright (c) 2014 Chandler Carruth\n *\n * Permission is hereby granted, free of charge, to any person obtaining a "...,
    Length = 1771}, {Filename = 0x7ffff6ccce61 "/bpftrace/include/float.h",
    Contents = 0x7ffff6cd7f20 <float_h> "/*===---- float.h - Characteristics of floating point types ", '-' <repeats 16 times>, "===\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy\n * of this software and associated do"...,
    Length = 5192}, {Filename = 0x7ffff6ccce7b "/bpftrace/include/limits.h",
    Contents = 0x7ffff6cd9380 <limits_h> "/*===---- limits.h - Standard header for integer sizes ", '-' <repeats 20 times>, "===*\\\n *\n * Copyright (c) 2009 Chris Lattner\n *\n * Permission is hereby granted, free of charge, to any person obtaining a co"...,
    Length = 3735}, {Filename = 0x7ffff6ccce96 "/bpftrace/include/stdarg.h",
    Contents = 0x7ffff6cda220 <stdarg_h> "/*===---- stdarg.h - Variable argument handling ", '-' <repeats 28 times>, "===\n *\n * Copyright (c) 2008 Eli Friedman\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"...,
    Length = 2025}, {Filename = 0x7ffff6ccceb1 "/bpftrace/include/stddef.h",
    Contents = 0x7ffff6cdaa20 <stddef_h> "/*===---- stddef.h - Basic type definitions ", '-' <repeats 32 times>, "===\n *\n * Copyright (c) 2008 Eli Friedman\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"..., Length = 4499}, {
    Filename = 0x7ffff6cccecc "/bpftrace/include/stdint.h",
    Contents = 0x7ffff6cdbbc0 <stdint_h> "/*===---- stdint.h - Standard header for sized integer types ", '-' <repeats 14 times>, "===*\\\n *\n * Copyright (c) 2009 Chris Lattner\n *\n * Permission is hereby granted, free of charge, to any person obtaining a co"..., Length = 23388}}

[dalehamel]

Interestingly, input appears to be the same in both the crashing and non-crashing version:

Crash:

$2 = {static npos = 18446744073709551615,
  _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>},
    _M_p = 0x7ffff8035840 "#include <linux/tcp.h>\n#include <net/sock.h>\n"},
  _M_string_length = 45, {
    _M_local_buf = "-\000\000\000\000\000\000\000?\000\000\000\000\000\000",
    _M_allocated_capacity = 45}}

No crash:

$1 = {static npos = 18446744073709551615,
  _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>},
    _M_p = 0x7ffff8035840 "#include <linux/tcp.h>\n#include <net/sock.h>\n"},
  _M_string_length = 45, {
    _M_local_buf = "-\000\000\000\000\000\000\000?\000\000\000\000\000\000",
    _M_allocated_capacity = 45}}

[dalehamel]

Single-stepping into clang::CIndexer::getClangResourcesPath, It looks like the crash is happening at https://github.com/llvm-mirror/clang/blob/master/tools/libclang/CIndexer.cpp#L74 when trying to access info.dli_fname

Here is the crashing version:

(gdb) s
Single stepping until exit from function _ZN5clang8CIndexer21getClangResourcesPathB5cxx11Ev,
which has no line number information.
stub_dladdr (addr=0x7ffff5848b6a <clang_createTranslationUnit>, info=0x7ffff52567a0)
    at src/ldso/dladdr.c:8
8       src/ldso/dladdr.c: No such file or directory.
(gdb) print info
$3 = (Dl_info *) 0x7ffff52567a0
(gdb) print info.dli_fname
$4 = 0x1500 <error: Cannot access memory at address 0x1500>

And here is the non-crashing version:

Thread 2 "bpftrace" hit Breakpoint 1, 0x00007ffff58532dc in clang::CIndexer::getClangResourcesPath[abi:cxx11]() ()
(gdb) s
Single stepping until exit from function _ZN5clang8CIndexer21getClangResourcesPathB5cxx11Ev,
which has no line number information.
stub_dladdr (addr=0x7ffff5848b6a <clang_createTranslationUnit>, info=0x7ffff52567a0)
    at src/ldso/dladdr.c:8
8       src/ldso/dladdr.c: No such file or directory.
(gdb) print info.dli_fname
$1 = 0x7ffff7ffd4bc <mal+252> ""

[ajor]

I've been this before, but think it's been fixed in later clang versions. Are you using Clang 5 by any chance?

Unfortunately we can't use a later version of Clang/LLVM in the Docker build until it's supported in Alpine. Other Linux distros don't package clang as a static library.
https://pkgs.alpinelinux.org/packages?name=llvm&branch=edge&arch=x86_64

That's not to say there isn't anything we can do about this on the bpftrace side though - no one's looked into that yet.

[dalehamel]

If i were to hazard a guess here, I'd say that the crash is related to musl's implementation of dladdr https://github.com/bpowers/musl/blob/master/src/ldso/dladdr.c

It seems to always return zero... so I'm not sure how this sometimes succeeds and sometimes fails.

Perhaps the memory offset (of something in the bpftrace process image) is affected by the amount of memory allocated to the script being parsed, and the value when examining info.dli_fname will either crash (if that memory address is out of bounds), or not crash (if it happens to be able to access the memory).

I built a dyamically linked bpftrace using ubuntu (off of master), and it is able to execute this test program just fine.

The same revision (c49b333c034a6d29a7ce90f565e27da1061af971), but built statically using apline fails to run this test program. If I modify it to include a no-op printf, it is able to run.

I think that this more-or-less proves that the issue is linking musl vs glibc.

[dalehamel]

Are you using Clang 5 by any chance?

Yes, both ubuntu and alpine it seems to be using libclang. For ubuntu, I followed the instructions precisely.

I've been this before

Ok, somewhat glad it's not just me as that should mean it's pretty easily reproduced. I was also a bit worried it might be related to my patch, but now I don't think so.

but think it's been fixed in later clang versions.

If my theory is correct, the related code here in clang hasn't been touched in between 6-9 years, but if I could figure out how alpine is building the current binary, perhaps building a package for a newer version would work.

I could also try to get a static build working using, say, Gentoo, which I believe supports static clang in the main ebuilds. Edit: Nope https://wiki.gentoo.org/wiki/Project:LLVM/Shared_libraries

That's not to say there isn't anything we can do about this on the bpftrace side though - no one's looked into that yet.

Yeah i'm not sure, there might be if we can figure out a way to bypass this problematic call. Perhaps something that could be done is to override the definition of dladdr to ensure that the struct member that it is failing to dereference at least gets assigned to something that won't blow up. I'm not sure that it actually needs the result of this call, it's just what is causing it to crash.

[ajor]

Yeah, looks like you're right about it not being related to the Clang version. It's also only the static CI build that shows these errors, not the dynamically linked LLVM5 one.

[dalehamel]

I'd need to compare the cmake flags vs another distro, but nothing jumps out as me as particularly special about how alpine is building llvm https://git.alpinelinux.org/cgit/aports/tree/main/llvm5/APKBUILD#n79

Their patches are also pretty mundane, I don't think any of these add any "secret sauce" to a static clang. The only thing that jumps out is https://git.alpinelinux.org/cgit/aports/tree/main/llvm5/APKBUILD#n176, which seems to just change the paths the existing static libraries are placed in. It looks like alpine is just packaging the individual static libs, not a shared static lib for llvm. If this is the case, then maybe all the ubuntu (or any other glibc distro) build needs in order to build statically is to have the static libs be in the correct path.

I'll see about getting a static build working with another distro, as that would be an acceptable solution to me here - just simply not using alpine for my static build target.

[joestringer]

I'm seeing something like this while running tcpconnect-tp on a bpf-next kernel (calls itself v5.6.0-rc2):

# bpftrace ~/bpf-perf-tools-book/originals/Ch10_Networking/tcpconnect-tp.bt
libclang: crash detected during parsing: {
  'source_filename' : 'definitions.h'
  'command_line_args' : ['clang', '-isystem', '/usr/local/include', '-isystem', '/bpftrace/include', '-isystem', '/usr/include', '-nostdinc', '-isystem', '/virtual/lib/clang/include', '-I/lib/modules/5.6.0-rc2-custom/build/arch/x86/include', '-I/lib/modules/5.6.0-rc2-custom/build/arch/x86/include/generated', '-I/lib/modules/5.6.0-rc2-custom/build/include', '-I/lib/modules/5.6.0-rc2-custom/build/include', '-I/lib/modules/5.6.0-rc2-custom/build/arch/x86/include/uapi', '-I/lib/modules/5.6.0-rc2-custom/build/arch/x86/include/generated/uapi', '-I/lib/modules/5.6.0-rc2-custom/build/include/uapi', '-I/lib/modules/5.6.0-rc2-custom/build/include/generated/uapi'
, '-include', '/lib/modules/5.6.0-rc2-custom/build/include/linux/kconfig.h', '-D__KERNEL__', '-D__BPF_TRACING__', '-D__HAVE_BUILTIN_BSWAP16__', '-D__HAVE_BUILTIN_BSWAP32__',
'-D__HAVE_BUILTIN_BSWAP64__', '-DKBUILD_MODNAME='"bpftrace"'', '-include', 'clang_workarounds.h'],
  'unsaved_files' : [('definitions.h', '...', 439), ('/bpftrace/include/__stddef_max_align_t.h', '...', 1771), ('/bpftrace/include/float.h', '...', 5192), ('/bpftrace/include/limits.h', '...', 3735), ('/bpftrace/include/stdarg.h', '...', 2025), ('/bpftrace/include/stddef.h', '...', 4499), ('/bpftrace/include/stdint.h', '...', 23390), ('/bpftrace/include/clang_workarounds.h', '...', 921)],
  'options' : 1,
}

This is with a bpftrace built statically using libc from the tree around 0.9.4:

$ bpftrace --version
bpftrace v0.9.4-dirty

[dalehamel]

@joestringer is your bpftrace build alpine using musl or is it using glibc? Did you build it yourself or is it from a distro?

Is it a fully static binary? What is the output of ldd?

[joestringer]

@dalehamel glibc, I couldn't get the musl one to work for me. I did build it myself, because at the time v0.9.4 had no binaries available and I need a newer version to run against newer kernels (older releases just broke in weird and wonderful ways with something 5.x-ish)

$ ldd $(which bpftrace)
        statically linked

[dalehamel]

@joestringer this is expected, and the reason why the static+glibc builds exist.

The Believe issue is that libdl cannot be used statically, and this issue will always be a problem for fully static builds so long as we use clang for header parsing, which under the hood is calling libdl from somewhere

[dalehamel]

@joestringer if you go to the actions tab you should be able to download a recent build like I’m talking about. The glibc2.23 linkage is the most portable

[joestringer]

Thanks @dalehamel , this fixes that issue for me (though the script seems broken but I can follow up on that elsewhere...)