Bcc: cannot attach kprobe in kernel 4.19

[egggHang]

HI BCC,
I got this error when run execsnoop and killsnoop in Centos 7 and Debian 9 with kernel 4.19, but everything is fine with the 4.18 one:

cannot attach kprobe, probe entry may not exist
Traceback (most recent call last):
  File "./execsnoop", line 168, in <module>
    b.attach_kprobe(event=execve_fnname, fn_name="syscall__execve")
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 606, in attach_kprobe
    (fn_name, event))
Exception: Failed to attach BPF program syscall__execve to kprobe sys_execve

I found this function called by _bpf_try_perf_event_open_with_probe_ in src/cc/libbpf.c failed, and the errer number is 2:
_syscall(__NR_perf_event_open, &attr, pid, cpu, -1 /* group_fd */, PERF_FLAG_FD_CLOEXEC);_

Any idea why this might happen?

Thank you

[yonghong-song]

The error number 2 is ENOENT.

#define ENOENT           2      /* No such file or directory */

Even if bpf_try_perf_event_open_with_probe failed, libbpf should still try to use debugfs based approach, do you know why it does not work?
Could you check whether there exists any kernel message for the error (dmesg output)?
Could you attach you config file here so in case further debug is needed?

[egggHang]

Sorry for my late reply.

1. I find when bpf_try_perf_event_open_with_probe failed as stated above, the code tries to open the file /sys/kernel/debug/tracing/kprobe_events using the function kfd = open(buf, O_WRONLY | O_APPEND, 0), which succeeds , and the _fd_ is 5. But code failed to write p:kprobes/p_sys_execve_bcc_19099 sys_execve into the same file using write(kfd, buf, strlen(buf), the errno is ENOENT, so the whole bpf_attach_kprobe returns -1. The problem is why it failed to write.

1. After i run _execsnoop_ or _killsnoop_ , the dmesg output would add this message:
   trace_kprobe: Probe definition must be started with 'p', 'r' or '-'.

2. Here is the config info

-- The C compiler identification is GNU 8.2.0
-- The CXX compiler identification is GNU 8.2.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Latest recognized Git tag is HEAD-HASH-NOTFOUND
-- Git HEAD is GITDIR-NOTFOUND
-- Revision is EAD-HASH-NOTFOUND-GITDIR-N
-- Performing Test HAVE_NO_PIE_FLAG
-- Performing Test HAVE_NO_PIE_FLAG - Success
-- Found BISON: /usr/bin/bison (found version "3.2.2")
-- Found FLEX: /usr/bin/flex (found version "2.6.4")
-- Found LLVM: /usr/lib/llvm-6.0/include 6.0.1
-- Found LibElf: /usr/lib/aarch64-linux-gnu/libelf.so
-- Performing Test ELF_GETSHDRSTRNDX
-- Performing Test ELF_GETSHDRSTRNDX - Success
-- Using static-libstdc++
-- Found LuaJIT: /usr/lib/aarch64-linux-gnu/libluajit-5.1.a;/usr/lib/aarch64-linux-gnu/libdl.so;/usr/lib/aarch64-linux-gnu/libm.so
-- Configuring done
-- Generating done

And some kernel features are enabled as below:

CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_NET_CLS_BPF=m
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
CONFIG_BPF_EVENTS=y

[yonghong-song]

On Tue, Dec 11, 2018 at 3:25 AM egggHang [e-mail] wrote:

Sorry for my late reply.

1.

I find when bpf_try_perf_event_open_with_probe failed as stated above,
the code tries to open the file /sys/kernel/debug/tracing/kprobe_events
using the function kfd = open(buf, O_WRONLY | O_APPEND, 0), which
succeeds , and the fd is 5. But code failed to write p:kprobes/p_sys_execve_bcc_19099
sys_execve into the same file using write(kfd, buf, strlen(buf), the
errno is ENOENT, so the whole bpf_attach_kprobe returns -1. The
problem is why it failed to write.

For 4.19, the function name sys_execve is wrong. The execsnoop.py or
killsnoop.py should find the syscall func name
__x64_sys_execve instead. Did you use an old bcc installation? Could you
check whether your execsnoop.py has the
following code?

b = BPF(text=bpf_text)
execve_fnname = b.get_syscall_fnname("execve")
b.attach_kprobe(event=execve_fnname, fn_name="syscall__execve")
b.attach_kretprobe(event=execve_fnname, fn_name="do_ret_sys_execve")

If the problem still exists, could you check why get_syscall_fnname() did
not return correct function name __x64_sys_execve?

>

1.
2.

After i run execsnoop or killsnoop , the dmesg output would add
this message:
trace_kprobe: Probe definition must be started with 'p', 'r' or '-'.

Not sure why. But the entry indeed started with "p"...

>

1.
2.

Here is the config info

-- The C compiler identification is GNU 8.2.0
-- The CXX compiler identification is GNU 8.2.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Latest recognized Git tag is HEAD-HASH-NOTFOUND
-- Git HEAD is GITDIR-NOTFOUND
-- Revision is EAD-HASH-NOTFOUND-GITDIR-N
-- Performing Test HAVE_NO_PIE_FLAG
-- Performing Test HAVE_NO_PIE_FLAG - Success
-- Found BISON: /usr/bin/bison (found version "3.2.2")
-- Found FLEX: /usr/bin/flex (found version "2.6.4")
-- Found LLVM: /usr/lib/llvm-6.0/include 6.0.1
-- Found LibElf: /usr/lib/aarch64-linux-gnu/libelf.so
-- Performing Test ELF_GETSHDRSTRNDX
-- Performing Test ELF_GETSHDRSTRNDX - Success
-- Using static-libstdc++
-- Found LuaJIT: /usr/lib/aarch64-linux-gnu/libluajit-5.1.a;/usr/lib/aarch64-linux-gnu/libdl.so;/usr/lib/aarch64-linux-gnu/libm.so
-- Configuring done
-- Generating done

And some kernel features are enabled as below:

CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_NET_CLS_BPF=m
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
CONFIG_BPF_EVENTS=y

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/iovisor/bcc/issues/2071#issuecomment-446168955, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ALq6ol_Ez8H0KF10K54lP7v85qTnDnxbks5u35YrgaJpZM4ZKk13
.

[egggHang]

For 4.19, the function name sys_execve is wrong. The execsnoop.py or
killsnoop.py should find the syscall func name
__x64_sys_execve instead. Did you use an old bcc installation? Could you
check whether your execsnoop.py has the
following code?

b = BPF(text=bpf_text)
execve_fnname = b.get_syscall_fnname("execve")
b.attach_kprobe(event=execve_fnname, fn_name="syscall__execve")
b.attach_kretprobe(event=execve_fnname, fn_name="do_ret_sys_execve")

The version of bcc I am using is 0.7.0 released on 5 Sep, which does have the code above. And I run bcc on an aarch64 device.

In fact, in CentOS/Debian with 4.18 kernel, the execsnoop.py or killsnoop.py also find the syscall func name sys_execve, and everything goes well, does it use __x64_sys_execve from kernel 4.19?

BTW, even if I fix the variable execve_fnname to __x64_sys_execve, the program ends up the same error in kernel 4.19, am I doing the right way?

[yonghong-song]

In fact, in CentOS/Debian with 4.18 kernel, the execsnoop.py or killsnoop.py also find the syscall func name sys_execve, and everything goes well, does it use __x64_sys_execve from kernel 4.19?

Right in 4.18, it should be sys_execve and in 4.19, it will be __x64_sys_execve.

BTW, even if I fix the variable execve_fnname to __x64_sys_execve, the program ends up the same error in kernel 4.19, am I doing the right way?

I did not spot any issues. May need to reproduce first to see what is the reason.
Do you have issues with other kprobe functions other than sys_*?

[egggHang]

Do you have issues with other kprobe functions other than sys_*?

Other kprobe functions like mark_buffer_dirty (cachestat)、do_sys_open（opensnoop）、blk_account_io_start(biosnoop) work well, but functions like sys_sync 、sys_execve、 sys_kill fails.

And I am using an Arm64 device , will it matter?

[egggHang]

Ohh, the right kprobe function name is __arm64_sys_*, which can be checked in /boot/System.map-***. after I modify this, the tool works well.
And I wonder why the function get_syscall_fnname get the wrong one.

[palmtenor]

Because the prefix is hard-coded https://github.com/iovisor/bcc/blob/master/src/python/bcc/__init__.py#L170

[egggHang]

Because the prefix is hard-coded https://github.com/iovisor/bcc/blob/master/src/python/bcc/__init__.py#L170

That's it. Adding an arm64 prefix like b"__arm64_sys_" in _syscall_prefixes in __init__.py solved this.

Thank you for all your help ~ @yonghong-song @palmtenor

[yonghong-song]

@egggHang Do you mind to submit a pull request to fix the issue on amd64 since you have a propre system to test it? Thanks!

[egggHang]

@yonghong-song I‘d like to, but need some try to do this. Just not familiar how to do it.

[yonghong-song]

The link below has some information about how to create a pull request
https://help.github.com/articles/creating-a-pull-request/
Maybe you can follow the guide there?

[egggHang]

2081

I have submitted the pull request, and I think this issue should be closed