Bcc: Exception: Failed to attach BPF program kprobe___copy_from_iter to kprobe _copy_from_iter

[aaron-ai]

I use debian 10 whose kernel version is 4.19.0-5-amd64, and I use kprobe on function _copy_from_iter, which should be copy_from_iter on 4.9. Because the symbol of copy_from_iter is not exposed on 4.19, so I use _copy_from_iter instead of it. when I use kprobe__copy_from_iter on 4.9, it works, but when I use kprobe___copy_from_iter(3 underlines) on 4.19.0-5-amd64, it will show an exception:

Exception: Failed to attach BPF program kprobe___copy_from_iter to kprobe _copy_from_iter.

here is my code:

int kprobe___copy_from_iter(struct pt_regs *ctx, void *addr, size_t bytes, struct iov_iter *i)
{
// do nothing here
    return 0;
}

[yonghong-song]

could you check for error message in dmesg? A little bit more detailed log can be found at /sys/kernel/debug/tracing/error_log.

[aaron-ai]

could you check for error message in dmesg? A little bit more detailed log can be found at /sys/kernel/debug/tracing/error_log.

Thanks, I used dmesg, it gives me the info below:

[   31.085841] trace_kprobe: Could not probe notrace function _copy_from_iter
[   31.087862] trace_kprobe: Could not probe notrace function _copy_from_iter
[   78.788978] trace_kprobe: Could not probe notrace function _copy_from_iter
[   78.790554] trace_kprobe: Could not probe notrace function _copy_from_iter
[  184.780877] trace_kprobe: Could not probe notrace function _copy_from_iter
[  184.782437] trace_kprobe: Could not probe notrace function _copy_from_iter
[35446.661824] trace_kprobe: Could not probe notrace function _copy_from_iter
[35446.663432] trace_kprobe: Could not probe notrace function _copy_from_iter
[38167.653602] trace_kprobe: Could not probe notrace function _copy_from_iter
[38167.655165] trace_kprobe: Could not probe notrace function _copy_from_iter

but I do not see the error_log file in /sys/kernel/debug/tracing/, should I do something to enable it ?

[palmtenor]

Do you have _copy_from_iter in your /proc/kallsyms?

[aaron-ai]

Do you have _copy_from_iter in your /proc/kallsyms?

yes (on 4.19.0-5-amd64).

$ cat /proc/kallsyms | grep _copy_from_iter
ffffffffaf5c6b90 T _copy_from_iter_full
ffffffffaf5c6dd0 T _copy_from_iter_full_nocache
ffffffffaf5c7020 T csum_and_copy_from_iter_full
ffffffffaf5c7bd0 T _copy_from_iter_flushcache
ffffffffaf5c8550 T _copy_from_iter
ffffffffaf5c8a10 T _copy_from_iter_nocache
ffffffffaf5c8cc0 T csum_and_copy_from_iter
ffffffffaf74e280 T dax_copy_from_iter
ffffffffb00d87c0 r __ksymtab__copy_from_iter
ffffffffb00d87c8 r __ksymtab__copy_from_iter_full
ffffffffb00d87d0 r __ksymtab__copy_from_iter_full_nocache
ffffffffb00d87d8 r __ksymtab__copy_from_iter_nocache
ffffffffb00d9d80 r __ksymtab_csum_and_copy_from_iter
ffffffffb00d9d88 r __ksymtab_csum_and_copy_from_iter_full
ffffffffb00e0ef0 r __ksymtab__copy_from_iter_flushcache
ffffffffb00e28f8 r __ksymtab_dax_copy_from_iter
ffffffffb0100f90 r __kstrtab_csum_and_copy_from_iter_full
ffffffffb0100fad r __kstrtab_csum_and_copy_from_iter
ffffffffb01010d3 r __kstrtab__copy_from_iter_full_nocache
ffffffffb01010f0 r __kstrtab__copy_from_iter_flushcache
ffffffffb010110b r __kstrtab__copy_from_iter_nocache
ffffffffb0101123 r __kstrtab__copy_from_iter_full
ffffffffb0101138 r __kstrtab__copy_from_iter
ffffffffb010defb r __kstrtab_dax_copy_from_iter
ffffffffc0526960 t dm_dax_copy_from_iter    [dm_mod]
ffffffffc052d6c0 t linear_dax_copy_from_iter    [dm_mod]
ffffffffc052dbc0 t stripe_dax_copy_from_iter    [dm_mod]

[yonghong-song]

In my environment, I have

[[email protected] /home/yhs]# trace.py '_copy_from_iter'
cannot attach kprobe, Invalid argument
Failed to attach BPF program probe__copy_from_iter_1 to kprobe _copy_from_iter
[[email protected] /home/yhs]# dmesg | tail
...
[1547427.099036] trace_kprobe: Could not probe notrace function _copy_from_iter
[[email protected] /home/yhs]#

The attachment failed as _copy_from_iter is marked as notrace by the kernel.
I am using 5.2 kernel, your old kernel may not have this issue, but may be other issues.

Do you have more detailed error messages in dmseg?

[aaron-ai]

In my environment, I have

[[email protected] /home/yhs]# trace.py '_copy_from_iter'
cannot attach kprobe, Invalid argument
Failed to attach BPF program probe__copy_from_iter_1 to kprobe _copy_from_iter
[[email protected] /home/yhs]# dmesg | tail
...
[1547427.099036] trace_kprobe: Could not probe notrace function _copy_from_iter
[[email protected] /home/yhs]#

The attachment failed as _copy_from_iter is marked as notrace by the kernel.
I am using 5.2 kernel, your old kernel may not have this issue, but may be other issues.

Do you have more detailed error messages in dmseg?

It is marked as notrace on 4.19.0-5-amd64, too. Thanks！！

[aaron-ai]

In my environment, I have

[[email protected] /home/yhs]# trace.py '_copy_from_iter'
cannot attach kprobe, Invalid argument
Failed to attach BPF program probe__copy_from_iter_1 to kprobe _copy_from_iter
[[email protected] /home/yhs]# dmesg | tail
...
[1547427.099036] trace_kprobe: Could not probe notrace function _copy_from_iter
[[email protected] /home/yhs]#

The attachment failed as _copy_from_iter is marked as notrace by the kernel.
I am using 5.2 kernel, your old kernel may not have this issue, but may be other issues.
Do you have more detailed error messages in dmseg?

It is marked as notrace on 4.19.0-5-amd64, too. Thanks！！

if I do not change the function I trace (_copy_from_iter), what should I do? enable CONFIG_KPROBE_EVENTS_ON_NOTRACE and recompile the kernel ? is this the only solution ?

[yonghong-song]

Enabling CONFIG_KPROBE_EVENTS_ON_NOTRACE probably the only solutiion, I guess.