Bpftrace: uprobe char pointer arguments, output abnormal

[b-ripper]

bpftrace --info

System
OS: Linux 4.19.110-perf-gef1b69c #1 SMP PREEMPT Thu Aug 20 17:08:35 CST 2020
Arch: aarch64

Build
version: v0.11.0-646fffc6
LLVM: 8
foreach_sym: yes
unsafe uprobe: no
bfd: no
bpf_attach_kfunc: no
bcc_usdt_addsem: no
libbpf: no
libbpf btf dump: no
libbpf btf dump type decl: no

Kernel helpers
probe_read: yes
probe_read_str: yes
probe_read_user: yes
probe_read_user_str: yes
probe_read_kernel: yes
probe_read_kernel_str: yes
get_current_cgroup_id: yes
send_signal: yes
override_return: yes

Kernel features
Instruction limit: -1
Loop support: no
btf (depends on Build:libbpf): no

Map types
hash: yes
percpu hash: yes
array: yes
percpu array: yes
stack_trace: yes
perf_event_array: yes

Probe types
kprobe: no
tracepoint: yes
perf_event: yes
kfunc: no

What reproduces the bug?

hello.cpp:

#include <stdio.h>

__attribute__ ((visibility ("default"))) int CallFunc(const char* pStr)
{
    printf("caller:%s\n", pStr);
    return 0;
}

hello.h:

#ifndef __HELLO_H__
#define __HELLO_H__

int CallFunc(const char* pStr);
#endif

test.cpp:

#include <stdio.h>
#include "hello.h"

int main()
{
    CallFunc("Session");    
    return 0;
}

root@localhost:/# bpftrace -e 'u:/libhello.so:"CallFunc(char const*)" { printf("arg0:%s\n", str(arg0));}'
Attaching 1 probe...
arg0:
^C

root@localhost:/# ./a.out 
caller:Session

arg0 output abnormal.

[fbs]

This looks similar to https://github.com/iovisor/bpftrace/issues/1494#issuecomment-684699992

[mmisono]

Maybe #614? since the arch is aarch64. We need to change the way to read user memory on some architectures.

[b-ripper]

structure pointer arguments also has the same problem, could not get correctly.

[danobi]

Can you show output with -kk flag on?

[b-ripper]

@danobi I found that there is no output, exec " ./a.out" will cause device hang dead, while doing:
bpftrace -kk -e 'u:/libhello.so:"CallFunc(char const*)" { printf("arg0:%s\n", str(arg0));}'

[b-ripper]

on my desktop, it works fine:
System
OS: Linux 5.4.0-42-generic #46~18.04.1-Ubuntu SMP Fri Jul 10 07:21:24 UTC 2020
Arch: x86_64

Build
version: v0.11.0-121-ga8919
LLVM: 6
foreach_sym: yes
unsafe uprobe: no
bfd: yes
bpf_attach_kfunc: yes
bcc_usdt_addsem: no
libbpf: no
libbpf btf dump: no
libbpf btf dump type decl: no

Kernel helpers
probe_read: yes
probe_read_str: yes
probe_read_user: no
probe_read_user_str: no
probe_read_kernel: no
probe_read_kernel_str: no
get_current_cgroup_id: yes
send_signal: yes
override_return: yes
get_boot_ns: no

Kernel features
Instruction limit: 1000000
Loop support: yes
btf (depends on Build:libbpf): no

Map types
hash: yes
percpu hash: yes
array: yes
percpu array: yes
stack_trace: yes
perf_event_array: yes

Probe types
kprobe: yes
tracepoint: yes
perf_event: yes
kfunc: no

[fbs]

Would it be possible for you to build from master and try with:

bpftrace -e 'u:/libhello.so:"CallFunc(char const*)" { printf("arg0:%s\n", str(uptr((uint8*)arg0)));}'

[b-ripper]

@fbs still not work on aarch64

[b-ripper]

@danobi I test on the other device:
bpftrace --info:

System
OS: Linux 5.4.57-qgki-consolidate-g7946ab2 #1 SMP PREEMPT Mon Sep 28 14:53:31 CST 2020
Arch: aarch64

Build
version: v0.11.1-e52519
LLVM: 8
foreach_sym: yes
unsafe uprobe: no
bfd: no
bpf_attach_kfunc: no
bcc_usdt_addsem: no
libbpf: no
libbpf btf dump: no
libbpf btf dump type decl: no

Kernel helpers
probe_read: yes
probe_read_str: yes
probe_read_user: no
probe_read_user_str: no
probe_read_kernel: no
probe_read_kernel_str: no
get_current_cgroup_id: yes
send_signal: yes
override_return: yes
get_boot_ns: no

Kernel features
Instruction limit: 1000000
Loop support: yes
btf (depends on Build:libbpf): no
map batch (depends on Build:libbpf): no

Map types
hash: yes
percpu hash: yes
array: yes
percpu array: yes
stack_trace: yes
perf_event_array: yes

Probe types
kprobe: yes
tracepoint: yes
perf_event: yes
kfunc: no

add -kk flag show the error:

bpftrace -kk -e 'u:/system/bin/uprobe_tester:"CallFunc(char const)" { printf("arg0:%s\n", str(uptr((uint8)arg0)));}'
Attaching 1 probe...
stdin:1:75-98: WARNING: Failed to probe_read_str: Bad address (-14)
u:/system/bin/uprobe_tester:"CallFunc(char const)" { printf("arg0:%s\n", str(uptr((uint8)arg0)));}
~~~~~~~
arg0:
^C

[fbs]

Did you see the last part of https://github.com/iovisor/bpftrace/issues/1494#issuecomment-684699992 ? Does it still happen with:

int main()
{
    CallFunc("Session"); 
    CallFunc("Session");    
    CallFunc("Session");       
    return 0;
}

[b-ripper]

@michalgr @dalehamel I was wondering if you could help to confirm that if this issue exist on your android(aarch64) device env? very thanks!

[b-ripper]

uprobe_tester： https://drive.google.com/file/d/1sOXPj57NhUKfrUtwhNlw_-wDj1bxYPUX/view?usp=sharing

[b-ripper]

Maybe #614? since the arch is aarch64. We need to change the way to read user memory on some architectures.

I'm not sure this is my local issue, or the kernel I use miss some functionality patches from upstream kernel source code, or the bpf reference functionality I built not adapted correctly.

@mmisono could you show me where is the reference source code of reading user memory on some architectures(like x86_64 or aarch64)? I want to have a check. very thanks!

[mmisono]

@b-ripper
I tried your program and I found that the problem also occurs on x86-64.

% sudo ./src/bpftrace -e 'u:./libhello.so:"CallFunc(char const*)" { printf("arg0:%s\n", str(arg0));}' -kk
Attaching 1 probe...
stdin:1:63-72: WARNING: Failed to probe_read_user_str: Bad address (-14)
u:./libhello.so:"CallFunc(char const*)" { printf("arg0:%s\n", str(arg0));}
                                                              ~~~~~~~~~
arg0:

"Bad address (-14)" means the address is wrong or a page fault occurs in most cases. It seems that the const string "Session" is not on the page. If I rewrote the program as

#include <stdio.h>
#include "hello.h"

int main()
{
    printf("%s\n", "Session"); // access the const string
    CallFunc("Session");    
    return 0;
}

or

#include <stdio.h>
#include "hello.h"

int main()
{
    const s[] = "Session"; // place const string in the stack
    CallFunc(&s[0]);    
    return 0;
}

then, bpftrace worked properly.

% sudo ./src/bpftrace -e 'u:./libhello.so:"CallFunc(char const*)" { printf("arg0:%s\n", str(arg0));}' -kk
Attaching 1 probe...
arg0:Session

[mmisono]

Ah, fbs mentioned the same things. This is a mystifying problem but I believe there is nothing that bpftrace can help.

Did you see the last part of #1494 (comment) ? Does it still happen with:

int main()
{
    CallFunc("Session"); 
    CallFunc("Session");    
    CallFunc("Session");       
    return 0;
}

[b-ripper]

@mmisono thanks for your reply!
I also test on x86_64 as I mensioned above earlier, it works fine on x86_64 desktop, but always fails on aarch64.
I don't know it's my local kernel problem(I used 4.19 and 5.4, both failed) or the issue of upstream kernel need to fix.
Do you have any suggestion for further check? file email to [email protected] mailing list for help? very thanks!

[mmisono]

There was another problem related to aarch64, but recently the fix was merged (#1504). Do you try the master version of bpftrace & my suggestion on aarch64? Do you still get the problem? If the problem still exists, I'm not sure where is the best place looking for help...

[mmisono]

@b-ripper
Ah, you need bpf_probe_read_user() helper function, and apparently, it's available from Linux 5.5 (https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md#helpers)

Probably bpftrace should emit a warning when trying to read user memory on architecture that requires bpf_probe_read_user() but that doesn't have it.

[b-ripper]

@mmisono yes, I tried the lastest master of bpftrace and the suggestion from fbs and your suggestion for test program. It still fails.
On my x86_64 (linux 5.4.0), it works fine. so I think it maybe related to the way difference of architectures.

@michalgr @dalehamel could you help to confirm that if this issue exist on your android(aarch64) device env? using the aarch64 built uprobe_tester： https://drive.google.com/file/d/1sOXPj57NhUKfrUtwhNlw_-wDj1bxYPUX/view?usp=sharing

very thanks

[b-ripper]

test.cpp in uprobe_tester:
int main() { char * msg = "Session"; printf("%s\n", msg); CallFunc("Session"); CallFunc("Session"); CallFunc("Session"); return 0; }

[mmisono]

@b-ripper
On x86_64, we don't need to use bpf_probe_read_user(), and we can simply use bpf_probe_read(). bpf_probe_read_user() is for architecture that have different address spaces for userspace and kernel. I don't know on which architecture we must use the function, but apparently, aarch64 and s390x are.

[b-ripper]

I applied the bpf_probe_read_user() related patches from the lastest upstream linux to my local 4.19 kernel, using the lastest master bpftrace, now it works.
thank you very much! @mmisono