Bcc: bpf_probe_read not work

[woodliu]

My Env is :Linux kernal 5.7.7, centos 7.6 operation system
I run this program , and i got this error :
HINT: The invalid mem access 'inv' error can happen if you try to dereference memory without first using bpf_probe_read_kernel() to copy it to the BPF stack. Sometimes the bpf_probe_read_kernel() is automatic by the bcc rewriter, other times you'll need to be explicit..
And when i change the bpf_probe_read to bpf_probe_read_kernel it works.
What is the difference between bpf_probe_read_kernel and bpf_probe_read, i am confused.
I noticed this issue 2986, it seems be reverted back to bpf_probe_read. But when i should choice which one to use?
Thank you!

[yonghong-song]

Which architecture you are using? x390? On x64, it should not happen as bpf_probe_read should work just fine although it may have a kernel warning if you try to read user memories.

I rolled back due to some user space applications used bpf_probe_read on x64 to read user memories. Changing to bpf_probe_read_kernel will suddenly break their application.

I will prepare a patch to use bpf_probe_read_kernel for s390, so tools will continue work on latest kernel. Once user space applications are all deprecating bpf_probe_read, we can move to bpf_probe_read_kernel for implicitly generated probe read again.

[woodliu]

user space applications used bpf_probe_read on x64 to read user memories. Changing to bpf_probe_read_kernel will suddenly break their application.

I will prepare a patch to use bpf_probe_read_kernel for s390, so tools will c

This is my architecture：

# uname -a
Linux iZuf68h2giq2s2go9sodhhZ 5.7.7-1.el7.elrepo.x86_64 #1 SMP Wed Jul 1 11:53:16 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux

I use Alibaba cloud ECS to learn the BCC.

[yonghong-song]

Okay x64. unexpected. The code is clearly trying to access kernel memory.
Could you check whether your config file has CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE? Could you post complete failure for the error with

 HINT: The invalid mem access 'inv' error can happen if you try to dereference memory without first using bpf_probe_read_kernel() to copy it to the BPF stack. Sometimes the bpf_probe_read_kernel() is automatic by the bcc rewriter, other times you'll need to be explicit..

I want to see exact what caused the failure so I can have a proper fix. Thanks!

[woodliu]

@yonghong-song This is the full stack

# ./traceicmpsoftirq.py
bpf: Failed to load program: Permission denied
0: (bf) r6 = r1
1: (79) r8 = *(u64 *)(r6 +112)
2: (85) call bpf_get_current_pid_tgid#14
3: (bf) r7 = r0
4: (b7) r1 = 0
5: (7b) *(u64 *)(r10 -8) = r1
last_idx 5 first_idx 0
regs=2 stack=0 before 4: (b7) r1 = 0
6: (7b) *(u64 *)(r10 -16) = r1
7: (7b) *(u64 *)(r10 -24) = r1
8: (85) call bpf_ktime_get_ns#5
9: (7b) *(u64 *)(r10 -40) = r0
10: (bf) r1 = r7
11: (77) r1 >>= 32
12: (63) *(u32 *)(r10 -32) = r1
13: (63) *(u32 *)(r10 -28) = r7
14: (bf) r1 = r10
15: (07) r1 += -24
16: (b7) r2 = 16
17: (85) call bpf_get_current_comm#16
last_idx 17 first_idx 9
regs=4 stack=0 before 16: (b7) r2 = 16
18: (69) r1 = *(u16 *)(r8 +178)
R8 invalid mem access 'inv'
processed 19 insns (limit 1000000) max_states_per_insn 0 total_states 1 peak_states 1 mark_read 1

HINT: The invalid mem access 'inv' error can happen if you try to dereference memory without first using bpf_probe_read_kernel() to copy it to the BPF stack. Sometimes the bpf_probe_read_kernel() is automatic by the bcc rewriter, other times you'll need to be explicit.

Traceback (most recent call last):
  File "./traceicmpsoftirq.py", line 66, in <module>
    b.attach_kprobe(event="icmp_echo", fn_name="probe_icmp")
  File "/usr/lib/python2.7/site-packages/bcc/__init__.py", line 673, in attach_kprobe
    fn = self.load_func(fn_name, BPF.KPROBE)
  File "/usr/lib/python2.7/site-packages/bcc/__init__.py", line 407, in load_func
    (func_name, errstr))
Exception: Failed to load BPF program probe_icmp: Permission denied

And this is the kernet config：

# cat /boot/config-`uname -r`|grep CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE
CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE=y

[woodliu]

@yonghong-song
And could you tell me when should i use this function to read data from kernel?
I find in the script /too/tcpconnect，there are some codes as below，why it can get value directly in ipv4.count, but it need read data from kernel in ipv6.count,thank you!

struct_init = { 'ipv4':
        { 'count' :
               """
               struct ipv4_flow_key_t flow_key = {};
               flow_key.saddr = skp->__sk_common.skc_rcv_saddr;
               flow_key.daddr = skp->__sk_common.skc_daddr;
               flow_key.dport = ntohs(dport);
               ipv4_count.increment(flow_key);""",
          'trace' :
               """
               struct ipv4_data_t data4 = {.pid = pid, .ip = ipver};
               data4.uid = bpf_get_current_uid_gid();
               data4.ts_us = bpf_ktime_get_ns() / 1000;
               data4.saddr = skp->__sk_common.skc_rcv_saddr;
               data4.daddr = skp->__sk_common.skc_daddr;
               data4.dport = ntohs(dport);
               bpf_get_current_comm(&data4.task, sizeof(data4.task));
               ipv4_events.perf_submit(ctx, &data4, sizeof(data4));"""
               },
        'ipv6':
        { 'count' :
               """
               struct ipv6_flow_key_t flow_key = {};
               bpf_probe_read_kernel(&flow_key.saddr, sizeof(flow_key.saddr),
                   skp->__sk_common.skc_v6_rcv_saddr.in6_u.u6_addr32);
               bpf_probe_read_kernel(&flow_key.daddr, sizeof(flow_key.daddr),
                   skp->__sk_common.skc_v6_daddr.in6_u.u6_addr32);
               flow_key.dport = ntohs(dport);
               ipv6_count.increment(flow_key);""",
          'trace' :
               """
               struct ipv6_data_t data6 = {.pid = pid, .ip = ipver};
               data6.uid = bpf_get_current_uid_gid();
               data6.ts_us = bpf_ktime_get_ns() / 1000;
               bpf_probe_read_kernel(&data6.saddr, sizeof(data6.saddr),
                   skp->__sk_common.skc_v6_rcv_saddr.in6_u.u6_addr32);
               bpf_probe_read_kernel(&data6.daddr, sizeof(data6.daddr),
                   skp->__sk_common.skc_v6_daddr.in6_u.u6_addr32);
               data6.dport = ntohs(dport);
               bpf_get_current_comm(&data6.task, sizeof(data6.task));
               ipv6_events.perf_submit(ctx, &data6, sizeof(data6));"""
               }
        }

[yonghong-song]

Okay, I see why it works for bpf_probe_read_kernel, but not bpf_probe_read. In current bcc rewriter, if the call is bpf_probe_read, the rewrite for its 3rd argument will be skipped. In the following case,

bpf_probe_read(&seq, sizeof(seq), &my_icmp_hdr(skb)->un.echo.sequence);

&my_icmp_hdr(skb)->un.echo.sequence actually requires a memory access. Using bpf_probe_read won't convert it to bpf_probe_read, while bpf_probe_read_kernel does not have restriction and it works.

The above limitation could be improved in the future. But In the near future, I still suggest to rewrite your code to something like

   addr = &my_icmp_hdr(skb)->un.echo.sequence;
   bpf_probe_read[_kernel](&seq, sizeof(seq), addr);

[yonghong-song]

For your question related to /tool/tcpconnect, using bpf_probe_read is to address some bcc specific code generation issue related to array's. For example, support you have a kernel array struct s { u32 addr[4]; } to represent a ipv6 address. you cannot do user_array = s->addr since it only copies addresses. Using user_array[0] = s->addr[0], user_array[1] = s->addr[1] is not nice either. That is why bpf_probe_read is used explicitly.

[woodliu]

For your question related to /tool/tcpconnect, using bpf_probe_read is to address some bcc specific code generation issue related to array's. For example, support you have a kernel array struct s { u32 addr[4]; } to represent a ipv6 address. you cannot do user_array = s->addr since it only copies addresses. Using user_array[0] = s->addr[0], user_array[1] = s->addr[1] is not nice either. That is why bpf_probe_read is used explicitly.

Thank you! So much help！