Bcc: Provenance of `struct pt_regs* ctx`?

[ChaosData]

Hi,

I'm having a hard time coming up with an explanation for some of the behavior I've been seeing around the first struct pt_regs* ctx argument to kprobes when using BCC and was wondering if any of you folks happened to know why some of it is happening as it appears to. I'm assuming x86_64 here, since that's what I'm running BCC on. :)

Some of the code in the kernel is a bit obtuse, so it's a bit hard to track where this thing exactly comes from when it enters the eBPF execution flow, but it looks like it may be the bpf_context for kprobes. Within the execution of a BCC eBPF kprobe, this struct contains an odd menagerie of values. As far as I can tell, the main set of argument registers (RDI as ctx->di, RSI as ctx->si, RDX as ctx->dx, RCX as ctx->cx, ...) are are valid. In contrast to this, values such as RBP (ctx->bp), RSP (ctx->sp), and RIP (ctx->ip) are appear to be kernel memory values; ctx->orig_ax also seems to be garbage.

This potentially seems in line with @4ast's comment in #751 noting that "In case of kprobe program type, struct pt_regs for program == struct pt_regs for kernel." The odd thing I've noticed (and note that the following magic number may be a bit specific to my kernel/kprobe) is that ctx->bp + 16 is the address of yet another struct pt_regs that appears to be the userspace one, where not only are the argument registers correct for the traced process, but all of the others I mentioned above. For clarity, I got the above + 16 by simply dumping memory from ctx->bp and observing that known values showed up there.

Ideally, I would like to know where in the kernel the eBPF ctx comes from, if this pt_regs chain relationship is something stable, and if there is a helper macro in BCC that can be used to follow such a chain and pull out the real userspace values.

Thanks.

[yonghong-song]

Are you kprobing a function like __x64_sys_<name>? On x86, the arguments are wrapped which conforms to what you described in the above. The following is a relevant bcc commit:

commit 2da34267fcae4485f4e05a17521214749f6f0edd
Author: yonghong-song <[email protected]>
Date:   Wed Jun 13 06:12:22 2018 -0700

    generate indirect parameter assignment if arch uses syscall wrapper (#1816)

    Fix issue #1802.

    On x64, the following commit (in 4.17) changed the raw parameter passed to
    the syscall entry function from a list of parameters supplied in user space
    to a single `pt_regs *` parameter. Also in 4.17, x64 syscall entry function
    is changed from `sys_<name>` to `__x64_sys_<name>`.

    ```
    commit fa697140f9a20119a9ec8fd7460cc4314fbdaff3
    Author: Dominik Brodowski <[email protected]>
    Date:   Thu Apr 5 11:53:02 2018 +0200

        syscalls/x86: Use 'struct pt_regs' based syscall calling convention for 64-bit syscalls

        Let's make use of ARCH_HAS_SYSCALL_WRAPPER=y on pure 64-bit x86-64 systems:

        Each syscall defines a stub which takes struct pt_regs as its only
        argument. It decodes just those parameters it needs, e.g:
...

[ChaosData]

kprobe__sys_timerfd_settime, but this is on a 4.15 kernel, so while this is good to know going forward, I'm not sure how much it applies to what I'm seeing.

But funny enough, as I was trying to track this down, I was spending a bit of time reading through the codegen in BTypeVisitor::genParamIndirectAssign added in that commit. It looked as if the <ctx> parameter declared by the user was being used to set up a __<ctx>, but I couldn't tell if the original <ctx> was being overridden.

[yonghong-song]

The original <ctx> is preserved and not overridden.

[ChaosData]

For reference, I figured out a way to do this "properly," task_pt_regs(current) (assuming user context). This does yield me the same address of the struct pt_regs I scanned out, so at least the previous method wasn't getting me a stale ABA copy or anything, but I haven't observed anything that would lead me to believe that the previous math is any sort of portable. Unfortunately, like many such kernel macros, the codegen has a hard time with it, so it needs to be deconstructed and embedded within the BCC C code as below.

struct task_struct* __current = (struct task_struct*)bpf_get_current_task();
void* __current_stack_page = __current->stack;
void* __ptr = __current_stack_page + THREAD_SIZE - TOP_OF_KERNEL_STACK_PADDING;
struct pt_regs* _tctx = ((struct pt_regs *)__ptr) - 1;

After that, I've also found it useful to simply copy it to a local eBPF stack-based struct pt_regs for dealing with similar codegen issues with pointer dereferences, but that's going to happen with non-eBPF pointers a lot of the time anyway.