We had issue with using uprobes with ebpf due to dynamic nature of golang stacks. Is there a proposal/way to address that?
shashankmjain
All 8 comments
What is ebpf? What are uprobes?
The Go compiler is committed to having stacks that grow, and therefore move, as needed. That is a key part of supporting programs that use a very large number of goroutines.
ianlancetaylor
on 25 Sep 2017
ebpf is the linux tracing mechanism and uprobes are dynamic probes for user space programs. The ebpf program attaches to a user probe and gets executed when that function is invoked. Here it seems the golang stack is resized and the return of the ebpf program gets a segmentation fault as it points to a invalid address
shashankmjain
on 25 Sep 2017
Sounds like ebpf/uprobes supports only processes having a pure C ABI with thread-static stacks .
cznic
on 25 Sep 2017
I don't see how we could support a dynamic probe executed by the kernel that refers to a local address on the stack.
ianlancetaylor
on 25 Sep 2017
Anyway for go to pause the moving of stacks, eg, by sending it a signal? That way the behavior could be disabled temporarily while the kernel did user-level instrumentation.
brendangregg
on 25 Sep 2017
The SIGSTOP signal would work.
There is no way to disable moving stacks other than stopping the program, though. There can't be: program execution could require additional stack space at any time.
ianlancetaylor
on 26 Sep 2017
FYI I posted a tentative solution that I'm experimenting with to work around this problem, and would appreciate comments and skepticisms: https://github.com/iovisor/bcc/issues/1320#issuecomment-407927542
Thanks
gianlucaborello
on 26 Jul 2018
The title and description is misleading. Uprobes do in fact work with Go programs. What does not work are uretprobes.
A uprobe is associated with a userspace program binary and offset. When a probe is added, Linux will load that program, save the old instruction at the absolute file offset and patch it with a trap instruction. When the probe is hit, the eBPF program will be executed in kernel space. This works fine with normal uprobes.
A uretprobe patches a function in a similar way at entry, but it will modify the return address on the stack to a trampoline function. Once hit, the EBPF program is executed and the instruction pointer is modified to the original return address again. If the stack changes, this will likely cause corruption and crashes.
uretprobes should not be used with Go programs.
Lekensteyn
on 21 Aug 2019
Related issues
bradfitz
·
3Comments
myitcv
·
3Comments
mingrammer
·
3Comments
michaelsafyan
·
3Comments
enoodle
·
3Comments
Most helpful comment
The title and description is misleading. Uprobes do in fact work with Go programs. What does not work are uretprobes.
A uprobe is associated with a userspace program binary and offset. When a probe is added, Linux will load that program, save the old instruction at the absolute file offset and patch it with a trap instruction. When the probe is hit, the eBPF program will be executed in kernel space. This works fine with normal uprobes.
A uretprobe patches a function in a similar way at entry, but it will modify the return address on the stack to a trampoline function. Once hit, the EBPF program is executed and the instruction pointer is modified to the original return address again. If the stack changes, this will likely cause corruption and crashes.
uretprobes should not be used with Go programs.