Rfcs: dtrace probes

[steveklabnik]

Issue by graydon
_Wednesday May 29, 2013 at 23:45 GMT_

_For earlier discussion, see https://github.com/rust-lang/rust/issues/6816_

_This issue was labelled with: A-instrumentation, A-libs, I-enhancement, P-low in the Rust repository_

---

I think rust should expose dtrace userland probes

Links:
http://www.solarisinternals.com/wiki/index.php/DTrace_Topics_USDT
https://wikis.oracle.com/display/DTrace/Statically+Defined+Tracing+for+User+Applications
https://wiki.freebsd.org/DTrace/userland

Some specific providers (including some dynamic):
https://github.com/chrisa/libusdt
https://github.com/chrisa/node-dtrace-provider
http://prefetch.net/projects/apache_modtrace/index.html
https://bugzilla.mozilla.org/show_bug.cgi?id=370906

Probably at least some for pervasive / standard-library actions:

• per-task CPU time, coroutine switches, lifecycle events
• heap allocations, GC, segmented stack growth
• message passing, IO ops / byte volumes
• conditions raised

Note that the "USDT" mechanism provides _static_ probes whereas things like libusdt (above) provide an API for dynamically registering probes at runtime. This is probably important, possibly even a good place to start, though it's slightly more expensive; ideally of course we should be able to provide both.

See also #6810 concerning metrics. I think there's some potential design-informing between the two. Possibly libusdt as a reporting sink for metrics?

[tromey]

Linux has its own form of static probe: https://sourceware.org/systemtap/wiki/UserSpaceProbeImplementation. They are very low cost and various tools, like gdb and systemtap, already know how to access them.

[cuviper]

@tromey see my probe crate for SystemTap SDT support, and in theory it could support implementations for other platforms too. It needs asm! though, which makes it nightly-only.

[Fiedzia]

Its been a while, I looked at it recently and I can't see any way to get it working on stable.
My attempts is here: https://github.com/Fiedzia/rust-usdt. It works, but only on nightly due to being compiler plugin and using asm, and even nightly removed some features I had before (access to mir pass). The only way for now I'd see would be to integrate it with compiler.

[jonhoo]

With the up-and-coming support for eBPF-based program tracing using tools like bpftrace, it would be _really_ cool to see this being picked up again. From this article, it looks like any kind of USDT probe would be immediately recognizable by eBPF-based tools. @cuviper can you comment any more on the state of libprobe since https://github.com/rust-lang/rust/pull/14031?

[cuviper]

@jonhoo AFAIK my probe crate still works, but only on nightly for inline asm (rust-lang/rust#29722). I don't know how to implement this otherwise, but rumor from the recent all-hands meeting is that asm might now have a path toward stabilization.

[jonhoo]

I wonder whether libprobe also works on Linux with eBPF now... I suspect it should. Have you tried that at all?

[cuviper]

I haven't tried with eBPF, but it ought to work -- I implemented it the same way as sys/sdt.h.

[jonhoo]

I can confirm that libprobe works with eBPF. Specifically, I added

#![feature(asm)]
#[macro_use]
#[no_link]
extern crate probe;

to inferno here. I then added annotations here:

if line.is_empty() {
    probe!(inferno_collapse_perf, post_stack);
    self.after_event();
} else {
    probe!(inferno_collapse_perf, in_stack);
    self.on_line(line.trim_end());
}

This then let me do this:

$ sudo bpftrace -e "usdt:$CARGO_TARGET_DIR/release/inferno-collapse-perf:inferno_collapse_perf:post_stack /@start[pid]/ { @ns[comm]= hist(nsecs - @start[pid]); delete(@start[pid]); } usdt:$CARGO_TARGET_DIR/release/inferno-collapse-perf:inferno_collapse_perf:post_stack { @start[pid] = nsecs; }"
Attaching 2 probes...
^C

@ns[inferno-collaps]:
[1K, 2K)           24014 |@@@@                                                |
[2K, 4K)          108620 |@@@@@@@@@@@@@@@@@@                                  |
[4K, 8K)          129485 |@@@@@@@@@@@@@@@@@@@@@                               |
[8K, 16K)         310007 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[16K, 32K)        244627 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@           |
[32K, 64K)         69413 |@@@@@@@@@@@                                         |
[64K, 128K)        21566 |@@@                                                 |
[128K, 256K)         152 |                                                    |
[256K, 512K)           1 |                                                    |
[512K, 1M)             2 |                                                    |

That's really neat!

The command is a _little_ awkward since bpftrace requires that the name of the binary matches the program name (https://github.com/iovisor/bpftrace/issues/328#issuecomment-459013046), and Rust identifiers cannot have - in them. We get around that by giving an explicit application name following https://github.com/iovisor/bpftrace/pull/413.

[jonhoo]

Also note that with https://github.com/iovisor/bpftrace/pull/444 (which is really https://github.com/iovisor/bpftrace/pull/280), the above can (?) be simplified greatly to:

$ sudo bpftrace -p $(pgrep inferno-collapse-perf) -e "\
usdt::post_stack /@start[pid]/ { @ns[comm]= hist(nsecs - @start[pid]); delete(@start[pid]); }\
usdt::post_stack { @start[pid] = nsecs; }"

[dalehamel]

Weird to come across an issue, linking my own pull requests. I'd love to see support for dtrace probes in Rust!

Should also check out libstapsdt, which can be used for dynamic support.

I'm not as familiar with rust as I am other runtimes, but it would be great to at least have an --enable-dtrace option when building Rust, offering functionality similar to the tools Sasha Goldshtein put together, (eg, ucalls.py to support "rustcalls.sh" sort of things).

I'm planning on doing a lot more eBPF USDT work in the next few months, I'd love to help however I can.