Bpftrace: Error finding location for probe (usdt)

[DavidVentura]

I started playing with bpftrace and bcc tools, but I cannot get bpftrace to work with usdt probes. bcc's trace works fine.

Issue:

$ sudo ./build/src/bpftrace  -e 'usdt:/home/david/git/bpftrace/a.out:testprobe { time("%H:%M:%S  "); printf("%-6d %d\n", pid, arg0); }'
Attaching 1 probe...
Error finding location for probe: usdt:/home/david/git/bpftrace/a.out:testprobe

bcc on the same binary

$ sudo ../../bcc/tools/trace.py   'u:/home/david/git/bpftrace/a.out:testprobe "%d" retval'
PID     TID     COMM            FUNC             -
7442    7442    a.out           testprobe        1547067909
7442    7442    a.out           testprobe        1547067910

example code from here

$ cat test.c                                                                                                                                                                                                                                                                                
#include <sys/sdt.h>                                                                                                                                                                                                                                                                                                         
#include <sys/time.h>                                                                                                                                                                                                                                                                                                        
#include <unistd.h>                                                                                                                                                                                                                                                                                                          
#include <stdio.h>                                                                                                                                                                                                                                                                                                           

long int myclock() {                                                                                                                                                                                                                                                                                                         
    struct timeval tv;                                                                                                                                                                                                                                                                                                       
    gettimeofday(&tv, NULL);                                                                                                                                                                                                                                                                                                 
    //printf("%ld\n", tv.tv_sec);
    DTRACE_PROBE1(test-app, testprobe, tv.tv_sec);
    return tv.tv_sec;
}

int main(int argc, char **argv)
{

    while(1) {
        myclock();
        sleep(1);
    }
    return 0;
}

Using an uretprobe on the function works though

$ sudo ./build/src/bpftrace  -e 'uretprobe:/home/david/git/bpftrace/a.out:myclock { time("%H:%M:%S  "); printf("%-6d %d\n", pid, retval); }'
Attaching 1 probe...
22:13:17  7660   1547068397
22:13:18  7660   1547068398

What am I doing wrong? I am on HEAD on both projects..

[FooBarWidget]

I have a similar problem, but instead of tracing my own binary I am trying to trace glibc on Ubuntu 18.04. According to the glibc malloc source there should be a probe called memory_arena_new:

 LIBC_PROBE (memory_arena_new, 2, a, size);

Indeed, readelf tells me that it is there:

~
stapsdt 0x0000003c NT_STAPSDT (SystemTap probe descriptors)
Provider: libc
Name: memory_arena_new
Location: 0x0000000000091bd0, Base: 0x00000000001bdd48, Semaphore: 0x0000000000000000
Arguments: 8@%rdx 8@%rbp
~

And yet:

~
$ sudo bpftrace -e 'usdt:/lib/x86_64-linux-gnu/libc-2.27.so:memory_arena_new { printf("memory arena new"); }'
Attaching 1 probe...
Error finding location for probe: usdt:/lib/x86_64-linux-gnu/libc-2.27.so:memory_arena_new
~

Am I doing something wrong? bpftrace commit ccac69c2239.

[FooBarWidget]

@DavidVentura After debugging bpftrace's source code, I found out why your example doesn't work. Your provide's provider name is 'test-app'. However, bpftrace looks for a probe whose provider name equals that of the binary, i.e. bpftrace is looking for your 'testprobe' under the provider name 'a.out'. This is apparent from attached_probe.cpp AttachedProbe::attach_usdt:

~c++
std::string provider_name = GetProviderFromPath(probe_.path);
err = bcc_usdt_get_location(ctx, provider_name.c_str(), probe_.attach_point.c_str(), 0, &loc);
if (err)
throw std::runtime_error("Error finding location for probe: " + probe_.name);
~

I fixed your issue by renaming your binary so that it matches the provider name (you could also do it the other way around, i.e. renaming the provider name to the binary's name).

In my case, the provider name is "libc". I fixed my problem my symlinking /lib/x86_64-linux-gnu/libc-2.27.so to /tmp/libc and then attaching the usdt probe to /tmp/libc instead.

This was not at all obvious. I think bpftrace should provide a way to specify the provider name, and making it more clear in which provider namespace it searched for a probe.

[dalehamel]

I have also run into this issue recently, and came up with similar conclusions to @FooBarWidget.

A more forgiving / flexible behavior might be to check for a given probe name against all providers, and only request that the provider be specified if there is duplication. This is less explicit though, and I could see there being issues with this.

I think that being able to manually specify the provider name, rather than depending on it being the basename of the path provided, is probably the best way forward here, as it isn't unreasonable to require the user to explicitly indicate the provider name. In this approach it would be harder to maintain backwards compatibility. One option would be to default to the current approach with a deprecation warning if no provider name is specified, and then deprecate in a future release.

So for example, where I currently write a probe as:

bpftrace -p PID -e 'usdt:/usr/bin/ruby:method__entry { printf("here\n"); }

Or, generically:

usdt:PATH:PROBE, which provider being inferred from the final segment of the path.

The new interface would be:

bpftrace -p PID -e 'usdt:/usr/bin/ruby:ruby:method__entry { printf("here\n"); } (explicitly indicating the ruby provider, even though it's the same as the final path segment in this case)

Or, generically:

usdt:PATH:PROVIDER:PROBE

I'm not sure if this would be an issue for how probes are generally handled though (having 4 arguments instead of 3). I'll need to check the code for that.

[brendangregg]

Thanks for the debugging (I just ran into this too). So how about:

bpftrace -p PID -e 'usdt:/usr/bin/ruby:method__entry { printf("here\n"); }

would instrument ruby:method__entry

bpftrace -p PID -e 'usdt:/usr/bin/ruby:hello:method__entry { printf("here\n"); }

would instrument hello:method__entry

ie, if the provider name is not provided, it defaults to the binary name. Sound ok?

[mmarchini]

Sounds good to me.

[brendangregg]

Anyone already made a start at coding it? :-)

[dalehamel]

ie, if the provider name is not provided, it defaults to the binary name. Sound ok?

i like that suggestion @brendangregg

Anyone already made a start at coding it? :-)

Not quite, but I did start poking around the code to see how this works and I think I have a rough idea of what needs to be done.

At the surface, it looks like the changes would just be to AttachPoint and a few related files:

https://github.com/iovisor/bpftrace/blob/master/src/ast/ast.h#L205-L220
https://github.com/iovisor/bpftrace/blob/master/src/parser.yy#L145-L150

And probably a couple of other minor changes elsewhere related to the conditional handling of the provider name. I'll keep pulling at this thread and see if i can come up with a rough cut for a PR.

[brendangregg]

This also needs to update bcc_usdt_get_argument() in ast/irbuilderbpf.cpp.

[dalehamel]

I have #413 up and ready for review

[williangaspar]

did https://github.com/iovisor/bpftrace/pull/413 solve it?

[jonhoo]

Also, how does #413 interact with #444 (which is really #280)? Should -p now really be all that's needed, even if the prove's provider doesn't match the name of the binary?