Bcc: option to show full paths for fileslower and related tools
I just used fileslower for the first time, it's really a cool tool. Thanks for that already!
But looking at the results, it leaves me guessing as to what file is actually being referenced:
Tracing sync read/writes slower than 100 ms
TIME(s) COMM TID D BYTES LAT(ms) FILENAME
3.215 akonadi_follow 8034 R 16384 324.60 agent_config_akonadi_followup...
3.218 akonadi_indexi 8044 R 16384 330.60 agent_config_akonadi_indexing...
3.270 akonadi_ical_r 8040 R 16384 379.21 agent_config_akonadi_ical_res...
3.338 akonadi_akonot 8029 R 16384 394.43 agent_config_akonadi_akonotes...
3.350 akonadi_maildi 8046 R 16384 457.47 agent_config_akonadi_maildisp...
3.354 akonadi_contac 8033 R 16384 419.19 agent_config_akonadi_contacts...
3.354 akonadi_vcard_ 8052 R 16384 413.45 agent_config_akonadi_vcard_re...
3.354 akonadi_maildi 8045 R 16384 404.17 agent_config_akonadi_maildir_...
3.367 akonadi_akonot 8028 R 16384 426.73 agent_config_akonadi_akonotes...
3.382 akonadi_akonot 8031 R 16384 487.54 agent_config_akonadi_akonotes...
3.405 akonadi_migrat 8048 R 16384 401.60 akonadi-migrationrc
3.418 akonadi_indexi 8044 R 16384 199.96 agent_config_akonadi_indexing...
3.418 akonadi_ical_r 8040 R 16384 108.69 akonadi_ical_resource_0rc
3.418 akonadi_notes_ 8050 R 16384 404.33 akonadi_notes_agentrc
3.425 akonadi_akonot 8030 R 16384 410.83 akonadi_akonotes_resource_2rc
3.454 akonadi_maildi 8046 R 16384 103.39 agent_config_akonadi_maildisp...
3.583 akonadi_ical_r 8040 R 524288 111.20 std.ics
3.620 akonadi_akonot 8031 R 16384 135.59 akonadi_akonotes_resource_3rc
3.637 akonadi_vcard_ 8052 R 16384 157.05 Default Address Bookrc
3.783 akonadi_maildi 8046 R 16384 272.56 specialmailcollectionsrc
3.800 akonadi_akonot 8029 R 16384 225.03 1343311121.R370.milian-kdab2
3.898 akonadi_vcard_ 8052 R 524288 261.01 std.vcf
3.898 akonadi_akonot 8028 R 16384 430.36 akonadi_akonotes_resource_0rc
3.933 akonadi_akonot 8029 R 16384 123.05 types
3.961 akonadi_indexi 8044 R 16384 457.26 baloorc
6.712 kmail 8185 R 4096 115.27 machine-id
I'd like to request two options to improve this situation:
- a way to increase the filename limit, such that I at least see the full filename instead of e.g.
agent_config_akonadi_akonotes... - an option to print the full path, instead of just the filename
If you guide me as to what would be required to implement this, I'd be willing to provide a patch. Just increasing the DNAME_INLINE_LEN in fileslower.py doesn't seem to produce correct results ;-)
Cheers
milianw
All 15 comments
@milianw I think the limit for this event is set from kernel itself : http://lxr.free-electrons.com/source/include/linux/dcache.h#L67 AFAIU, there is not much that can be done at VFS layer, unless we do major work in tracking file's lifecycle higher up, right from the open syscall which can give the filename up to 256 chars. Is there any other hack around this anyone?
tuxology
on 5 Apr 2017
Can the inode be mapped back to a full path/filename somehow maybe? Just putting it out as an idea, not sure this is feasible. I mean if we report back the inode to python and resolve it with some overhead there, it would still be better than the status quo.
milianw
on 5 Apr 2017
Or what about the d_name.name? what does that contain?
milianw
on 5 Apr 2017
fileslower is currently using d_name.name.
Regarding full path:
We could use a map to cache an inode->path on open(), and that might even be largely stable (tracing probes & structures that don't change much), but, that means adding more probes.
fileslower already has access to the dentry and an inode. We can, with a "-f" (full) or "-l" (long) option, do an unrolled loop to walk back several directory levels and build up the path. We might need to get a bit creative with the limited string functions, eg, have a char[256] that we populate in reverse directory order, and have python switch it back around the right way. I think that would be fine because it's fileslower, and only printing events of interest (and not every single VFS call), so this unrolled loop and char[256] buffers would only be done when needed.
I just took a look, and doing a reverse path walk from the dentry is also how SystemTap does it. :-)
brendangregg
on 6 Apr 2017
Hey,
I hope no one minds me commenting on this. I had a similar thought as @milianw that it would be cool to see the full file path. I tried out a few things and one of them was using a function in the kernel that already exists, d_path. I quickly found out you cant use any kernel function. I also tried to use a recursive function in a bpf program to get the path and got illegal instruction errors.
I thought it might be interesting to write a helper function that would basically wrap the existing d_path function. I uploaded two rough patches for the kernel and for bcc to use the new helper function along with a small example program. https://drive.google.com/drive/folders/0B40yY9I_bXsBYnJDTjdaTUlVdGs?usp=sharing
If it's something worth pursuing I could clean it up and try to have these merged?
tejom
on 13 Apr 2017
@tuxology is my idea above something this project would be interested in?
tejom
on 19 Apr 2017
@tejom I would like this feature personally. Maybe name it bpf_return_dpath_str as bpf_return_str can be too generic. I think @4ast is more suitable to decide if adding a separate function to kernel just for the path string is interesting.
tuxology
on 19 Apr 2017
i haven't looked at the kernel diff, since google drive isn't the place for code reviews.
I'm also not sure how it's possible to guarantee safety of such helper.
I think @brendangregg had working dpath returning function that was completely done on bcc side using probe_read. I think we can put that into bcc 'library' of useful always_inline functions?
Like bpf_log2() in src/cc/export/helpers.h
4ast
on 19 Apr 2017
Thanks @tuxology and @4ast . I uploaded it onto google drive just as a quick place to share it. It's a pretty rough patch just to demonstrate the idea and see if it's worth the time to make it better. I can put the kernel diff somewhere else so its easier to look at. I'm just not sure where? I figured the kernel mailing list would be the place for that, but I didn't want to do that if this wasn't a direction to go in.
If there is a better way to accomplish getting the path or if it won't be safe then I won't bother with this patch then.
tejom
on 19 Apr 2017
Documentation/networking/netdev-FAQ.txt is a good first step.
for kernel upstream submission it has to be over email with proper SOB and follow the process.
To share rough idea it can be a link to github commit in your cloned kernel tree or email to iovisor-dev or netdev lists (depending on how many people you want to hear feedback from)
4ast
on 19 Apr 2017
Ok thanks, I'll send it to the mailing list for feedback and if that goes well I'll go through the process for submitting upstream.
tejom
on 20 Apr 2017
I looked at the kernel patch: it looks like it provides a new BPF_FUNC_return_str, which just calls d_path().
d_path() calls rcu_read_lock() and other functions. I haven't dug in specifically, but I thought we needed to be very careful of what is executed in the BPF environment. Is it safe to call these?
It's minor, but the name BPF_FUNC_return_str is not illustrative. I'd have called it BPF_FUNC_walk_path.
I still wonder if we can, to start with, implement this just in bcc as an unrolled loop with existing BPF instructions.
brendangregg
on 20 Apr 2017
Thanks for looking. I would fix the naming of the functions and variables if I actually went to submit it. (Along with other problems like space alignment and missing comments). I just wanted to share a quick patch to see if there were any concerns like the one you mentioned.
tejom
on 20 Apr 2017
I still wonder if we can, to start with, implement this just in bcc as an unrolled loop with existing BPF instructions.
@brendangregg have you had a chance to look into this further or found a way to do it?
schu
on 19 May 2017
I just took a look, and doing a reverse path walk from the dentry is also how SystemTap does it. :-)
In case someone goes down the path of looking at how SystemTap does it: https://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=blob;f=tapset/linux/dentry.stp;h=4e73532f24b96ee67be3adb4f7c16d5608f8f608;hb=HEAD#l184
zoidbergwill
on 4 Jul 2019
Related issues
alvenwong
路
7Comments
mithunk18
路
7Comments
brendangregg
路
4Comments
banh-gao
路
8Comments
saobao913
路
8Comments
Most helpful comment
fileslower is currently using d_name.name.
Regarding full path:
We could use a map to cache an inode->path on open(), and that might even be largely stable (tracing probes & structures that don't change much), but, that means adding more probes.
fileslower already has access to the dentry and an inode. We can, with a "-f" (full) or "-l" (long) option, do an unrolled loop to walk back several directory levels and build up the path. We might need to get a bit creative with the limited string functions, eg, have a char[256] that we populate in reverse directory order, and have python switch it back around the right way. I think that would be fine because it's fileslower, and only printing events of interest (and not every single VFS call), so this unrolled loop and char[256] buffers would only be done when needed.
I just took a look, and doing a reverse path walk from the dentry is also how SystemTap does it. :-)