fd -HI faster on root-dir then without -HI
According the docs, fd ignores hidden en (vcs)-ignored files. This should make it faster. One can disable this feature with -HI.
Home-dir
Indeed, when running within my home directory, it is faster without -HI:
Cold run
````
$ hyperfine --prepare 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches' 'sudo fdfind -HI ".[0-9].jpg$" ~' 'sudo fdfind "[0-9].jpg$" ~'
Benchmark #1: sudo fdfind -HI ".[0-9].jpg$" ~
Time (mean ยฑ ฯ): 286.5 ms ยฑ 5.4 ms [User: 277.2 ms, System: 454.1 ms]
Range (min โฆ max): 279.2 ms โฆ 294.7 ms 10 runs
Benchmark #2: sudo fdfind "[0-9].jpg$" ~
Time (mean ยฑ ฯ): 155.2 ms ยฑ 5.3 ms [User: 196.8 ms, System: 245.9 ms]
Range (min โฆ max): 148.1 ms โฆ 166.0 ms 10 runs
Summary
'sudo fdfind "[0-9].jpg$" ~' ran
1.85 ยฑ 0.07 times faster than 'sudo fdfind -HI ".*[0-9].jpg$" ~'
````
Warm run
````
$ hyperfine --warmup 3 'sudo fdfind -HI ".[0-9].jpg$" ~' 'sudo fdfind "[0-9].jpg$" ~'
Benchmark #1: sudo fdfind -HI ".[0-9].jpg$" ~
Time (mean ยฑ ฯ): 96.9 ms ยฑ 4.9 ms [User: 329.8 ms, System: 291.7 ms]
Range (min โฆ max): 93.1 ms โฆ 111.7 ms 26 runs
Warning: The first benchmarking run for this command was significantly slower than the rest (111.7 ms). This could be caused by (filesystem) caches that were not filled until after the first run. You should consider using the '--warmup' option to fill those caches before the actual benchmark. Alternatively, use the '--prepare' option to clear the caches before each timing run.
Benchmark #2: sudo fdfind "[0-9].jpg$" ~
Time (mean ยฑ ฯ): 62.8 ms ยฑ 2.7 ms [User: 200.0 ms, System: 167.8 ms]
Range (min โฆ max): 59.3 ms โฆ 71.3 ms 47 runs
Summary
'sudo fdfind "[0-9].jpg$" ~' ran
1.54 ยฑ 0.10 times faster than 'sudo fdfind -HI ".*[0-9].jpg$" ~'
````
Number of results
````
$ sudo fdfind -HI ".*[0-9].jpg$" ~ | wc -l
10228
$ sudo fdfind "[0-9].jpg$" ~ | wc -l
10212
````
Root-dir
Unfortunately, when running the same command on my root-dir, the reverse is true: with -HI it is (slightly) faster:
Cold run
````
$ hyperfine --prepare 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches' 'sudo fdfind -HI ".[0-9].jpg$" /' 'sudo fdfind "[0-9].jpg$" /'
Benchmark #1: sudo fdfind -HI ".[0-9].jpg$" /
Time (mean ยฑ ฯ): 2.983 s ยฑ 0.083 s [User: 2.891 s, System: 10.862 s]
Range (min โฆ max): 2.898 s โฆ 3.160 s 10 runs
Benchmark #2: sudo fdfind "[0-9].jpg$" /
Time (mean ยฑ ฯ): 3.228 s ยฑ 0.055 s [User: 3.737 s, System: 12.508 s]
Range (min โฆ max): 3.160 s โฆ 3.321 s 10 runs
Summary
'sudo fdfind -HI ".*[0-9].jpg$" /' ran
1.08 ยฑ 0.04 times faster than 'sudo fdfind "[0-9].jpg$" /'
````
Warm run
````
$ hyperfine --warmup 3 'sudo fdfind -HI ".[0-9].jpg$" /' 'sudo fdfind "[0-9].jpg$" /'
Benchmark #1: sudo fdfind -HI ".[0-9].jpg$" /
Time (mean ยฑ ฯ): 1.156 s ยฑ 0.009 s [User: 3.109 s, System: 4.089 s]
Range (min โฆ max): 1.147 s โฆ 1.172 s 10 runs
Benchmark #2: sudo fdfind "[0-9].jpg$" /
Time (mean ยฑ ฯ): 1.448 s ยฑ 0.010 s [User: 3.891 s, System: 5.367 s]
Range (min โฆ max): 1.431 s โฆ 1.462 s 10 runs
Summary
'sudo fdfind -HI ".*[0-9].jpg$" /' ran
1.25 ยฑ 0.01 times faster than 'sudo fdfind "[0-9].jpg$" /'
````
Number of results
````
$ sudo fdfind -HI ".*[0-9].jpg$" / | wc -l
10585
$ sudo fdfind "[0-9].jpg$" / | wc -l
10569
````
System specs
Hardware
- CPU: Intel i7-8565U (8) @ 4.600GHz
- Storage: Samsung SSD 970 PRO 1TB
RAM: 2x 32GB @ 2666MHz from Samsung (about 3GB in use while testing)
Software
OS: Ubuntu 19.04 x86_64
- Kernel: 5.0.0-15-generic
- Shell: bash 5.0.3
- fd-version: 7.2.0 (installed with apt)
OS:
BartG95
All 12 comments
Thank you for the detailed report.
I would also expect non--HI searches to be faster IF a large part of the search tree can be pruned, that is, if there are many files beneath hidden directories and ignored directories.
On the other hand, non--HI searches also need to perform extra work to parse .gitignore files and to perform glob matching. If you have a lot of different or complicated .(git)ignore files in the search tree, the -HI search might actually be faster.
You could try to search for large .(git)ignore files to see if there are any candidates that might slow down the non--HI search:
fd -HI --size +10kb '^\.(git|fd)?ignore$' /
sharkdp
on 8 Jun 2019
All .(git)ignores outside my home-dir (those inside my home-dir don't matter since they are included in both scenarios):
$ sudo fd -HI -E '/home/bart/*' '^\.(git|fd)?ignore$' /
/usr/src/tuxedo_keyboard-1/.gitignore
/usr/src/linux-headers-5.0.0-16/scripts/.gitignore
/usr/src/linux-headers-5.0.0-15/scripts/.gitignore
/usr/src/linux-headers-5.0.0-16/scripts/gcc-plugins/.gitignore
/usr/src/linux-headers-5.0.0-16/scripts/dtc/.gitignore
/usr/src/linux-headers-5.0.0-16/scripts/genksyms/.gitignore
/usr/src/linux-headers-5.0.0-16/scripts/mod/.gitignore
/usr/src/linux-headers-5.0.0-16/scripts/basic/.gitignore
/usr/src/linux-headers-5.0.0-16/scripts/kconfig/.gitignore
/usr/src/linux-headers-5.0.0-15/scripts/gcc-plugins/.gitignore
/usr/src/linux-headers-5.0.0-15/scripts/dtc/.gitignore
/usr/src/linux-headers-5.0.0-15/scripts/genksyms/.gitignore
/usr/src/linux-headers-5.0.0-15/scripts/mod/.gitignore
/usr/src/linux-headers-5.0.0-15/scripts/basic/.gitignore
/usr/src/linux-headers-5.0.0-15/scripts/kconfig/.gitignore
/usr/src/linux-headers-5.0.0-16/scripts/selinux/genheaders/.gitignore
/usr/src/linux-headers-5.0.0-16/scripts/selinux/mdp/.gitignore
/usr/src/linux-headers-5.0.0-16/scripts/gdb/linux/.gitignore
/usr/src/linux-headers-5.0.0-16/scripts/kconfig/lxdialog/.gitignore
/usr/src/linux-headers-5.0.0-15/scripts/selinux/genheaders/.gitignore
/usr/src/linux-headers-5.0.0-15/scripts/selinux/mdp/.gitignore
/usr/src/linux-headers-5.0.0-15/scripts/gdb/linux/.gitignore
/usr/src/linux-headers-5.0.0-15/scripts/kconfig/lxdialog/.gitignore
/usr/lib/nodejs/iconv-lite/generation/source-data/.gitignore
$ sudo fd -HI -E '/home/bart/*' '^\.(git|fd)?ignore$' / -x bat | wc -l
116
$ sudo fd -HI -E '/home/bart/*' '^\.(git|fd)?ignore$' / -X bat --paging never
โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ File: /usr/src/tuxedo_keyboard-1/.gitignore
โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1 โ # Prerequisites
2 โ *.d
3 โ
4 โ # Object files
5 โ *.o
6 โ *.ko
7 โ *.obj
8 โ *.elf
9 โ
10 โ # Kernel Module Compile Results
11 โ *.mod*
12 โ *.cmd
13 โ .tmp_versions/
14 โ modules.order
15 โ Module.symvers
16 โ Mkfile.old
17 โ
18 โ .vscode/
โโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ File: /usr/src/linux-headers-5.0.0-16/scripts/.gitignore
โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1 โ #
2 โ # Generated files
3 โ #
4 โ bin2c
5 โ conmakehash
6 โ kallsyms
7 โ pnmtologo
8 โ unifdef
9 โ recordmcount
10 โ sortextable
11 โ asn1_compiler
12 โ extract-cert
13 โ sign-file
14 โ insert-sys-cert
โโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ File: /usr/src/linux-headers-5.0.0-15/scripts/.gitignore
โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1 โ #
2 โ # Generated files
3 โ #
4 โ bin2c
5 โ conmakehash
6 โ kallsyms
7 โ pnmtologo
8 โ unifdef
9 โ recordmcount
10 โ sortextable
11 โ asn1_compiler
12 โ extract-cert
13 โ sign-file
14 โ insert-sys-cert
โโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ File: /usr/src/linux-headers-5.0.0-16/scripts/dtc/.gitignore
โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1 โ dtc
โโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ File: /usr/src/linux-headers-5.0.0-16/scripts/gcc-plugins/.gitignore
โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1 โ randomize_layout_seed.h
โโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ File: /usr/src/linux-headers-5.0.0-16/scripts/genksyms/.gitignore
โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1 โ genksyms
โโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ File: /usr/src/linux-headers-5.0.0-16/scripts/mod/.gitignore
โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1 โ elfconfig.h
2 โ mk_elfconfig
3 โ modpost
4 โ devicetable-offsets.h
โโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ File: /usr/src/linux-headers-5.0.0-16/scripts/basic/.gitignore
โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1 โ fixdep
โโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ File: /usr/src/linux-headers-5.0.0-16/scripts/kconfig/.gitignore
โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1 โ #
2 โ # Generated files
3 โ #
4 โ *.moc
5 โ *conf-cfg
6 โ
7 โ #
8 โ # configuration programs
9 โ #
10 โ conf
11 โ mconf
12 โ nconf
13 โ qconf
14 โ gconf
โโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ File: /usr/src/linux-headers-5.0.0-15/scripts/dtc/.gitignore
โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1 โ dtc
โโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ File: /usr/src/linux-headers-5.0.0-15/scripts/gcc-plugins/.gitignore
โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1 โ randomize_layout_seed.h
โโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ File: /usr/src/linux-headers-5.0.0-15/scripts/genksyms/.gitignore
โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1 โ genksyms
โโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ File: /usr/src/linux-headers-5.0.0-15/scripts/mod/.gitignore
โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1 โ elfconfig.h
2 โ mk_elfconfig
3 โ modpost
4 โ devicetable-offsets.h
โโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ File: /usr/src/linux-headers-5.0.0-15/scripts/basic/.gitignore
โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1 โ fixdep
โโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ File: /usr/src/linux-headers-5.0.0-15/scripts/kconfig/.gitignore
โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1 โ #
2 โ # Generated files
3 โ #
4 โ *.moc
5 โ *conf-cfg
6 โ
7 โ #
8 โ # configuration programs
9 โ #
10 โ conf
11 โ mconf
12 โ nconf
13 โ qconf
14 โ gconf
โโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ File: /usr/src/linux-headers-5.0.0-16/scripts/selinux/mdp/.gitignore
โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1 โ # Generated file
2 โ mdp
โโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ File: /usr/src/linux-headers-5.0.0-16/scripts/selinux/genheaders/.gitignore
โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1 โ genheaders
โโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ File: /usr/src/linux-headers-5.0.0-16/scripts/gdb/linux/.gitignore
โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1 โ *.pyc
2 โ *.pyo
3 โ constants.py
โโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ File: /usr/src/linux-headers-5.0.0-16/scripts/kconfig/lxdialog/.gitignore
โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1 โ #
2 โ # Generated files
3 โ #
4 โ lxdialog
โโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ File: /usr/src/linux-headers-5.0.0-15/scripts/selinux/genheaders/.gitignore
โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1 โ genheaders
โโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ File: /usr/src/linux-headers-5.0.0-15/scripts/selinux/mdp/.gitignore
โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1 โ # Generated file
2 โ mdp
โโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ File: /usr/src/linux-headers-5.0.0-15/scripts/gdb/linux/.gitignore
โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1 โ *.pyc
2 โ *.pyo
3 โ constants.py
โโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ File: /usr/src/linux-headers-5.0.0-15/scripts/kconfig/lxdialog/.gitignore
โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1 โ #
2 โ # Generated files
3 โ #
4 โ lxdialog
โโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ File: /usr/lib/nodejs/iconv-lite/generation/source-data/.gitignore
โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1 โ # This directory will contain temp files needed to create encoding data files.
2 โ
3 โ # Don't commit any files in this directory
4 โ *
5 โ
6 โ # Except this file
7 โ !.gitignore
โโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
BartG95
on 8 Jun 2019
I restarted my laptop, then removed fd-find (7.2.0) via apt, then installed fd (7.3.0) via cargo.
- All tests take ~20% longer
- The difference between
fd -HIandfdis very similar to the previous tests (including the difference between home-dir vs root-dir).
BartG95
on 8 Jun 2019
I also compared all variations of including -H and -I
````
$ hyperfine --warmup 3 'sudo fd -HI ".[0-9].jpg$" /' 'sudo fd -H ".[0-9].jpg$" /' 'sudo fd -I ".[0-9].jpg$" /' 'sudo fd "[0-9].jpg$" /'
Benchmark #1: sudo fd -HI ".[0-9].jpg$" /
Time (mean ยฑ ฯ): 1.315 s ยฑ 0.009 s [User: 3.819 s, System: 4.406 s]
Range (min โฆ max): 1.297 s โฆ 1.324 s 10 runs
Benchmark #2: sudo fd -H ".*[0-9].jpg$" /
Time (mean ยฑ ฯ): 1.725 s ยฑ 0.006 s [User: 4.915 s, System: 6.268 s]
Range (min โฆ max): 1.717 s โฆ 1.735 s 10 runs
Benchmark #3: sudo fd -I ".*[0-9].jpg$" /
Time (mean ยฑ ฯ): 1.265 s ยฑ 0.009 s [User: 3.641 s, System: 4.225 s]
Range (min โฆ max): 1.250 s โฆ 1.278 s 10 runs
Benchmark #4: sudo fd "[0-9].jpg$" /
Time (mean ยฑ ฯ): 1.638 s ยฑ 0.009 s [User: 4.539 s, System: 5.965 s]
Range (min โฆ max): 1.623 s โฆ 1.653 s 10 runs
Summary
'sudo fd -I ".[0-9].jpg$" /' ran
1.04 ยฑ 0.01 times faster than 'sudo fd -HI ".[0-9].jpg$" /'
1.30 ยฑ 0.01 times faster than 'sudo fd "[0-9].jpg$" /'
1.36 ยฑ 0.01 times faster than 'sudo fd -H ".*[0-9].jpg$" /'
````
BartG95
on 8 Jun 2019
So there is a slight win when hidden files are excluded:
- On fd vs fd -H: 1.638 vs 1.725: 5% faster
- On fd -I vs fd -HI: 1.265 vs 1.315: 4% faster
There is however a slight decrease in performance when ignored files are excluded:
- On fd vs fd -I: 1.638 vs 1.265: 29% slower
- On fd -H vs fd -HI: 1.725 vs 1.315: 31% slower.
BartG95
on 8 Jun 2019
If (complex) ignore files make such a difference, then why do I only see this when searching the root-dir and not when searching the home-dir?
All ignore files in my home-dir amount to ~2000 rules, those in my root-dir only 116.
When searching my root-dir, I also include my home-dir, so the number of rules only increase by ~5%.
BartG95
on 8 Jun 2019
Maybe it is not the ignore files themselves, but the amount of files that the ignore file applies to:
````
bart@:/usr/src/linux-headers-5.0.0-15
$ fd | wc -l
17439
bart@:/usr/src/linux-headers-5.0.0-15
$ fd -HI | wc -l
17450
bart@:/usr/src/linux-headers-5.0.0-15
$ hyperfine 'fd' 'fd -HI'
Benchmark #1: fd
Time (mean ยฑ ฯ): 36.5 ms ยฑ 4.2 ms [User: 158.8 ms, System: 89.0 ms]
Range (min โฆ max): 27.5 ms โฆ 46.0 ms 99 runs
Benchmark #2: fd -HI
Time (mean ยฑ ฯ): 28.3 ms ยฑ 2.5 ms [User: 135.6 ms, System: 50.8 ms]
Range (min โฆ max): 24.6 ms โฆ 34.6 ms 93 runs
Summary
'fd -HI' ran
1.29 ยฑ 0.19 times faster than 'fd'
````
BartG95
on 8 Jun 2019
This demonstrates pretty well that there is almost nothing that can be pruned from the search tree.
I think it's pretty clear that fd -I will be faster than fd in this case. The pure fd version needs to check for .gitignore, .fdignore, .ignore and .git/info/exclude files in every possible directory. There are about 5000 in my version of the linux headers.
We can check this by running strace -f -c (-f to follow all threads, -c to print statistics):
fd -I
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
27,84 0,329149 364 904 185 futex
21,31 0,251904 25 9776 getdents64
10,25 0,121188 24 4895 openat
9,93 0,117449 5 20239 write
9,79 0,115758 23 4889 4887 stat
9,79 0,115729 23 4895 close
9,65 0,114144 23 4894 fstat
fd
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
34,04 0,634243 25 24465 19559 openat
21,84 0,406844 286 1422 257 futex
13,95 0,259963 26 9776 getdents64
9,12 0,169883 8 20239 write
6,80 0,126617 25 4906 close
6,62 0,123370 25 4900 4898 stat
6,54 0,121889 24 4894 fstat
Notice the huge difference in openat syscalls for the pure fd command, including 19559 errors. That's in line with the 4 checks per directory.
sharkdp
on 10 Jun 2019
Thanks for you for the clarification.
For future generations: complex ignore files, or ingore files applied to a large file tree, can slow down fd a little (25%). In such a case you can use fd -I instead, with the implication that some files won't be ignored.
In any case, it is still way faster than find.
BartG95
on 10 Jun 2019
I'm going to reopen this ticket since there could potentially be a chance for improvement here.
This stack overflow answer seems to indicate that we could maybe speed up the .gitignore-existence-check by using stat first instead of directly opening the file (if the results can be transferred to our setup). In a typical setup, I would think that the number of directories with a .gitignore/.ignore/.rgignore/.. file is much smaller than the number of directories without such a file. If a file-existence check is really faster than directly trying to open the file, this would imply that it is worth doing if (file exists) { open(file) } instead of directly trying to open(โฆ) the file.
All of this would probably have to be done inside the ignore crate and not within fd itself.
sharkdp
on 10 Jun 2019
As for the latter point, see my PR here: https://github.com/BurntSushi/ripgrep/pull/1381
sharkdp
on 23 Sep 2019
My PR has been merged and the new ignore release is included in the latest version of fd. We should now see these performance improvements.
sharkdp
on 2 Apr 2020
Related issues
blueray453
ยท
4Comments
hungptit
ยท
3Comments
sharkdp
ยท
4Comments
sharkdp
ยท
3Comments
matu3ba
ยท
4Comments