Fd: Proposal: add `--filter <cmd>` option

[sharkdp]

This is a proposal to add a --filter <cmd>/-f <cmd> command-line option that would work very similar to the filter option in (functional) programming languages. It would be implemented in a similar way like --exec <cmd> and execute the given command for every path. Instead of showing the output of the command, it would filter the search results based on the exit status of the command (i.e. only include the ones which succeed with exit code 0).

In addition, we could also add a --reject <cmd>/-r <cmd> option that would do the opposite.

This would allow us to do things like:

# Show all C++ source files that contain the word TODO
fd -e cpp -e h -f grep TODO

# Show all entries that are NOT directories (this can not be done with `--type`)
fd -r test -d

# Only show files which exist and the current user has write permission
fd … test -w

# Only show files that are newer than 'reference.txt'
fd … -f test {} -nt reference.txt

# Only show files which have a diff with respect to the Git index
fd … -f git diff --exit-code

# Show all *.jpg files with a mime type that does NOT match image/jpeg
fd -e jpg -f sh -c '[ $(file -b --mime-type "{}") != "image/jpeg" ]'

I'd be happy to get some feedback on this idea or hear about other ways on how this could be used.

[tavianator]

Your last suggestion is vulnerable to shell injection, or just strange behaviour if you have filenames with strange characters. You could do

fd -e jpg -f sh -c '[ $(file -b --mime-type "$1") != "image/jpeg" ]' sh {}

instead to fix it.

[sharkdp]

Your last suggestion is vulnerable to shell injection

I was afraid it would be. Thank you for the hint. I didn't know that sh could be invoked with sh -c command_string [command_name [argument...]].

Unfortunately, the different levels of quoting and replacements which are involved here would probably keep users from actually running more complex shell (or other scripting language) commands with --filter <cmd>.

[tavianator]

@sharkdp Yeah it's a little complex. Maybe some syntax that does it for you would be nice, like

fd --filter-shell '[ $(file -b --mime-type "$1") != "image/jpeg" ]'

and you could do the sh -c '<command>' sh <file> dance for the user.

But having all of --exec, --exec-shell, --exec-batch, --exec-batch-shell, --filter, --filter-shell, etc. might be a bit much.

[crides]

I found this today when I was checking for mime type support in fd. Using shell filtering is definitely an interesting idea, but I would rather use the original find for that, as it would probably require more complex logics when you need to use other programs to check the file, which kind of contradicts fd's philosophy of "simplicity".

Another way to do it is just to use -x --exec itself as a predicate, like what find does.

[sharkdp]

I found this today when I was checking for mime type support in fd. Using shell filtering is definitely an interesting idea, but I would rather use the original find for that, as it would probably require more complex logics when you need to use other programs to check the file, which kind of contradicts fd's philosophy of "simplicity".

Thank you for the feedback. I tend to agree with you. In the meantime, I'm leaning towards not implementing this feature.

Another way to do it is just to use -x --exec itself as a predicate, like what find does.

You mean via something like --exec sh -c "condition && echo {}" (modulo quoting etc.)?

[crides]

You mean via something like --exec sh -c "condition && echo {}" (modulo quoting etc.)?

Yes, but only in terms of the actual result. What I meant is that the actual usage should only be --exec condition.... If you look into man find under the option -exec, you can see that the -exec command would return true or false base on the return value of the command it executes, and the return value can then be used in other logics to filter the files.

[tavianator]

fd's command line is not compositional, so there's not really a way to write "if --exec ... then print" like you can with find -exec ... \; -print. Printing by default would be annoying if you're interested in the output of the commands you're executing -- e.g. fd -x echo {} would print everything twice. I think manually writing -x sh -c '... && echo "$1"' sh is not too bad.

[Ordoviz]

This functionality can also be achieved in a Unix-y way with sor, for example:

fd --type file | sor 'grep --quiet TODO'

Since sor is not a common tool, a --filter option could still be useful.

[FranklinYu]

@Ordoviz sor is a very creative idea, avoiding the hassle for dealing with multiple predicates. It works well enough for grep and git diff cases, but for simple metadata tests (such as file permission and size) it is an order of magnitude slower, as mentioned in its ReadMe.

Actually I wish there is a better implementation of sor, targetting simple tests and do it in-process. Let’s call it test-filter; it should be called like

fd --type file | test-filter -w

This should only show files which exist and the current user has write permission, while only two processes are spawned. How about adding that binary in this project? It’s similar to walk and sor being in the same project; the output (coloring) part can be shared.