Fd: Excessive memory usage.

[WloHu]

I've run fd -Ht d '3493' on directory which contains 92GB of Jenkins backup files to find logs for given build number.

I would consider memory usage of fd an insanity not exactly because of the amount (32% of 8GB RAM) but the fact that memory isn't freed during run. For comparison find -type d -name '*3493*' works slower but at least it doesn't eat up more and more memory during run.

I also found out that I'm not the only one who reported increasing memory usage https://github.com/sharkdp/fd/issues/378.

[sharkdp]

Thank you for your feedback.

If possible, could you please run rg --files on that folder and see if it suffers from a similar problem? If so, this is an inherent behavior of the ignore crate. If not, it could be a bug in fd.

[jonashaag]

I see the same behaviour in rg --files

[jonashaag]

Here's an example of a deeply nested directory structure where you can test it on. https://www.dropbox.com/s/i95knl4w60dd48o/files.tar.bz2?dl=0

Be aware it contains a giant amount of files and directories and will take multiple minutes to extract and even more minutes to delete :-)

[sharkdp]

@jonashaag Thank you very much for the feedback and for providing a reproducible test case (I haven't tested it so far).

If this is indeed a bug, it should probably be reported upstream.

[WloHu]

I will also try to test it and let you know about the results. If that's a rg issue I will report it and link this issue.

[sharkdp]

@WloHu Any update on this?

[data-man]

Maybe jwalk is better.

[WloHu]

Sorry. I forgot about this. Should I still inform fd devs or test the issue?

[sharkdp]

Similar to #378, this has been (unexpectedly) fixed since Rust 1.36 came out. If I compile fd 7.4.0 with Rust 1.35, I get a peak memory usage of 221 MB when searching the test directory provided by @jonashaag:

❯ /usr/bin/time -f "%M" fd non-existing files2
221160

If I compile it with Rust 1.36 or newer, peak memory consumption drops significantly to 16 MB:

❯ /usr/bin/time -f "%M" fd non-existing files2
15928

This is still true for the latest version of fd.