lsd should run asynchronously/as a stream/as a generator

[Berrysoft]

lsd now runs synchronously, that is, it reads all the files and directories metadata into memory and outputs them. If the directory contains many files, it costs much more time to run lsd --tree than tree.

[Peltoche]

Hi @Berrysoft ,

The synchronicity have been a recent a design choice. The goal was to try to keep the code easy to understand in order to make the contributions easier. The first versions was not asynchronous but was working as a stream. Back then the code was complicated and several features like the file ordering was not possible so we probably loose some few micro-seconds but I think it was worth it.

That said, if you have some propositions for an asynchronous/stream architecture which would make the codebase not too complicated. We are interested.

[Berrysoft]

I'm not very familiar with Rust, but it seems that Rust supports generator, with keyword yield. Have you tried it?

[jeffw387]

I can imagine that if you didn't care about sorting, you could fetch matches and display them one by one as they became available. It seems like displaying sorted output would resynchronize at the sort point, making it much less meaningful there. What kind of scale of benefits might we expect from this enhancement?

I did some very unscientific comparisons between tree and lsd --tree and it did seem like tree started to display output before lsd --tree, but they both finished fairly quickly for my /usr dir. Without proper benchmarking I can't really distinguish which is finishing first.

For my personal use case if there's a long list of files I'm usually scrolling to review them after the output is finished as opposed to watching them go by during initial output since the initial output moves so quickly.