Node: readdir should have {type:'file'} option - should allow user to only retrieve files not folders etc

[ORESoftware]

With this SO question in mind:

https://stackoverflow.com/questions/51333163/nodejs-readdir-only-find-files/51333279

I know that readdir is some c utitlity on Unix. It would be possibly more efficient for users to pre-filter with readdir instead of running lstatSync or statSync on each item from the array.

Is it possible for the lower level routines to filter things out efficiently before sending back?

[devsnek]

we should be able to filter anything on this list so seems reasonable imo

typedef enum {
    UV_DIRENT_UNKNOWN,
    UV_DIRENT_FILE,
    UV_DIRENT_DIR,
    UV_DIRENT_LINK,
    UV_DIRENT_FIFO,
    UV_DIRENT_SOCKET,
    UV_DIRENT_CHAR,
    UV_DIRENT_BLOCK
} uv_dirent_type_t;

edit: even better, we could just pass back { name, type }

[silverwind]

Note that scandir would provide similar functionality, if it is ever implemented.

[ORESoftware]

Related question - https://github.com/nodejs/help/issues/1471

it would be nice if we stream the items in a folder, instead of reading them all at once, useful for very large directories.

[ORESoftware]

in the meantime, I suppose find will work on Unix/MacOS, but the purpose of a core Node.js utility would be more x-platform-ness.

const rl = require('readline');
const cp = require('child_process');

const k = cp.spawn('bash');
k.stdin.end(`find . -maxdepth 1 -type f`);

rl.createInterface({
  input: k.stdout,
})
.on('line', v => {
    // new file path
});

[silverwind]

I consider this one fixed as of https://github.com/nodejs/node/pull/22020 which provides a efficient way of reading files without any additional stat calls involved. Benchmarks show it's about twice to thrice as fast as readdir with stat. The API might not be as nice as you might have hoped, but it can be wrapped in a userland module pretty easily.

[silverwind]

Oh, and forgot to mention it, but this new withFileTypes should become available in the next minor release 10.10.

[ORESoftware]

@silverwind nice, but what would be nice is to only retrieve files or dirs, but yeah I guess for userland you can do that for the user, but say you have a folder with 10,000 files and 3 folders and you only want the folders or whatever, sucks that you have to read in 10,003 things.

[silverwind]

I'm not aware of a more efficient low-level method for your case. It's just how file systems work, often times, you have to crawl them.