Node: Startup time optimization - specifying location of node_modules

[ORESoftware]

I am guessing that node.js startup could be optimized if we specify the (only) location of node_modules. I believe that the fs gets peppered with stat requests on startup, but if we had something like this:

$ node --nm="./node_modules" dist/main.js

then when non-relative require statements were evaluated, they could jump to the location specified by --nm, this might save hundreds of stat calls on startup?

I would venture to guess that 95% of npm projects only have one relevant node_modules dir, so walking up the filesystem for nested files could be optimized.

I would guess that there may already be some optimization for this in place, where directories are marked as not having a node_modules dir within them on first pass, but maybe there's more optimization to be had?

[DiegoRBaquero]

This sounds nice. Have you done any benchmarking with a possible modified version allowing this?

[ORESoftware]

@DiegoRBaquero I am guessing it would on average only save ~5-10ms on startup, but that would be a nice boost, for lambda functions in tha cloud etc.

It depends on the current situation, if for every require() call, it walks up the fs looking for node_modules in each dir, then that would be a lot of stat calls, and we could really optimize that. But if there is a system in place that caches the location of node_modules, then it would just read that first before looking in the fs.

I am guessing that currently it will create a hash of filepaths and if the filepath exists in the hash already, it will know if node_modules exists there or not. However one problem with that technique is one package might exist in that node_modules dir, but another might be further up the tree, so maybe it's very unoptimized atm.

[devsnek]

related https://github.com/bahmutov/cache-require-paths

[ORESoftware]

@devsnek yeah the cache-require-paths package is neat, but it only works on the second run, not the first. I think it would be nice to have something work on the first execution. also a bit tricky when dependencies shift locations in node_modules even if package.json doesn't change. Maybe there is a way to get a hash of the contents in node_modules, but that would take a lot of computational effort lol, so prohibitive.

[TimothyGu]

This could also be tied to the import maps proposal, which could have much of the same effect.