Deno: Deno fmt "error: Too many open files (os error 24)"

[caspervonb]

This was fixed in the dprint cli a while back (used to happen with tools/format.py right when we switched) but seem this is still an issue with deno fmt.

$ deno fmt
/Users/caspervonb/Projects/deno/std/http/testdata/simple_https_server.ts
/Users/caspervonb/Projects/deno/std/http/testdata/file_server_as_library.ts
/Users/caspervonb/Projects/deno/std/hash/_wasm/build.ts
/Users/caspervonb/Projects/deno/std/hash/_wasm/wasm.js
error: Too many open files (os error 24)

Bumped the limit up to 1000 here.

[caspervonb]

Remember any of the details here @dsherret?

[dsherret]

This might have to do with deno fmt using task::spawn_blocking:

Tokio will spawn more blocking threads when they are requested through this function until the upper limit configured on the Builder is reached. This limit is very large by default, because spawn_blocking is often used for various kinds of IO operations that cannot be performed asynchronously. When you run CPU-bound code using spawn_blocking, you should keep this large upper limit in mind; to run your CPU-bound computations on only a few threads, you should use a separate thread pool such as rayon rather than configuring the number of blocking threads.

So it's probably creating a large number of threads instead of just using a few.

[magurotuna]

@caspervonb Are you on macOS? What input do you get when running ulimit -n?

I guess that a naive solution is to set a limit of the number of spawned threads via tokio::runtime::Builder::max_threads.
Another solution may be making I/O operations asynchronous using the following traits:

• tokio::io::AsyncReadExt
• tokio::io::AsyncWriteExt

Making I/O operations asynchronous would allow tokio to run them in the core threads, whose number is equal to the number of CPU cores by default, instead of the blocking threads. So opening too many files at the same time would be avoided.

references:

• tokio - cpu-bound-tasks-and-blocking-code
• https://github.com/tokio-rs/tokio/issues/2309

[dsherret]

Yeah, he’s on Mac.

I think we just need to use a thread pool. It would be really easy if we used Rayon as per Tokio’s recommendation and that would simplify the code a lot. I’ll open up a PR tomorrow (today).