Pkg.jl: Limit how many threads used during precompile

[Moelf]

ran into this error today on a Linux machine with 2 CPUs, 28 cores, reporting 56 threads.

(@v1.6) pkg> precompile
┌ Error: Pkg.precompile error
│   exception =
│    IOError: could not spawn `....julia-1.6.0-rc1/bin/julia -Cnative -Jjulia-1.6.0-rc1/lib/julia/sys.so -g1 -O0 --output-ji .....julia/compiled/v1.6/MacroTools/jl_KkrSks --output-incremental=yes --startup-file=no --history-file=no --warn-overwrite=yes --color=yes --eval 'eval(Meta.parse(read(stdin,String)))'`: resource temporarily unavailable (EAGAIN)
│    Stacktrace:
│      [1] _spawn_primitive(file::String, cmd::Cmd, stdio::Vector{Any})
│        @ Base ./process.jl:99
│      [2] #639
│        @ ./process.jl:112 [inlined]
│      [3] setup_stdios(f::Base.var"#639#640"{Cmd}, stdios::Vector{Any})
│        @ Base ./process.jl:196
│      [4] _spawn

└ @ Pkg.API /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:1012

This is on a shared system with many people. I personally would think it makes sense to add a limit so that CPU usage during precompile is also bounded by <= JULIA_NUM_THREADS since if the user doesn't want to occupy too much resource during computation, they likely don't want to during precompile. cc @IanButterworth thanks for the help on Slack.

[IanButterworth]

To elaborate a little from the slack discussion.

There is always ENV["JULIA_NUM_PRECOMPILE_TASKS"] which can be set, which @Moelf reported fixed the problem when set to 28.

[Moelf]

That solves the problem for me but I think in general:

1. precompile error can be intimidating -- the user can feel they haven't even started doing anything yet.
2. even if it didn't error, it can silently occupy too many resources on a shared system.
3. the total precompile time is limited by the deepest dependency which in the end is just the tail of 1-2 pkgs running for a few more levels. I don't think 15 vs. 20 threads give much improvement over all.

[IanButterworth]

I'm far from knowledgeable on this kind of thing, but perhaps Sys.CPU_THREADS should be taught to return a more meaningful number, if it's not paying attention to some sys admin enforced limit. That would fix the auto task count

[StefanKarpinski]

Sys.CPU_THREADS by definition gives the number of logical threads (with hyperthreading), not physical cores (without hypterthreading). Long discussion in https://github.com/JuliaLang/julia/issues/33409 of adding Sys.CPU_CORES which gives the number of physical cores instead of logical threads and using that for the number of OpenBLAS threads. Of course that got bogged down in how to implement detecting the number of physical cores... Maybe the safest thing to do here is just assume that there is two-way hyperthreading and use Sys.CPU_THREADS ÷ 2? Either that or just have some reasonable cap. Capping concurrent precompilation at 16 or 32 or something like that seems reasonable.

[IanButterworth]

Because of https://github.com/JuliaLang/Pkg.jl/issues/2323 we already limit on Windows to (Sys.CPU_THREADS ÷ 2) + 1 as of https://github.com/JuliaLang/Pkg.jl/pull/2366
I don't think we need to do the same on other platforms. I haven't seen issues like that reported since we did that fix.

Though adding a hard limit of 16 does seem reasonable. I share @Moelf's experience that the long dependency tail is the thing that mainly affects the duration.

Plus, users on unusual machines with many cores that do actually want to throw everything at it could override it via setting ENV["JULIA_NUM_PRECOMPILE_TASKS"]

If anything, there's only so many visible rows in a terminal window.. More than 16 active UX elements would be a bit overwhelming.