Dune: Detect core numbers and set default concurrent job numberes

[objmagic]

I am wondering if we can set default concurrent job numbers based on number of cores detected.

Or is it recommended to use jobs opam variable ("jbuilder" "build" "-p" name "-j" jobs)? But it seems opam's max job numbers is set only if it is compiled from source code.

I am happy to send a PR if maintainers think this change is good to have.

[jeremiedimino]

Well, for opam, I think opam should definitely be in control of the number of jobs since it might build multiple packages in parallel. One idea was to have opam know a bit more about Dune, so that it can build multiple Dune packages at once and get maximum parallelism (/cc @avsm @altGr).

That said, we'd be happy to set the default based on the number of cores for non-opam builds.

[AltGr]

"jbuilder" "build" "-p" name "-j" jobs

Many have adopted this, but it's actually not really consistent: the jobs variable is set from the value of opam's own --jobs parameter, which is the number of processes __opam__ will run in parallel. I am not sure how to improve, though, should we make them two different settings ?

By default, opam initialises the value with (num cores - 1)

[AltGr]

But it seems opam's max job numbers is set only if it is compiled from source code.

Not sure I get what you mean ?

[jeremiedimino]

Many have adopted this, but it's actually not really consistent: the jobs variable is set from the value of opam's own --jobs parameter, which is the number of processes opam will run in parallel. I am not sure how to improve, though, should we make them two different settings ?

Oh... I thought opam would split this number between the processes that it ran concurrently. In practice, what can individual packages do with the jobs value that was passed to opam itself?

[objmagic]

@AltGr For example, I have 24 cores on my machine. I installed opam binary and jobs field in ~/.opam/config is 4. Then opam cannot build more than 4 jobs in parallel until I change this field to 24.

[objmagic]

Many have adopted this, but it's actually not really consistent

Yes I am confused as well. Two "jobs" here have different meaning.

[objmagic]

@AltGr I just tried opam 2.0 rc, and jobs numbers are initialized to 23 after binary installation. This looks good. But with opam 1.2 binary installation, jobs are initialized to 4.

[AltGr]

For example, I have 24 cores on my machine. I installed opam binary and jobs field in ~/.opam/config is 4. Then opam cannot build more than 4 jobs in parallel until I change this field to 24.

Yes. So following @diml's comment, an idea would be to initialise the jobs variable, as visible by packages, to num_cores / OPAMJOBS.
So running opam install foo -j 4 on a 24-core machine would build at most 4 packages concurrently, and pass them a jobs=6 variable, resulting typically in -j 6.

[AltGr]

@objmagic indeed, the number was fixed in 1.2 IIRC.
If we follow the idea in my last post, we should probably change the default value for OPAMJOBS, though, unless having OPAMJOBS=num_cores - 1 and make jobs = 1 is a good compromise ?
E.g. on a typical 6-core desktop, I'd probably prefer OPAMJOBS=3 and make jobs = 2.

Another possibility would be to have opam dynamically compute the make jobs depending on the max numbers of package builds that may run concurrently when the build starts (that would e.g. allow ocaml to build on all cores)

[jeremiedimino]

Another possibility would be to have opam dynamically compute the make jobs depending on the max numbers of package builds that may run concurrently when the build starts (that would e.g. allow ocaml to build on all cores)

That's seems good indeed. There are even ways to make the number of jobs dynamic while the command is running, such as with GNU make jobserver. Though that's probably a lot of work...

[UnixJunkie]

OPAMJOBS should default to num_cores, not num_cores - 1.
If people are not happy with that, they should 'nice opam ...'.

[UnixJunkie]

@AltGr the standard meaning of the -j N option in unix CLI tools is "use at max N cores".
Please don't change that.
opam should control the number of cores which are allowed to be used by each build so that the computer
doesn't become overloaded.
Using all the cores at once is not overloading, this is maximum efficiency, hence I don't like the num_cores - 1, this might make me wait longer, for no good reason.

[AltGr]

num_cores - 1 was a request by @avsm :angel:. Apparently macos has some responsivity issues under load ?
I actually use 6 for 4 cores, accounting for disk accesses and all, it works well...

[AltGr]

@UnixJunkie so with OPAMJOBS set to num_cores, you would have a default for the jobs variable to 1 ?
(Unless we use the dynamic setting, but that would be tricky and certainly won't be for 2.0)

[UnixJunkie]

At least, with a default of 1, there is no risk to overload a machine.

[UnixJunkie]

If the dynamic setting is with a proper DAG and maximizing the throughput at which opam will build and install things, I am eager to see that in action.

[AltGr]

parallel processing of actions is based on a DAG ; but it would require a feedback from the scheduler back up to the layer of package action processing to implement that dynamic variable, which might not be doable with a slight interface change...

[AltGr]

Sorry, getting back to this a little late

@AltGr the standard meaning of the -j N option in unix CLI tools is "use at max N cores".
Please don't change that.

With this in mind, we should have (where OPAMJ is opam's -j argument value, opamjobs the number of parallel build jobs opam may spawn, and makejobs the value of the jobs variable passed to package builds):

1. opamjobs = min(OPAMJ, num_cores) and makejobs = max(1, OPAMJ / opamjobs) rather than
2. opamjobs = OPAMJ and makejobs = max(1, num_cores / opamjobs) as was proposed earlier.

In case 1, you specify the maximal total number of jobs, and opam will never spawn more processes than cores, and provide remaining ressources allowed by -j to the subprocesses.

In case 2, you specify the number of jobs opam can spawn directly with -j, and ressources allowed to each process are determined by the number of cores remaining.

Not sure which is the most obvious to the user ? The first seems to have the benefit that if I want *2 overload, I can specify -j $((2* num_cores)), and have opam spawn num_cores processes with -j 2 each, while the second limits makejobs to 1 as soon as opam is allowed to use all cores.

[bobot]

There are even ways to make the number of jobs dynamic while the command is running, such as with GNU make jobserver. Though that's probably a lot of work...

Supporting the protocol of make jobserver is not very hard. The only difficulty is to be able to wait for its children and read the pipe at the same time.

[jeremiedimino]

The only difficulty is to be able to wait for its children and read the pipe at the same time.

I suppose we could just use system threads

[jeremiedimino]

BTW, independently of the opam discussion, I think we should also make the default number of jobs be auto-detected, so that when we type jbuilder in the terminal we get good performances without having to write -j ... or write a configuration file. I googled a bit and it seems that to detect the number of cpus the process can use one should try in order the following commands:

• nproc
• getconf _NPROCESSORS_ONLN
• getconf NPROCESSORS_ONLN

and use the environment variable NUMBER_OF_PROCESSORS on Windows.

[UnixJunkie]

I don't think -j by default is a good idea.

On 31 Mar 2018, at 12:11, Jérémie Dimino [e-mail] wrote:

BTW, independently of the opam discussion, I think we should also make the default number of jobs be auto-detected, so that when we type jbuilder in the terminal we get good performances without having to write -j ... or write a configuration file. I googled a bit and it seems that to detect the number of cpus the process can use one should try in order the following commands:

nproc
getconf _NPROCESSORS_ONLN
getconf NPROCESSORS_ONLN
and use the environment variable NUMBER_OF_PROCESSORS on Windows.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[UnixJunkie]

@AltGr maybe this can be useful to opam https://github.com/let-def/ocamp

[jeremiedimino]

I don't think -j by default is a good idea.

Why not? The current default is -j 4 BTW, not -j 1.

[UnixJunkie]

For such behaviour, I think the unix-way is to have a .rc file (like opam's ~/.opam/config for example).

[rgrinberg]

@UnixJunkie we already have such an .rc file. It should be dune/config in your xdg dir. This should be somewhere in the docs btw, I'll double check that this info is easy to find.

[jeremiedimino]

Yh, basically since we added the configuration file, I keep on writing a ~/.config/dune/config file with (jobs ...) on every machine I work on in order to get faster builds. This makes me think that using the maximum number of jobs available is a better default.

I think the main reason make uses -j1 by default is because most makefiles don't support parallel builds. We don't have this issue with Dune.

[UnixJunkie]

OK, pigz (parallel gzip) has its default set to the number of online processors. Maybe that's the way of the future.

[OvermindDL1]

I think the main reason make uses -j1 by default is because most makefiles don't support parallel builds.

Precisely.

This is also why build systems like ninja do default to multi-core building as their dependency resolution ensures separation between parts that don't mix (as an aside, the BS project has made their build system around ninja and it is by far the fastest ocaml build system out that I've seen, ninja is also the fastest I've seen in the C++ world, where I use it routinely because of that).

[jeremiedimino]

This is implemented in master