Hydra: [Feature Request] [Submitit] [array_parallelism]

[emilemathieu]

🚀 Feature Request

Hi all!

Would there be a way to limit the number of map tasks that can be executed in parallel?

In submitit one can use the parameter slurm_array_parallelism as in the job array example.
Passing additional_parameters: {"array_parallelism": 2} does not work since it's not a slurm argument.

Cheers,
Emile

[omry]

This sounds like a nice addition, want to try to put out a pull request?

[emilemathieu]

Would the idea to add in SlurmQueueConf a parameter array_parallelism: int = 256?

[jieru-hu]

Would the idea to add in SlurmQueueConf a parameter array_parallelism: int = 256?

Yes, looks like a good place to add the config :)

[shagunsodhani]

Closed via https://github.com/facebookresearch/hydra/pull/1137