Hydra: [Launcher: Local parallel sweep]

[emilemathieutmp]

🚀 Feature Request

Is it doable to execute sweeps locally in parallel (with a variable like ntasks_per_node in config.yaml)?
Additionally, being able to specify a list of gpu indices and run the sweep in parallel on those (relying on the env var CUDA_VISIBLE_DEVICES) could be useful.

Motivation

Is your feature request related to a problem? Please describe.
The motivation is that without a slurm system configured, a simple parallel launcher would allow easier and faster computations than executing several times the same command.

Pitch

Describe the solution you'd like
A launcher object allowing a sweep in parallel.

Describe alternatives you've considered
I tried adapting the BasicLauncher using joblib but did not succeed so far.
One issue being that I get a Invalid plugin error because my launcher class does start neither with hydra_plugins nor with hydra._internal.core_plugins..

Additional context

Thanks fo the great library! :)

[omry]

Hi Emile, I am not planning currently to extend the basic sweeper for that because we are not far from the first launcher plugin that will support it using Ray.

See discussion around #152.

@BadrYoubiIdrissi , Will your Ray launcher plugin support local execution?
When are you planning to send a PR? :)

[omry]

@emilemathieutmp, by the way - I am happy to see you are still using Hydra :)
Please join the chat if you have general questions, I would love to hear what you are up to now.

[emilemathieutmp]

Thanks @omry for reaching back :)

I eventually succeeded in implementing a parallel Launcher based on joblib (can be found here)

I still have a few issues:

• How to avoid the Invalid plugin error because my launcher class does start neither with hydra_plugins nor with hydra._internal.core_plugins. ?
• I don't know why but the OmegaConf cannot resolve an inter_type now at some point which I hard fixed with something like value = datetime.datetime.now().strftime(inter_key).

If you have any hint that'd be great! :)

[omry]

Great!

• Currently I require that Hydra plugins are in hydra_plugins module. This allows for automatic discovery of installed plugins.
• Hard to say, will have to dig deeper to understand. this kind of area is a bit rough around the edges and require attention to details.
  If you can come up with a reproducible example I might have time to take a look.

If you are willing to spend the time to make this generic and tested, I can consider making it an official plugin

[omry]

Closing this as you have a solution, feel free to send a PR with the plugin.

[omry]

The new plugin is published. It supports Hydra 1.0.0 which is not yet released. you can try it by checkout out Hydra from master and installing the plugin with pip install.

Plugin website page.