Ludwig: hyper parameter search using RAY

[BenMacKenzie]

it would be nice to use a tool like RAY to search over hyper-parameters and elements of model architecture. I have a simple prototype that involves annotating the search space directly in the model yaml file. However.... i had to comment out this statement on line 388 in model.py

signal.signal(signal.SIGINT, self.set_epochs_to_1_or_quit)

I think the problem is related to the fact that RAY will build and train each permutation of the model outside in a separate thread and signal handlers in python must be run in main thread (I am not a python expert)

Just food for thought. It looks like the signal handler allows for more graceful shutdown in the event of a failure, which i can live without.

[w4nderlust]

I can't live without it unfortunately :) It's really useful in many cases when you run experiment and you want the prediction results to be generated.

That said, having an integration with some hyperparameter search tooling would be really cool, are you considering contributing it? In that case we can try to find a clean solution. One option could be a simple flag that, if on, would remove the assignment of the sigint signal, and in the case ray is used, it would be on.

[BenMacKenzie]

ok..a switch would be perfect. Will definitely contribute the hyper parameter search back.

[BenMacKenzie]

i can share a jupyter notebook that illustrates a very basic approach.

[w4nderlust]

That would a great starting point!

[BenMacKenzie]

ray_search.zip

hope this helps!

[marvin-hansen]

Hyperparameter optimization can be computationally expensive. My last experiments with Bayesian optimization took quite some time. While searching for a way to accelerate things, I stumbled across BOHB that combines bandit-based methods with Bayesian optimization.

Essential to the bandit method is the "successive halving" technique, that is, after running all of a set of pipelines for a small budget, it drops the worse half of the pipelines and doubles the budget of the remaining pipelines.

The combined approach reduces computational time by one magnitude when used on a single machine, but more excitingly, it scales across many GPU nodes and their experiments show 2 - 3 magnitudes of performance improvements while required a lot less GPU's less compared to other techniques.

I am not able to contribute code because of my ongoing project and other commitments, but it would be great if anyone can take a closer look into this area to determine whether adding BOHB as high-performance hyper-parameter optimizer or model generator is feasible and makes sense.

Repo:
https://github.com/automl/HpBandSter

Paper:
https://ml.informatik.uni-freiburg.de/papers/18-ICML-BOHB.pdf

Blog:
https://www.automl.org/blog_bohb/

"Successive Halving"
https://www.automl.org/blog-2nd-automl-challenge/

Auto-Pytorch
https://github.com/automl/Auto-PyTorch

[w4nderlust]

ray_search.zip

hope this helps!

Getting back to this after a long time. We implemented a hyperparameter optimization component. it doesn't yet itnegrate with Ray, but that integration will be our next step.
Thank you for sharing your example!