Hyperopt: simple multiprocess enhancement for speedup

[zhuoqiang]

hyperopt could use mongodb as backend for multiprocess concurrent search, however it is a little complex to setup.

I am wondering if the basic trial could take advantage of python's build-in multiprocess to speedup.

More specific, is it possible to run the following loop in parallel using a multiprocess pool:

https://github.com/hyperopt/hyperopt/blob/master/hyperopt/fmin.py#L82-L111

        for trial in self.trials._dynamic_trials:
            if trial['state'] == base.JOB_STATE_NEW:
                trial['state'] == base.JOB_STATE_RUNNING
                now = coarse_utcnow()
                trial['book_time'] = now
                trial['refresh_time'] = now
                spec = base.spec_from_misc(trial['misc'])
                ctrl = base.Ctrl(self.trials, current_trial=trial)
                try:
                    result = self.domain.evaluate(spec, ctrl)
                except Exception as e:
                    logger.info('job exception: %s' % str(e))
                    trial['state'] = base.JOB_STATE_ERROR
                    trial['misc']['error'] = (str(type(e)), str(e))
                    trial['refresh_time'] = coarse_utcnow()
                    if not self.catch_eval_exceptions:
                        # -- JOB_STATE_ERROR means this trial
                        #    will be removed from self.trials.trials
                        #    by this refresh call.
                        self.trials.refresh()
                        raise
                else:
                    trial['state'] = base.JOB_STATE_DONE
                    trial['result'] = result
                    trial['refresh_time'] = coarse_utcnow()
                N -= 1
                if N == 0:
                    break
self.trials.refresh()

[jaberg]

I doubt it. You have to manage some shared database to gather together the
results. The database could be in-memory Python objects, but you'd still
have to add functionality to hyperopt so that the multiprocessing
subprocesses contribute their results back to a single process / memory
space that contains the database. I don't recall that the default Trials
class does anything like this.

On Sat, Jan 14, 2017 at 9:37 PM, ZHUO Qiang [e-mail]
wrote:

hyperopt could use mongodb as backend for multiprocess concurrent search,
however it is a little complex to setup.

I am wondering if the basic trial could take advantage of python's
build-in multiprocess to speedup.

More specific, is it possible to run the following loop in parallel using
a multiprocess pool:

https://github.com/hyperopt/hyperopt/blob/master/hyperopt/fmin.py#L82-L111

    for trial in self.trials._dynamic_trials:
        if trial['state'] == base.JOB_STATE_NEW:
            trial['state'] == base.JOB_STATE_RUNNING
            now = coarse_utcnow()
            trial['book_time'] = now
            trial['refresh_time'] = now
            spec = base.spec_from_misc(trial['misc'])
            ctrl = base.Ctrl(self.trials, current_trial=trial)
            try:
                result = self.domain.evaluate(spec, ctrl)
            except Exception as e:
                logger.info('job exception: %s' % str(e))
                trial['state'] = base.JOB_STATE_ERROR
                trial['misc']['error'] = (str(type(e)), str(e))
                trial['refresh_time'] = coarse_utcnow()
                if not self.catch_eval_exceptions:
                    # -- JOB_STATE_ERROR means this trial
                    #    will be removed from self.trials.trials
                    #    by this refresh call.
                    self.trials.refresh()
                    raise
            else:
                trial['state'] = base.JOB_STATE_DONE
                trial['result'] = result
                trial['refresh_time'] = coarse_utcnow()
            N -= 1
            if N == 0:
                breakself.trials.refresh()

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/hyperopt/hyperopt/issues/282, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAKdDA9KUMvZWcdiJa3x65kK8Kluk-aYks5rSYaBgaJpZM4LjznR
.

[jaberg]

Consider looking at the Trials class, and the MongoTrials class and adding
a new one. It would a be a significant job.

On Mon, Jan 16, 2017 at 8:01 AM, James Bergstra [e-mail]
wrote:

I doubt it. You have to manage some shared database to gather together the
results. The database could be in-memory Python objects, but you'd still
have to add functionality to hyperopt so that the multiprocessing
subprocesses contribute their results back to a single process / memory
space that contains the database. I don't recall that the default Trials
class does anything like this.

On Sat, Jan 14, 2017 at 9:37 PM, ZHUO Qiang [e-mail]
wrote:

hyperopt could use mongodb as backend for multiprocess concurrent search,
however it is a little complex to setup.

I am wondering if the basic trial could take advantage of python's
build-in multiprocess to speedup.

More specific, is it possible to run the following loop in parallel using
a multiprocess pool:

https://github.com/hyperopt/hyperopt/blob/master/hyperopt/fm
in.py#L82-L111

    for trial in self.trials._dynamic_trials:
        if trial['state'] == base.JOB_STATE_NEW:
            trial['state'] == base.JOB_STATE_RUNNING
            now = coarse_utcnow()
            trial['book_time'] = now
            trial['refresh_time'] = now
            spec = base.spec_from_misc(trial['misc'])
            ctrl = base.Ctrl(self.trials, current_trial=trial)
            try:
                result = self.domain.evaluate(spec, ctrl)
            except Exception as e:
                logger.info('job exception: %s' % str(e))
                trial['state'] = base.JOB_STATE_ERROR
                trial['misc']['error'] = (str(type(e)), str(e))
                trial['refresh_time'] = coarse_utcnow()
                if not self.catch_eval_exceptions:
                    # -- JOB_STATE_ERROR means this trial
                    #    will be removed from self.trials.trials
                    #    by this refresh call.
                    self.trials.refresh()
                    raise
            else:
                trial['state'] = base.JOB_STATE_DONE
                trial['result'] = result
                trial['refresh_time'] = coarse_utcnow()
            N -= 1
            if N == 0:
                breakself.trials.refresh()

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/hyperopt/hyperopt/issues/282, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAKdDA9KUMvZWcdiJa3x65kK8Kluk-aYks5rSYaBgaJpZM4LjznR
.

[mrocklin]

Just to throw another hat in the ring, Dask's distributed scheduler might be an easy way to both distribute work and to coordinate parameters.

http://distributed.readthedocs.org

[superaja]

Hello -

Following up on mrocklin's comment, can dask be considered for hyperopt (fmin) parallelization ?

Thanks so much!

AD

[richardliaw]

@superaja Maybe instead of the default HyperOpt parallelism mechanism (or dask), try using HyperOpt with Tune?

Tune can use HyperOpt to search but manages the asynchronous parallel execution for you and works with any learning framework.

from hyperopt import hp
import ray
from ray import tune
from ray.tune.suggest import HyperOptSearch


def train_model(config):
    model = Model(config)  # create model with Hyperparameters
    for i in range(100):
        accuracy = model.train()
        tune.report(mean_accuracy=accuracy)

space = {
    'width': hp.uniform('width', 0, 20),
    'height': hp.uniform('height', -100, 100),
    'activation': hp.choice("activation", ["relu", "tanh"])
}
# this wraps hyperopt TPE
algo = HyperOptSearch(space, max_concurrent=32, metric="neg_mean_loss")
tune.run(train_model, num_samples=10, stop={"training_iteration": 100}, search_alg=algo)