Hyperopt: Show parameter value along with best loss

[VinodKumar9576]

Is there any functionality to show parameter values in the progress bar itself for each evaluation along with best loss as shown below.
If you can see, my hyper parameter is 'K' and i want each K value to show in each evaluation along with loss.

@bjkomer can you please help me on this.

[bjkomer]

I don't think there is an easy built-in way of doing that right now. You can pass your own callback as the show_progressbar parameter, but there doesn't look to be a clean way of getting that information to the callback.

The easiest workaround would be to write a simple progress bar yourself, that way you can format it any way you want. Something like this:

from hyperopt import hp, tpe, fmin, Trials
from tqdm import trange
import hyperopt

def objective(x):
    return {'loss':x**2,'status':hyperopt.STATUS_OK}

space = hp.uniform('K', -10, 10)

max_evals = 100

trials = Trials()
with trange(max_evals) as t:
    for e in t:
        best = fmin(fn=objective, space=space, algo=tpe.suggest, max_evals=e+1,
                   trials=trials, show_progressbar=False)
        t.set_postfix(K=best['K'], best_loss=min(trials.losses()))

[VinodKumar9576]

Thank you!

[marload]

@maxpumperla @marctorsoc

Personally, I think this feature will help HyperOpt. What do you think?

[VinodKumar9576]

Yes it will, when we make function and it errors out due to some issue after hyper opt found the best values, we have to run the algo again as the function failed to return the best values. If those values got printed as default, we can use them without having to run the algo again.

[marctorsoc]

Yes it will, when we make function and it errors out due to some issue after hyper opt found the best values, we have to run the algo again as the function failed to return the best values. If those values got printed as default, we can use them without having to run the algo again.

I'm not sure why there would be an error between "finding the best values" and "returning them". The steps there seem quite robust

This looks great for one parameter, but can become quite cumbersome if you have a grid of HPs. I would lean towards a log message (with default=false) after each trial, and logging only when a "new best" is found. There you can show the loss, some extra metrics if you want to, and the point in the search space for the best. Potentially, I would add some patience so that it's not too noisy for the first N trials. Otherwise I'm not sure how useful for the user this would be