Hyperopt: Accuracy Difference is high between Hyperopt and GridSearch?

[hanzigs]

Hi,
Working on a Kaggle Competition on Mechanisms of Action (MoA) Prediction

Initially used GridsearchCV for hyperparameters for XGBoost and i got

XGB_Hyperparamerters = {
    'n_estimators' : [100,200,300,400,500,600,700,800,900,1000],
    'eta' : [0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0],
    'gamma' : [1,2,3,4,5,6,7,8,9,10],
    'max_depth' : [3,4,5,6,7,8,9,10],
    'min_child_weight' : [1,2,3,4,5,6,7,8,9,10],
    'subsample' : [0.5,0.6,0.7,0.8,0.9,1.0],
    'colsample_bytree' : [0.5,0.6,0.7,0.8,0.9,1.0],
    'colsample_bylevel' : [0.5,0.6,0.7,0.8,0.9,1.0],
    'colsample_bynode' : [0.5,0.6,0.7,0.8,0.9,1.0],
    'max_leaves' : [1,2,3,4,5,6,7,8,9,10] 
}  
best_hyperparameters = {'n_estimators': 700,
 'eta': 0.2,
 'gamma': 3,
 'max_depth': 5,
 'min_child_weight': 5,
 'subsample': 0.9,
 'colsample_bytree': 0.6,
 'colsample_bylevel': 0.8,
 'colsample_bynode': 0.8,
 'max_leaves': 1}

For this Model i got an PRAUC as 0.7911

And then tried Hyperopt

from hyperopt import STATUS_OK, Trials, fmin, hp, tpe
space={'n_estimators': hp.uniform('n_estimators', 100,1000),
       'eta': hp.uniform('eta', 0.1,1.0),
       'gamma': hp.uniform('gamma', 1,10),
       'max_depth': hp.quniform("max_depth", 3, 18, 1),
       'min_child_weight' : hp.quniform('min_child_weight', 0, 10, 1),
       'subsample' : hp.uniform('subsample', 0.5,1),
       'colsample_bytree' : hp.uniform('colsample_bytree', 0.5,1),
       'colsample_bylevel' : hp.uniform('colsample_bylevel', 0.5,1),
       'colsample_bynode' : hp.uniform('colsample_bynode', 0.5,1),
       'max_leaves' : hp.uniform('max_leaves', 1,10),
       'seed': 0
    }
def objective(space):
    clf=XGBClassifier(booster='gbtree', objective='binary:logistic', base_score=0.5, random_state=567,
                      n_estimators = int(space['n_estimators']),
                      eta = int(space['eta']),
                      gamma = int(space['gamma']),
                      max_depth = int(space['max_depth']), 
                      min_child_weight=int(space['min_child_weight']),
                      subsample = int(space['subsample']),
                      colsample_bytree=int(space['colsample_bytree']),
                      colsample_bylevel=int(space['colsample_bylevel']),
                      colsample_bynode=int(space['colsample_bynode']),
                      max_leaves = int(space['max_leaves']))
    evaluation = [(X_train, y_train), (X_valid, y_valid)]   
    clf.fit(X_train, y_train,eval_set=evaluation, eval_metric="auc",early_stopping_rounds=10,verbose=False)
    pred = clf.predict(X_valid)
    accuracy = accuracy_score(y_valid, pred>0.5)
    print ("SCORE:", accuracy)
    return {'loss': -accuracy, 'status': STATUS_OK }
trials = Trials()
best_hyperparams = fmin(fn = objective,space = space,algo = tpe.suggest,max_evals = 100,trials = trials)

and i got

best_hyperparams = {'colsample_bylevel': 0.6801340788149979,
 'colsample_bynode': 0.8192495779929423,
 'colsample_bytree': 0.6045563047355242,
 'eta': 0.7461000892301534,
 'gamma': 6.858645345356081,
 'max_depth': 14,
 'max_leaves': 8,
 'min_child_weight': 7.0,
 'n_estimators': 893,
 'subsample': 0.9733931260434276}

i got PRAUC as 0.2682 which is very poor

I dont understand why?
GridSearch took 5 Hrs, but hyperopt took less than 2mins which is good, but accuracy is not.

[bjkomer]

For your gridsearch you tried 1,036,800,000 parameters, but you told hyperopt to only try 100.

You shouldn't need nearly as many parameters with hyperopt to get comparable performance to the gridsearch, but you definitely need more than that.

[hanzigs]

May i know how to set that
Thanks

[bjkomer]

Yes, max_evals is the total number of evaluations to run.

If you want to run a lot and don't know how long it will take, a good idea is to split it into a loop and periodically save, so you can kill it and easily run from where you left off. E.g:

batch_size = 100
for evals in range(0, max_evals, batch_size):
    best_hyperparams = fmin(fn=objective, space=space, algo=tpe.suggest, max_evals=evals, trials=trials)
    # save trials here: e.g. pickle.dump(trials)

[hanzigs]

So, in order to get nearly same as GridSearch, i have to set (just saying) max_evals = 1036800000 some thing like that.

[hanzigs]

With that long GridSearch, the values came as 700,0.6,0.7....
But with less time in hyperopt i got values as 'n_estimators': 893 and decimals in other parameters,

This made me carried away that it did the full search, didn't expect the max_evals = 100 could narrow down the space to that decimals.

Thanks

[bjkomer]

You won't need to go that high. I would expect that to beat gridsearch but will take forever. Completely guessing, but start with something like 10,000 and go from there as you need.

[hanzigs]

Thank you very much for the prompt answers