Ax: [Question(s)] What is the recommended Sobol/Bayesian step ratio? How many Sobol steps should I run in parallel?

Created on 17 Dec 2019  路  2Comments  路  Source: facebook/Ax

I have a CPU with 44 threads. Using the service API, I could evaluate 44 Sobol steps in parallel (multiprocessing). Then I could continue with another sequential 10-15 steps for the Bayesian optimization.
Do you think the large number of Sobol steps is somehow skewing my result?
Should I run so many Sobol steps in parallel?
Thanks.
Sample code:

...
from joblib import Parallel, delayed
...
no_sobol_steps, no_bayesian_steps = 44, 15
...
#Initialize client and add generation strategies if needed (Models: BOTORCH,GPEI).
gs = GenerationStrategy(steps=[
        GenerationStep(Models.SOBOL, num_arms=no_sobol_steps),
        GenerationStep(Models.GPEI, num_arms=-1)])
ax = AxClient(generation_strategy=gs)
...
ax.create_experiment(...)
...
def evaluate(parameters):
...
trials_to_evaluate = {}
for i in range(no_sobol_steps): #run them in parallel
    parameters, trial_index = ax.get_next_trial()
    trials_to_evaluate[trial_index] = parameters
def parallel(i):
    result = {i: evaluate(trials_to_evaluate[i])}
    return result
# invoke joblib multiprocessing
par_results = Parallel(n_jobs=-2, verbose=55)(delayed(parallel)(i) for i in range(no_sobol_steps))
for i in range(no_sobol_steps):
    ax.complete_trial(x, par_results[i][i]) 

for i in range(no_bayesian_steps):
   parameters, trial_index = ax.get_next_trial()
   ax.complete_trial(trial_index=trial_index, raw_data=evaluate(parameters))  
question

Most helpful comment

Hi @covrig ! There's no problem at all with running lots of Sobol trials, so if you have the computing power, go for it! In general we say you should run at least min(5, number of parameters) of Sobol trials, but again there's no harm in running more.

Does that answer your question? cc @lena-kashtelyan in case she has anything to add.

All 2 comments

Hi @covrig ! There's no problem at all with running lots of Sobol trials, so if you have the computing power, go for it! In general we say you should run at least min(5, number of parameters) of Sobol trials, but again there's no harm in running more.

Does that answer your question? cc @lena-kashtelyan in case she has anything to add.

Thank you for the fast reply. This solves my question.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

wooohoooo picture wooohoooo  路  4Comments

Irynei picture Irynei  路  5Comments

ugurmengilli picture ugurmengilli  路  3Comments

dqii picture dqii  路  5Comments

richarddwang picture richarddwang  路  4Comments