I have a CPU with 44 threads. Using the service API, I could evaluate 44 Sobol steps in parallel (multiprocessing). Then I could continue with another sequential 10-15 steps for the Bayesian optimization.
Do you think the large number of Sobol steps is somehow skewing my result?
Should I run so many Sobol steps in parallel?
Thanks.
Sample code:
...
from joblib import Parallel, delayed
...
no_sobol_steps, no_bayesian_steps = 44, 15
...
#Initialize client and add generation strategies if needed (Models: BOTORCH,GPEI).
gs = GenerationStrategy(steps=[
GenerationStep(Models.SOBOL, num_arms=no_sobol_steps),
GenerationStep(Models.GPEI, num_arms=-1)])
ax = AxClient(generation_strategy=gs)
...
ax.create_experiment(...)
...
def evaluate(parameters):
...
trials_to_evaluate = {}
for i in range(no_sobol_steps): #run them in parallel
parameters, trial_index = ax.get_next_trial()
trials_to_evaluate[trial_index] = parameters
def parallel(i):
result = {i: evaluate(trials_to_evaluate[i])}
return result
# invoke joblib multiprocessing
par_results = Parallel(n_jobs=-2, verbose=55)(delayed(parallel)(i) for i in range(no_sobol_steps))
for i in range(no_sobol_steps):
ax.complete_trial(x, par_results[i][i])
for i in range(no_bayesian_steps):
parameters, trial_index = ax.get_next_trial()
ax.complete_trial(trial_index=trial_index, raw_data=evaluate(parameters))
Hi @covrig ! There's no problem at all with running lots of Sobol trials, so if you have the computing power, go for it! In general we say you should run at least min(5, number of parameters) of Sobol trials, but again there's no harm in running more.
Does that answer your question? cc @lena-kashtelyan in case she has anything to add.
Thank you for the fast reply. This solves my question.
Most helpful comment
Hi @covrig ! There's no problem at all with running lots of Sobol trials, so if you have the computing power, go for it! In general we say you should run at least
min(5, number of parameters)of Sobol trials, but again there's no harm in running more.Does that answer your question? cc @lena-kashtelyan in case she has anything to add.