Optuna: Hyperparams confidence interval estimation

[TheoLvs]

Hello,

Thanks for the great work on Optuna library,
It's definitely a game-changer compare to other options, I just switched to Optuna in many of my personal & professional projects.

I do have one question, is there a way of computing parameters confidence intervals? Maybe with a specific sampler?

Thanks a lot,
Theo

[HideakiImamura]

Hi Theo!

Thank you for your insightful question. I'm very glad to hear that you've started to use Optuna.

In current Optuna implementation, there is no way for computing parameter confidence intervals because estimating confidence of parameters is difficult. There are some reasons why it is difficult.

• [1] For any samplers, we can compute how the optimized hyperparameter performs well after hyperparameter optimization by conducting some trials under the optimized hyperparameter. However, in order to obtain a sufficiently accurate confidence interval, this procedure requires a large number of trials besides the hyperparameter optimization. A snippet of the code I envision would look something like this:

def obj(params):
  return SOMETHING

def objective(trial):
  params = trial.suggest_SOMETHING
  return obj(params)

N_TRIALS_FOR_OPTIMIZATION = 100
N_TRIALS_FOR_VALIDATION = 30

study = optuna.study.create_study()
study.optimize(objective, n_trials=N_TRIALS_FOR_OPTIMIZATION)
best_params = study.best_params

values_for_best_params = []
for _ in range(N_TRIALS_FOR_VALIDATION):
  values_for_best_params.append(obj)
standard_deviation = numpy.std(values_for_best_params)

• [2] By making a specific sampler, we might compute the hyperparameter confidence through the optimization, but there is no research or implementation to do it to the best of my knowledge because we need more trials than that of above.

BTW, I greatly appreciate your insight about hyperparameter confidence. We will consider that topic in the near future.

[TheoLvs]

Hello,

Thanks a lot for your quick answer :)

That's what I had in mind indeed, I computed the std of the parameters yielding x%-quantile of the best values in a study. Which can be sufficient as you mentioned to have intervals even if it's not proper statistical confidence interval.

I'm working on a Bayesian sampler with pyro using Stochastic Variational Inference and MCMC, I'll let you know if 1. it works 2. I think it could be integrated in Optuna.

Also FYI, we are using Optuna to calibrate epidemiology parameters to help French government response to COVID19 crisis 💪

Closing the issue for now then.
Best,
Theo