Scikit-optimize: Stopping criteria

Created on 23 Mar 2016 · 6Comments · Source: scikit-optimize/scikit-optimize

What is a good stopping criteria for blackbox optimization?

API New feature

Source

MechCoder

Most helpful comment

Most implementations of CMA-ES (another black-box optimizer) use the following stopping criteria (e.g. this one):

max_iter / maximum number of function evaluations
minimum fitness value (f(x) < threshold)
_differences_ of function values (does not mean that we have an improvement, exploration is small near convergence, |f(xt) - f(xt-1)| < threshold)
difference of N best function values
difference of evaluated parameter vectors (||xt - xt-1|| < threshold)

AlexanderFabisch on 23 May 2016

👍3

All 6 comments

Right now, I just let it run it upto max_iter

MechCoder on 23 Mar 2016

We might add something like an improvement threshold between consecutive iterations, but this may be tricky to get correctly, since BO trades off exploitation for exploitation, and therefore it is far from guaranteed that the best optimum found so far will improve from one iteration to another.

glouppe on 2 Apr 2016

Most implementations of CMA-ES (another black-box optimizer) use the following stopping criteria (e.g. this one):

max_iter / maximum number of function evaluations
minimum fitness value (f(x) < threshold)
_differences_ of function values (does not mean that we have an improvement, exploration is small near convergence, |f(xt) - f(xt-1)| < threshold)
difference of N best function values
difference of evaluated parameter vectors (||xt - xt-1|| < threshold)

AlexanderFabisch on 23 May 2016

👍3

Came up in #324. When working on this we should also investigate why there are so many warnings about a point already being evaluated in the unittests. My guess would be that with a convergence criteria like proposed here we would stop instead of continuing to evaluate the objective.