Ax: Error Message:Specified maximum draws (10000) exhausted, without finding sufficiently many (1) candidates.

[algharak]

Trying to use Ax to fo hyperparameter tuning of an XGBOOST model with the following space:

Ax_par=[{"name": "max_depth", "type": "choice", "values": [6,7], "value_type": "int"},
{"name": "min_child_weight", "type": "choice", "values": [1,2], "value_type": "int"}]

The following is the error message I received. Any assistance you can provide will be great.

File "/usr/local/lib/python3.7/site-packages/ax/service/managed_loop.py", line 166, in full_run
self.run_trial()
File "/usr/local/lib/python3.7/site-packages/ax/service/managed_loop.py", line 145, in run_trial
experiment=self.experiment
File "/usr/local/lib/python3.7/site-packages/ax/modelbridge/generation_strategy.py", line 345, in gen
keywords=get_function_argument_names(model.gen),
File "/usr/local/lib/python3.7/site-packages/ax/modelbridge/base.py", line 616, in gen
model_gen_options=model_gen_options,
File "/usr/local/lib/python3.7/site-packages/ax/modelbridge/random.py", line 92, in _gen
rounding_func=transform_callback(self.parameters, self.transforms),
File "/usr/local/lib/python3.7/site-packages/ax/models/random/sobol.py", line 119, in gen
rounding_func=rounding_func,
File "/usr/local/lib/python3.7/site-packages/ax/models/random/base.py", line 118, in gen
existing_points=self.generated_points,
File "/usr/local/lib/python3.7/site-packages/ax/models/model_utils.py", line 108, in rejection_sample
f"Specified maximum draws ({max_draws}) exhausted, without "
ValueError: Specified maximum draws (10000) exhausted, without finding sufficiently many (1) candidates.

[ldworkin]

Hi @algharak ! Can you provide a full code example that would allow me to reproduce this?

[algharak]

Thanks for your quick response. I did find out the problem and has to do with my choice of parameters.

[sandys]

@algharak could you paste a code example that you got working with xgboost + Ax. we are looking to get the same working.

[eytan]

Hopefully somebody can answer your question, but if you are tuning HPs I’d
recommend avoiding choice parameters and sticking with ranges (and floats)
as much as possible. Since ax uses Bayesian optimization It can very
efficiently explore the space. Your config here is just exploring a 2x2
combinations of parameters, for which you don’t really need ax for :) you
might get less errors if you try using the appropriate range types and more
values.

Best
E

On Thu, May 7, 2020 at 11:57 AM algharak [e-mail] wrote:

Trying to use Ax to fo hyperparameter tuning of an XGBOOST model with the
following space:

Ax_par=[{"name": "max_depth", "type": "choice", "values": [6,7],
"value_type": "int"},
{"name": "min_child_weight", "type": "choice", "values": [1,2],
"value_type": "int"}]
The following is the error message I received. Any assistance you can
provide will be great.

File "/usr/local/lib/python3.7/site-packages/ax/service/managed_loop.py",
line 166, in full_run
self.run_trial()
File "/usr/local/lib/python3.7/site-packages/ax/service/managed_loop.py",
line 145, in run_trial
experiment=self.experiment
File
"/usr/local/lib/python3.7/site-packages/ax/modelbridge/generation_strategy.py",
line 345, in gen
keywords=get_function_argument_names(model.gen),
File "/usr/local/lib/python3.7/site-packages/ax/modelbridge/base.py", line
616, in gen
model_gen_options=model_gen_options,
File "/usr/local/lib/python3.7/site-packages/ax/modelbridge/random.py",
line 92, in _gen
rounding_func=transform_callback(self.parameters, self.transforms),
File "/usr/local/lib/python3.7/site-packages/ax/models/random/sobol.py",
line 119, in gen
rounding_func=rounding_func,
File "/usr/local/lib/python3.7/site-packages/ax/models/random/base.py",
line 118, in gen
existing_points=self.generated_points,
File "/usr/local/lib/python3.7/site-packages/ax/models/model_utils.py",
line 108, in rejection_sample
f"Specified maximum draws ({max_draws}) exhausted, without "
ValueError: Specified maximum draws (10000) exhausted, without finding
sufficiently many (1) candidates.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/facebook/Ax/issues/313, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAW34KGAGNOAZA452OKIZ3RQMADBANCNFSM4M3SK3UA
.

[ldworkin]

Was able to verify that this does indeed break on a very simple example:

from ax import optimize
from ax.utils.measurement.synthetic_functions import branin

best_parameters, values, experiment, model = optimize(
    parameters=[
        {"name": "max_depth", "type": "choice", "values": [6,7], "value_type": "int"},
        {"name": "min_child_weight", "type": "choice", "values": [1,2], "value_type": "int"},
    ],
    evaluation_function=lambda p: 10,
    minimize=True,
)

@eytan 's right that this is a bit of an abuse of the optimize API, but we should at least fail in a more elegant way.

[lena-kashtelyan]

@ldworkin, agreed that we should exit gracefully in this case!

@algharak, just FYI, the error was just trying to indicate that your search space was exhausted and we couldn't produce the next trial from our Sobol sampler (since there are only 4 possible parameter combinations in the search space in this case). We'll work on exiting gracefully in this case!

[lena-kashtelyan]

The fix for this is now on master and will be included in the next stable version release.

[algharak]

I am good