Scikit-optimize: TypeError: %d format: a number is required, not tuple - when changing n_iter

Created on 3 Apr 2018  路  3Comments  路  Source: scikit-optimize/scikit-optimize

When running opt.fit(X_train, y_train) with n_iter=10 (or higher) I get the error TypeError: %d format: a number is required, not tuple. With n_iter=9 or less, it works (with only a few convergence warnings).
EDIT: seen with version 0.5.2

# minimal example
from skopt import BayesSearchCV
from sklearn.model_selection import train_test_split
from sklearn.neural_network.multilayer_perceptron import MLPRegressor
import itertools

XS = [[1], [2], [3], [4], [5], [6]]
ys = [2, 3, 4, 5, 6, 7]
X_train, X_test, y_train, y_test = train_test_split(XS, ys, train_size=0.75, random_state=1)

opt = BayesSearchCV(
    MLPRegressor(),
    {
        'hidden_layer_sizes': [x for x in itertools.product((10, 100), repeat=2)]
    },
    n_iter=10
)

opt.fit(X_train, y_train)

Full error log

/home/atarust/.local/lib/python3.5/site-packages/sklearn/model_selection/_split.py:2026: FutureWarning: From version 0.21, test_size will always complement train_size unless both are specified.
  FutureWarning)
/home/atarust/.local/lib/python3.5/site-packages/sklearn/neural_network/multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.
  % self.max_iter, ConvergenceWarning)
/home/atarust/.local/lib/python3.5/site-packages/sklearn/neural_network/multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.
  % self.max_iter, ConvergenceWarning)
/home/atarust/.local/lib/python3.5/site-packages/sklearn/neural_network/multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.
  % self.max_iter, ConvergenceWarning)
/home/atarust/.local/lib/python3.5/site-packages/sklearn/neural_network/multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.
  % self.max_iter, ConvergenceWarning)
/home/atarust/.local/lib/python3.5/site-packages/sklearn/neural_network/multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.
  % self.max_iter, ConvergenceWarning)
/home/atarust/.local/lib/python3.5/site-packages/sklearn/neural_network/multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.
  % self.max_iter, ConvergenceWarning)
/home/atarust/.local/lib/python3.5/site-packages/sklearn/neural_network/multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.
  % self.max_iter, ConvergenceWarning)
/home/atarust/.local/lib/python3.5/site-packages/sklearn/neural_network/multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.
  % self.max_iter, ConvergenceWarning)
/home/atarust/.local/lib/python3.5/site-packages/sklearn/neural_network/multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.
  % self.max_iter, ConvergenceWarning)
/home/atarust/.local/lib/python3.5/site-packages/sklearn/neural_network/multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.
  % self.max_iter, ConvergenceWarning)
/home/atarust/.local/lib/python3.5/site-packages/sklearn/neural_network/multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.
  % self.max_iter, ConvergenceWarning)
Traceback (most recent call last):
  File "/home/atarust/eclipse-workspace/learnSkiKit/src/minExample.py", line 19, in <module>
    opt.fit(X_train, y_train)
  File "/home/atarust/.local/lib/python3.5/site-packages/skopt/searchcv.py", line 654, in fit
    groups=groups, n_points=n_points_adjusted
  File "/home/atarust/.local/lib/python3.5/site-packages/skopt/searchcv.py", line 541, in _step
    params = optimizer.ask(n_points=n_points)
  File "/home/atarust/.local/lib/python3.5/site-packages/skopt/optimizer/optimizer.py", line 372, in ask
    opt._tell(x, y_lie)
  File "/home/atarust/.local/lib/python3.5/site-packages/skopt/optimizer/optimizer.py", line 486, in _tell
    est.fit(self.space.transform(self.Xi), self.yi)
  File "/home/atarust/.local/lib/python3.5/site-packages/skopt/learning/gaussian_process/gpr.py", line 195, in fit
    super(GaussianProcessRegressor, self).fit(X, y)
  File "/home/atarust/.local/lib/python3.5/site-packages/sklearn/gaussian_process/gpr.py", line 217, in fit
    self.kernel_.bounds))]
  File "/home/atarust/.local/lib/python3.5/site-packages/sklearn/gaussian_process/gpr.py", line 454, in _constrained_optimization
    fmin_l_bfgs_b(obj_func, initial_theta, bounds=bounds)
  File "/home/atarust/.local/lib/python3.5/site-packages/scipy/optimize/lbfgsb.py", line 199, in fmin_l_bfgs_b
    **opts)
  File "/home/atarust/.local/lib/python3.5/site-packages/scipy/optimize/lbfgsb.py", line 335, in _minimize_lbfgsb
    f, g = func_and_grad(x)
  File "/home/atarust/.local/lib/python3.5/site-packages/scipy/optimize/lbfgsb.py", line 285, in func_and_grad
    f = fun(x, *args)
  File "/home/atarust/.local/lib/python3.5/site-packages/scipy/optimize/optimize.py", line 292, in function_wrapper
    return function(*(wrapper_args + args))
  File "/home/atarust/.local/lib/python3.5/site-packages/scipy/optimize/optimize.py", line 63, in __call__
    fg = self.fun(x, *args)
  File "/home/atarust/.local/lib/python3.5/site-packages/sklearn/gaussian_process/gpr.py", line 209, in obj_func
    theta, eval_gradient=True)
  File "/home/atarust/.local/lib/python3.5/site-packages/sklearn/gaussian_process/gpr.py", line 412, in log_marginal_likelihood
    K, K_gradient = kernel(self.X_train_, eval_gradient=True)
  File "/home/atarust/.local/lib/python3.5/site-packages/sklearn/gaussian_process/kernels.py", line 679, in __call__
    K1, K1_gradient = self.k1(X, Y, eval_gradient=True)
  File "/home/atarust/.local/lib/python3.5/site-packages/sklearn/gaussian_process/kernels.py", line 754, in __call__
    K2, K2_gradient = self.k2(X, Y, eval_gradient=True)
  File "/home/atarust/.local/lib/python3.5/site-packages/skopt/learning/gaussian_process/kernels.py", line 391, in __call__
    (X.shape, len(length_scale)))
TypeError: %d format: a number is required, not tuple
picture Atarust

Most helpful comment

Hi @Atarust ,
the reason you see an exception is because as bounds for space hidden_layer_sizes, you provide [(10, 10), (10, 100), (100, 10), (100, 100)].

BayesSearchCV expects that any parameter of the underlying estimator is of a Categorical, Integer or Real type. The values that you specify in search space are typically either left and right bound of dimensions, or a list of categorical values to choose from. hidden_layer_sizes is a rare exception in that it should be an array of values, which is currently not supported directly by BayesSearchCV.

One way to work around this is to wrap MLPRegressor, e.g.:

# minimal example
from skopt import BayesSearchCV
from skopt.space import Integer
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPRegressor
from sklearn.base import BaseEstimator, RegressorMixin
import itertools

XS = [[1], [2], [3], [4], [5], [6]]
ys = [2, 3, 4, 5, 6, 7]
X_train, X_test, y_train, y_test = train_test_split(XS, ys, train_size=0.75, random_state=1)

# Ugly wrapper around MLPRegressor; The ugliness is to make parameters of the estimator be recognized properly
class MLPWrapper(BaseEstimator, RegressorMixin):
    def __init__(self, layer1=10, layer2=10, layer3=10):
        self.layer1 = layer1
        self.layer2 = layer2
        self.layer3 = layer3

    def fit(self, X, y):
        model = MLPRegressor(
            hidden_layer_sizes=[self.layer1, self.layer2, self.layer3]
        )
        model.fit(X, y)
        self.model = model
        return self

    def predict(self, X):
        return self.model.predict(X)

    def score(self, X, y):
        return self.model.score(X, y)


opt = BayesSearchCV(
    estimator=MLPWrapper(),
    search_spaces={
        'layer1': Integer(10, 100),
        'layer2': Integer(10, 100),
        'layer3': Integer(10, 100)
    },
    n_iter=11
)

opt.fit(X_train, y_train)

If you have a better solution let us know. I am closing this for now - feel free to reopen.

picture iaroslav-ai on 9 Jun 2018
👍3

All 3 comments

Which version of scikit-optimize are you using? I remember we recently changed something in that area of the code.
Thanks for the minimal example! Maybe we should add it as a test to make sure this bug doesn't come back.

betatim picture betatim on 7 Apr 2018

I am using the current release 0.5.2

>>> import skopt
>>> skopt.__version__
'0.5.2'
Atarust picture Atarust on 7 Apr 2018

Hi @Atarust ,
the reason you see an exception is because as bounds for space hidden_layer_sizes, you provide [(10, 10), (10, 100), (100, 10), (100, 100)].

BayesSearchCV expects that any parameter of the underlying estimator is of a Categorical, Integer or Real type. The values that you specify in search space are typically either left and right bound of dimensions, or a list of categorical values to choose from. hidden_layer_sizes is a rare exception in that it should be an array of values, which is currently not supported directly by BayesSearchCV.

One way to work around this is to wrap MLPRegressor, e.g.:

# minimal example
from skopt import BayesSearchCV
from skopt.space import Integer
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPRegressor
from sklearn.base import BaseEstimator, RegressorMixin
import itertools

XS = [[1], [2], [3], [4], [5], [6]]
ys = [2, 3, 4, 5, 6, 7]
X_train, X_test, y_train, y_test = train_test_split(XS, ys, train_size=0.75, random_state=1)

# Ugly wrapper around MLPRegressor; The ugliness is to make parameters of the estimator be recognized properly
class MLPWrapper(BaseEstimator, RegressorMixin):
    def __init__(self, layer1=10, layer2=10, layer3=10):
        self.layer1 = layer1
        self.layer2 = layer2
        self.layer3 = layer3

    def fit(self, X, y):
        model = MLPRegressor(
            hidden_layer_sizes=[self.layer1, self.layer2, self.layer3]
        )
        model.fit(X, y)
        self.model = model
        return self

    def predict(self, X):
        return self.model.predict(X)

    def score(self, X, y):
        return self.model.score(X, y)


opt = BayesSearchCV(
    estimator=MLPWrapper(),
    search_spaces={
        'layer1': Integer(10, 100),
        'layer2': Integer(10, 100),
        'layer3': Integer(10, 100)
    },
    n_iter=11
)

opt.fit(X_train, y_train)

If you have a better solution let us know. I am closing this for now - feel free to reopen.

iaroslav-ai picture iaroslav-ai on 9 Jun 2018
👍3
Was this page helpful?
0 / 5 - 0 ratings

Related issues

MassimilianoGrassiDataScience picture MassimilianoGrassiDataScience  路  4Comments
nunocesarsa picture nunocesarsa  路  6Comments
MechCoder picture MechCoder  路  6Comments
arose13 picture arose13  路  5Comments
betatim picture betatim  路  3Comments