Scikit-optimize: ValueError: array must not contain infs or NaNs

[NaSed]

I am getting this error stems from skopt while I am sure the objective function does not generate nan or inf. I have tested the code on my friend's machine and it works but it generates the below error on my machine. DO you have any idea why it happens?

Traceback (most recent call last):
File "/project/6004228/DR_Project/Code/Residual_v3.py", line 248, in
verbose=0)
File "/project/6004228/DR_Project/Code/TF220/lib/python3.6/site-packages/skopt/optimizer/gp.py", line 295, in gp_minimize
callback=callback, n_jobs=n_jobs, model_queue_size=model_queue_size)
File "/project/6004228/DR_Project/Code/TF220/lib/python3.6/site-packages/skopt/optimizer/base.py", line 303, in base_minimize
result = optimizer.tell(next_x, next_y)
File "/project/6004228/DR_Project/Code/TF220/lib/python3.6/site-packages/skopt/optimizer/optimizer.py", line 483, in tell
return self._tell(x, y, fit=fit)
File "/project/6004228/DR_Project/Code/TF220/lib/python3.6/site-packages/skopt/optimizer/optimizer.py", line 526, in _tell
est.fit(self.space.transform(self.Xi), self.yi)
File "/project/6004228/DR_Project/Code/TF220/lib/python3.6/site-packages/skopt/learning/gaussian_process/gpr.py", line 195, in fit
super(GaussianProcessRegressor, self).fit(X, y)
File "/project/6004228/DR_Project/Code/TF220/lib/python3.6/site-packages/sklearn/gaussian_process/_gpr.py", line 234, in fit
self.kernel_.bounds))]
File "/project/6004228/DR_Project/Code/TF220/lib/python3.6/site-packages/sklearn/gaussian_process/_gpr.py", line 503, in _constrained_optimization
bounds=bounds)
File "/project/6004228/DR_Project/Code/TF220/lib/python3.6/site-packages/scipy/optimize/_minimize.py", line 610, in minimize
callback=callback, *options)
File "/project/6004228/DR_Project/Code/TF220/lib/python3.6/site-packages/scipy/optimize/lbfgsb.py", line 345, in _minimize_lbfgsb
f, g = func_and_grad(x)
File "/project/6004228/DR_Project/Code/TF220/lib/python3.6/site-packages/scipy/optimize/lbfgsb.py", line 295, in func_and_grad
f = fun(x, *args)
File "/project/6004228/DR_Project/Code/TF220/lib/python3.6/site-packages/scipy/optimize/optimize.py", line 327, in function_wrapper
return function((wrapper_args + args))
File "/project/6004228/DR_Project/Code/TF220/lib/python3.6/site-packages/scipy/optimize/optimize.py", line 65, in __call__
fg = self.fun(x, *args)
File "/project/6004228/DR_Project/Code/TF220/lib/python3.6/site-packages/sklearn/gaussian_process/_gpr.py", line 225, in obj_func
theta, eval_gradient=True, clone_kernel=False)
File "/project/6004228/DR_Project/Code/TF220/lib/python3.6/site-packages/sklearn/gaussian_process/_gpr.py", line 476, in log_marginal_likelihood
alpha = cho_solve((L, True), y_train) # Line 3
File "/project/6004228/DR_Project/Code/TF220/lib/python3.6/site-packages/scipy/linalg/decomp_cholesky.py", line 196, in cho_solve
b1 = asarray_chkfinite(b)
File "/project/6004228/DR_Project/Code/TF220/lib/python3.6/site-packages/numpy/lib/function_base.py", line 499, in asarray_chkfinite
"array must not contain infs or NaNs")
ValueError: array must not contain infs or NaNs

I have tried this with numpy 1.18.4, pandas is 1.0.3, tensorflow is 2.2.0. I have tried several versions of skopt since dev version 0.8 to skopt 0.7.3.

[MarcAzar]

I'm facing the same issue. After creating the Optimizer class and invoking ask() for the first time, I get the same error as above. This happened only after I updated my environment from scikit-learn==0.22.2.post1 to scikit-learn==0.23.0.

There are some API changes between the two releases. A quick look at scikit-learn commits, I see that _gpr.py has two changes. Can anyone confirm if scikit-optimize is working under scikit-learn==0.23.0?

[dmoklaf]

Same problem here, happens with scikit-learn==0.23.1
Same workaround than @MarcAzar : the issue disappears when reverting to scikit-learn==0.22.1
So the bug is indeed coming from some API subtleties between skopt and scikit-learn

[bartolkaruza]

I'm getting this same issue, but I can also prevent this using less n_points for some reason.

scikit-learn: 0.23.2
scikit-optimize: 0.8.1
Python 3.8.4

So with the below optimizer:

optimizer = Optimizer(
        dimensions=[Real(0.5, 10.0), Real(0.1, 5.0), Real(0.5, 5.0), Integer(1, 50), Integer(1, 50)],
        random_state=1,
        base_estimator='gp'
    )

If I ask at n_points=9 the function works fine, but with n_points=10 it throws the ValueError every time.

Here is the check I used to make sure it's not just a higher probability of throwing the ValueError:

count_9 = 0
 for i in range(100):
    try:
        optimizer.ask(n_points=9)
    except:
        count_9 = count_9 + 1

count_10 = 0
for i in range(100):
    try:
        optimizer.ask(n_points=10)
    except:
        count_10 = count_10 + 1

print(count_9, count_10)

Output: 0 100

[nxorable]

Possibly related:
https://github.com/scikit-learn/scikit-learn/pull/19703
sklearn 0.24.2 release bugfix