Cuml: [BUG] SVC with Kernel poly and degree > 30 leads to CudaAPIError

[magehex]

Describe the bug
SVC with Kernel poly and degree > 30 leads to CudaAPIError

SMO error: NaN found during fitting
ERROR:Call to cuPointerGetAttribute results in CUDA_ERROR_INVALID_VALUE
Traceback (most recent call last):
  File "<input>", line 12, in <module>
  File "cuml/svm/svm.pyx", line 440, in cuml.svm.svm.SVC.fit
  File "cuml/svm/svm.pyx", line 361, in cuml.svm.svm.SVC._unpack_model
  File "/home/magehex/anaconda3/envs/reinbo/lib/python3.7/site-packages/cuml/utils/numba_utils.py", line 168, in device_array_from_ptr
    devptr = cuda.driver.get_devptr_for_active_ctx(ptr)
  File "/home/magehex/anaconda3/envs/reinbo/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 1750, in get_devptr_for_active_ctx
    driver.cuPointerGetAttribute(byref(devptr), attr, ptr)
  File "/home/magehex/anaconda3/envs/reinbo/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 294, in safe_cuda_api_call
    self._check_error(fname, retcode)
  File "/home/magehex/anaconda3/envs/reinbo/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 329, in _check_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [1] Call to cuPointerGetAttribute results in CUDA_ERROR_INVALID_VALUE

Steps/Code to reproduce bug

import numpy as np
from cuml.svm import SVC
from sklearn.datasets.samples_generator import make_blobs


X, y = make_blobs(n_samples=100000, centers=2, n_features=380)
X = X.astype(np.float32)
y = y.astype(np.int32)

clf = SVC(kernel='poly', degree=30)

clf.fit(X,y)

Expected behavior
No error

Environment details (please complete the following information):

• Environment location: [Desktop PC]
• Linux Distro/Architecture: [ubuntu 18.04 amd64]
• GPU Model/Driver: [Quadro P600 and driver 418.87.01]
• CUDA: [10.1]
• Method of cuDF & cuML install: [conda]
• conda install -c rapidsai-nightly -c nvidia -c conda-forge
• -c defaults rapids=0.11 python=3.7 cudatoolkit=10.1

Additional context
Not sure if it is a bug or gpu memory is not enough. Could you explain this please @oyilmaz-nvidia

[cjnolet]

Tagging @tfeher for insight (the author of SVM in cuML).

[tfeher]

@magehex thank you for reporting. The root cause is that the kernel function values for your input parameters cannot be represented in fp32. Unfortunately this error is not treated gracefully at the moment, I have opened Bug #1352 to improve that. Here I will focus on the numerical problem.

Using poly kernel, the kernel values are defined as
K = np.pow(gamma*np.dot(x,y.T) + coef0, degree).

The default value for gamma is currently auto which sets gamma = 1 / n_features, so in this case gamma = 1/380. This follows the current stable branch of Scikit learn (0.21), which will change in 0.22. I will improve the documentation of cuML to clarify which implementation do we have.

Let's calculate some of the values that goes in as the first argument for the power function:

a = np.dot(X, X[:1000,:].T)/380
scipy.stats.describe(a.ravel())
DescribeResult(nobs=100000000, minmax=(-2.33288, 35.919594), mean=16.19433, variance=267.11047, skewness=0.003026929683983326, kurtosis=-1.9934726468547659)

If I take the max and raise it to the 30th power then we get around 4.5e45 which is outside range that fp32 can represent. This causes the training to fail with SMO error: NaN found during fitting.

Workarounds: use gamma='scale', or use data type fp64. It is enough to convert X, y is converted under the hood to the same data type as X.

Note: just noticed that we treat the 'scale', differently than Sklearn #1353. It will be fixed shortly. Until that you can initialize the gamma parameter explicitly gamma = 1/(n_features*X.var()), to get the same behavior as Sklearn's 'scale' option.

[tfeher]

@cjnolet The related issues have been fixed, I believe this can be closed as well.