Cuml: [BUG] Running solver in a for-loop leads to error

[magehex]

Describe the bug
I want to run a solver in a for-loop and got following error:

Got CUSOLVER error 7 at /conda/conda-bld/libcuml_1572860950777/work/cpp/src/common/cumlHandle.cpp:275
CUSOLVER_STATUS_INTERNAL_ERROR

Steps/Code to reproduce bug

from sklearn.metrics import make_scorer
from sklearn.metrics import mean_squared_error
from cuml.linear_model import Lasso
from sklearn.datasets import load_boston

X, y = load_boston(return_X_y=True)

lr = Lasso()

score = make_scorer(mean_squared_error)

for i in range(0,10000):
    lr.fit(X, y)
    score(lr, X, y)

Expected behavior
No error

Environment details (please complete the following information):

• Environment location: [Desktop PC]
• Linux Distro/Architecture: [ubuntu 18.04 amd64]
• GPU Model/Driver: [Quadro P600 and driver 418.87.01]
• CUDA: [10.1]
• Method of cuDF & cuML install: [conda]
  conda install -c rapidsai-nightly -c nvidia -c conda-forge \
  -c defaults rapids=0.11 python=3.7 cudatoolkit=10.1

[dantegd]

Interesting, @oyilmaz-nvidia would you know why this could happen?

[oyilmaz-nvidia]

Himm, really interesting. Need to analyze the code and memory usage. Will get back to you in couple of days.

[magehex]

There is another error if I execute:

from cuml.svm import SVC
import time
from sklearn.datasets.samples_generator import make_blobs
from sklearn.metrics import make_scorer
from sklearn.metrics import accuracy_score

# generate 2d classification dataset
X, y = make_blobs(n_samples=1000, centers=2, n_features=20)

score = make_scorer(accuracy_score)


for i in range(1,1000):
    C = 1
    tol = 1e-3
    kernel = 'rbf'
    gamma = 0.01

    cumlSVC = SVC(kernel=kernel, C=C, tol=tol, gamma=gamma)

    cumlSVC.fit(X, y)
    score(cumlSVC, X, y)

end = time.time()
print(end - start)

Error:

Exception ignored in: <finalize object at 0x7ff5fa5af6d0; dead>
Traceback (most recent call last):
  File "/home/magehex/anaconda3/envs/reinbo/lib/python3.7/site-packages/numba/utils.py", line 669, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/home/magehex/anaconda3/envs/reinbo/lib/python3.7/site-packages/rmm/rmm.py", line 220, in finalizer
    librmm.rmm_free(handle, stream)
  File "rmm/_lib/lib.pyx", line 241, in rmm._lib.lib.rmm_free
  File "rmm/_lib/lib.pyx", line 230, in rmm._lib.lib.c_free
  File "rmm/_lib/lib.pyx", line 47, in rmm._lib.lib.check_error
rmm.rmm.RMMError: RMM_ERROR_CUDA_ERROR
Traceback (most recent call last):
  File "<input>", line 23, in <module>
  File "cuml/svm/svm.pyx", line 433, in cuml.svm.svm.SVC.fit
RuntimeError: Exception occured! file=/conda/conda-bld/libcuml_1572860950777/work/cpp/include/cuml/common/cuml_allocator.hpp line=109: FAIL: call='cudaMalloc(&ptr, n)'. Reason:out of memory
Obtained 44 stack frames
#0 in /home/magehex/anaconda3/envs/reinbo/lib/python3.7/site-packages/cuml/common/../../../../libcuml++.so(_ZN8MLCommon9Exception16collectCallStackEv+0x3e) [0x7ff67a03c66e]
#1 in /home/magehex/anaconda3/envs/reinbo/lib/python3.7/site-packages/cuml/common/../../../../libcuml++.so(_ZN8MLCommon9ExceptionC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x80) [0x7ff67a03d180]
#2 in /home/magehex/anaconda3/envs/reinbo/lib/python3.7/site-packages/cuml/common/../../../../libcuml++.so(_ZN8MLCommon22defaultDeviceAllocator8allocateEmP11CUstream_st+0x102) [0x7ff67a03d452]
#3 in /home/magehex/anaconda3/envs/reinbo/lib/python3.7/site-packages/cuml/common/../../../../libcuml++.so(_ZN8MLCommon11buffer_baseIdNS_15deviceAllocatorEE7reserveEmP11CUstream_st+0xd5) [0x7ff67a04a845]
#4 in /home/magehex/anaconda3/envs/reinbo/lib/python3.7/site-packages/cuml/common/../../../../libcuml++.so(_ZN8MLCommon5Cache5CacheIdLi32EEC2ESt10shared_ptrINS_15deviceAllocatorEEP11CUstream_stifb+0x9a3) [0x7ff67a2ca323]
#5 in /home/magehex/anaconda3/envs/reinbo/lib/python3.7/site-packages/cuml/common/../../../../libcuml++.so(_ZN2ML3SVM9SmoSolverIdE5SolveEPdiiS3_PS3_PiS4_PS5_S3_ii+0x38a) [0x7ff67a2a2aaa]
#6 in /home/magehex/anaconda3/envs/reinbo/lib/python3.7/site-packages/cuml/common/../../../../libcuml++.so(_ZN2ML3SVM6svcFitIdEEvRKNS_10cumlHandleEPT_iiS6_RKNS0_12svmParameterERN8MLCommon6Matrix12KernelParamsERNS0_8svmModelIS5_EE+0x758) [0x7ff67a28b968]
#7 in /home/magehex/anaconda3/envs/reinbo/lib/python3.7/site-packages/cuml/svm/svm.cpython-37m-x86_64-linux-gnu.so(+0x24173) [0x7ff5fb62e173]
#8 in /home/magehex/anaconda3/envs/reinbo/lib/python3.7/site-packages/cuml/svm/svm.cpython-37m-x86_64-linux-gnu.so(+0x25feb) [0x7ff5fb62ffeb]
#9 in /home/magehex/anaconda3/envs/reinbo/bin/python(_PyObject_FastCallKeywords+0x49b) [0x560d68c6319b]
#10 in /home/magehex/anaconda3/envs/reinbo/bin/python(_PyEval_EvalFrameDefault+0x52e6) [0x560d68cc84d6]
#11 in /home/magehex/anaconda3/envs/reinbo/bin/python(_PyEval_EvalCodeWithName+0x2f9) [0x560d68c0b1b9]
#12 in /home/magehex/anaconda3/envs/reinbo/bin/python(PyEval_EvalCodeEx+0x44) [0x560d68c0c094]
#13 in /home/magehex/anaconda3/envs/reinbo/bin/python(PyEval_EvalCode+0x1c) [0x560d68c0c0bc]
#14 in /home/magehex/anaconda3/envs/reinbo/bin/python(+0x1daeb0) [0x560d68cd2eb0]
#15 in /home/magehex/anaconda3/envs/reinbo/bin/python(_PyMethodDef_RawFastCallKeywords+0xe9) [0x560d68c5bab9]
#16 in /home/magehex/anaconda3/envs/reinbo/bin/python(_PyCFunction_FastCallKeywords+0x21) [0x560d68c5bd51]
#17 in /home/magehex/anaconda3/envs/reinbo/bin/python(_PyEval_EvalFrameDefault+0x4784) [0x560d68cc7974]
#18 in /home/magehex/anaconda3/envs/reinbo/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x560d68c5b1ab]
#19 in /home/magehex/anaconda3/envs/reinbo/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x560d68cc3893]
#20 in /home/magehex/anaconda3/envs/reinbo/bin/python(_PyEval_EvalCodeWithName+0x2f9) [0x560d68c0b1b9]
#21 in /home/magehex/anaconda3/envs/reinbo/bin/python(_PyFunction_FastCallKeywords+0x325) [0x560d68c5b3d5]
#22 in /home/magehex/anaconda3/envs/reinbo/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x560d68cc3893]
#23 in /home/magehex/anaconda3/envs/reinbo/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x560d68c5b1ab]
#24 in /home/magehex/anaconda3/envs/reinbo/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x560d68cc3893]
#25 in /home/magehex/anaconda3/envs/reinbo/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x560d68c5b1ab]
#26 in /home/magehex/anaconda3/envs/reinbo/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x560d68cc3893]
#27 in /home/magehex/anaconda3/envs/reinbo/bin/python(_PyEval_EvalCodeWithName+0x2f9) [0x560d68c0b1b9]
#28 in /home/magehex/anaconda3/envs/reinbo/bin/python(_PyFunction_FastCallKeywords+0x325) [0x560d68c5b3d5]
#29 in /home/magehex/anaconda3/envs/reinbo/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x560d68cc3893]
#30 in /home/magehex/anaconda3/envs/reinbo/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x560d68c5b1ab]
#31 in /home/magehex/anaconda3/envs/reinbo/bin/python(_PyEval_EvalFrameDefault+0x416) [0x560d68cc3606]
#32 in /home/magehex/anaconda3/envs/reinbo/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x560d68c5b1ab]
#33 in /home/magehex/anaconda3/envs/reinbo/bin/python(_PyEval_EvalFrameDefault+0x4b29) [0x560d68cc7d19]
#34 in /home/magehex/anaconda3/envs/reinbo/bin/python(_PyEval_EvalCodeWithName+0x2f9) [0x560d68c0b1b9]
#35 in /home/magehex/anaconda3/envs/reinbo/bin/python(PyEval_EvalCodeEx+0x44) [0x560d68c0c094]
#36 in /home/magehex/anaconda3/envs/reinbo/bin/python(PyEval_EvalCode+0x1c) [0x560d68c0c0bc]
#37 in /home/magehex/anaconda3/envs/reinbo/bin/python(+0x22a9e4) [0x560d68d229e4]
#38 in /home/magehex/anaconda3/envs/reinbo/bin/python(PyRun_FileExFlags+0xa1) [0x560d68d2cde1]
#39 in /home/magehex/anaconda3/envs/reinbo/bin/python(PyRun_SimpleFileExFlags+0x1c3) [0x560d68d2cfd3]
#40 in /home/magehex/anaconda3/envs/reinbo/bin/python(+0x2360c5) [0x560d68d2e0c5]
#41 in /home/magehex/anaconda3/envs/reinbo/bin/python(_Py_UnixMain+0x3c) [0x560d68d2e1ec]
#42 in /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7ff6ac0c5b97]
#43 in /home/magehex/anaconda3/envs/reinbo/bin/python(+0x1daf7d) [0x560d68cd2f7d]

[oyilmaz-nvidia]

Created this PR to fix the problem with lasso;
https://github.com/rapidsai/cuml/pull/1342

Bug will be fixed when the PR is merged.

BTW, if you'd like to run batched Lasso, please create a feature request. We can actually speed it up with multiple streams.

Closing this issue. Thanks again for this bug report. It really helped us.

[magehex]

Thank you for fixing this bug @oyilmaz-nvidia . Did you also fix the bug of SVC?

[oyilmaz-nvidia]

There will be a different PR for the SVC.