Cuml: [QST] Best practices to work with big datasets in RandomForestRegressor
What is your question?
Hi, I have a sample script here that reads in a DF of 500000000 rows and 20 columns. This is just a example to mimic some real data that is being used.
On this example the system is hitting GPU OOM errors.
import numpy as np
import pandas as pd
from cuml.dask.common import utils as dask_utils
from dask.distributed import Client, wait
from dask_cuda import LocalCUDACluster
import dask_cudf
import dask.dataframe as dd
from cuml.dask.ensemble import RandomForestRegressor as cumlDaskRF
cluster = LocalCUDACluster()
c = Client(cluster)
workers = c.has_what().keys()
n_workers = len(workers)
n_streams = 8
n_partitions = n_workers
# Desired parameters
max_depth = 50
n_trees = 100
rows, cols = 500000000, 20
cols_names = ["C{}".format(i) for i in range(1,cols+1)]
cols_names_train = cols_names
cols_names_train.remove('C2')
# Generate fake data for example's sake
x = np.random.random((rows, cols))
df = pd.DataFrame(x, columns=["C{}".format(i) for i in range(1,cols+1)])
df_dask = dask_cudf.from_dask_dataframe(dd.from_pandas(df, npartitions=n_partitions))
X_train_dask, y_train_dask = dask_utils.persist_across_workers(c, [df_dask[cols_names_train], df_dask['C2']], workers=workers)
cuml_model = cumlDaskRF(max_depth=max_depth, n_estimators=n_trees,n_streams=n_streams,workers = workers)
cuml_model.fit(X_train_dask, y_train_dask)
wait(cuml_model.rfs)
--------------------------------------------------------------------------- RuntimeError Traceback (most recent call last)in 1 cuml_model = cumlDaskRF(max_depth=max_depth, n_estimators=n_trees,n_streams=n_streams,workers = workers) ----> 2 cuml_model.fit(X_train_dask, y_train_dask) 3 4 wait(cuml_model.rfs) /opt/conda/lib/python3.7/site-packages/cuml/dask/ensemble/randomforestregressor.py in fit(self, X, y) 362 363 wait(futures) --> 364 raise_exception_from_futures(futures) 365 366 return self /opt/conda/lib/python3.7/site-packages/cuml/dask/common/utils.py in raise_exception_from_futures(futures) 139 if errs: 140 raise RuntimeError("%d of %d worker jobs failed: %s" % ( --> 141 len(errs), len(futures), ", ".join(map(str, errs)) 142 )) 143 RuntimeError: 16 of 16 worker jobs failed: Exception occured! file=/conda/conda-bld/libcuml_1583811942451/work/cpp/include/cuml/common/cuml_allocator.hpp line=109: FAIL: call='cudaMalloc(&ptr, n)'. Reason:out of memory Obtained 29 stack frames #0 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9Exception16collectCallStackEv+0x3e) [0x7f3c2c01339e] #1 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9ExceptionC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x80) [0x7f3c2c013eb0] #2 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN8MLCommon22defaultDeviceAllocator8allocateEmP11CUstream_st+0x102) [0x7f3abb8fcf52] #3 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN15TemporaryMemoryIddE17LevelMemAllocatorEiifiiiib+0x1247) [0x7f3abb9af757] #4 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML11rfRegressorIdE3fitERKNS_10cumlHandleEPKdiiPdRPNS_20RandomForestMetaDataIddEE+0x900) [0x7f3abbb3da60] #5 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML3fitERKNS_10cumlHandleERPNS_20RandomForestMetaDataIddEEPdiiS7_NS_9RF_paramsE+0x204) [0x7f3abbb224a4] #6 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28446) [0x7f3c21104446] #7 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28b1b) [0x7f3c21104b1b] #8 in /opt/conda/bin/python(_PyObject_FastCallKeywords+0x49b) [0x55d29bc3e8fb] #9 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x52f8) [0x55d29bca26e8] #10 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #11 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #12 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #13 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #14 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #15 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #16 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #17 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #18 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #19 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #20 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #21 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #22 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #23 in /opt/conda/bin/python(_PyObject_Call_Prepend+0x63) [0x55d29bc05e53] #24 in /opt/conda/bin/python(PyObject_Call+0x6e) [0x55d29bbf8dbe] #25 in /opt/conda/bin/python(+0x223817) [0x55d29bcf5817] #26 in /opt/conda/bin/python(+0x1de788) [0x55d29bcb0788] #27 in /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f3c79bb26db] #28 in /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f3c798db88f] , Exception occured! file=/conda/conda-bld/libcuml_1583811942451/work/cpp/include/cuml/common/cuml_allocator.hpp line=109: FAIL: call='cudaMalloc(&ptr, n)'. Reason:out of memory Obtained 29 stack frames #0 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9Exception16collectCallStackEv+0x3e) [0x7f3c4e01339e] #1 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9ExceptionC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x80) [0x7f3c4e013eb0] #2 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN8MLCommon22defaultDeviceAllocator8allocateEmP11CUstream_st+0x102) [0x7f3ac38fcf52] #3 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN15TemporaryMemoryIddE17LevelMemAllocatorEiifiiiib+0x1247) [0x7f3ac39af757] #4 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML11rfRegressorIdE3fitERKNS_10cumlHandleEPKdiiPdRPNS_20RandomForestMetaDataIddEE+0x900) [0x7f3ac3b3da60] #5 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML3fitERKNS_10cumlHandleERPNS_20RandomForestMetaDataIddEEPdiiS7_NS_9RF_paramsE+0x204) [0x7f3ac3b224a4] #6 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28446) [0x7f3c45304446] #7 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28b1b) [0x7f3c45304b1b] #8 in /opt/conda/bin/python(_PyObject_FastCallKeywords+0x49b) [0x55d29bc3e8fb] #9 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x52f8) [0x55d29bca26e8] #10 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #11 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #12 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #13 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #14 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #15 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #16 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #17 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #18 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #19 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #20 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #21 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #22 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #23 in /opt/conda/bin/python(_PyObject_Call_Prepend+0x63) [0x55d29bc05e53] #24 in /opt/conda/bin/python(PyObject_Call+0x6e) [0x55d29bbf8dbe] #25 in /opt/conda/bin/python(+0x223817) [0x55d29bcf5817] #26 in /opt/conda/bin/python(+0x1de788) [0x55d29bcb0788] #27 in /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f3c79bb26db] #28 in /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f3c798db88f] , Exception occured! file=/conda/conda-bld/libcuml_1583811942451/work/cpp/include/cuml/common/cuml_allocator.hpp line=109: FAIL: call='cudaMalloc(&ptr, n)'. Reason:out of memory Obtained 29 stack frames #0 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9Exception16collectCallStackEv+0x3e) [0x7f3c5a66339e] #1 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9ExceptionC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x80) [0x7f3c5a663eb0] #2 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN8MLCommon22defaultDeviceAllocator8allocateEmP11CUstream_st+0x102) [0x7f3aa759cf52] #3 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN15TemporaryMemoryIddE17LevelMemAllocatorEiifiiiib+0x1247) [0x7f3aa764f757] #4 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML11rfRegressorIdE3fitERKNS_10cumlHandleEPKdiiPdRPNS_20RandomForestMetaDataIddEE+0x900) [0x7f3aa77dda60] #5 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML3fitERKNS_10cumlHandleERPNS_20RandomForestMetaDataIddEEPdiiS7_NS_9RF_paramsE+0x204) [0x7f3aa77c24a4] #6 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28446) [0x7f3c472a2446] #7 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28b1b) [0x7f3c472a2b1b] #8 in /opt/conda/bin/python(_PyObject_FastCallKeywords+0x49b) [0x55d29bc3e8fb] #9 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x52f8) [0x55d29bca26e8] #10 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #11 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #12 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #13 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #14 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #15 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #16 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #17 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #18 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #19 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #20 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #21 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #22 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #23 in /opt/conda/bin/python(_PyObject_Call_Prepend+0x63) [0x55d29bc05e53] #24 in /opt/conda/bin/python(PyObject_Call+0x6e) [0x55d29bbf8dbe] #25 in /opt/conda/bin/python(+0x223817) [0x55d29bcf5817] #26 in /opt/conda/bin/python(+0x1de788) [0x55d29bcb0788] #27 in /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f3c79bb26db] #28 in /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f3c798db88f] , Exception occured! file=/conda/conda-bld/libcuml_1583811942451/work/cpp/include/cuml/common/cuml_allocator.hpp line=109: FAIL: call='cudaMalloc(&ptr, n)'. Reason:out of memory Obtained 29 stack frames #0 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9Exception16collectCallStackEv+0x3e) [0x7f3c3971639e] #1 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9ExceptionC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x80) [0x7f3c39716eb0] #2 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN8MLCommon22defaultDeviceAllocator8allocateEmP11CUstream_st+0x102) [0x7f3ac38fcf52] #3 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN15TemporaryMemoryIddE17LevelMemAllocatorEiifiiiib+0x1247) [0x7f3ac39af757] #4 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML11rfRegressorIdE3fitERKNS_10cumlHandleEPKdiiPdRPNS_20RandomForestMetaDataIddEE+0x900) [0x7f3ac3b3da60] #5 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML3fitERKNS_10cumlHandleERPNS_20RandomForestMetaDataIddEEPdiiS7_NS_9RF_paramsE+0x204) [0x7f3ac3b224a4] #6 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28446) [0x7f3c39318446] #7 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28b1b) [0x7f3c39318b1b] #8 in /opt/conda/bin/python(_PyObject_FastCallKeywords+0x49b) [0x55d29bc3e8fb] #9 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x52f8) [0x55d29bca26e8] #10 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #11 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #12 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #13 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #14 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #15 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #16 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #17 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #18 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #19 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #20 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #21 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #22 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #23 in /opt/conda/bin/python(_PyObject_Call_Prepend+0x63) [0x55d29bc05e53] #24 in /opt/conda/bin/python(PyObject_Call+0x6e) [0x55d29bbf8dbe] #25 in /opt/conda/bin/python(+0x223817) [0x55d29bcf5817] #26 in /opt/conda/bin/python(+0x1de788) [0x55d29bcb0788] #27 in /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f3c79bb26db] #28 in /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f3c798db88f] , Exception occured! file=/conda/conda-bld/libcuml_1583811942451/work/cpp/include/cuml/common/cuml_allocator.hpp line=109: FAIL: call='cudaMalloc(&ptr, n)'. Reason:out of memory Obtained 29 stack frames #0 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9Exception16collectCallStackEv+0x3e) [0x7f3c5565539e] #1 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9ExceptionC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x80) [0x7f3c55655eb0] #2 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN8MLCommon22defaultDeviceAllocator8allocateEmP11CUstream_st+0x102) [0x7f3aab8fcf52] #3 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN15TemporaryMemoryIddE17LevelMemAllocatorEiifiiiib+0x1247) [0x7f3aab9af757] #4 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML11rfRegressorIdE3fitERKNS_10cumlHandleEPKdiiPdRPNS_20RandomForestMetaDataIddEE+0x900) [0x7f3aabb3da60] #5 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML3fitERKNS_10cumlHandleERPNS_20RandomForestMetaDataIddEEPdiiS7_NS_9RF_paramsE+0x204) [0x7f3aabb224a4] #6 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28446) [0x7f3c430c3446] #7 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28b1b) [0x7f3c430c3b1b] #8 in /opt/conda/bin/python(_PyObject_FastCallKeywords+0x49b) [0x55d29bc3e8fb] #9 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x52f8) [0x55d29bca26e8] #10 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #11 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #12 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #13 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #14 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #15 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #16 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #17 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #18 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #19 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #20 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #21 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #22 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #23 in /opt/conda/bin/python(_PyObject_Call_Prepend+0x63) [0x55d29bc05e53] #24 in /opt/conda/bin/python(PyObject_Call+0x6e) [0x55d29bbf8dbe] #25 in /opt/conda/bin/python(+0x223817) [0x55d29bcf5817] #26 in /opt/conda/bin/python(+0x1de788) [0x55d29bcb0788] #27 in /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f3c79bb26db] #28 in /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f3c798db88f] , Exception occured! file=/conda/conda-bld/libcuml_1583811942451/work/cpp/include/cuml/common/cuml_allocator.hpp line=109: FAIL: call='cudaMalloc(&ptr, n)'. Reason:out of memory Obtained 29 stack frames #0 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9Exception16collectCallStackEv+0x3e) [0x7f3c5ce6839e] #1 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9ExceptionC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x80) [0x7f3c5ce68eb0] #2 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN8MLCommon22defaultDeviceAllocator8allocateEmP11CUstream_st+0x102) [0x7f3aaf59cf52] #3 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN15TemporaryMemoryIddE17LevelMemAllocatorEiifiiiib+0x1247) [0x7f3aaf64f757] #4 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML11rfRegressorIdE3fitERKNS_10cumlHandleEPKdiiPdRPNS_20RandomForestMetaDataIddEE+0x900) [0x7f3aaf7dda60] #5 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML3fitERKNS_10cumlHandleERPNS_20RandomForestMetaDataIddEEPdiiS7_NS_9RF_paramsE+0x204) [0x7f3aaf7c24a4] #6 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28446) [0x7f3c51325446] #7 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28b1b) [0x7f3c51325b1b] #8 in /opt/conda/bin/python(_PyObject_FastCallKeywords+0x49b) [0x55d29bc3e8fb] #9 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x52f8) [0x55d29bca26e8] #10 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #11 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #12 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #13 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #14 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #15 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #16 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #17 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #18 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #19 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #20 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #21 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #22 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #23 in /opt/conda/bin/python(_PyObject_Call_Prepend+0x63) [0x55d29bc05e53] #24 in /opt/conda/bin/python(PyObject_Call+0x6e) [0x55d29bbf8dbe] #25 in /opt/conda/bin/python(+0x223817) [0x55d29bcf5817] #26 in /opt/conda/bin/python(+0x1de788) [0x55d29bcb0788] #27 in /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f3c79bb26db] #28 in /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f3c798db88f] , Exception occured! file=/conda/conda-bld/libcuml_1583811942451/work/cpp/include/cuml/common/cuml_allocator.hpp line=109: FAIL: call='cudaMalloc(&ptr, n)'. Reason:out of memory Obtained 29 stack frames #0 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9Exception16collectCallStackEv+0x3e) [0x7f3c576ce39e] #1 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9ExceptionC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x80) [0x7f3c576ceeb0] #2 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN8MLCommon22defaultDeviceAllocator8allocateEmP11CUstream_st+0x102) [0x7f3abb59cf52] #3 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN15TemporaryMemoryIddE17LevelMemAllocatorEiifiiiib+0x1247) [0x7f3abb64f757] #4 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML11rfRegressorIdE3fitERKNS_10cumlHandleEPKdiiPdRPNS_20RandomForestMetaDataIddEE+0x900) [0x7f3abb7dda60] #5 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML3fitERKNS_10cumlHandleERPNS_20RandomForestMetaDataIddEEPdiiS7_NS_9RF_paramsE+0x204) [0x7f3abb7c24a4] #6 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28446) [0x7f3c57310446] #7 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28b1b) [0x7f3c57310b1b] #8 in /opt/conda/bin/python(_PyObject_FastCallKeywords+0x49b) [0x55d29bc3e8fb] #9 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x52f8) [0x55d29bca26e8] #10 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #11 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #12 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #13 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #14 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #15 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #16 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #17 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #18 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #19 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #20 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #21 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #22 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #23 in /opt/conda/bin/python(_PyObject_Call_Prepend+0x63) [0x55d29bc05e53] #24 in /opt/conda/bin/python(PyObject_Call+0x6e) [0x55d29bbf8dbe] #25 in /opt/conda/bin/python(+0x223817) [0x55d29bcf5817] #26 in /opt/conda/bin/python(+0x1de788) [0x55d29bcb0788] #27 in /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f3c79bb26db] #28 in /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f3c798db88f] , Exception occured! file=/conda/conda-bld/libcuml_1583811942451/work/cpp/include/cuml/common/cuml_allocator.hpp line=109: FAIL: call='cudaMalloc(&ptr, n)'. Reason:out of memory Obtained 29 stack frames #0 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9Exception16collectCallStackEv+0x3e) [0x7f3c52e6339e] #1 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9ExceptionC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x80) [0x7f3c52e63eb0] #2 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN8MLCommon22defaultDeviceAllocator8allocateEmP11CUstream_st+0x102) [0x7f3aaf59cf52] #3 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN15TemporaryMemoryIddE17LevelMemAllocatorEiifiiiib+0x1247) [0x7f3aaf64f757] #4 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML11rfRegressorIdE3fitERKNS_10cumlHandleEPKdiiPdRPNS_20RandomForestMetaDataIddEE+0x900) [0x7f3aaf7dda60] #5 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML3fitERKNS_10cumlHandleERPNS_20RandomForestMetaDataIddEEPdiiS7_NS_9RF_paramsE+0x204) [0x7f3aaf7c24a4] #6 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28446) [0x7f3c452b7446] #7 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28b1b) [0x7f3c452b7b1b] #8 in /opt/conda/bin/python(_PyObject_FastCallKeywords+0x49b) [0x55d29bc3e8fb] #9 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x52f8) [0x55d29bca26e8] #10 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #11 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #12 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #13 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #14 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #15 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #16 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #17 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #18 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #19 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #20 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #21 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #22 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #23 in /opt/conda/bin/python(_PyObject_Call_Prepend+0x63) [0x55d29bc05e53] #24 in /opt/conda/bin/python(PyObject_Call+0x6e) [0x55d29bbf8dbe] #25 in /opt/conda/bin/python(+0x223817) [0x55d29bcf5817] #26 in /opt/conda/bin/python(+0x1de788) [0x55d29bcb0788] #27 in /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f3c79bb26db] #28 in /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f3c798db88f] , Exception occured! file=/conda/conda-bld/libcuml_1583811942451/work/cpp/include/cuml/common/cuml_allocator.hpp line=109: FAIL: call='cudaMalloc(&ptr, n)'. Reason:out of memory Obtained 29 stack frames #0 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9Exception16collectCallStackEv+0x3e) [0x7f3c5f20939e] #1 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9ExceptionC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x80) [0x7f3c5f209eb0] #2 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN8MLCommon22defaultDeviceAllocator8allocateEmP11CUstream_st+0x102) [0x7f3abb8fcf52] #3 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN15TemporaryMemoryIddE17LevelMemAllocatorEiifiiiib+0x1247) [0x7f3abb9af757] #4 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML11rfRegressorIdE3fitERKNS_10cumlHandleEPKdiiPdRPNS_20RandomForestMetaDataIddEE+0x900) [0x7f3abbb3da60] #5 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML3fitERKNS_10cumlHandleERPNS_20RandomForestMetaDataIddEEPdiiS7_NS_9RF_paramsE+0x204) [0x7f3abbb224a4] #6 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28446) [0x7f3c5b291446] #7 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28b1b) [0x7f3c5b291b1b] #8 in /opt/conda/bin/python(_PyObject_FastCallKeywords+0x49b) [0x55d29bc3e8fb] #9 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x52f8) [0x55d29bca26e8] #10 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #11 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #12 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #13 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #14 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #15 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #16 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #17 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #18 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #19 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #20 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #21 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #22 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #23 in /opt/conda/bin/python(_PyObject_Call_Prepend+0x63) [0x55d29bc05e53] #24 in /opt/conda/bin/python(PyObject_Call+0x6e) [0x55d29bbf8dbe] #25 in /opt/conda/bin/python(+0x223817) [0x55d29bcf5817] #26 in /opt/conda/bin/python(+0x1de788) [0x55d29bcb0788] #27 in /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f3c79bb26db] #28 in /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f3c798db88f] , Exception occured! file=/conda/conda-bld/libcuml_1583811942451/work/cpp/include/cuml/common/cuml_allocator.hpp line=109: FAIL: call='cudaMalloc(&ptr, n)'. Reason:out of memory Obtained 29 stack frames #0 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9Exception16collectCallStackEv+0x3e) [0x7f3c54e3939e] #1 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9ExceptionC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x80) [0x7f3c54e39eb0] #2 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN8MLCommon22defaultDeviceAllocator8allocateEmP11CUstream_st+0x102) [0x7f3ac38fcf52] #3 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN15TemporaryMemoryIddE17LevelMemAllocatorEiifiiiib+0x1247) [0x7f3ac39af757] #4 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML11rfRegressorIdE3fitERKNS_10cumlHandleEPKdiiPdRPNS_20RandomForestMetaDataIddEE+0x900) [0x7f3ac3b3da60] #5 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML3fitERKNS_10cumlHandleERPNS_20RandomForestMetaDataIddEEPdiiS7_NS_9RF_paramsE+0x204) [0x7f3ac3b224a4] #6 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28446) [0x7f3c49299446] #7 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28b1b) [0x7f3c49299b1b] #8 in /opt/conda/bin/python(_PyObject_FastCallKeywords+0x49b) [0x55d29bc3e8fb] #9 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x52f8) [0x55d29bca26e8] #10 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #11 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #12 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #13 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #14 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #15 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #16 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #17 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #18 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #19 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #20 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #21 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #22 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #23 in /opt/conda/bin/python(_PyObject_Call_Prepend+0x63) [0x55d29bc05e53] #24 in /opt/conda/bin/python(PyObject_Call+0x6e) [0x55d29bbf8dbe] #25 in /opt/conda/bin/python(+0x223817) [0x55d29bcf5817] #26 in /opt/conda/bin/python(+0x1de788) [0x55d29bcb0788] #27 in /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f3c79bb26db] #28 in /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f3c798db88f] , Exception occured! file=/conda/conda-bld/libcuml_1583811942451/work/cpp/include/cuml/common/cuml_allocator.hpp line=109: FAIL: call='cudaMalloc(&ptr, n)'. Reason:out of memory Obtained 29 stack frames #0 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9Exception16collectCallStackEv+0x3e) [0x7f3c5b60c39e] #1 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9ExceptionC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x80) [0x7f3c5b60ceb0] #2 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN8MLCommon22defaultDeviceAllocator8allocateEmP11CUstream_st+0x102) [0x7f3ab38fcf52] #3 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN15TemporaryMemoryIddE17LevelMemAllocatorEiifiiiib+0x1247) [0x7f3ab39af757] #4 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML11rfRegressorIdE3fitERKNS_10cumlHandleEPKdiiPdRPNS_20RandomForestMetaDataIddEE+0x900) [0x7f3ab3b3da60] #5 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML3fitERKNS_10cumlHandleERPNS_20RandomForestMetaDataIddEEPdiiS7_NS_9RF_paramsE+0x204) [0x7f3ab3b224a4] #6 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28446) [0x7f3c57248446] #7 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28b1b) [0x7f3c57248b1b] #8 in /opt/conda/bin/python(_PyObject_FastCallKeywords+0x49b) [0x55d29bc3e8fb] #9 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x52f8) [0x55d29bca26e8] #10 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #11 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #12 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #13 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #14 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #15 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #16 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #17 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #18 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #19 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #20 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #21 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #22 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #23 in /opt/conda/bin/python(_PyObject_Call_Prepend+0x63) [0x55d29bc05e53] #24 in /opt/conda/bin/python(PyObject_Call+0x6e) [0x55d29bbf8dbe] #25 in /opt/conda/bin/python(+0x223817) [0x55d29bcf5817] #26 in /opt/conda/bin/python(+0x1de788) [0x55d29bcb0788] #27 in /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f3c79bb26db] #28 in /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f3c798db88f] , Exception occured! file=/conda/conda-bld/libcuml_1583811942451/work/cpp/include/cuml/common/cuml_allocator.hpp line=109: FAIL: call='cudaMalloc(&ptr, n)'. Reason:out of memory Obtained 29 stack frames #0 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9Exception16collectCallStackEv+0x3e) [0x7f3c5840b39e] #1 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9ExceptionC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x80) [0x7f3c5840beb0] #2 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN8MLCommon22defaultDeviceAllocator8allocateEmP11CUstream_st+0x102) [0x7f3aa759cf52] #3 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN15TemporaryMemoryIddE17LevelMemAllocatorEiifiiiib+0x1247) [0x7f3aa764f757] #4 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML11rfRegressorIdE3fitERKNS_10cumlHandleEPKdiiPdRPNS_20RandomForestMetaDataIddEE+0x900) [0x7f3aa77dda60] #5 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML3fitERKNS_10cumlHandleERPNS_20RandomForestMetaDataIddEEPdiiS7_NS_9RF_paramsE+0x204) [0x7f3aa77c24a4] #6 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28446) [0x7f3c48637446] #7 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28b1b) [0x7f3c48637b1b] #8 in /opt/conda/bin/python(_PyObject_FastCallKeywords+0x49b) [0x55d29bc3e8fb] #9 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x52f8) [0x55d29bca26e8] #10 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #11 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #12 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #13 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #14 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #15 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #16 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #17 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #18 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #19 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #20 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #21 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #22 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #23 in /opt/conda/bin/python(_PyObject_Call_Prepend+0x63) [0x55d29bc05e53] #24 in /opt/conda/bin/python(PyObject_Call+0x6e) [0x55d29bbf8dbe] #25 in /opt/conda/bin/python(+0x223817) [0x55d29bcf5817] #26 in /opt/conda/bin/python(+0x1de788) [0x55d29bcb0788] #27 in /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f3c79bb26db] #28 in /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f3c798db88f] , Exception occured! file=/conda/conda-bld/libcuml_1583811942451/work/cpp/include/cuml/common/cuml_allocator.hpp line=109: FAIL: call='cudaMalloc(&ptr, n)'. Reason:out of memory Obtained 29 stack frames #0 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9Exception16collectCallStackEv+0x3e) [0x7f3c5ce6b39e] #1 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9ExceptionC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x80) [0x7f3c5ce6beb0] #2 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN8MLCommon22defaultDeviceAllocator8allocateEmP11CUstream_st+0x102) [0x7f3ab359cf52] #3 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN15TemporaryMemoryIddE17LevelMemAllocatorEiifiiiib+0x1247) [0x7f3ab364f757] #4 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML11rfRegressorIdE3fitERKNS_10cumlHandleEPKdiiPdRPNS_20RandomForestMetaDataIddEE+0x900) [0x7f3ab37dda60] #5 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML3fitERKNS_10cumlHandleERPNS_20RandomForestMetaDataIddEEPdiiS7_NS_9RF_paramsE+0x204) [0x7f3ab37c24a4] #6 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28446) [0x7f3c43268446] #7 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28b1b) [0x7f3c43268b1b] #8 in /opt/conda/bin/python(_PyObject_FastCallKeywords+0x49b) [0x55d29bc3e8fb] #9 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x52f8) [0x55d29bca26e8] #10 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #11 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #12 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #13 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #14 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #15 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #16 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #17 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #18 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #19 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #20 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #21 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #22 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #23 in /opt/conda/bin/python(_PyObject_Call_Prepend+0x63) [0x55d29bc05e53] #24 in /opt/conda/bin/python(PyObject_Call+0x6e) [0x55d29bbf8dbe] #25 in /opt/conda/bin/python(+0x223817) [0x55d29bcf5817] #26 in /opt/conda/bin/python(+0x1de788) [0x55d29bcb0788] #27 in /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f3c79bb26db] #28 in /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f3c798db88f] , Exception occured! file=/conda/conda-bld/libcuml_1583811942451/work/cpp/include/cuml/common/cuml_allocator.hpp line=109: FAIL: call='cudaMalloc(&ptr, n)'. Reason:out of memory Obtained 29 stack frames #0 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9Exception16collectCallStackEv+0x3e) [0x7f3c5b66239e] #1 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9ExceptionC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x80) [0x7f3c5b662eb0] #2 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN8MLCommon22defaultDeviceAllocator8allocateEmP11CUstream_st+0x102) [0x7f3aa759cf52] #3 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN15TemporaryMemoryIddE17LevelMemAllocatorEiifiiiib+0x1247) [0x7f3aa764f757] #4 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML11rfRegressorIdE3fitERKNS_10cumlHandleEPKdiiPdRPNS_20RandomForestMetaDataIddEE+0x900) [0x7f3aa77dda60] #5 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML3fitERKNS_10cumlHandleERPNS_20RandomForestMetaDataIddEEPdiiS7_NS_9RF_paramsE+0x204) [0x7f3aa77c24a4] #6 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28446) [0x7f3c49ff1446] #7 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28b1b) [0x7f3c49ff1b1b] #8 in /opt/conda/bin/python(_PyObject_FastCallKeywords+0x49b) [0x55d29bc3e8fb] #9 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x52f8) [0x55d29bca26e8] #10 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #11 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #12 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #13 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #14 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #15 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #16 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #17 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #18 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #19 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #20 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #21 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #22 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #23 in /opt/conda/bin/python(_PyObject_Call_Prepend+0x63) [0x55d29bc05e53] #24 in /opt/conda/bin/python(PyObject_Call+0x6e) [0x55d29bbf8dbe] #25 in /opt/conda/bin/python(+0x223817) [0x55d29bcf5817] #26 in /opt/conda/bin/python(+0x1de788) [0x55d29bcb0788] #27 in /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f3c79bb26db] #28 in /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f3c798db88f] , Exception occured! file=/conda/conda-bld/libcuml_1583811942451/work/cpp/include/cuml/common/cuml_allocator.hpp line=109: FAIL: call='cudaMalloc(&ptr, n)'. Reason:out of memory Obtained 29 stack frames #0 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9Exception16collectCallStackEv+0x3e) [0x7f3bcf61f39e] #1 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9ExceptionC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x80) [0x7f3bcf61feb0] #2 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN8MLCommon22defaultDeviceAllocator8allocateEmP11CUstream_st+0x102) [0x7f3ac38fcf52] #3 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN15TemporaryMemoryIddE17LevelMemAllocatorEiifiiiib+0x1247) [0x7f3ac39af757] #4 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML11rfRegressorIdE3fitERKNS_10cumlHandleEPKdiiPdRPNS_20RandomForestMetaDataIddEE+0x900) [0x7f3ac3b3da60] #5 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML3fitERKNS_10cumlHandleERPNS_20RandomForestMetaDataIddEEPdiiS7_NS_9RF_paramsE+0x204) [0x7f3ac3b224a4] #6 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28446) [0x7f3bc46d6446] #7 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28b1b) [0x7f3bc46d6b1b] #8 in /opt/conda/bin/python(_PyObject_FastCallKeywords+0x49b) [0x55d29bc3e8fb] #9 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x52f8) [0x55d29bca26e8] #10 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #11 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #12 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #13 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #14 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #15 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #16 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #17 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #18 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #19 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #20 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #21 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #22 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #23 in /opt/conda/bin/python(_PyObject_Call_Prepend+0x63) [0x55d29bc05e53] #24 in /opt/conda/bin/python(PyObject_Call+0x6e) [0x55d29bbf8dbe] #25 in /opt/conda/bin/python(+0x223817) [0x55d29bcf5817] #26 in /opt/conda/bin/python(+0x1de788) [0x55d29bcb0788] #27 in /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f3c79bb26db] #28 in /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f3c798db88f] , Exception occured! file=/conda/conda-bld/libcuml_1583811942451/work/cpp/include/cuml/common/cuml_allocator.hpp line=109: FAIL: call='cudaMalloc(&ptr, n)'. Reason:out of memory Obtained 29 stack frames #0 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9Exception16collectCallStackEv+0x3e) [0x7f3c6565939e] #1 in /opt/conda/lib/python3.7/site-packages/cuml/utils/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9ExceptionC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x80) [0x7f3c65659eb0] #2 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN8MLCommon22defaultDeviceAllocator8allocateEmP11CUstream_st+0x102) [0x7f3ab759cf52] #3 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN15TemporaryMemoryIddE17LevelMemAllocatorEiifiiiib+0x1247) [0x7f3ab764f757] #4 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML11rfRegressorIdE3fitERKNS_10cumlHandleEPKdiiPdRPNS_20RandomForestMetaDataIddEE+0x900) [0x7f3ab77dda60] #5 in /opt/conda/lib/python3.7/site-packages/cuml/utils/../../../../libcuml++.so(_ZN2ML3fitERKNS_10cumlHandleERPNS_20RandomForestMetaDataIddEEPdiiS7_NS_9RF_paramsE+0x204) [0x7f3ab77c24a4] #6 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28446) [0x7f3c4f2c4446] #7 in /opt/conda/lib/python3.7/site-packages/cuml/ensemble/randomforestregressor.cpython-37m-x86_64-linux-gnu.so(+0x28b1b) [0x7f3c4f2c4b1b] #8 in /opt/conda/bin/python(_PyObject_FastCallKeywords+0x49b) [0x55d29bc3e8fb] #9 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x52f8) [0x55d29bca26e8] #10 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #11 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #12 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #13 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #14 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #15 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #16 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #17 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x1e42) [0x55d29bc9f232] #18 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #19 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #20 in /opt/conda/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x55d29bc35ccb] #21 in /opt/conda/bin/python(_PyEval_EvalFrameDefault+0x6a3) [0x55d29bc9da93] #22 in /opt/conda/bin/python(_PyFunction_FastCallDict+0x10b) [0x55d29bbe756b] #23 in /opt/conda/bin/python(_PyObject_Call_Prepend+0x63) [0x55d29bc05e53] #24 in /opt/conda/bin/python(PyObject_Call+0x6e) [0x55d29bbf8dbe] #25 in /opt/conda/bin/python(+0x223817) [0x55d29bcf5817] #26 in /opt/conda/bin/python(+0x1de788) [0x55d29bcb0788] #27 in /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f3c79bb26db] #28 in /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f3c798db88f]
bug
nikiforov-sm
All 62 comments
@nikiforov-sm correct me if I am wrong, but that's 10^10 data-points? With float64, which numpy implicitly uses, that's 80 GB of data. How many GPUs are you using and what are the specifications of those?
divyegala
on 9 Apr 2020
@divyegala I'm using dask_cudf dataframe to train the dask_cuml.RandomForest on DGX-2 with 16 GPU.
nikiforov-sm
on 9 Apr 2020
@nikiforov-sm could you tell me what version of cuml/RAPIDS you are using and other environment details like CUDA version, driver version, etc.? Also, for completeness, are those 32 GB V100s in your DGX-2 or 16 GB?
divyegala
on 9 Apr 2020
Also tagging @Salonijain27 for thoughts/visibility
divyegala
on 9 Apr 2020
Hi @divyegala ,
cuml version is 0.13.0a+1003.g4a7070f, GPUs with 32GB
nikiforov-sm
on 9 Apr 2020
Hi @nikiforov-sm , at the moment, by looking at the model parameters such as number of trees, max_depth and the size of dataset, the model will be running out of memory, afaik .
We will create a feature request to update the RF code to reduce the memory consumed for deeper and larger forests.
At the moment i would recommend breaking up the dataset used on this model and training 2 or 3 models with it. Then running predict on all of the models and taking the mean of their predicted value as the final result.
Or you can also try increasing the number of partitions per worker and using float32 dataset.
Salonijain27
on 9 Apr 2020
Hi @Salonijain27,
Thank you for the update.
I’ve tested different number of partitions per worker – 1, 100, 1000 with float32 dataset with OOM.
nikiforov-sm
on 9 Apr 2020
@nikiforov-sm - with a random dataset and max_depth = 50 for regression, we could in the worst case have up to one leaf node per sample, so nearly 100M nodes total per tree * 100 trees = 10Bln nodes.
The training or inference data can be distributed over multiple GPUs, but the actual forest structure itself must be able to fit in a single GPU's memory. Unfortunately, even if we aren't quite in the worst possible case, that size of a tree may not fit in a GPU's memory.
By far the biggest lever to reduce memory consumption would be to reduce max_depth for trees. Increasing min_rows_per_node would also help to reduce tree size. @Salonijain27 's proposed method above (training separate models in parallel with fewer trees) is also a potential workaround.
Additionally, converting to float32 is strongly recommended - it will not only decrease memory consumption but also improve performance and allow full use of the FIL inference library for inference.
To the extent that data loading is an issue, loading data natively onto the workers with dask_cudf (or a gpu-based data generation function like cuml.dask.make_regression) will also help reduce load time for large datasets. @miroenev will follow up with additional details and some sample code.
JohnZed
on 14 Apr 2020
adding @jamesmaki who did all the great work behind our experiments!
miroenev
on 15 Apr 2020
We were able to successfully train depth 25 trees with 200M rows and 20 columns on 2 V100s.
We swapped out the data generation to a chunked dask array function using dask.array.random that will create 4 partitions per worker and cast that array to float32 dtype.
x = da.random.random((rows, cols), chunks=(rows//(n_workers*4) ,cols)).astype('float32')
Also removed pandas and numpy completely by changing the dask_cudf load to:
df_dask = dask_cudf.from_dask_dataframe(dd.from_array(x))
This helped resolve memory problems stemming from a very large pandas dataframe wtih float64 dtypes being converted into a dask dataframe with only one partition per worker.
We also made some hyperparameter changes to manage memory usage:
max_depth=25
Tree depth has a large impact on memory usage. The code below will likely run at higher depth on a DGX2, but a depth of 50 is still extreme. I would recommend starting smaller and letting hyperparameter optimization determine the value (if you are getting better results at a depth of 17 compared to a depth of 30, it doesn't make sense to train again with depth 50).n_streams=1
This is the number of parallel streams used for forest building. Higher values ofn_streamswill result in faster training up to a point but requires higher memory usage.min_rows_per_node
Made little difference on this example because of the random data but will likely be a valuable for tree regularization on real problems and should increase overall model performance by reducing over-fitting.
We used the jupyter dashboards as we were profiling this workload (very helpful to understand memory utilization during data generation and model training).
This code took 14 minutes and 24 seconds to run on 2 V100s:
````python
from cuml.dask.common import utils as dask_utils
from dask.distributed import Client, wait
from dask_cuda import LocalCUDACluster
import dask_cudf
import dask.dataframe as dd
import dask.array as da
from cuml.dask.ensemble import RandomForestRegressor as cumlDaskRF
cluster = LocalCUDACluster()
c = Client(cluster)
workers = c.has_what().keys()
n_workers = len(workers)
n_streams = 1
Desired parameters
max_depth = 25
n_trees = 100
rows, cols = 200_000_000, 20
random array with 4 chunks per worker and cast to float32 instead of float64
x = da.random.random((rows, cols), chunks=(rows//(n_workers*4) ,cols)).astype('float32')
df_dask = dask_cudf.from_dask_dataframe(dd.from_array(x))
X_train_dask, y_train_dask = dask_utils.persist_across_workers(c, [df_dask.drop(2,axis=1), df_dask[2]], workers=workers)
cuml_model = cumlDaskRF(max_depth=max_depth, n_estimators=n_trees,n_streams=n_streams,workers = workers)
cuml_model.fit(X_train_dask, y_train_dask)
wait(cuml_model.rfs)
````
JamesMaki
on 15 Apr 2020
@JohnZed , @JamesMaki
Thank you for this update. It is very helpful example.
We are using grafana utility to monitor GPU memory and utilization in NVIDIA DGX2.
@JamesMaki example works, thank you!
There is no OOM with rows, cols = 200_000_000, 90
There is OOM with rows, cols = 500_000_000, 20.
But we need to train forest on dataset with 500_000_000 rows and 90 columns.
With this numbers of rows and columns I have OOM exception.
Is there single solution only - split our dataset to the small datasets and train several models with averaging their forecast to the result forecast?
nikiforov-sm
on 15 Apr 2020
Can you try again with 20 partitions per worker? I try to avoid having partitions that are much larger than 1 GB.
x = da.random.random((rows, cols), chunks=(rows//(n_workers*20) ,cols)).astype('float32')
There shouldn't be an issue loading onto the GPUs with 500_000_000 rows and 90 columns but I'm not sure there will be enough memory left to handle the overhead of training RandomForest. You could test with max_depth=3, n_streams=1, n_estimators=16 so each GPU trains just a single very simple tree.
A few other options instead of training separate models are:
- Random sampling (should work well given how large your dataset is)
````python
40% random sample
df_sample = df_dask.sample(frac=0.4)
or 50%, 30%, 20% split
df_dask1, df_dask2, df_dask3 = df_dask.random_split([0.5, 0.3, 0.2])
````
- Dimensionality reduction with PCA to reduce the number of columns - example here and docs here. This works well distributed with dask_cudf.
I would recommend a combination of 1 and 2.
JamesMaki
on 15 Apr 2020
@JamesMaki Thank you!
Can you try again with 20 partitions per worker? I try to avoid having partitions that are much larger than 1 GB.
Yes, the same issue with OOM with 20 partitions per worker like this:
x = da.random.random((rows, cols), chunks=(rows//(n_workers*20) ,cols)).astype('float32')
1. Random sampling (should work well given how large your dataset is)After sampling dataframe fit routine fails with exception
---------------------------------------------------------------------------
StopIteration Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/cuml/dask/common/dask_df_utils.py in extract_ddf_partitions(ddf, client)
48 for key, workers in who_has.items():
---> 49 worker = first(workers)
50 worker_map[key_to_part_dict[key]] = worker
/opt/conda/lib/python3.7/site-packages/toolz/itertoolz.py in first(seq)
375 """
--> 376 return next(iter(seq))
377
StopIteration:
The above exception was the direct cause of the following exception:
RuntimeError Traceback (most recent call last)
<ipython-input-1-97fffc19489b> in <module>
29 #min_rows_per_node=100
30 )
---> 31 cuml_model.fit(X_train_dask, y_train_dask)
32
33 wait(cuml_model.rfs)
/opt/conda/lib/python3.7/site-packages/cuml/dask/ensemble/randomforestregressor.py in fit(self, X, y)
331 c = default_client()
332
--> 333 X_futures = workers_to_parts(c.sync(extract_ddf_partitions, X))
334 y_futures = workers_to_parts(c.sync(extract_ddf_partitions, y))
335
/opt/conda/lib/python3.7/site-packages/distributed/client.py in sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
767 else:
768 return sync(
--> 769 self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
770 )
771
/opt/conda/lib/python3.7/site-packages/distributed/utils.py in sync(loop, func, callback_timeout, *args, **kwargs)
333 if error[0]:
334 typ, exc, tb = error[0]
--> 335 raise exc.with_traceback(tb)
336 else:
337 return result[0]
/opt/conda/lib/python3.7/site-packages/distributed/utils.py in f()
317 if callback_timeout is not None:
318 future = gen.with_timeout(timedelta(seconds=callback_timeout), future)
--> 319 result[0] = yield future
320 except Exception as exc:
321 error[0] = sys.exc_info()
/opt/conda/lib/python3.7/site-packages/tornado/gen.py in run(self)
733
734 try:
--> 735 value = future.result()
736 except Exception:
737 exc_info = sys.exc_info()
/opt/conda/lib/python3.7/site-packages/tornado/gen.py in run(self)
746 exc_info = None
747 else:
--> 748 yielded = self.gen.send(value)
749
750 except (StopIteration, Return) as e:
RuntimeError: generator raised StopIteration
2. Dimensionality reduction with PCA to reduce the number of columns - example [here](https://github.com/rapidsai/cuml/blob/branch-0.13/notebooks/pca_demo.ipynb) and docs [here](https://docs.rapids.ai/api/cuml/stable/api.html#principal-component-analysis). This works well distributed with dask_cudf.
from cuml.dask.common import utils as dask_utils
from dask.distributed import Client, wait
from dask_cuda import LocalCUDACluster
import dask_cudf
import dask.dataframe as dd
import dask.array as da
from cuml.dask.ensemble import RandomForestRegressor as cumlDaskRF
from cuml.decomposition import PCA as cuPCA
cluster = LocalCUDACluster()
c = Client(cluster)
workers = c.has_what().keys()
n_workers = len(workers)
n_streams = 1
n_components = 20
whiten = False
svd_solver = "full"
random_state = 777
# Desired parameters
max_depth = 25
n_trees = 20
rows, cols = 500_000_000, 90
#random array with 4 chunks per worker and cast to float32 instead of float64
x = da.random.random((rows, cols), chunks=(rows//(n_workers*20) ,cols)).astype('float32')
df_dask = dask_cudf.from_dask_dataframe(dd.from_array(x))
X_train_dask, y_train_dask = dask_utils.persist_across_workers(c, [df_dask.drop(2,axis=1), df_dask[2]], workers=workers)
pca_cuml = cuPCA(n_components=n_components,
svd_solver=svd_solver,
whiten=whiten,
random_state=random_state)
X_train_dask = pca_cuml.fit_transform(X_train_dask)
fails with exception
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-1-ef74c4306432> in <module>
34 whiten=whiten,
35 random_state=random_state)
---> 36 X_train_dask = pca_cuml.fit_transform(X_train_dask)
37
38 cuml_model = cumlDaskRF(max_depth=max_depth, n_estimators=n_trees,n_streams=n_streams,workers = workers,
cuml/decomposition/pca.pyx in cuml.decomposition.pca.PCA.fit_transform()
cuml/decomposition/pca.pyx in cuml.decomposition.pca.PCA.fit()
/opt/conda/lib/python3.7/site-packages/cuml/utils/input_utils.py in input_to_cuml_array(X, order, deepcopy, check_dtype, convert_to_dtype, check_cols, check_rows, fail_on_order)
147 else:
148 msg = "X matrix format " + str(X.__class__) + " not supported"
--> 149 raise TypeError(msg)
150
151 if check_dtype:
TypeError: X matrix format <class 'dask_cudf.core.DataFrame'> not supported
Is dask_cudf dataframe supported by cuML PCA?
nikiforov-sm
on 16 Apr 2020
Are those first two errors using the original max_depth=50, n_streams=8, and n_estimators=100? I'm happy the data is loading onto the GPUs but I don't think the model will train with those parameters. Did you test with the very simple parameters max_depth=3, n_streams=1, n_estimators=16? I think we should get a simple model to train and gradually increase complexity from there until we start hitting the OOM errors. Can you please post the full code you used with your sampling attempt?
For PCA the distributed version is in the dask module so instead of from from cuml.decomposition import PCA use from cuml.dask.decomposition import PCA.
JamesMaki
on 16 Apr 2020
This error was with script:
from cuml.dask.common import utils as dask_utils
from dask.distributed import Client, wait
from dask_cuda import LocalCUDACluster
import dask_cudf
import dask.dataframe as dd
import dask.array as da
from cuml.dask.ensemble import RandomForestRegressor as cumlDaskRF
cluster = LocalCUDACluster()
c = Client(cluster)
workers = c.has_what().keys()
n_workers = len(workers)
n_streams = 1
# Desired parameters
max_depth = 25
n_trees = 20
rows, cols = 500_000_000, 90
#random array with 20 chunks per worker and cast to float32 instead of float64
x = da.random.random((rows, cols), chunks=(rows//(n_workers*20) ,cols)).astype('float32')
df_dask = dask_cudf.from_dask_dataframe(dd.from_array(x))
# apply sampling
df_dask = df_dask.sample(frac=0.4)
X_train_dask, y_train_dask = dask_utils.persist_across_workers(c, [df_dask.drop(2,axis=1), df_dask[2]], workers=workers)
cuml_model = cumlDaskRF(max_depth=max_depth, n_estimators=n_trees,n_streams=n_streams,workers = workers)
cuml_model.fit(X_train_dask, y_train_dask)
wait(cuml_model.rfs)
With max_depth=3, n_streams=1, n_estimators=16 the same exception.
nikiforov-sm
on 16 Apr 2020
With PCA from cuml.dask.decomposition error in script
from cuml.dask.common import utils as dask_utils
from dask.distributed import Client, wait
from dask_cuda import LocalCUDACluster
import dask_cudf
import dask.dataframe as dd
import dask.array as da
from cuml.dask.ensemble import RandomForestRegressor as cumlDaskRF
from cuml.dask.decomposition import PCA as cuPCA
cluster = LocalCUDACluster()
c = Client(cluster)
workers = c.has_what().keys()
n_workers = len(workers)
n_streams = 1
n_components = 20
whiten = False
svd_solver = "full"
random_state = 777
# Desired parameters
max_depth = 25
n_trees = 20
rows, cols = 500_000_000, 90
#random array with 20 chunks per worker and cast to float32 instead of float64
x = da.random.random((rows, cols), chunks=(rows//(n_workers*20) ,cols)).astype('float32')
df_dask = dask_cudf.from_dask_dataframe(dd.from_array(x))
X_train_dask, y_train_dask = dask_utils.persist_across_workers(c, [df_dask.drop(2,axis=1), df_dask[2]], workers=workers)
pca_cuml = cuPCA(n_components=n_components,
svd_solver=svd_solver,
whiten=whiten,
random_state=random_state)
X_train_dask = pca_cuml.fit_transform(X_train_dask)
cuml_model = cumlDaskRF(max_depth=max_depth, n_estimators=n_trees,n_streams=n_streams,workers = workers)
cuml_model.fit(X_train_dask, y_train_dask)
wait(cuml_model.rfs)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-1-e7faa386646b> in <module>
33 svd_solver=svd_solver,
34 whiten=whiten,
---> 35 random_state=random_state)
36 X_train_dask = pca_cuml.fit_transform(X_train_dask)
37
/opt/conda/lib/python3.7/site-packages/cuml/dask/decomposition/pca.py in __init__(self, client, **kwargs)
155 def __init__(self, client=None, **kwargs):
156
--> 157 super(PCA, self).__init__(PCA._create_pca, client, **kwargs)
158 self.noise_variance_ = None
159
/opt/conda/lib/python3.7/site-packages/cuml/dask/decomposition/base.py in __init__(self, model_func, client, **kwargs)
35 Constructor for distributed decomposition model
36 """
---> 37 super(BaseDecomposition, self).__init__(client, **kwargs)
38 self._model_func = model_func
39
/opt/conda/lib/python3.7/site-packages/cuml/dask/common/base.py in __init__(self, client, **kwargs)
35 self.client = default_client() if client is None else client
36
---> 37 patch_cupy_sparse_serialization(self.client)
38
39 self.kwargs = kwargs
/opt/conda/lib/python3.7/site-packages/cuml/dask/common/utils.py in patch_cupy_sparse_serialization(client)
182 serialize_mat_descriptor)
183
--> 184 patch_func()
185 client.run(patch_func)
186
/opt/conda/lib/python3.7/site-packages/cuml/dask/common/utils.py in patch_func()
171 dask_deserialize, register_generic
172
--> 173 register_generic(Base, "cuda", cuda_serialize, cuda_deserialize)
174 register_generic(Base, "dask", dask_serialize, dask_deserialize)
175
TypeError: register_generic() takes 1 positional argument but 4 were given
I'm working with cuml version '0.13.0a+1120.g8a658ab'
nikiforov-sm
on 16 Apr 2020
It looks like cudf.DataFrame.sample is not implemented yet so that is why part 1 failed. The code I provided was for regular dask dataframes backed by pandas, sorry about that. https://github.com/rapidsai/cudf/issues/1426
You can simulate sampling on your random data with df_dask = df_dask[df_dask[1] <= 0.4] or df_dask = df_dask.query('1 <= 0.4')
I'll see if I can re-produce the PCA error. Thanks!
JamesMaki
on 16 Apr 2020
Distributed PCA and TruncatedSVD are both working for me with 200_000_000 rows and 20 columns. I think you will need a combination of one of random sampling followed by one of these algorithms to get your dataset down to a manageable size without losing information.
JamesMaki
on 16 Apr 2020
@JamesMaki Thanks! Could you show example with distributed PCA?
I have an error on initializing PCA with parameters.
nikiforov-sm
on 16 Apr 2020
Both of these worked for me, I just kept the default parameters.
python
from cuml.dask.decomposition import PCA
pca = PCA(n_components = 5)
df_dask = pca.fit_transform(df_dask)
or
python
from cuml.dask.decomposition import TruncatedSVD
tsvd = TruncatedSVD(n_components = 5)
df_dask = tsvd.fit_transform(df_dask)
JamesMaki
on 16 Apr 2020
Thanks! What version of cuml do you use?
nikiforov-sm
on 16 Apr 2020
I’m also on 0.13
JamesMaki
on 16 Apr 2020
I’m also on 0.13
On my version I have an error:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-1-f7568bf303fc> in <module>
33 # Apply PCA
34 #pca_cuml = cuPCA(n_components=n_components,svd_solver=svd_solver,whiten=whiten,random_state=random_state)
---> 35 pca = cuPCA(n_components = n_components)
36 X_train_dask = pca_cuml.fit_transform(X_train_dask)
37
/opt/conda/lib/python3.7/site-packages/cuml/dask/decomposition/pca.py in __init__(self, client, **kwargs)
155 def __init__(self, client=None, **kwargs):
156
--> 157 super(PCA, self).__init__(PCA._create_pca, client, **kwargs)
158 self.noise_variance_ = None
159
/opt/conda/lib/python3.7/site-packages/cuml/dask/decomposition/base.py in __init__(self, model_func, client, **kwargs)
35 Constructor for distributed decomposition model
36 """
---> 37 super(BaseDecomposition, self).__init__(client, **kwargs)
38 self._model_func = model_func
39
/opt/conda/lib/python3.7/site-packages/cuml/dask/common/base.py in __init__(self, client, **kwargs)
35 self.client = default_client() if client is None else client
36
---> 37 patch_cupy_sparse_serialization(self.client)
38
39 self.kwargs = kwargs
/opt/conda/lib/python3.7/site-packages/cuml/dask/common/utils.py in patch_cupy_sparse_serialization(client)
182 serialize_mat_descriptor)
183
--> 184 patch_func()
185 client.run(patch_func)
186
/opt/conda/lib/python3.7/site-packages/cuml/dask/common/utils.py in patch_func()
171 dask_deserialize, register_generic
172
--> 173 register_generic(Base, "cuda", cuda_serialize, cuda_deserialize)
174 register_generic(Base, "dask", dask_serialize, dask_deserialize)
175
TypeError: register_generic() takes 1 positional argument but 4 were given
Does PCA work with 200_000_000 rows and 20 columns? There may be an issue with your install of RAPIDS. I can run it on 2x V100s so a DGX-2 should do it very easily.
My full cuML version is: 0.13.0a+1351.g424c3f6
You may have an older version from before the stable release of 0.13
JamesMaki
on 16 Apr 2020
No, PCA doesn't work with any numbers of rows and columns.
My cuML version is: 0.13.0a+1120.g8a658ab
nikiforov-sm
on 16 Apr 2020
You should be able to update from that unstable version to the stable release with:
conda install -y -c rapidsai rapids=0.13
You could also use the RAPIDS 0.13 container from here: https://ngc.nvidia.com/catalog/containers/nvidia:rapidsai:rapidsai/tags
This should solve some problems.
JamesMaki
on 16 Apr 2020
I thought of two more things I wanted to include as you test today:
- You could sample directly from the dask dataframe before it is passed to dask_cudf.
python ddf = dd.from_array(x) ddf = ddf.sample(frac=0.5) #50% sample df_dask = dask_cudf.from_dask_dataframe(ddf) - You can use dask_ml train_test_split and sample your dask_cudf dataframe that way as well.
python from dask_ml.model_selection import train_test_split X_train, y_train, _, _ = train_test_split(df_dask.drop(2, axis=1), df_dask[2], train_size = 0.5) #50% sample
JamesMaki
on 17 Apr 2020
@JamesMaki , I have an error on predict in the following script on cuML version 0.13.0a+1351.g424c3f6.dirty:
from cuml.dask.common import utils as dask_utils
from dask.distributed import Client, wait
from dask_cuda import LocalCUDACluster
import dask_cudf
import dask.dataframe as dd
import dask.array as da
from cuml.dask.ensemble import RandomForestRegressor as cumlDaskRF
from cuml.dask.decomposition import PCA as cuPCA
cluster = LocalCUDACluster()
c = Client(cluster)
workers = c.has_what().keys()
n_workers = len(workers)
n_streams = 1
# Desired parameters
max_depth = 3
n_trees = 16
rows, cols = 200_000_000, 20
#random array with 20 chunks per worker and cast to float32 instead of float64
x = da.random.random((rows, cols), chunks=(rows//(n_workers*20) ,cols)).astype('float32')
df_dd = dd.from_array(x)
df_dask = dask_cudf.from_dask_dataframe(df_dd)
X_train_dask, y_train_dask = dask_utils.persist_across_workers(c, [df_dask.drop(2,axis=1), df_dask[2]], workers=workers)
cuml_model = cumlDaskRF(max_depth=max_depth, n_estimators=n_trees,n_streams=n_streams,workers = workers)
cuml_model.fit(X_train_dask, y_train_dask)
wait(cuml_model.rfs)
x_test, y_test = df_dd.drop(2,axis=1).to_dask_array().compute(), df_dd[2].to_dask_array().compute()
cuml_y_pred = cuml_model.predict(x_test)
mse = mean_squared_error(y_test, cuml_y_pred)
print(mse)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-1-7d7251c3f4be> in <module>
33
34 x_test, y_test = df_dd.drop(2,axis=1).to_dask_array().compute(), df_dd[2].to_dask_array().compute()
---> 35 cuml_y_pred = cuml_model.predict(x_test)
36 mse = mean_squared_error(y_test, cuml_y_pred)
37 print(mse)
/opt/conda/envs/rapids/lib/python3.6/site-packages/cuml/dask/ensemble/randomforestregressor.py in predict(self, X, predict_model, algo, convert_dtype, fil_sparse_format, delayed)
469 convert_dtype=convert_dtype,
470 fil_sparse_format=fil_sparse_format,
--> 471 delayed=delayed)
472 return preds
473
/opt/conda/envs/rapids/lib/python3.6/site-packages/cuml/dask/ensemble/randomforestregressor.py in _predict_using_fil(self, X, predict_model, algo, convert_dtype, fil_sparse_format, delayed)
475 convert_dtype=True, fil_sparse_format='auto',
476 delayed=True):
--> 477 self._concat_treelite_models()
478 data = DistributedDataHandler.single(X, client=self.client)
479 self.datatype = data.datatype
/opt/conda/envs/rapids/lib/python3.6/site-packages/cuml/dask/ensemble/randomforestregressor.py in _concat_treelite_models(self)
332 last_worker = w
333 all_tl_mod_handles = []
--> 334 model = self.rfs[last_worker].result()
335 for n in range(len(self.workers)):
336 all_tl_mod_handles.append(model._tl_model_handles(mod_bytes[n]))
/opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/client.py in result(self, timeout)
215
216 # shorten error traceback
--> 217 result = self.client.sync(self._result, callback_timeout=timeout, raiseit=False)
218 if self.status == "error":
219 typ, exc, tb = result
/opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/client.py in sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
778 else:
779 return sync(
--> 780 self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
781 )
782
/opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/utils.py in sync(loop, func, callback_timeout, *args, **kwargs)
346 if error[0]:
347 typ, exc, tb = error[0]
--> 348 raise exc.with_traceback(tb)
349 else:
350 return result[0]
/opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/utils.py in f()
330 if callback_timeout is not None:
331 future = asyncio.wait_for(future, callback_timeout)
--> 332 result[0] = yield future
333 except Exception as exc:
334 error[0] = sys.exc_info()
/opt/conda/envs/rapids/lib/python3.6/site-packages/tornado/gen.py in run(self)
733
734 try:
--> 735 value = future.result()
736 except Exception:
737 exc_info = sys.exc_info()
/opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/client.py in _result(self, raiseit)
240 return exception
241 else:
--> 242 result = await self.client._gather([self])
243 return result[0]
244
/opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/client.py in _gather(self, futures, errors, direct, local_worker)
1779 else:
1780 self._gather_future = future
-> 1781 response = await future
1782
1783 if response["status"] == "error":
/opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/client.py in _gather_remote(self, direct, local_worker)
1830
1831 else: # ask scheduler to gather data for us
-> 1832 response = await retry_operation(self.scheduler.gather, keys=keys)
1833
1834 return response
/opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/utils_comm.py in retry_operation(coro, operation, *args, **kwargs)
389 delay_min=retry_delay_min,
390 delay_max=retry_delay_max,
--> 391 operation=operation,
392 )
/opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/utils_comm.py in retry(coro, count, delay_min, delay_max, jitter_fraction, retry_on_exceptions, operation)
377 delay *= 1 + random.random() * jitter_fraction
378 await asyncio.sleep(delay)
--> 379 return await coro()
380
381
/opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/core.py in send_recv_from_rpc(**kwargs)
755 name, comm.name = comm.name, "ConnectionPool." + key
756 try:
--> 757 result = await send_recv(comm=comm, op=key, **kwargs)
758 finally:
759 self.pool.reuse(self.addr, comm)
/opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/core.py in send_recv(comm, reply, serializers, deserializers, **kwargs)
538 await comm.write(msg, serializers=serializers, on_error="raise")
539 if reply:
--> 540 response = await comm.read(deserializers=deserializers)
541 else:
542 response = None
/opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/comm/tcp.py in read(self, deserializers)
210 try:
211 msg = await from_frames(
--> 212 frames, deserialize=self.deserialize, deserializers=deserializers
213 )
214 except EOFError:
/opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/comm/utils.py in from_frames(frames, deserialize, deserializers)
73 res = await offload(_from_frames)
74 else:
---> 75 res = _from_frames()
76
77 return res
/opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/comm/utils.py in _from_frames()
59 try:
60 return protocol.loads(
---> 61 frames, deserialize=deserialize, deserializers=deserializers
62 )
63 except EOFError:
/opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/protocol/core.py in loads(frames, deserialize, deserializers)
122 fs = decompress(head, fs)
123 fs = merge_frames(head, fs)
--> 124 value = _deserialize(head, fs, deserializers=deserializers)
125 else:
126 value = Serialized(head, fs)
/opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/protocol/serialize.py in deserialize(header, frames, deserializers)
266 )
267 dumps, loads, wants_context = families[name]
--> 268 return loads(header, frames)
269
270
/opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/protocol/serialize.py in serialization_error_loads(header, frames)
78 def serialization_error_loads(header, frames):
79 msg = "\n".join([ensure_bytes(frame).decode("utf8") for frame in frames])
---> 80 raise TypeError(msg)
81
82
TypeError: Could not serialize object of type RandomForestRegressor.
Traceback (most recent call last):
File "/opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/protocol/pickle.py", line 38, in dumps
result = pickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL)
File "cuml/ensemble/randomforestregressor.pyx", line 304, in cuml.ensemble.randomforestregressor.RandomForestRegressor.__getstate__
File "cuml/ensemble/randomforestregressor.pyx", line 389, in cuml.ensemble.randomforestregressor.RandomForestRegressor._get_protobuf_bytes
File "cuml/ensemble/randomforestregressor.pyx", line 379, in cuml.ensemble.randomforestregressor.RandomForestRegressor._obtain_treelite_handle
RuntimeError: Exception occured! file=/conda/conda-bld/libcuml_1585623076737/work/cpp/src/randomforest/randomforest.cu line=291: TREELITE FAIL: call='TreeliteLoadProtobufModel(filename, model)'. Reason:[13:54:27] /conda/conda-bld/libcuml_1585623076737/work/cpp/build/treelite/src/treelite/src/frontend/protobuf.cc:79: Check failed: protomodel.has_num_feature() num_feature must exist
PCA reduce dimensions only on fit_transform.
It's not reducing dimensions on transform method.
pca = cuPCA(n_components = n_components)
print(x_train.shape)
pca = pca.fit(x_train)
x_train = pca.transform(x_train)
print(x_train.shape)
return
(Delayed('int-da06d31c-c48a-492e-9e8e-e1de5d1ca772'), 89)
(Delayed('int-6af6f8b8-23b3-432e-bfff-c14bf448ee29'), 89)
I have to use transform method for test-dataset.
nikiforov-sm
on 22 Apr 2020
@nikiforov-sm Looking into the error that you have posted:
https://github.com/rapidsai/cuml/issues/1998#issuecomment-617107720
Salonijain27
on 22 Apr 2020
@Salonijain27 Thank you for response. Did you manage to reproduce the error?
nikiforov-sm
on 23 Apr 2020
@nikiforov-sm Thank you for providing the test script, I am able to reproduce the error with it. Looking into whats causing this error.
Salonijain27
on 23 Apr 2020
@nikiforov-sm, I have opened an issue for the error :
https://github.com/rapidsai/cuml/issues/1998#issuecomment-617107720
2141
Salonijain27
on 28 Apr 2020
@Salonijain27 Thank you for the update.
Did you manage to reproduce the error with PCA?
nikiforov-sm
on 28 Apr 2020
@nikiforov-sm @Salonijain27
I was able to reproduce the PCA error, this will probably need to be opened as a bug as well. Could you use Truncated-SVD instead for now? It seems to be working correctly.
Edit: Opened the bug here: https://github.com/rapidsai/cuml/issues/2157
JamesMaki
on 28 Apr 2020
@nikiforov-sm yes, i am able to reproduce the PCA error mentioned in : https://github.com/rapidsai/cuml/issues/1998#issuecomment-617805886
Salonijain27
on 29 Apr 2020
@JamesMaki @Salonijain27 Thank you for your updates.
I can use Truncated-SVD instead of PCA and it's successfully reduce columns, but predict is not working as I described in #1998 (comment)
nikiforov-sm
on 29 Apr 2020
@nikiforov-sm the fix for the random forest bug #2141 has been merged
Salonijain27
on 30 Apr 2020
@Salonijain27 Thank you! Should I update cuML version upto 0.14?
nikiforov-sm
on 1 May 2020
@nikiforov-sm yes please, you can use the nightly release of cuml-0.14
Salonijain27
on 4 May 2020
1. You can use dask_ml train_test_split and sample your dask_cudf dataframe that way as well.from dask_ml.model_selection import train_test_split X_train, y_train, _, _ = train_test_split(df_dask.drop(2, axis=1), df_dask[2], train_size = 0.5) #50% sample
It seems to be not working correctly with shuffle=True:
from cuml.dask.common import utils as dask_utils
from dask.distributed import Client, wait
from dask_cuda import LocalCUDACluster
import dask_cudf
import dask.dataframe as dd
import dask.array as da
from cuml.dask.ensemble import RandomForestRegressor as cumlDaskRF
from dask_ml.model_selection import train_test_split
cluster = LocalCUDACluster()
c = Client(cluster)
workers = c.has_what().keys()
n_workers = len(workers)
n_streams = 1
# Desired parameters
max_depth = 3
n_trees = 16
rows, cols = 100_000, 20
# Prepare datasets
#random array with 20 chunks per worker and cast to float32 instead of float64
x = da.random.random((rows, cols), chunks=(rows//(n_workers*20),cols)).astype('float32')
df_dd = dd.from_array(x)
df_dask = dask_cudf.from_dask_dataframe(df_dd)
X_train, X_test, y_train, y_test = train_test_split(df_dask.drop(2,axis=1), df_dask[2], shuffle=True, train_size = 0.8, random_state=31)
X_train_dask, y_train_dask = dask_utils.persist_across_workers(c, [X_train, y_train], workers=workers)
# Train model
cuml_model = cumlDaskRF(max_depth=max_depth, n_estimators=n_trees,n_streams=n_streams,workers = workers)
cuml_model.fit(X_train_dask, y_train_dask)
wait(cuml_model.rfs)
# Evaluate model
x_test, y_test = dask_utils.persist_across_workers(c, [X_test, y_test], workers=workers)
y_test = y_test.to_dask_array().compute().get()
cuml_y_pred = cuml_model.predict(x_test)
cuml_y_pred = cuml_y_pred.to_dask_array().compute().get()
from sklearn.metrics import mean_squared_error
mse = mean_squared_error(y_test, cuml_y_pred)
print(mse)
---------------------------------------------------------------------------
StopIteration Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/cuml/dask/common/part_utils.py in _extract_partitions(dask_obj, client)
156 raise gen.Return([(first(who_has[key]), part)
--> 157 for key, part in key_to_part])
/opt/conda/lib/python3.7/site-packages/cuml/dask/common/part_utils.py in <listcomp>(.0)
156 raise gen.Return([(first(who_has[key]), part)
--> 157 for key, part in key_to_part])
/opt/conda/lib/python3.7/site-packages/toolz/itertoolz.py in first(seq)
375 """
--> 376 return next(iter(seq))
377
StopIteration:
The above exception was the direct cause of the following exception:
RuntimeError Traceback (most recent call last)
<ipython-input-1-7c367e094750> in <module>
30 # Train model
31 cuml_model = cumlDaskRF(max_depth=max_depth, n_estimators=n_trees,n_streams=n_streams,workers = workers)
---> 32 cuml_model.fit(X_train_dask, y_train_dask)
33 wait(cuml_model.rfs)
34
/opt/conda/lib/python3.7/site-packages/cuml/dask/ensemble/randomforestregressor.py in fit(self, X, y, convert_dtype)
375 will increase memory used for the method.
376 """
--> 377 data = DistributedDataHandler.create((X, y), client=self.client)
378 self.datatype = data.datatype
379 futures = list()
/opt/conda/lib/python3.7/site-packages/cuml/dask/common/input_utils.py in create(cls, data, client)
109 validate_dask_array(data)
110
--> 111 gpu_futures = client.sync(_extract_partitions, data, client)
112
113 workers = tuple(set(map(lambda x: x[0], gpu_futures)))
/opt/conda/lib/python3.7/site-packages/distributed/client.py in sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
778 else:
779 return sync(
--> 780 self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
781 )
782
/opt/conda/lib/python3.7/site-packages/distributed/utils.py in sync(loop, func, callback_timeout, *args, **kwargs)
345 if error[0]:
346 typ, exc, tb = error[0]
--> 347 raise exc.with_traceback(tb)
348 else:
349 return result[0]
/opt/conda/lib/python3.7/site-packages/distributed/utils.py in f()
329 if callback_timeout is not None:
330 future = asyncio.wait_for(future, callback_timeout)
--> 331 result[0] = yield future
332 except Exception as exc:
333 error[0] = sys.exc_info()
/opt/conda/lib/python3.7/site-packages/tornado/gen.py in run(self)
733
734 try:
--> 735 value = future.result()
736 except Exception:
737 exc_info = sys.exc_info()
/opt/conda/lib/python3.7/site-packages/tornado/gen.py in run(self)
746 exc_info = None
747 else:
--> 748 yielded = self.gen.send(value)
749
750 except (StopIteration, Return) as e:
RuntimeError: generator raised StopIteration
But it works with shuffle=False.
Version = 0.14.0a+2432.g612e973
nikiforov-sm
on 13 May 2020
The same error if we use dask_dataframe.sample to get 50% of dataset to the CUDA:
from cuml.dask.common import utils as dask_utils
from dask.distributed import Client, wait
from dask_cuda import LocalCUDACluster
import dask_cudf
import dask.dataframe as dd
import dask.array as da
from cuml.dask.ensemble import RandomForestRegressor as cumlDaskRF
from dask_ml.model_selection import train_test_split
cluster = LocalCUDACluster()
c = Client(cluster)
workers = c.has_what().keys()
n_workers = len(workers)
n_streams = 1
# Desired parameters
max_depth = 3
n_trees = 16
rows, cols = 100_000, 20
# Prepare datasets
#random array with 20 chunks per worker and cast to float32 instead of float64
x = da.random.random((rows, cols), chunks=(rows//(n_workers*20),cols)).astype('float32')
df_dd = dd.from_array(x)
# Get random 50% rows
df_dd = df_dd.sample(frac=0.5)
df_dask = dask_cudf.from_dask_dataframe(df_dd)
X_train, X_test, y_train, y_test = train_test_split(df_dask.drop(2,axis=1), df_dask[2], shuffle=True, train_size = 0.8, random_state=31)
X_train_dask, y_train_dask = dask_utils.persist_across_workers(c, [X_train, y_train], workers=workers)
# Train model
cuml_model = cumlDaskRF(max_depth=max_depth, n_estimators=n_trees,n_streams=n_streams,workers = workers)
cuml_model.fit(X_train_dask, y_train_dask)
wait(cuml_model.rfs)
---------------------------------------------------------------------------
StopIteration Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/cuml/dask/common/part_utils.py in _extract_partitions(dask_obj, client)
156 raise gen.Return([(first(who_has[key]), part)
--> 157 for key, part in key_to_part])
/opt/conda/lib/python3.7/site-packages/cuml/dask/common/part_utils.py in <listcomp>(.0)
156 raise gen.Return([(first(who_has[key]), part)
--> 157 for key, part in key_to_part])
/opt/conda/lib/python3.7/site-packages/toolz/itertoolz.py in first(seq)
375 """
--> 376 return next(iter(seq))
377
StopIteration:
The above exception was the direct cause of the following exception:
RuntimeError Traceback (most recent call last)
<ipython-input-1-0580ec3663aa> in <module>
32 # Train model
33 cuml_model = cumlDaskRF(max_depth=max_depth, n_estimators=n_trees,n_streams=n_streams,workers = workers)
---> 34 cuml_model.fit(X_train_dask, y_train_dask)
35 wait(cuml_model.rfs)
36
/opt/conda/lib/python3.7/site-packages/cuml/dask/ensemble/randomforestregressor.py in fit(self, X, y, convert_dtype)
375 will increase memory used for the method.
376 """
--> 377 data = DistributedDataHandler.create((X, y), client=self.client)
378 self.datatype = data.datatype
379 futures = list()
/opt/conda/lib/python3.7/site-packages/cuml/dask/common/input_utils.py in create(cls, data, client)
109 validate_dask_array(data)
110
--> 111 gpu_futures = client.sync(_extract_partitions, data, client)
112
113 workers = tuple(set(map(lambda x: x[0], gpu_futures)))
/opt/conda/lib/python3.7/site-packages/distributed/client.py in sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
778 else:
779 return sync(
--> 780 self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
781 )
782
/opt/conda/lib/python3.7/site-packages/distributed/utils.py in sync(loop, func, callback_timeout, *args, **kwargs)
345 if error[0]:
346 typ, exc, tb = error[0]
--> 347 raise exc.with_traceback(tb)
348 else:
349 return result[0]
/opt/conda/lib/python3.7/site-packages/distributed/utils.py in f()
329 if callback_timeout is not None:
330 future = asyncio.wait_for(future, callback_timeout)
--> 331 result[0] = yield future
332 except Exception as exc:
333 error[0] = sys.exc_info()
/opt/conda/lib/python3.7/site-packages/tornado/gen.py in run(self)
733
734 try:
--> 735 value = future.result()
736 except Exception:
737 exc_info = sys.exc_info()
/opt/conda/lib/python3.7/site-packages/tornado/gen.py in run(self)
746 exc_info = None
747 else:
--> 748 yielded = self.gen.send(value)
749
750 except (StopIteration, Return) as e:
RuntimeError: generator raised StopIteration
@JamesMaki @Salonijain27
Given all the methods currently used to reduce the size of the original dataset (500_000_000 rows * 100 columns) due to an OOM error, I have two questions:
- Will the OOM error be fixed when using a cluster of two DGX-2 with full dataset? Can the cluster use shared memory?
- Are you considering a revision to reduce the size of artifacts (metadata) for cuml_dask.RandomForest that are stored in GPU memory or store these artifacts outside of the GPU?
nikiforov-sm
on 13 May 2020
@JamesMaki @Salonijain27
The errors mentioned here and here are also present in version 0.14.0a+2783.g1e5a5dd
nikiforov-sm
on 18 May 2020
Here: https://github.com/rapidsai/cuml/issues/1998#issuecomment-627767083 and https://github.com/rapidsai/cuml/issues/1998#issuecomment-627900649 , you are passing dask_cudf dataframe and dask_cudf series into the dask-ml function train_test_split. The train_test_split function uses a series function .sample (https://docs.dask.org/en/latest/dataframe-api.html?highlight=series#dask.dataframe.Series.sample) to obtain random samples while shuffling. Dask_cudf Series does not have this function. Therefore, when shuffle=True . you get an error.
Salonijain27
on 19 May 2020
@nikiforov-sm to follow up on your questions:
(1) Yes, a cluster of two DGX-2s should roughly double the amount of data you can use for training. (Note that for inference the full tree still needs to be able to fit in each worker though, so there will always be limits for extreme large tree size.)
(2) We're always looking for ways to improve memory consumption for RF and related algorithms. We have upcoming changes planned (early 0.15 cycle) that will improve _inference_ memory consumption, but we don't yet have an 0.14/early-0.15 memory reduction planned for training. This is an area we're interested in pursuing further especially based on these examples, but unfortunately there is not a firm timeline set yet for possible memory usage reductions. We are planning support for cupy-backed dask array inputs in an upcoming release, and I believe this will be a significant step to reduce memory overhead here. I will reply on this bug when that roadmap is set to share the details.
JohnZed
on 20 May 2020
@Salonijain27 @JohnZed
Thank you for the updates.
nikiforov-sm
on 20 May 2020
Due to the lack of the ability to use shuffle in train_test_split, we switched to shuffling dask_dataframe.
But now we have an error in predicting if we train the model for the collection of data from a file:
from cuml.dask.common import utils as dask_utils
from dask.distributed import Client, wait
from dask_cuda import LocalCUDACluster
import dask_cudf
import dask.dataframe as dd
import dask.array as da
from cuml.dask.ensemble import RandomForestRegressor as cumlDaskRF
from dask_ml.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
cluster = LocalCUDACluster(threads_per_worker=1)
c = Client(cluster)
workers = c.has_what().keys()
n_workers = len(workers)
n_streams = 1
# Desired parameters
max_depth = 3
n_trees = 16
rows, cols = 1_000, 20
# Prepare datasets
#random array with 20 chunks per worker and cast to float32 instead of float64
x = da.random.random((rows, cols), chunks=(rows//(n_workers*20),cols)).astype('float32')
dfd = dd.from_array(x)
# Save and load using csv-file
dfd.compute().to_csv('/mnt/ml/data/test0.csv',index=False)
dfd = dd.read_csv('/mnt/ml/data/test0.csv')
# Split dataset to train and test
dfd_train, _, dfd_test = dfd.random_split([0.3, 0.6, 0.1])
dfcd_train = dask_cudf.from_dask_dataframe(dfd_train)
dfcd_test = dask_cudf.from_dask_dataframe(dfd_test)
# Persist
X_train, y_train = dask_utils.persist_across_workers(c, [dfcd_train.drop('2',axis=1), dfcd_train['2']], workers=workers)
X_test, y_test = dask_utils.persist_across_workers(c, [dfcd_test.drop('2',axis=1), dfcd_test['2']], workers=workers)
y_test = y_test.to_dask_array().compute().get()
# Train model
cuml_model = cumlDaskRF(max_depth=max_depth, n_estimators=n_trees,n_streams=n_streams,workers = workers)
cuml_model.fit(X_train, y_train)
wait(cuml_model.rfs)
# Predict
cuml_y_pred = cuml_model.predict(X_test)
# Evaluate
y_pred = cuml_y_pred.to_dask_array().compute().get()
mse = mean_squared_error(y_test, y_pred)
print(f'mse={mse}')
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-1-ee75b10b30f6> in <module>
40 wait(cuml_model.rfs)
41 # Predict
---> 42 cuml_y_pred = cuml_model.predict(X_test)
43 # Evaluate
44 y_pred = cuml_y_pred.to_dask_array().compute().get()
/opt/conda/lib/python3.7/site-packages/cuml/dask/ensemble/randomforestregressor.py in predict(self, X, predict_model, algo, convert_dtype, fil_sparse_format, delayed)
256 convert_dtype=convert_dtype,
257 fil_sparse_format=fil_sparse_format,
--> 258 delayed=delayed)
259 return preds
260
/opt/conda/lib/python3.7/site-packages/cuml/dask/ensemble/randomforestregressor.py in predict_using_fil(self, X, delayed, **kwargs)
261 def predict_using_fil(self, X, delayed, **kwargs):
262 if self.local_model is None:
--> 263 self.local_model = self._concat_treelite_models()
264 return self._predict_using_fil(X=X,
265 delayed=delayed,
/opt/conda/lib/python3.7/site-packages/cuml/dask/ensemble/base.py in _concat_treelite_models(self)
97 dask.delayed(_get_protobuf_bytes)
98 (self.rfs[w]))
---> 99 mod_bytes = self.client.compute(model_protobuf_futures, sync=True)
100 last_worker = w
101 all_tl_mod_handles = []
/opt/conda/lib/python3.7/site-packages/distributed/client.py in compute(self, collections, sync, optimize_graph, workers, allow_other_workers, resources, retries, priority, fifo_timeout, actors, traverse, **kwargs)
2869
2870 if sync:
-> 2871 result = self.gather(futures)
2872 else:
2873 result = futures
/opt/conda/lib/python3.7/site-packages/distributed/client.py in gather(self, futures, errors, direct, asynchronous)
1965 direct=direct,
1966 local_worker=local_worker,
-> 1967 asynchronous=asynchronous,
1968 )
1969
/opt/conda/lib/python3.7/site-packages/distributed/client.py in sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
814 else:
815 return sync(
--> 816 self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
817 )
818
/opt/conda/lib/python3.7/site-packages/distributed/utils.py in sync(loop, func, callback_timeout, *args, **kwargs)
345 if error[0]:
346 typ, exc, tb = error[0]
--> 347 raise exc.with_traceback(tb)
348 else:
349 return result[0]
/opt/conda/lib/python3.7/site-packages/distributed/utils.py in f()
329 if callback_timeout is not None:
330 future = asyncio.wait_for(future, callback_timeout)
--> 331 result[0] = yield future
332 except Exception as exc:
333 error[0] = sys.exc_info()
/opt/conda/lib/python3.7/site-packages/tornado/gen.py in run(self)
733
734 try:
--> 735 value = future.result()
736 except Exception:
737 exc_info = sys.exc_info()
/opt/conda/lib/python3.7/site-packages/distributed/client.py in _gather(self, futures, errors, direct, local_worker)
1824 exc = CancelledError(key)
1825 else:
-> 1826 raise exception.with_traceback(traceback)
1827 raise exc
1828 if errors == "skip":
/opt/conda/lib/python3.7/site-packages/cuml/dask/ensemble/base.py in _get_protobuf_bytes()
178
179 def _get_protobuf_bytes(model):
--> 180 return model._get_protobuf_bytes()
cuml/ensemble/randomforestregressor.pyx in cuml.ensemble.randomforestregressor.RandomForestRegressor._get_protobuf_bytes()
cuml/ensemble/randomforestregressor.pyx in cuml.ensemble.randomforestregressor.RandomForestRegressor._obtain_treelite_handle()
cuml/common/base.pyx in cuml.common.base.Base.__getattr__()
AttributeError:
We have error with unpickling pickled cuml.dask.RFR.
But I found [FEA] Saving and loading of distributed models #1841 and it's open.
Are there any alternative solutions for saving and loading pre-trained cuml.dask.RFR?
nikiforov-sm
on 24 May 2020
Sorry, but at the moment the direct pickling and unpickling of dask RF models is not supported.
Also, I am looking into the error posted above :
https://github.com/rapidsai/cuml/issues/1998#issuecomment-633206582
Salonijain27
on 27 May 2020
One option is to call the internal dask rf python function _concat_treelite_models and then pickle the model returned. ex:
model_to_be_pickled = dask_cuml_mod._concat_treelite_models()
filename = 'finalized_model.sav'
pickle.dump(model_to_be_pickled , open(filename, 'wb'))
The output of _concat_treelite_models is the model used to predict the labels on different workers in dask RF.
Once this model is pickled you can unpickle the model and then treat it as a non dask RF model. You can then pass chunks of/ entire (depending on the size of the test dataset) test dataset through the unpickled model for prediction.
loaded_model = pickle.load(open(filename, 'rb'))
preds = loaded_model.predict(X_test)
@Salonijain27
Thank you for this update.
Option to call _concat_treelite_models works.
nikiforov-sm
on 29 May 2020
@Salonijain27
Did you manage to reproduce the error with predicting on shuffled dask_dataframe?
nikiforov-sm
on 1 Jun 2020
Hi, @nikiforov-sm,
I was able to reproduce the error. The problem was that when you load the dataset from the csv file, the data returned is of dtype=np.float64. I created a PR #2099 to throw an assertion error when float64 data is used for Dask RF
Salonijain27
on 1 Jun 2020
Hi, @Salonijain27
Thank you for the update.
The same error produced if I have dtype=float32 by reading csv with .astype('float32'):
dfd = dd.read_csv('/mnt/ml/data/test0.csv').astype('float32')
instead
dfd = dd.read_csv('/mnt/ml/data/test0.csv')
Hi, @Salonijain27 , @JamesMaki
I have the same error with dataframe with dtypes=float32 only.
Here is the test script.
from cuml.dask.common import utils as dask_utils
from dask.distributed import Client, wait
from dask_cuda import LocalCUDACluster
import dask_cudf
import dask.dataframe as dd
import dask.array as da
from cuml.dask.ensemble import RandomForestRegressor as cumlDaskRF
from dask_ml.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
cluster = LocalCUDACluster(threads_per_worker=1, local_directory='/mnt/ml/data/dask')
c = Client(cluster)
workers = c.has_what().keys()
n_workers = len(workers)
n_streams = 1
# Desired parameters
max_depth = 3
n_trees = 16
rows, cols = 1_000, 20
# Prepare datasets
#random array with 20 chunks per worker and cast to float32 instead of float64
x = da.random.random((rows, cols), chunks=(rows//(n_workers*20),cols)).astype('float32')
dfd = dd.from_array(x)
# Save and load using csv-file
dfd.compute().to_csv('/mnt/ml/data/test0.csv',index=False)
dfd = dd.read_csv('/mnt/ml/data/test0.csv', dtype={str(c):'float32' for c in range(cols)})
# Split dataset to train and test
dfd_train, _, dfd_test = dfd.random_split([0.3, 0.6, 0.1])
dfcd_train = dask_cudf.from_dask_dataframe(dfd_train)
dfcd_test = dask_cudf.from_dask_dataframe(dfd_test)
# Persist
X_train, y_train = dask_utils.persist_across_workers(c, [dfcd_train.drop('2',axis=1), dfcd_train['2']], workers=workers)
X_test, y_test = dask_utils.persist_across_workers(c, [dfcd_test.drop('2',axis=1), dfcd_test['2']], workers=workers)
y_test = y_test.to_dask_array().compute().get()
# Train model
cuml_model = cumlDaskRF(max_depth=max_depth, n_estimators=n_trees,n_streams=n_streams,workers = workers)
cuml_model.fit(X_train, y_train)
wait(cuml_model.rfs)
# Predict
cuml_y_pred = cuml_model.predict(X_test)
# Evaluate
y_pred = cuml_y_pred.to_dask_array().compute().get()
mse = mean_squared_error(y_test, y_pred)
print(f'mse={mse}')
Is there workaround to get shuffled part of dataset into training process?
nikiforov-sm
on 8 Jun 2020
👀1
Hi @nikiforov-sm ,
The issue in the above example is that on saving your data to the csv file you are saving all the partitions into a single file. Therefore, when the data is read from the file it contains only a single partition. The data should have atleast the same number of partitions as the workers you are using to fit and predict the Dask Random Forest model.
The data should be saved and loaded using:
x = da.random.random((rows, cols), chunks=(rows//(n_workers*10),cols)).astype('float32')
dfd = dd.from_array(x)
# Save and load using csv-file
dfd.to_csv('data/test-*.csv',index=False)
dfd = dd.read_csv('data/test-*.csv', dtype={str(c):'float32' for c in range(cols)})
On increasing the number of chunks to (rows//(n_workers*20),cols) the code seems to have
RuntimeError: UnownedMemory requires explicit device ID for a null pointer.
issue. I am exploring whats causing the issue.
Salonijain27
on 10 Jun 2020
When we use chunks=(rows//(n_workers*20),cols) we see the error :
RuntimeError: UnownedMemory requires explicit device ID for a null pointer.
because the X_test is too small to be divided into 40 partitions. This causes some partition to have X_test of shape (0, 19). While trying to access partition of shape (0, 19) we get the above mentioned error. Increasing the size of the dataset or decreasing the number of chunks/partitions created will solve this issue.
Salonijain27
on 10 Jun 2020
Hi @Salonijain27 ,
Thank you for this update, it works!
Is there workaround to get feature importance from fitted model of dask_cuml.RandomForestRegressor?
nikiforov-sm
on 12 Jun 2020
👀1
Hi @nikiforov-sm ,
I have created a PR which will let you print the detailed dask RF model #2439. This would print the features used to split each node, the split value and the index value calculated.
I am also working on a PR to create a function which would return a list containing features used to split each node.
Salonijain27
on 18 Jun 2020
Hi @Salonijain27 ,
Thank you for this update!
nikiforov-sm
on 18 Jun 2020
@nikiforov-sm to follow up on your questions:
(1) Yes, a cluster of two DGX-2s should roughly double the amount of data you can use for training. (Note that for inference the full tree still needs to be able to fit in each worker though, so there will always be limits for extreme large tree size.)
(2) We're always looking for ways to improve memory consumption for RF and related algorithms. We have upcoming changes planned (early 0.15 cycle) that will improve _inference_ memory consumption, but we don't yet have an 0.14/early-0.15 memory reduction planned for training. This is an area we're interested in pursuing further especially based on these examples, but unfortunately there is not a firm timeline set yet for possible memory usage reductions. We are planning support for cupy-backed dask array inputs in an upcoming release, and I believe this will be a significant step to reduce memory overhead here. I will reply on this bug when that roadmap is set to share the details.
Hi @JohnZed ,
Do you have news about these improvements to reduce the random forest metadata size in the gpu-memory?
nikiforov-sm
on 25 Jun 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
benjha
·
6Comments
beckernick
·
8Comments
vinayprabhu
·
9Comments
drscotthawley
·
9Comments
miguelusque
·
3Comments