Cuml: [BUG] Fallback from RF experimental backend causes hang
Describe the bug
If an RF classifier using the new experimental backend is instantiated with options that cause a fallback to the default backend, it produces a hang that will not respond to SIGINT.
Steps/Code to reproduce bug
import cupy as cp
import time
from cuml.ensemble import RandomForestClassifier
x_shape = (10, 10)
y_shape = (x_shape[0], 1)
x = cp.random.random(x_shape, dtype=cp.float32)
y = cp.random.randint(2, size=y_shape)
rf_old = RandomForestClassifier(
max_depth=10,
use_experimental_backend=False
)
rf_new = RandomForestClassifier(
max_depth=10,
use_experimental_backend=True
)
print("Starting old fit...")
start = time.time()
rf_old.fit(x, y)
print("Old fit time: {:.2f}".format(time.time() - start))
print("Starting new fit...")
start = time.time()
rf_new.fit(x, y)
print("New fit time: {:.2f}".format(time.time() - start))
This produces a hang, but adding max_features=1.0 eliminates it.
Expected behavior
The classifier should not hang, even if its parameters are not currently supported
Environment details (please complete the following information):
- Environment location: Bare metal
- Linux Distro/Architecture: Ubuntu 20.04/amd64
- GPU Model/Driver: Quadro RTX 8000/450.66
- CUDA: 10.1
- Method of cuDF & cuML install: conda nightly and reproduced from source on branch-0.17
conda environment:
# packages in environment at /home/whicks/anaconda3/envs/blog:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_gnu conda-forge
abseil-cpp 20200225.2 he1b5a44_2 conda-forge
aiohttp 3.6.3 py38h1e0a361_0 conda-forge
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
argon2-cffi 20.1.0 py38h1e0a361_2 conda-forge
arrow-cpp 1.0.1 py38h41c2bc5_14_cuda conda-forge
arrow-cpp-proc 1.0.1 cuda conda-forge
async-timeout 3.0.1 py_1000 conda-forge
async_generator 1.10 py_0 conda-forge
attrs 20.2.0 pyh9f0ad1d_0 conda-forge
aws-c-common 0.4.59 he1b5a44_0 conda-forge
aws-c-event-stream 0.1.6 h84e28f3_5 conda-forge
aws-checksums 0.1.9 he252421_2 conda-forge
aws-sdk-cpp 1.8.63 h9b98462_0 conda-forge
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backports 1.0 py_2 conda-forge
backports.functools_lru_cache 1.6.1 py_0 conda-forge
bleach 3.2.1 pyh9f0ad1d_0 conda-forge
bokeh 2.2.2 py38h32f6830_0 conda-forge
boost 1.72.0 py38h1e42940_1 conda-forge
boost-cpp 1.72.0 h8e57a91_0 conda-forge
brotli 1.0.9 he1b5a44_2 conda-forge
brotlipy 0.7.0 py38h8df0ef7_1001 conda-forge
bzip2 1.0.8 h516909a_3 conda-forge
c-ares 1.16.1 h516909a_3 conda-forge
ca-certificates 2020.6.20 hecda079_0 conda-forge
cairo 1.16.0 hcf35c78_1003 conda-forge
certifi 2020.6.20 py38h924ce5b_2 conda-forge
cffi 1.14.3 py38h1bdcb99_1 conda-forge
cfitsio 3.470 h0392d32_7 conda-forge
chardet 3.0.4 py38h924ce5b_1008 conda-forge
click 7.1.2 pyh9f0ad1d_0 conda-forge
click-plugins 1.1.1 py_0 conda-forge
cligj 0.5.0 py_0 conda-forge
cloudpickle 1.6.0 py_0 conda-forge
colorcet 2.0.1 py_0 conda-forge
cryptography 3.1.1 py38hb23e4d4_1 conda-forge
cudatoolkit 10.1.243 h6bb024c_0 nvidia
cudf 0.16.0a201019 cuda_10.1_py38_ga6296e34b8_1978 rapidsai-nightly
cudf_kafka 0.16.0a201019 py38_ga6296e34b8_1978 rapidsai-nightly
cudnn 7.6.0 cuda10.1_0 nvidia
cugraph 0.16.0a201019 py38_g14606fc8_485 rapidsai-nightly
cuml 0.16.0a201017 cuda10.1_py38_g48a5416c8_873 rapidsai-nightly
cupy 8.0.0 py38h68d73f9_2 conda-forge
curl 7.71.1 he644dc0_8 conda-forge
cusignal 0.16.0a201019 py37_gc3c9d71_190 rapidsai-nightly
cuspatial 0.16.0a201019 py38_g487b88f_109 rapidsai-nightly
custreamz 0.16.0a201019 py38_ga6296e34b8_1978 rapidsai-nightly
cuxfilter 0.16.0a201019 py38_g468968d_124 rapidsai-nightly
cycler 0.10.0 py_2 conda-forge
cyrus-sasl 2.1.27 h063b49f_1 conda-forge
cytoolz 0.11.0 py38h1e0a361_1 conda-forge
dask 2.30.0 py_0 conda-forge
dask-core 2.30.0 py_0 conda-forge
dask-cuda 0.16.0a201019 py38_96 rapidsai-nightly
dask-cudf 0.16.0a201019 py38_ga6296e34b8_1978 rapidsai-nightly
datashader 0.11.1 pyh9f0ad1d_0 conda-forge
datashape 0.5.4 py_1 conda-forge
dbus 1.13.6 h7a60e0d_1 conda-forge
decorator 4.4.2 py_0 conda-forge
defusedxml 0.6.0 py_0 conda-forge
distributed 2.30.0 py38h32f6830_1 conda-forge
dlpack 0.3 he1b5a44_1 conda-forge
entrypoints 0.3 py38h32f6830_1002 conda-forge
expat 2.2.9 he1b5a44_2 conda-forge
faiss-proc 1.0.0 cuda conda-forge
fastavro 1.0.0.post1 py38h1e0a361_1 conda-forge
fastrlock 0.5 py38h950e882_1 conda-forge
fiona 1.8.13 py38h033e0f6_1 conda-forge
fontconfig 2.13.1 h86ecdb6_1001 conda-forge
freetype 2.10.3 he06d7ca_0 conda-forge
freexl 1.0.5 h516909a_1002 conda-forge
fsspec 0.8.4 py_0 conda-forge
gdal 3.0.4 py38h172510d_10 conda-forge
geopandas 0.8.1 py_0 conda-forge
geos 3.8.1 he1b5a44_0 conda-forge
geotiff 1.6.0 h05acad5_0 conda-forge
gettext 0.19.8.1 hf34092f_1003 conda-forge
gflags 2.2.2 he1b5a44_1004 conda-forge
giflib 5.2.1 h516909a_2 conda-forge
glib 2.66.1 he1b5a44_1 conda-forge
glog 0.4.0 h49b9bf7_3 conda-forge
grpc-cpp 1.32.0 h7997a97_1 conda-forge
gst-plugins-base 1.14.5 h0935bb2_2 conda-forge
gstreamer 1.14.5 h36ae1b5_2 conda-forge
hdf4 4.2.13 hf30be14_1003 conda-forge
hdf5 1.10.6 nompi_h1022a3e_1110 conda-forge
heapdict 1.0.1 py_0 conda-forge
icu 64.2 he1b5a44_1 conda-forge
idna 2.10 pyh9f0ad1d_0 conda-forge
importlib-metadata 2.0.0 py_1 conda-forge
importlib_metadata 2.0.0 1 conda-forge
ipykernel 5.3.4 py38h1cdfbd6_1 conda-forge
ipython 7.18.1 py38h1cdfbd6_1 conda-forge
ipython_genutils 0.2.0 py_1 conda-forge
ipywidgets 7.5.1 pyh9f0ad1d_1 conda-forge
jedi 0.17.2 py38h32f6830_1 conda-forge
jinja2 2.11.2 pyh9f0ad1d_0 conda-forge
joblib 0.17.0 py_0 conda-forge
jpeg 9d h516909a_0 conda-forge
json-c 0.13.1 hbfbb72e_1002 conda-forge
jsonschema 3.2.0 py_2 conda-forge
jupyter-server-proxy 1.5.0 py_0 conda-forge
jupyter_client 6.1.7 py_0 conda-forge
jupyter_core 4.6.3 py38h32f6830_2 conda-forge
jupyterlab_pygments 0.1.2 pyh9f0ad1d_0 conda-forge
kaggle 1.5.8 pypi_0 pypi
kealib 1.4.13 h33137a7_1 conda-forge
kiwisolver 1.2.0 py38hbf85e49_1 conda-forge
krb5 1.17.1 hfafb76e_3 conda-forge
lcms2 2.11 hbd6801e_0 conda-forge
ld_impl_linux-64 2.35 h769bd43_9 conda-forge
libblas 3.8.0 17_openblas conda-forge
libcblas 3.8.0 17_openblas conda-forge
libclang 9.0.1 default_hde54327_0 conda-forge
libcudf 0.16.0a201019 cuda10.1_ga6296e34b8_1978 rapidsai-nightly
libcudf_kafka 0.16.0a201019 ga6296e34b8_1978 rapidsai-nightly
libcugraph 0.16.0a201019 cuda10.1_g14606fc8_485 rapidsai-nightly
libcuml 0.16.0a201017 cuda10.1_g48a5416c8_873 rapidsai-nightly
libcumlprims 0.16.0a200930 cuda10.1_g1c28023_35 rapidsai-nightly
libcurl 7.71.1 hcdd3856_8 conda-forge
libcuspatial 0.16.0a201019 cuda10.1_g487b88f_109 rapidsai-nightly
libdap4 3.20.6 h1d1bd15_1 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libevent 2.1.10 hcdb4288_3 conda-forge
libfaiss 1.6.3 he68dc02_3_cuda conda-forge
libffi 3.2.1 he1b5a44_1007 conda-forge
libgcc-ng 9.3.0 h5dbcf3e_17 conda-forge
libgcrypt 1.8.4 hf484d3e_1000 conda-forge
libgdal 3.0.4 he6a97d6_10 conda-forge
libgfortran-ng 9.3.0 he4bcb1c_17 conda-forge
libgfortran5 9.3.0 he4bcb1c_17 conda-forge
libglib 2.66.1 h0dae87d_1 conda-forge
libgomp 9.3.0 h5dbcf3e_17 conda-forge
libgpg-error 1.39 he1b5a44_0 conda-forge
libgsasl 1.8.0 2 conda-forge
libhwloc 2.3.0 h3c4fd83_0 conda-forge
libiconv 1.16 h516909a_0 conda-forge
libkml 1.3.0 hd79254b_1012 conda-forge
liblapack 3.8.0 17_openblas conda-forge
libllvm10 10.0.1 he513fc3_3 conda-forge
libllvm9 9.0.1 he513fc3_1 conda-forge
libnetcdf 4.7.4 nompi_h84807e1_105 conda-forge
libnghttp2 1.41.0 h8cfc5f6_2 conda-forge
libntlm 1.4 h516909a_1002 conda-forge
libopenblas 0.3.10 pthreads_h4812303_5 conda-forge
libpng 1.6.37 hed695b0_2 conda-forge
libpq 12.3 h5513abc_2 conda-forge
libprotobuf 3.13.0.1 h8b12597_0 conda-forge
librdkafka 1.5.0 h40bdf00_0 conda-forge
librmm 0.16.0a201019 cuda10.1_gb7cdbc4_399 rapidsai-nightly
libsodium 1.0.18 h516909a_1 conda-forge
libspatialindex 1.9.3 he1b5a44_3 conda-forge
libspatialite 4.3.0a h2482549_1038 conda-forge
libssh2 1.9.0 hab1572f_5 conda-forge
libstdcxx-ng 9.3.0 h2ae2ef3_17 conda-forge
libthrift 0.13.0 h5aa387f_6 conda-forge
libtiff 4.1.0 hc7e4089_6 conda-forge
libutf8proc 2.5.0 h516909a_2 conda-forge
libuuid 2.32.1 h14c3975_1000 conda-forge
libuv 1.34.0 h516909a_0 conda-forge
libwebp 1.1.0 h56121f0_4 conda-forge
libwebp-base 1.1.0 h516909a_3 conda-forge
libxcb 1.13 h14c3975_1002 conda-forge
libxgboost 1.2.0dev.rapidsai0.16 cuda10.1_2 rapidsai-nightly
libxkbcommon 0.10.0 he1b5a44_0 conda-forge
libxml2 2.9.10 hee79883_0 conda-forge
llvmlite 0.34.0 py38h4f45e52_2 conda-forge
locket 0.2.0 py_2 conda-forge
lz4-c 1.9.2 he1b5a44_3 conda-forge
markdown 3.3.1 pyh9f0ad1d_0 conda-forge
markupsafe 1.1.1 py38h8df0ef7_2 conda-forge
matplotlib 3.3.2 py38h32f6830_1 conda-forge
matplotlib-base 3.3.2 py38h4d1ce4f_1 conda-forge
mistune 0.8.4 py38h1e0a361_1002 conda-forge
msgpack-python 1.0.0 py38hbf85e49_2 conda-forge
multidict 4.7.5 py38h1e0a361_2 conda-forge
multipledispatch 0.6.0 py_0 conda-forge
munch 2.5.0 py_0 conda-forge
nbclient 0.5.1 py_0 conda-forge
nbconvert 6.0.7 py38h32f6830_1 conda-forge
nbformat 5.0.8 py_0 conda-forge
nccl 2.7.8.1 h51cf6c1_1 conda-forge
ncurses 6.2 he1b5a44_2 conda-forge
nest-asyncio 1.4.1 py_0 conda-forge
networkx 2.5 py_0 conda-forge
nodejs 13.13.0 hf5d1a2b_0 conda-forge
notebook 6.1.4 py38h32f6830_1 conda-forge
nspr 4.29 he1b5a44_1 conda-forge
nss 3.58 h27285de_1 conda-forge
numba 0.51.2 py38hc5bc63f_0 conda-forge
numpy 1.19.2 py38hf89b668_1 conda-forge
nvtx 0.2.1 py38h1e0a361_2 conda-forge
olefile 0.46 pyh9f0ad1d_1 conda-forge
openjpeg 2.3.1 h981e76c_3 conda-forge
openssl 1.1.1h h516909a_0 conda-forge
orc 1.6.5 hd3605a7_0 conda-forge
packaging 20.4 pyh9f0ad1d_0 conda-forge
pandas 1.1.3 py38hc5bc63f_2 conda-forge
pandoc 2.11.0.2 hd18ef5c_0 conda-forge
pandocfilters 1.4.2 py_1 conda-forge
panel 0.9.7 py_0 conda-forge
param 1.9.3 py_0 conda-forge
parquet-cpp 1.5.1 2 conda-forge
parso 0.7.1 pyh9f0ad1d_0 conda-forge
partd 1.1.0 py_0 conda-forge
pcre 8.44 he1b5a44_0 conda-forge
pexpect 4.8.0 pyh9f0ad1d_2 conda-forge
pickleshare 0.7.5 py_1003 conda-forge
pillow 8.0.0 py38h9776b28_0 conda-forge
pip 20.2.4 py_0 conda-forge
pixman 0.38.0 h516909a_1003 conda-forge
poppler 0.87.0 h4190859_1 conda-forge
poppler-data 0.4.9 1 conda-forge
postgresql 12.3 h8573dbc_2 conda-forge
proj 7.0.0 h966b41f_5 conda-forge
prometheus_client 0.8.0 pyh9f0ad1d_0 conda-forge
prompt-toolkit 3.0.8 py_0 conda-forge
psutil 5.7.2 py38h8df0ef7_1 conda-forge
pthread-stubs 0.4 h14c3975_1001 conda-forge
ptyprocess 0.6.0 py_1001 conda-forge
py-xgboost 1.2.0dev.rapidsai0.16 cuda10.1py38_2 rapidsai-nightly
pyarrow 1.0.1 py38h31aad1c_14_cuda conda-forge
pycparser 2.20 pyh9f0ad1d_2 conda-forge
pyct 0.4.6 py_0 conda-forge
pyct-core 0.4.6 py_0 conda-forge
pydeck 0.4.1 pyh9f0ad1d_0 conda-forge
pyee 7.0.4 pyh9f0ad1d_0 conda-forge
pygments 2.7.1 py_0 conda-forge
pynvml 8.0.4 py_1 conda-forge
pyopenssl 19.1.0 py_1 conda-forge
pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge
pyppeteer 0.2.2 py_1 conda-forge
pyproj 2.6.1.post1 py38h7521cb9_0 conda-forge
pyqt 5.12.3 py38ha8c2ead_3 conda-forge
pyqt5-sip 4.19.18 pypi_0 pypi
pyqtchart 5.12 pypi_0 pypi
pyqtwebengine 5.12.1 pypi_0 pypi
pyrsistent 0.17.3 py38h1e0a361_1 conda-forge
pysocks 1.7.1 py38h924ce5b_2 conda-forge
python 3.8.6 h852b56e_0_cpython conda-forge
python-confluent-kafka 1.5.0 py38h1e0a361_0 conda-forge
python-dateutil 2.8.1 py_0 conda-forge
python-slugify 4.0.1 pypi_0 pypi
python_abi 3.8 1_cp38 conda-forge
pytz 2020.1 pyh9f0ad1d_0 conda-forge
pyviz_comms 0.7.6 pyh9f0ad1d_0 conda-forge
pyyaml 5.3.1 py38h8df0ef7_1 conda-forge
pyzmq 19.0.2 py38ha71036d_2 conda-forge
qt 5.12.5 hd8c4c69_1 conda-forge
rapids 0.16.0a201019 cuda10.1_py38_g44e3a1e_80 rapidsai-nightly
rapids-xgboost 0.16.0a201019 cuda10.1_py38_g44e3a1e_80 rapidsai-nightly
re2 2020.10.01 he1b5a44_0 conda-forge
readline 8.0 he28a2e2_2 conda-forge
requests 2.24.0 pyh9f0ad1d_0 conda-forge
rmm 0.16.0a201019 cuda_10.1_py38_gb7cdbc4_399 rapidsai-nightly
rtree 0.9.4 py38h08f867b_1 conda-forge
scikit-learn 0.23.2 py38h519568a_1 conda-forge
scipy 1.5.2 py38hd648e70_2 conda-forge
send2trash 1.5.0 py_0 conda-forge
setuptools 49.6.0 py38h924ce5b_2 conda-forge
shapely 1.7.1 py38hc7361b7_1 conda-forge
simpervisor 0.3 py_1 conda-forge
six 1.15.0 pyh9f0ad1d_0 conda-forge
slugify 0.0.1 pypi_0 pypi
snappy 1.1.8 he1b5a44_3 conda-forge
sortedcontainers 2.2.2 pyh9f0ad1d_0 conda-forge
spdlog 1.7.0 hc9558a2_2 conda-forge
sqlite 3.33.0 h4cf870e_1 conda-forge
streamz 0.6.0 pyh9f0ad1d_0 conda-forge
tbb 2020.2 hc9558a2_0 conda-forge
tblib 1.6.0 py_0 conda-forge
terminado 0.9.1 py38h32f6830_1 conda-forge
testpath 0.4.4 py_0 conda-forge
text-unidecode 1.3 pypi_0 pypi
threadpoolctl 2.1.0 pyh5ca1d4c_0 conda-forge
tiledb 1.7.7 h8efa9f0_3 conda-forge
tk 8.6.10 hed695b0_1 conda-forge
toolz 0.11.1 py_0 conda-forge
tornado 6.0.4 py38h1e0a361_2 conda-forge
tqdm 4.50.2 pyh9f0ad1d_0 conda-forge
traitlets 5.0.5 py_0 conda-forge
treelite 0.93 py38h950e882_2 conda-forge
treelite-runtime 0.93 pypi_0 pypi
typing_extensions 3.7.4.3 py_0 conda-forge
tzcode 2020a h516909a_0 conda-forge
ucx 1.8.1+g6b29558 ha5db111_0 rapidsai-nightly
ucx-py 0.16.0a201019 py38_g6b29558_187 rapidsai-nightly
urllib3 1.24.3 pypi_0 pypi
wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge
webencodings 0.5.1 py_1 conda-forge
websockets 8.1 py38h1e0a361_2 conda-forge
wheel 0.35.1 pyh9f0ad1d_0 conda-forge
widgetsnbextension 3.5.1 pyh9f0ad1d_3 conda-forge
xarray 0.16.1 py_0 conda-forge
xerces-c 3.2.2 h8412b87_1004 conda-forge
xgboost 1.2.0dev.rapidsai0.16 cuda10.1py38_2 rapidsai-nightly
xorg-kbproto 1.0.7 h14c3975_1002 conda-forge
xorg-libice 1.0.10 h516909a_0 conda-forge
xorg-libsm 1.2.3 h84519dc_1000 conda-forge
xorg-libx11 1.6.12 h516909a_0 conda-forge
xorg-libxau 1.0.9 h14c3975_0 conda-forge
xorg-libxdmcp 1.1.3 h516909a_0 conda-forge
xorg-libxext 1.3.4 h516909a_0 conda-forge
xorg-libxrender 0.9.10 h516909a_1002 conda-forge
xorg-renderproto 0.11.1 h14c3975_1002 conda-forge
xorg-xextproto 7.3.0 h14c3975_1002 conda-forge
xorg-xproto 7.0.31 h14c3975_1007 conda-forge
xz 5.2.5 h516909a_1 conda-forge
yaml 0.2.5 h516909a_0 conda-forge
yarl 1.6.2 py38h1e0a361_0 conda-forge
zeromq 4.3.3 he1b5a44_2 conda-forge
zict 2.0.0 py_0 conda-forge
zipp 3.3.1 py_0 conda-forge
zlib 1.2.11 h516909a_1010 conda-forge
zstd 1.4.5 h6597ccf_2 conda-forge
Source build:
- cmake 3.17.0
- gcc 7.5.0
- Hash: 1295202a13875d05118775cd89a4199d038ce765
- If method of install is [from source], provide versions of
cmake&gcc/g++and commit hash of build
bug
wphicks
👍1
All 8 comments
@vinaydes I know you've been working on the new backend quite a bit. Any ideas on where this might be coming from?
wphicks
on 19 Oct 2020
Additional context: Setting max_features to 1 rather than 1.0 still produces the hang
wphicks
on 19 Oct 2020
@wphicks Indeed, I was able to reproduce the issue on my machine. Let me take a look.
hcho3
on 19 Oct 2020
👍1
Assigning myself to find a cause. I will also write a patch.
hcho3
on 20 Oct 2020
Thanks @wphicks for reporting the issue. @hcho3 has already found the cause and fix for which he will create a PR soon. Thanks @hcho3 馃憤
vinaydes
on 20 Oct 2020
👍1
Looks like the hang is in the logger module. When log APIs are called in a multi-threaded part of code, I see a hang. Disabling multi-threading in RF, by commenting out #pragma omp parallel ... lines in randomforest_impl.cuh solved the issue. Commenting out Logging statements also solves the issue. @hcho3's PR #3021 moved logging from multi-threaded to single threaded, thus the hang is avoided. While we should go ahead with the PR, we still need to debug logger + multi-threading hang as this might occur again.
vinaydes
on 20 Oct 2020
Relates to #3024
JohnZed
on 20 Oct 2020
PR #3021 works around the root cause of this issue. The root cause (bug in the logger in multi-threaded environment) is tracked separately with Issue #3027.
vinaydes
on 21 Oct 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
divyegala
路
7Comments
teju85
路
7Comments
vinayprabhu
路
9Comments
miguelusque
路
3Comments
gumdropsteve
路
3Comments