Cudf: [BUG] is_monotonic crashes with memory-pool enabled
Describe the bug
Using cudf.set_allocator(pool=True) is causing a segmentation fault in certain calls to is_monotonic (see simple reproducer below). The failure seems to occur within libcudf's is_sorted.
Steps/Code to reproduce bug
import cudf
import numpy as np
cudf.set_allocator(pool=True)
df = cudf.DataFrame({"a": np.arange(10)})
df.a.is_monotonic
Output: Segmentation fault
The script finishes as expected when using cudf.set_allocator(pool=False)
Note that I originally discovered this problem while modifying the dask_cudf join implementation to use that of main-line Dask. In that case, I sometimes get the same Segmentation fault as above, but sometimes get a bit more information:
...
File "/home/nfs/rzamora/workspace/cudf-dask-devel/dask-bugfixes/dask/dataframe/methods.py", line 89, in boundary_slice
if kind == "loc" and not df.index.is_monotonic:
File "/home/nfs/rzamora/workspace/cudf-dask-devel/cudf-bugfixes/python/cudf/cudf/core/index.py", line 797, in is_monotonic
return self._values.is_monotonic
File "/home/nfs/rzamora/workspace/cudf-dask-devel/cudf-bugfixes/python/cudf/cudf/core/column/column.py", line 778, in is_monotonic
return self.is_monotonic_increasing
File "/home/nfs/rzamora/workspace/cudf-dask-devel/cudf-bugfixes/python/cudf/cudf/core/column/numerical.py", line 404, in is_monotonic_increasing
[self]
File "cudf/_lib/issorted.pyx", line 17, in cudf._lib.issorted.issorted
File "cudf/_lib/issorted.pyx", line 41, in cudf._lib.issorted.issorted
RuntimeError: after reduction step 2: invalid resource handle
Expected behavior
I wouldn't expect a legal is_monotonic call (like in the reproducer above) to crash.
Environment overview (please complete the following information)
- Environment location: Bare-metal
- Method of cuDF install: from source
Environment details
Click here to see environment details
**git***
commit d8ba85a1ca0d114ffcffd39ef58739961ad598d5 (HEAD)
Merge: b40a6d3 c4a72f6
Author: Jake Hemstad <[email protected]>
Date: Thu Oct 24 23:27:40 2019 -0500
Merge pull request #3065 from kaatish/fea-select-concat-table
[REVIEW] Select and Concat for cudf exp table
**git submodules***
b165e1fb11eeea64ccf95053e40f2424312599cc thirdparty/cub (v1.7.1)
63f644be44201467e3938d59ed9d89cc8725c35d thirdparty/jitify (remotes/origin/feature/api_v2_v0.10)
5654f2bcc8972d513cf9cedf28510c1b6613aeeb thirdparty/libcudacxx (0.8.1-82-g5654f2b)
+a4ace1ce72e2e5376c4e3965e632357e7d55e0b6 thirdparty/libcudacxx/libcxx (heads/master)
***OS Information***
DGX_NAME="DGX Server"
DGX_PRETTY_NAME="NVIDIA DGX Server"
DGX_SWBUILD_DATE="2018-03-20"
DGX_SWBUILD_VERSION="3.1.6"
DGX_COMMIT_ID="1b0f58ecbf989820ce745a9e4836e1de5eea6cfd"
DGX_SERIAL_NUMBER=QTFCOU8310024
DGX_OTA_VERSION="3.1.7"
DGX_OTA_DATE="Thu Sep 27 20:07:53 PDT 2018"
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.5 LTS"
NAME="Ubuntu"
VERSION="16.04.5 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.5 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
Linux dgx16 4.4.0-135-generic #161-Ubuntu SMP Mon Aug 27 10:45:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
***GPU Information***
Fri Oct 25 12:02:51 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 396.44 Driver Version: 396.44 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-SXM2... On | 00000000:06:00.0 Off | 0 |
| N/A 41C P0 46W / 300W | 11MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla V100-SXM2... On | 00000000:07:00.0 Off | 0 |
| N/A 44C P0 46W / 300W | 11MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 Tesla V100-SXM2... On | 00000000:0A:00.0 Off | 0 |
| N/A 44C P0 46W / 300W | 11MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 Tesla V100-SXM2... On | 00000000:0B:00.0 Off | 0 |
| N/A 42C P0 46W / 300W | 11MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 4 Tesla V100-SXM2... On | 00000000:85:00.0 Off | 0 |
| N/A 44C P0 47W / 300W | 11MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 5 Tesla V100-SXM2... On | 00000000:86:00.0 Off | 0 |
| N/A 43C P0 45W / 300W | 11MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 6 Tesla V100-SXM2... On | 00000000:89:00.0 Off | 0 |
| N/A 45C P0 45W / 300W | 11MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 7 Tesla V100-SXM2... On | 00000000:8A:00.0 Off | 0 |
| N/A 41C P0 44W / 300W | 11MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
***CPU***
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 79
Model name: Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz
Stepping: 1
CPU MHz: 2211.000
CPU max MHz: 3600.0000
CPU min MHz: 1200.0000
BogoMIPS: 4391.73
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 51200K
NUMA node0 CPU(s): 0-19,40-59
NUMA node1 CPU(s): 20-39,60-79
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt ssbd ibrs ibpb stibp kaiser tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdseed adx smap xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts flush_l1d
***CMake***
/home/nfs/rzamora/miniconda3/envs/cudf_11_dev/bin/cmake
cmake version 3.15.4
CMake suite maintained and supported by Kitware (kitware.com/cmake).
***g++***
/usr/bin/g++
g++ (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
***nvcc***
***Python***
/home/nfs/rzamora/miniconda3/envs/cudf_11_dev/bin/python
Python 3.7.3
***Environment Variables***
PATH : /home/nfs/rzamora/.vscode-server/bin/6ab598523be7a800d7f3eb4d92d7ab9a66069390/bin:/home/nfs/rzamora/bin:/home/nfs/rzamora/.local/bin:/home/nfs/rzamora/miniconda3/envs/cudf_11_dev/bin:/home/nfs/rzamora/miniconda3/condabin:/home/nfs/rzamora/.vscode-server/bin/6ab598523be7a800d7f3eb4d92d7ab9a66069390/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
LD_LIBRARY_PATH :
NUMBAPRO_NVVM : /usr/local/cuda-9.2/nvvm/lib64/libnvvm.so
NUMBAPRO_LIBDEVICE : /usr/local/cuda-9.2/nvvm/libdevice
CONDA_PREFIX : /home/nfs/rzamora/miniconda3/envs/cudf_11_dev
PYTHON_PATH :
***conda packages***
/home/nfs/rzamora/miniconda3/condabin/conda
# packages in environment at /home/nfs/rzamora/miniconda3/envs/cudf_11_dev:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main
aiohttp 3.6.2 pypi_0 pypi
alabaster 0.7.12 py_0 conda-forge
appdirs 1.4.3 py_1 conda-forge
arrow-cpp 0.15.0 py37h090bef1_1 conda-forge
asn1crypto 1.2.0 py37_0 conda-forge
aspy.yaml 1.3.0 py_0 conda-forge
async-timeout 3.0.1 pypi_0 pypi
atomicwrites 1.3.0 py_0 conda-forge
attrs 19.3.0 py_0 conda-forge
babel 2.7.0 py_0 conda-forge
backcall 0.1.0 py_0 conda-forge
black 19.3b0 py_0 conda-forge
bleach 3.1.0 py_0 conda-forge
bokeh 1.3.4 py37_0 conda-forge
boost-cpp 1.70.0 h8e57a91_2 conda-forge
brotli 1.0.7 he1b5a44_1000 conda-forge
bzip2 1.0.8 h516909a_1 conda-forge
c-ares 1.15.0 h516909a_1001 conda-forge
ca-certificates 2019.10.16 0
cached-property 1.5.1 py_0 conda-forge
certifi 2019.9.11 py37_0
cffi 1.13.1 py37h8022711_0 conda-forge
cfgv 2.0.1 py_0 conda-forge
chardet 3.0.4 py37_1003 conda-forge
click 7.0 py_0 conda-forge
cloudpickle 1.2.2 py_0 conda-forge
cmake 3.15.4 hf94ab9c_0 conda-forge
cmake_setuptools 0.1.3 py_0 rapidsai-nightly
commonmark 0.9.1 py_0 conda-forge
cryptography 2.7 py37h72c5cf5_0 conda-forge
cudatoolkit 9.2 0 nvidia
cudf 0.11.0a0+1730.g539416f.dirty dev_0 <develop>
cudnn 7.6.0 cuda9.2_0 nvidia
cupy 6.3.0 py37h6adefed_0 rapidsai
curl 7.65.3 hf8cf82a_0 conda-forge
cython 0.29.13 py37he1b5a44_0 conda-forge
cytoolz 0.10.0 py37h516909a_0 conda-forge
dask 2.6.0+7.g94c6b16 dev_0 <develop>
dask-cuda 0.10.0a1+9.g3f1a4f2 dev_0 <develop>
dask-cudf 0.11.0a0+1730.g539416f.dirty dev_0 <develop>
decorator 4.4.0 py_0 conda-forge
defusedxml 0.6.0 py_0 conda-forge
distributed 2.6.0+7.g876bca0 pypi_0 pypi
dlpack 0.2 he1b5a44_1 conda-forge
docutils 0.15.2 py37_0 conda-forge
double-conversion 3.1.5 he1b5a44_1 conda-forge
editdistance 0.5.3 py37hf484d3e_0 conda-forge
entrypoints 0.3 py37_1000 conda-forge
expat 2.2.5 he1b5a44_1004 conda-forge
fastavro 0.22.5 py37h516909a_0 conda-forge
fastrlock 0.4 py37hf484d3e_1000 conda-forge
flake8 3.7.8 py37_1 conda-forge
flatbuffers 1.11.0 he1b5a44_0 conda-forge
freetype 2.10.0 he983fc9_1 conda-forge
fsspec 0.5.2 py_0 conda-forge
future 0.18.1 py37_0 conda-forge
gflags 2.2.2 he1b5a44_1001 conda-forge
glog 0.4.0 he1b5a44_1 conda-forge
gmp 6.1.2 hf484d3e_1000 conda-forge
grpc-cpp 1.23.0 h18db393_0 conda-forge
heapdict 1.0.1 py_0 conda-forge
hypothesis 4.41.3 py37_0 conda-forge
icu 64.2 he1b5a44_1 conda-forge
identify 1.4.7 py_0 conda-forge
idna 2.8 py37_1000 conda-forge
imagesize 1.1.0 py_0 conda-forge
importlib_metadata 0.23 py37_0 conda-forge
ipykernel 5.1.3 py37h5ca1d4c_0 conda-forge
ipython 7.8.0 py37h5ca1d4c_0 conda-forge
ipython_genutils 0.2.0 py_1 conda-forge
isort 4.3.21 py37_0 conda-forge
jedi 0.15.1 py37_0 conda-forge
jinja2 2.10.3 py_0 conda-forge
jpeg 9c h14c3975_1001 conda-forge
json5 0.8.5 py_0
jsonschema 3.1.1 py37_0 conda-forge
jupyter-server-proxy 1.1.0 pypi_0 pypi
jupyter_client 5.3.3 py37_1 conda-forge
jupyter_core 4.5.0 py_0 conda-forge
jupyterlab 1.1.4 pyhf63ae98_0
jupyterlab-nvdashboard 0.1.11 pypi_0 pypi
jupyterlab_server 1.0.6 py_0
krb5 1.16.3 h05b26f9_1001 conda-forge
libblas 3.8.0 14_openblas conda-forge
libcblas 3.8.0 14_openblas conda-forge
libcurl 7.65.3 hda55be3_0 conda-forge
libedit 3.1.20170329 hf8c457e_1001 conda-forge
libevent 2.1.10 h72c5cf5_0 conda-forge
libffi 3.2.1 he1b5a44_1006 conda-forge
libgcc-ng 9.1.0 hdf63c60_0
libgfortran-ng 7.3.0 hdf63c60_2 conda-forge
liblapack 3.8.0 14_openblas conda-forge
libopenblas 0.3.7 h6e990d7_2 conda-forge
libpng 1.6.37 hed695b0_0 conda-forge
libprotobuf 3.8.0 h8b12597_0 conda-forge
librmm 0.11.0a191024 cuda9.2_123 rapidsai-nightly
libsodium 1.0.17 h516909a_0 conda-forge
libssh2 1.8.2 h22169c7_2 conda-forge
libstdcxx-ng 9.1.0 hdf63c60_0
libtiff 4.0.10 hfc65ed5_1004 conda-forge
libuv 1.33.1 h516909a_0 conda-forge
llvmlite 0.29.0 py37hfd453ef_1 conda-forge
locket 0.2.0 py_2 conda-forge
lz4-c 1.8.3 he1b5a44_1001 conda-forge
markdown 3.0.1 pypi_0 pypi
markupsafe 1.1.1 py37h14c3975_0 conda-forge
mccabe 0.6.1 py_1 conda-forge
mistune 0.8.4 py37h14c3975_1000 conda-forge
more-itertools 7.2.0 py_0 conda-forge
msgpack-python 0.6.2 py37hc9558a2_0 conda-forge
multidict 4.5.2 pypi_0 pypi
nbconvert 5.6.0 py37_1 conda-forge
nbformat 4.4.0 py_1 conda-forge
nbsphinx 0.4.3 py_0 conda-forge
nccl 2.4.6.1 cuda9.2_0 nvidia
ncurses 6.1 hf484d3e_1002 conda-forge
nodeenv 1.3.3 py_0 conda-forge
nodejs 10.13.0 he6710b0_0
notebook 6.0.1 py37_0 conda-forge
npm 0.1.1 pypi_0 pypi
numba 0.45.1 py37hb3f55d8_0 conda-forge
numpy 1.17.3 py37h95a1406_0 conda-forge
numpydoc 0.9.1 py_0 conda-forge
nvstrings-cudaunknown 0.0.0.dev0 pypi_0 pypi
olefile 0.46 py_0 conda-forge
openssl 1.1.1d h7b6447c_3
optional-django 0.1.0 pypi_0 pypi
packaging 19.2 py_0 conda-forge
pandas 0.24.2 py37hb3f55d8_0 conda-forge
pandoc 1.19.2 0 conda-forge
pandocfilters 1.4.2 py_1 conda-forge
parquet-cpp 1.5.1 2 conda-forge
parso 0.5.1 py_0 conda-forge
partd 1.0.0 py_0 conda-forge
pexpect 4.7.0 py37_0 conda-forge
pickleshare 0.7.5 py37_1000 conda-forge
pillow 6.2.1 py37h6b7be26_0 conda-forge
pip 19.3.1 py37_0 conda-forge
pluggy 0.13.0 py37_0 conda-forge
pre_commit 1.18.1 py37_0 conda-forge
prometheus_client 0.7.1 py_0 conda-forge
prompt_toolkit 2.0.10 py_0 conda-forge
psutil 5.6.3 py37h516909a_0 conda-forge
ptyprocess 0.6.0 py_1001 conda-forge
py 1.8.0 py_0 conda-forge
pyarrow 0.15.0 py37h8b68381_1 conda-forge
pycodestyle 2.5.0 py_0 conda-forge
pycparser 2.19 py37_1 conda-forge
pyflakes 2.1.1 py_0 conda-forge
pygments 2.4.2 py_0 conda-forge
pynvml 8.0.3 pypi_0 pypi
pyopenssl 19.0.0 py37_0 conda-forge
pyparsing 2.4.2 py_0 conda-forge
pyrsistent 0.15.4 py37h516909a_0 conda-forge
pysocks 1.7.1 py37_0 conda-forge
pytest 5.2.1 py37_0 conda-forge
python 3.7.3 h33d41f4_1 conda-forge
python-dateutil 2.8.0 py_0 conda-forge
pytz 2019.3 py_0 conda-forge
pyyaml 5.1.2 py37h516909a_0 conda-forge
pyzmq 18.1.0 py37h1768529_0 conda-forge
rapidjson 1.1.0 he1b5a44_1002 conda-forge
re2 2019.09.01 he1b5a44_0 conda-forge
readline 8.0 hf8c457e_0 conda-forge
recommonmark 0.6.0 py_0 conda-forge
requests 2.22.0 py37_1 conda-forge
rhash 1.3.6 h14c3975_1001 conda-forge
rmm 0.11.0a191024 py37_123 rapidsai-nightly
send2trash 1.5.0 py_0 conda-forge
setuptools 41.4.0 py37_0 conda-forge
simpervisor 0.3 pypi_0 pypi
six 1.12.0 py37_1000 conda-forge
snakeviz 2.0.1 pypi_0 pypi
snappy 1.1.7 he1b5a44_1002 conda-forge
snowballstemmer 2.0.0 py_0 conda-forge
sortedcontainers 2.1.0 py_0 conda-forge
sphinx 2.2.0 py_0 conda-forge
sphinx-markdown-tables 0.0.10 pypi_0 pypi
sphinx_rtd_theme 0.4.3 py_0 conda-forge
sphinxcontrib-applehelp 1.0.1 py_0 conda-forge
sphinxcontrib-devhelp 1.0.1 py_0 conda-forge
sphinxcontrib-htmlhelp 1.0.2 py_0 conda-forge
sphinxcontrib-jsmath 1.0.1 py_0 conda-forge
sphinxcontrib-qthelp 1.0.2 py_0 conda-forge
sphinxcontrib-serializinghtml 1.1.3 py_0 conda-forge
sphinxcontrib-websupport 1.1.2 py_0 conda-forge
sqlite 3.30.1 hcee41ef_0 conda-forge
streamz 0.5.2 pypi_0 pypi
tblib 1.4.0 py_0 conda-forge
terminado 0.8.2 py37_0 conda-forge
testpath 0.4.2 py_1001 conda-forge
thrift-cpp 0.12.0 hf3afdfd_1004 conda-forge
tk 8.6.9 hed695b0_1003 conda-forge
toml 0.10.0 py_0 conda-forge
toolz 0.10.0 py_0 conda-forge
tornado 6.0.3 py37h516909a_0 conda-forge
traitlets 4.3.3 py37_0 conda-forge
uriparser 0.9.3 he1b5a44_1 conda-forge
urllib3 1.25.6 py37_0 conda-forge
virtualenv 16.7.5 py_0 conda-forge
wcwidth 0.1.7 py_1 conda-forge
webencodings 0.5.1 py_1 conda-forge
wheel 0.33.6 py37_0 conda-forge
xz 5.2.4 h14c3975_1001 conda-forge
yaml 0.1.7 h14c3975_1001 conda-forge
yarl 1.3.0 pypi_0 pypi
zeromq 4.3.2 he1b5a44_2 conda-forge
zict 1.0.0 py_0 conda-forge
zipp 0.6.0 py_0 conda-forge
zlib 1.2.11 h516909a_1006 conda-forge
zstd 1.4.3 h3b9ef0a_0 conda-forge
rjzamora
All 21 comments
Note that the bug doesn't seem to show up in a jupyter notebook unless I run in a loop, e.g:
import cudf
import numpy as np
import sys
cudf.set_allocator(pool=True)
sort_check = True
for i in range(1000):
print("step", i)
sys.stdout.flush()
df = cudf.DataFrame({"a": np.arange(10)})
sort_check = df.a.is_monotonic and sort_check
print("sort_check", sort_check)
Just a note... The is_monotonic segfault is actually an is_sorted problem. The following also crashes:
import cudf
import cudf._lib as libcudf
import numpy as np
cudf.set_allocator(pool=True)
df = cudf.DataFrame({"a": np.arange(10)})
libcudf.issorted.issorted([df.a._column])
The segfault seems to happen here: https://github.com/rapidsai/cudf/blob/411e0c56e57a4344caf75de260f73ecec967e3dc/cpp/src/predicates/is_sorted.cu#L62
rjzamora
on 29 Oct 2019
@pentschev and I came across this error when trying out cudf+ucx
quasiben
on 30 Oct 2019
cc @kkraus14
jakirkham
on 31 Oct 2019
Has anyone run it through cuda-gdb/cuda-memcheck with a Debug build?
jrhemstad
on 31 Oct 2019
Nevermind. I already see what the problem is.
jrhemstad
on 31 Oct 2019
🎉2
👍1
Not yet. We looked with gdb yesterday and came upon the thrust line @rjzamora foundd
quasiben
on 31 Oct 2019
Can you verify that https://github.com/rapidsai/cudf/pull/3265 fixes your repro?
jrhemstad
on 31 Oct 2019
checking it out now
quasiben
on 31 Oct 2019
Having some issues on my workstation -- apologies for the delay
quasiben
on 31 Oct 2019
Thanks @jrhemstad !
rjzamora
on 31 Oct 2019
@jrhemstad, finally got around to testing and i'm still seeing the bug with rmm:
In [1]: import cudf
In [2]: cudf.__version__
Out[2]: '0.11.0a+1899.g7cf09ed65'
In [3]: cudf.set_allocator("default", pool=True, initial_pool_size=int(2.4e8))
In [4]: cdf = cudf.Series([1,5,3])
In [5]: cdf.is_monotonic_increasing
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-6-81d579b97f21> in <module>
----> 1 cdf.is_monotonic_increasing
/datasets/bzaitlen/miniconda3/envs/cudf_dev10.1/lib/python3.7/site-packages/cudf/core/series.py in is_monotonic_increasing(self)
2557 @property
2558 def is_monotonic_increasing(self):
-> 2559 return self._column.is_monotonic_increasing
2560
2561 @property
/datasets/bzaitlen/miniconda3/envs/cudf_dev10.1/lib/python3.7/site-packages/cudf/core/column/numerical.py in is_monotonic_increasing(self)
396 else:
397 self._is_monotonic_increasing = libcudf.issorted.issorted(
--> 398 [self]
399 )
400 return self._is_monotonic_increasing
cudf/_lib/issorted.pyx in cudf._lib.issorted.issorted()
cudf/_lib/issorted.pyx in cudf._lib.issorted.issorted()
RuntimeError: after reduction step 2: cudaErrorContextIsDestroyed: context is destroyed
I don't know what to tell you on that one. CUDA context creation/destruction is completely outside of libcudf's domain.
Is cuDF/Dask manually creating a CUDA context?
jrhemstad
on 1 Nov 2019
dask-cuda will create a cuda context. But there is no dask/dask-cuda here. @kkraus14 does using the rmm pool create a cuda context ?
cc @pentschev in case he has thoughts
quasiben
on 1 Nov 2019
No, RMM does not create a context. Numba does, because it uses the Driver API. We had this discussion on slack the other day.
harrism
on 1 Nov 2019
right right, you did say that.
However, we are still getting an cudaErrorInvalidResourceHandle with PR #3265 in. Should we open an issue up in RMM or re-open this issue ? Again, there is no dask in the reproducible examples above -- just cudf and rmm
quasiben
on 1 Nov 2019
Ah, actually, I see another problem in is_sorted. It doesn't have anything to do with the context though, but we'll see if it fixes it anyways.
jrhemstad
on 1 Nov 2019
Give https://github.com/rapidsai/cudf/pull/3277 a spin.
jrhemstad
on 1 Nov 2019
Building it now. Thank you again!
quasiben
on 1 Nov 2019
That works!
quasiben
on 1 Nov 2019
🎉2
3277 is resolving my join bug as well :)
rjzamora
on 1 Nov 2019
🎉1
Was this page helpful?
0 / 5 - 0 ratings
Related issues
yasmina-altair
路
3Comments
galipremsagar
路
3Comments
ayushdg
路
3Comments
kkraus14
路
3Comments
galipremsagar
路
3Comments
Most helpful comment
Nevermind. I already see what the problem is.