Cuml: [BUG] cuml.dask NearestNeighbots pytest failure in CUDA 11
Describe the bug
Running pytests in a CUDA 11 conda environment, the following tests are failing for me:
FAILED cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[True-5-1-10-5-30-1000.0] - assert 0.0523 <= 0.003
FAILED cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[True-5-1-10-5-30-10000.0] - assert 0.07282 <= 0.003
FAILED cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[True-5-5-10-5-30-10000.0] - assert 0.07282 <= 0.003
FAILED cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[True-10-1-10-5-30-1000.0] - assert 0.0523 <= 0.003
FAILED cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[True-10-1-10-5-30-10000.0] - assert 0.07282 <= 0.003
FAILED cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[True-10-5-10-5-30-10000.0] - assert 0.07282 <= 0.003
FAILED cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[False-5-1-10-5-30-1000.0] - assert 0.0523 <= 0.003
FAILED cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[False-5-1-10-5-30-10000.0] - assert 0.07282 <= 0.003
FAILED cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[False-5-5-10-5-30-10000.0] - assert 0.07282 <= 0.003
FAILED cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[False-10-1-10-5-30-1000.0] - assert 0.0523 <= 0.003
FAILED cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[False-10-1-10-5-30-10000.0] - assert 0.07282 <= 0.003
FAILED cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[False-10-5-10-5-30-10000.0] - assert 0.07282 <= 0.003
The particular assertion test failing can be seen here:
@pytest.mark.parametrize("nrows", [unit_param(100),
unit_param(1e3),
unit_param(1e4),
quality_param(1e6),
stress_param(5e8)])
@pytest.mark.parametrize("ncols", [10, 30])
@pytest.mark.parametrize("nclusters", [unit_param(5), quality_param(10),
stress_param(15)])
@pytest.mark.parametrize("n_neighbors", [unit_param(10), quality_param(4),
stress_param(100)])
@pytest.mark.parametrize("n_parts", [unit_param(1), unit_param(5),
quality_param(7), stress_param(50)])
@pytest.mark.parametrize("streams_per_handle", [5, 10])
@pytest.mark.parametrize("reverse_worker_order", [True, False])
def test_compare_skl(nrows, ncols, nclusters, n_parts, n_neighbors,
streams_per_handle, reverse_worker_order, client):
from cuml.dask.neighbors import NearestNeighbors as daskNN
from sklearn.datasets import make_blobs
nrows = _scale_rows(client, nrows)
X, y = make_blobs(n_samples=int(nrows),
n_features=ncols,
centers=nclusters,
random_state=0)
X = X.astype(np.float32)
X_cudf = _prep_training_data(client, X, n_parts, reverse_worker_order)
from dask.distributed import wait
wait(X_cudf)
dist = np.array([len(v) for v in client.has_what().values()])
assert np.all(dist == dist[0])
cumlModel = daskNN(n_neighbors=n_neighbors,
streams_per_handle=streams_per_handle)
cumlModel.fit(X_cudf)
out_d, out_i = cumlModel.kneighbors(X_cudf)
local_i = np.array(out_i.compute().as_gpu_matrix(), dtype="int64")
sklModel = KNeighborsClassifier(n_neighbors=n_neighbors).fit(X, y)
skl_y_hat = sklModel.predict(X)
y_hat, _ = predict(local_i, y, n_neighbors)
sk_d, sk_i = sklModel.kneighbors(X)
sk_i = sk_i.astype("int64")
assert array_equal(local_i[:, 0], np.arange(nrows))
diff = sk_i-local_i
n_diff = len(diff[diff > 0])
perc_diff = n_diff / (nrows * n_neighbors)
> assert perc_diff <= 3e-3
E assert 0.07282 <= 0.003
Steps/Code to reproduce bug
Build RMM, cuDF and cuML from source using CUDA 11.0 (update 1), then run the pytests under dask/test_nearest_neighbors.py
Expected behavior
No errors
Environment details (please complete the following information):
- Environment location: Bare metal
- Linux Distro/Architecture: Ubuntu 20.04
- GPU Model/Driver: 2070 Super
- CUDA: 11.0
- Method of cuDF & cuML install: RMM, cuDF and cuML from source
Additional context
Result of print_env:
Click here to see environment details
**git***
commit fcb8ebb28e771ff624c767ad0aff1393bb3aaf96 (HEAD -> fea-ext-cuda11-libcuml++-support, origin/fea-ext-cuda11-libcuml++-support)
Author: Thejaswi Rao <[email protected]>
Date: Mon Aug 10 03:37:30 2020 +0000
FIX clang-format fixes
**git submodules***
***OS Information***
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04 LTS"
NAME="Ubuntu"
VERSION="20.04 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
Linux schwifty1 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
***GPU Information***
Mon Aug 10 08:39:53 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.57 Driver Version: 450.57 CUDA Version: 11.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce RTX 207... Off | 00000000:0B:00.0 On | N/A |
| 41% 41C P0 33W / 215W | 463MiB / 7959MiB | 1% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1615 G /usr/lib/xorg/Xorg 72MiB |
| 0 N/A N/A 8826 G /usr/lib/xorg/Xorg 235MiB |
| 0 N/A N/A 9033 G /usr/bin/gnome-shell 110MiB |
+-----------------------------------------------------------------------------+
***CPU***
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 43 bits physical, 48 bits virtual
CPU(s): 32
On-line CPU(s) list: 0-31
Thread(s) per core: 2
Core(s) per socket: 16
Socket(s): 1
NUMA node(s): 1
Vendor ID: AuthenticAMD
CPU family: 23
Model: 113
Model name: AMD Ryzen 9 3950X 12-Core Processor
Stepping: 0
Frequency boost: disabled
CPU MHz: 3999.715
CPU max MHz: 4700.0000
CPU min MHz: 2200.0000
BogoMIPS: 7999.99
Virtualization: AMD-V
L1d cache: 512 KiB
L1i cache: 512 KiB
L2 cache: 8 MiB
L3 cache: 64 MiB
NUMA node0 CPU(s): 0-31
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Full AMD retpoline, IBPB conditional, STIBP conditional, RSB filling
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate sme ssbd mba sev ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip rdpid overflow_recov succor smca
***CMake***
/home/galahad/miniconda3/envs/c11-buildall/bin/cmake
cmake version 3.14.5
CMake suite maintained and supported by Kitware (kitware.com/cmake).
***g++***
/usr/bin/g++
g++ (Ubuntu 8.4.0-3ubuntu2) 8.4.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
***nvcc***
/usr/local/cuda-11.0/bin/nvcc
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Wed_Jul_22_19:09:09_PDT_2020
Cuda compilation tools, release 11.0, V11.0.221
Build cuda_11.0_bu.TC445_37.28845127_0
***Python***
/home/galahad/miniconda3/envs/c11-buildall/bin/python
Python 3.7.8
***Environment Variables***
PATH : /usr/local/cuda-11.0/bin:/usr/local/cuda-11.0/NsightCompute-2020.3.2:/home/galahad/miniconda3/envs/c11-buildall/bin:/home/galahad/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
LD_LIBRARY_PATH : /usr/local/cuda-11.0/lib64
NUMBAPRO_NVVM :
NUMBAPRO_LIBDEVICE :
CONDA_PREFIX : /home/galahad/miniconda3/envs/c11-buildall
PYTHON_PATH :
***conda packages***
/home/galahad/miniconda3/condabin/conda
# packages in environment at /home/galahad/miniconda3/envs/c11-buildall:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_gnu conda-forge
abseil-cpp 20200225.2 he1b5a44_2 conda-forge
alabaster 0.7.12 py_0 conda-forge
appdirs 1.4.3 py_1 conda-forge
argon2-cffi 20.1.0 py37h8f50634_1 conda-forge
arrow-cpp 0.17.1 py37h1234567_11_cuda conda-forge
arrow-cpp-proc 1.0.0 cuda conda-forge
attrs 19.3.0 py_0 conda-forge
aws-sdk-cpp 1.7.164 hc831370_1 conda-forge
babel 2.8.0 py_0 conda-forge
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backports 1.0 py_2 conda-forge
backports.functools_lru_cache 1.6.1 py_0 conda-forge
black 19.10b0 py_4 conda-forge
bleach 3.1.5 pyh9f0ad1d_0 conda-forge
bokeh 2.1.1 py37hc8dfbb8_0 conda-forge
boost-cpp 1.72.0 h7b93d67_2 conda-forge
brotli 1.0.7 he1b5a44_1004 conda-forge
brotlipy 0.7.0 py37h8f50634_1000 conda-forge
bzip2 1.0.8 h516909a_2 conda-forge
c-ares 1.16.1 h516909a_0 conda-forge
ca-certificates 2020.6.20 hecda079_0 conda-forge
certifi 2020.6.20 py37hc8dfbb8_0 conda-forge
cffi 1.14.1 py37h2b28604_0 conda-forge
cfgv 3.2.0 py_0 conda-forge
chardet 3.0.4 py37hc8dfbb8_1006 conda-forge
clang 8.0.1 hc9558a2_2 conda-forge
clang-tools 8.0.1 hc9558a2_2 conda-forge
clangxx 8.0.1 2 conda-forge
click 7.1.2 pyh9f0ad1d_0 conda-forge
cloudpickle 1.5.0 py_0 conda-forge
cmake 3.14.5 hf94ab9c_0 conda-forge
cmake_setuptools 0.1.3 py_0 rapidsai
commonmark 0.9.1 py_0 conda-forge
cryptography 3.0 py37hb09aad4_0 conda-forge
cudatoolkit 11.0.194 h6bb024c_0 nvidia
cudf 0.15.0a0+4108.g70182b231 pypi_0 pypi
cudnn 8.0.0 cuda11.0_0 nvidia
cuml 0.2.0+12971.gfcb8ebb28 pypi_0 pypi
cupy 8.0.0dev py37h78336d7_0 nvidia
curl 7.71.1 he644dc0_4 conda-forge
cython 0.29.21 py37h3340039_0 conda-forge
cytoolz 0.10.1 py37h516909a_0 conda-forge
dask 2.22.0+9.g14c53510 pypi_0 pypi
dask-cuda 0.15.0a200809 py37_90 rapidsai-nightly
dask-cudf 0.15.0a0+4108.g70182b231 pypi_0 pypi
dask-glm 0.2.0 py_1 conda-forge
dask-ml 1.6.0 py_0 conda-forge
decorator 4.4.2 py_0 conda-forge
defusedxml 0.6.0 py_0 conda-forge
distlib 0.3.1 pyh9f0ad1d_0 conda-forge
distributed 2.22.0+15.g2d97609b pypi_0 pypi
dlpack 0.3 he1b5a44_1 conda-forge
docutils 0.16 py37hc8dfbb8_1 conda-forge
double-conversion 3.1.5 he1b5a44_2 conda-forge
doxygen 1.8.18 hd1b7508_0 conda-forge
editdistance 0.5.3 py37h3340039_0 conda-forge
entrypoints 0.3 py37hc8dfbb8_1001 conda-forge
expat 2.2.9 he1b5a44_2 conda-forge
faiss-proc 1.0.0 cuda rapidsai-nightly
fastavro 0.24.0 py37h8f50634_0 conda-forge
fastrlock 0.5 py37h3340039_0 conda-forge
filelock 3.0.12 pyh9f0ad1d_0 conda-forge
flake8 3.8.3 py_1 conda-forge
flatbuffers 1.12.0 he1b5a44_0 conda-forge
freetype 2.10.2 he06d7ca_0 conda-forge
fsspec 0.8.0 py_0 conda-forge
future 0.18.2 py37hc8dfbb8_1 conda-forge
gflags 2.2.2 he1b5a44_1004 conda-forge
glog 0.4.0 h49b9bf7_3 conda-forge
gmp 6.2.0 he1b5a44_2 conda-forge
grpc-cpp 1.30.2 heedbac9_0 conda-forge
heapdict 1.0.1 py_0 conda-forge
hypothesis 5.23.11 py_0 conda-forge
icu 67.1 he1b5a44_0 conda-forge
identify 1.4.25 pyh9f0ad1d_0 conda-forge
idna 2.10 pyh9f0ad1d_0 conda-forge
imagesize 1.2.0 py_0 conda-forge
importlib-metadata 1.7.0 py37hc8dfbb8_0 conda-forge
importlib_metadata 1.7.0 0 conda-forge
iniconfig 1.0.1 pyh9f0ad1d_0 conda-forge
ipykernel 5.3.4 py37h43977f1_0 conda-forge
ipython 7.17.0 py37hc6149b9_0 conda-forge
ipython_genutils 0.2.0 py_1 conda-forge
isort 5.3.2 py37hc8dfbb8_0 conda-forge
jedi 0.17.2 py37hc8dfbb8_0 conda-forge
jinja2 2.11.2 pyh9f0ad1d_0 conda-forge
joblib 0.16.0 py_0 conda-forge
jpeg 9d h516909a_0 conda-forge
jsonschema 3.2.0 py37hc8dfbb8_1 conda-forge
jupyter_client 6.1.6 py_0 conda-forge
jupyter_core 4.6.3 py37hc8dfbb8_1 conda-forge
krb5 1.17.1 hfafb76e_2 conda-forge
lcms2 2.11 hbd6801e_0 conda-forge
ld_impl_linux-64 2.34 hc38a660_9 conda-forge
libblas 3.8.0 17_openblas conda-forge
libcblas 3.8.0 17_openblas conda-forge
libcurl 7.71.1 hcdd3856_4 conda-forge
libedit 3.1.20191231 h46ee950_1 conda-forge
libev 4.33 h516909a_0 conda-forge
libevent 2.1.10 hcdb4288_1 conda-forge
libfaiss 1.6.3 h328c4c8_1_cuda rapidsai-nightly
libffi 3.2.1 he1b5a44_1007 conda-forge
libgcc-ng 9.3.0 h24d8f2e_14 conda-forge
libgfortran-ng 7.5.0 hdf63c60_14 conda-forge
libgomp 9.3.0 h24d8f2e_14 conda-forge
libhwloc 2.1.0 h3c4fd83_0 conda-forge
libiconv 1.15 h516909a_1006 conda-forge
liblapack 3.8.0 17_openblas conda-forge
libllvm8 8.0.1 hc9558a2_0 conda-forge
libllvm9 9.0.1 he513fc3_1 conda-forge
libnghttp2 1.41.0 hab1572f_1 conda-forge
libopenblas 0.3.10 pthreads_hb3c22a3_4 conda-forge
libpng 1.6.37 hed695b0_1 conda-forge
libprotobuf 3.12.4 h8b12597_0 conda-forge
libsodium 1.0.18 h516909a_0 conda-forge
libssh2 1.9.0 hab1572f_5 conda-forge
libstdcxx-ng 9.3.0 hdf63c60_14 conda-forge
libtiff 4.1.0 hc7e4089_6 conda-forge
libuv 1.38.0 h516909a_0 conda-forge
libwebp-base 1.1.0 h516909a_3 conda-forge
libxml2 2.9.10 h72b56ed_2 conda-forge
llvmlite 0.33.0 py37h5202443_1 conda-forge
locket 0.2.0 py_2 conda-forge
lz4-c 1.9.2 he1b5a44_1 conda-forge
markdown 3.2.2 py_0 conda-forge
markupsafe 1.1.1 py37h8f50634_1 conda-forge
mccabe 0.6.1 py_1 conda-forge
mistune 0.8.4 py37h8f50634_1001 conda-forge
more-itertools 8.4.0 py_0 conda-forge
msgpack-python 1.0.0 py37h99015e2_1 conda-forge
multipledispatch 0.6.0 py_0 conda-forge
nbconvert 5.6.1 py37hc8dfbb8_1 conda-forge
nbformat 5.0.7 py_0 conda-forge
nbsphinx 0.7.1 pyh9f0ad1d_0 conda-forge
nccl 2.7.8.1 h4962215_0 nvidia
ncurses 6.2 he1b5a44_1 conda-forge
nodeenv 1.4.0 pyh9f0ad1d_0 conda-forge
notebook 6.1.1 py37hc8dfbb8_0 conda-forge
numba 0.50.1 py37h0da4684_1 conda-forge
numpy 1.19.1 py37h8960a57_0 conda-forge
numpydoc 1.1.0 pyh9f0ad1d_0 conda-forge
olefile 0.46 py_0 conda-forge
openssl 1.1.1g h516909a_1 conda-forge
packaging 20.4 pyh9f0ad1d_0 conda-forge
pandas 1.0.5 py37h0da4684_0 conda-forge
pandoc 1.19.2 0 conda-forge
pandocfilters 1.4.2 py_1 conda-forge
parquet-cpp 1.5.1 2 conda-forge
parso 0.7.1 pyh9f0ad1d_0 conda-forge
partd 1.1.0 py_0 conda-forge
pathspec 0.8.0 pyh9f0ad1d_0 conda-forge
patsy 0.5.1 py_0 conda-forge
pexpect 4.8.0 py37hc8dfbb8_1 conda-forge
pickleshare 0.7.5 py37hc8dfbb8_1001 conda-forge
pillow 7.2.0 py37h718be6c_1 conda-forge
pip 20.2.1 py_0 conda-forge
pluggy 0.13.1 py37hc8dfbb8_2 conda-forge
pre-commit 2.6.0 py37hc8dfbb8_0 conda-forge
pre_commit 2.6.0 0 conda-forge
prometheus_client 0.8.0 pyh9f0ad1d_0 conda-forge
prompt-toolkit 3.0.5 py_1 conda-forge
psutil 5.7.2 py37h8f50634_0 conda-forge
ptyprocess 0.6.0 py_1001 conda-forge
py 1.9.0 pyh9f0ad1d_0 conda-forge
pyarrow 0.17.1 py37h1234567_11_cuda conda-forge
pycodestyle 2.6.0 pyh9f0ad1d_0 conda-forge
pycparser 2.20 pyh9f0ad1d_2 conda-forge
pyflakes 2.2.0 pyh9f0ad1d_0 conda-forge
pygments 2.6.1 py_0 conda-forge
pynvml 8.0.4 py_1 conda-forge
pyopenssl 19.1.0 py_1 conda-forge
pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge
pyrsistent 0.16.0 py37h8f50634_0 conda-forge
pysocks 1.7.1 py37hc8dfbb8_1 conda-forge
pytest 6.0.1 py37hc8dfbb8_0 conda-forge
pytest-timeout 1.4.2 pyh9f0ad1d_0 conda-forge
python 3.7.8 h6f2ec95_1_cpython conda-forge
python-dateutil 2.8.1 py_0 conda-forge
python_abi 3.7 1_cp37m conda-forge
pytz 2020.1 pyh9f0ad1d_0 conda-forge
pyyaml 5.3.1 py37h8f50634_0 conda-forge
pyzmq 19.0.2 py37hac76be4_0 conda-forge
rapidjson 1.1.0 he1b5a44_1002 conda-forge
re2 2020.07.06 he1b5a44_1 conda-forge
readline 8.0 he28a2e2_2 conda-forge
recommonmark 0.6.0 py_0 conda-forge
regex 2020.7.14 py37h8f50634_0 conda-forge
requests 2.24.0 pyh9f0ad1d_0 conda-forge
rhash 1.3.6 h14c3975_1001 conda-forge
rmm 0.15.0 pypi_0 pypi
scikit-learn 0.23.2 py37h6785257_0 conda-forge
scipy 1.5.2 py37hb14ef9d_0 conda-forge
send2trash 1.5.0 py_0 conda-forge
setuptools 49.3.0 py37hc8dfbb8_0 conda-forge
six 1.15.0 pyh9f0ad1d_0 conda-forge
snappy 1.1.8 he1b5a44_3 conda-forge
snowballstemmer 2.0.0 py_0 conda-forge
sortedcontainers 2.2.2 pyh9f0ad1d_0 conda-forge
spdlog 1.7.0 hc9558a2_2 conda-forge
sphinx 3.2.0 py_0 conda-forge
sphinx-copybutton 0.3.0 pyh9f0ad1d_0 conda-forge
sphinx-markdown-tables 0.0.15 pypi_0 pypi
sphinx_rtd_theme 0.5.0 pyh9f0ad1d_0 conda-forge
sphinxcontrib-applehelp 1.0.2 py_0 conda-forge
sphinxcontrib-devhelp 1.0.2 py_0 conda-forge
sphinxcontrib-htmlhelp 1.0.3 py_0 conda-forge
sphinxcontrib-jsmath 1.0.1 py_0 conda-forge
sphinxcontrib-qthelp 1.0.3 py_0 conda-forge
sphinxcontrib-serializinghtml 1.1.4 py_0 conda-forge
sphinxcontrib-websupport 1.2.3 pyh9f0ad1d_0 conda-forge
sqlite 3.32.3 hcee41ef_1 conda-forge
statsmodels 0.11.1 py37h8f50634_2 conda-forge
streamz 0.5.4 pypi_0 pypi
tbb 2020.1 hc9558a2_0 conda-forge
tblib 1.6.0 py_0 conda-forge
terminado 0.8.3 py37hc8dfbb8_1 conda-forge
testpath 0.4.4 py_0 conda-forge
threadpoolctl 2.1.0 pyh5ca1d4c_0 conda-forge
thrift-cpp 0.13.0 h62aa4f2_2 conda-forge
tk 8.6.10 hed695b0_0 conda-forge
toml 0.10.1 pyh9f0ad1d_0 conda-forge
toolz 0.10.0 py_0 conda-forge
tornado 6.0.4 py37h8f50634_1 conda-forge
traitlets 4.3.3 py37hc8dfbb8_1 conda-forge
treelite 0.92 py37h023e13c_2 conda-forge
treelite-runtime 0.92 pypi_0 pypi
typed-ast 1.4.1 py37h516909a_0 conda-forge
typing_extensions 3.7.4.2 py_0 conda-forge
ucx 1.8.1+g6b29558 ha5db111_0 rapidsai-nightly
ucx-py 0.15.0a200809+g6b29558 py37_167 rapidsai-nightly
umap-learn 0.4.6 py37hc8dfbb8_0 conda-forge
urllib3 1.25.10 py_0 conda-forge
virtualenv 20.0.20 py37hc8dfbb8_1 conda-forge
wcwidth 0.2.5 pyh9f0ad1d_1 conda-forge
webencodings 0.5.1 py_1 conda-forge
wheel 0.34.2 py_1 conda-forge
xz 5.2.5 h516909a_1 conda-forge
yaml 0.2.5 h516909a_0 conda-forge
zeromq 4.3.2 he1b5a44_3 conda-forge
zict 2.0.0 py_0 conda-forge
zipp 3.1.0 py_0 conda-forge
zlib 1.2.11 h516909a_1006 conda-forge
zstd 1.4.5 h6597ccf_2 conda-forge
bug
dantegd
All 3 comments
Note: the numeric error seems larger in a 2xv100 system:
nrows = 20000.0, ncols = 10, nclusters = 5, n_parts = 1, n_neighbors = 10, streams_per_handle = 5, reverse_worker_order = True
client = <Client: 'tcp://127.0.0.1:36119' processes=2 threads=2, memory=49.16 GB>
@pytest.mark.parametrize("nrows", [unit_param(100),
unit_param(1e3),
unit_param(1e4),
quality_param(1e6),
stress_param(5e8)])
@pytest.mark.parametrize("ncols", [10, 30])
@pytest.mark.parametrize("nclusters", [unit_param(5), quality_param(10),
stress_param(15)])
@pytest.mark.parametrize("n_neighbors", [unit_param(10), quality_param(4),
stress_param(100)])
@pytest.mark.parametrize("n_parts", [unit_param(1), unit_param(5),
quality_param(7), stress_param(50)])
@pytest.mark.parametrize("streams_per_handle", [5, 10])
@pytest.mark.parametrize("reverse_worker_order", [True, False])
def test_compare_skl(nrows, ncols, nclusters, n_parts, n_neighbors,
streams_per_handle, reverse_worker_order, client):
from cuml.dask.neighbors import NearestNeighbors as daskNN
from sklearn.datasets import make_blobs
nrows = _scale_rows(client, nrows)
X, y = make_blobs(n_samples=int(nrows),
n_features=ncols,
centers=nclusters,
random_state=0)
X = X.astype(np.float32)
X_cudf = _prep_training_data(client, X, n_parts, reverse_worker_order)
from dask.distributed import wait
wait(X_cudf)
dist = np.array([len(v) for v in client.has_what().values()])
assert np.all(dist == dist[0])
cumlModel = daskNN(n_neighbors=n_neighbors,
streams_per_handle=streams_per_handle)
cumlModel.fit(X_cudf)
out_d, out_i = cumlModel.kneighbors(X_cudf)
local_i = np.array(out_i.compute().as_gpu_matrix(), dtype="int64")
sklModel = KNeighborsClassifier(n_neighbors=n_neighbors).fit(X, y)
skl_y_hat = sklModel.predict(X)
y_hat, _ = predict(local_i, y, n_neighbors)
sk_d, sk_i = sklModel.kneighbors(X)
sk_i = sk_i.astype("int64")
assert array_equal(local_i[:, 0], np.arange(nrows))
diff = sk_i-local_i
n_diff = len(diff[diff > 0])
perc_diff = n_diff / (nrows * n_neighbors)
> assert perc_diff <= 3e-3
E assert 0.13213 <= 0.003
When it鈥檚 the indices that don鈥檛 match up like this, the culprit is often in the translations. For example, each partition has an offset and it performs its own local kNN with indices that start at 0 and adjusts those indices based on its offset. If these translations are incorrect, or are not adjusted at all, the indices arrays can have huge delta like this (which will often grow with the number of GPUs). It would be strange if this particular problem was just happening for CUDA 11 and not other CUDA versions, though, unless maybe there鈥檚 a different dependency somewhere that鈥檚 changing how the partitions get assigned to the translation offsets.
If this was only related to the CUDA version (and everything else is the same, including FAISS versions), I鈥檇 expect the indices to match within the tiny potential 0.3% margin for error which can arise simply from nondeterminism in conflicting distances.
Best way to see of this is happening is to take a look at the actual indices being computed in the test. It鈥檚 often pretty obvious when the problem is related to the translations because you鈥檒l see a pattern in the resulting indices (for example, an index is exactly 1000 less than it should be).
cjnolet
on 11 Aug 2020
👍2
Should be solved by FAISS patch, but we will check within CI
JohnZed
on 17 Aug 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
beckernick
路
8Comments
benjha
路
6Comments
beckernick
路
10Comments
lk1983823
路
3Comments
wphicks
路
8Comments
Most helpful comment
When it鈥檚 the indices that don鈥檛 match up like this, the culprit is often in the translations. For example, each partition has an offset and it performs its own local kNN with indices that start at 0 and adjusts those indices based on its offset. If these translations are incorrect, or are not adjusted at all, the indices arrays can have huge delta like this (which will often grow with the number of GPUs). It would be strange if this particular problem was just happening for CUDA 11 and not other CUDA versions, though, unless maybe there鈥檚 a different dependency somewhere that鈥檚 changing how the partitions get assigned to the translation offsets.
If this was only related to the CUDA version (and everything else is the same, including FAISS versions), I鈥檇 expect the indices to match within the tiny potential 0.3% margin for error which can arise simply from nondeterminism in conflicting distances.
Best way to see of this is happening is to take a look at the actual indices being computed in the test. It鈥檚 often pretty obvious when the problem is related to the translations because you鈥檒l see a pattern in the resulting indices (for example, an index is exactly 1000 less than it should be).