What is your question?
After building libcuml and proceeding to build cuml using the build.sh script, I am getting the next error:
Third party modules found succesfully in the libcuml++ build folder.
running build_ext
building 'cuml.cluster.dbscan' extension
/sw/summit/gcc/6.4.0/bin/gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I../cpp/src -I../cpp/include -I../cpp/external -I../cpp/src_prims -I../cpp/build/cutlass/src/cutlass -I../cpp/build/cub/src/cub -I../cpp/build/faiss/src/ -I../cpp/build/treelite/src/treelite/include -I../cpp/comms/std/src -I../cpp/comms/std/include -I/sw/summit/cuda/10.1.168/include -I/gpfs/alpine/proj-shared/stf011/benjha/rapids_build_0.11/anaconda3/envs/rapids_0.11/lib/python3.7/site-packages/numpy/core/include -I/gpfs/alpine/proj-shared/stf011/benjha/rapids_build_0.11/anaconda3/envs/rapids_0.11/include -I/gpfs/alpine/proj-shared/stf011/benjha/rapids_build_0.11/anaconda3/envs/rapids_0.11/include/cumlprims -I/gpfs/alpine/proj-shared/stf011/benjha/rapids_build_0.11/anaconda3/envs/rapids_0.11/include/python3.7m -c cuml/cluster/dbscan.cpp -o build/temp.linux-ppc64le-3.7/cuml/cluster/dbscan.o -std=c++11
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
/sw/summit/gcc/6.4.0/bin/g++ -pthread -shared -B /gpfs/alpine/proj-shared/stf011/benjha/rapids_build_0.11/anaconda3/envs/rapids_0.11/compiler_compat -L/gpfs/alpine/proj-shared/stf011/benjha/rapids_build_0.11/anaconda3/envs/rapids_0.11/lib -Wl,-rpath=/gpfs/alpine/proj-shared/stf011/benjha/rapids_build_0.11/anaconda3/envs/rapids_0.11/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-ppc64le-3.7/cuml/cluster/dbscan.o -L/gpfs/alpine/proj-shared/stf011/benjha/rapids_build_0.11/anaconda3/envs/rapids_0.11/lib/python3.7/site-packages -Wl,-R/sw/summit/cuda/10.1.168/lib64 -Wl,-R/gpfs/alpine/proj-shared/stf011/benjha/rapids_build_0.11/anaconda3/envs/rapids_0.11/lib -lcuda -lcuml++ -lcumlcomms -lnccl -lrmm -lcumlprims -o build/lib.linux-ppc64le-3.7/cuml/cluster/dbscan.cpython-37m-powerpc64le-linux-gnu.so
/gpfs/alpine/proj-shared/stf011/benjha/rapids_build_0.11/anaconda3/envs/rapids_0.11/compiler_compat/ld: cannot find -lcumlprims
collect2: error: ld returned 1 exit status
error: command '/sw/summit/gcc/6.4.0/bin/g++' failed with exit status 1
The question here is when libcumlprims is generated and where it should exist ? A search to that file in my cuml's source code directory and conda environment showed no results.
My environment is as follows:
./print_env.sh
<details><summary>Click here to see environment details</summary><pre>
**git***
commit 3498c7e7b6a246eb7b65517214de4b5aa3deb3cf (HEAD, origin/branch-0.11, branch-0.11)
Author: Ray Douglass <[email protected]>
Date: Tue Dec 10 15:57:55 2019 -0500
Update CHANGELOG.md
**git submodules***
b165e1fb11eeea64ccf95053e40f2424312599cc thirdparty/cub (v1.7.1)
63f644be44201467e3938d59ed9d89cc8725c35d thirdparty/jitify (remotes/origin/feature/api_v2_v0.10)
39125e0e476b960c2001f1ec76a3441335ff91b2 thirdparty/libcudacxx (0.8.1-94-g39125e0)
08bc464bd8f4d779e4294305aa7dadebcebcc507 thirdparty/libcudacxx/libcxx (remotes/origin/master-6-g08bc464)
***OS Information***
NAME="Red Hat Enterprise Linux Server"
VERSION="7.6 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.6"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.6 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.6:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.6"
Red Hat Enterprise Linux Server release 7.6 (Maipo)
Red Hat Enterprise Linux Server release 7.6 (Maipo)
Linux login2 4.14.0-115.8.1.el7a.ppc64le #1 SMP Thu May 9 14:45:13 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux
***GPU Information***
Mon Dec 23 17:04:20 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.67 Driver Version: 418.67 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-SXM2... On | 00000035:03:00.0 Off | 0 |
| N/A 32C P0 35W / 300W | 0MiB / 16130MiB | 0% E. Process |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
***CPU***
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 128
On-line CPU(s) list: 0-127
Thread(s) per core: 4
Core(s) per socket: 16
Socket(s): 2
NUMA node(s): 6
Model: 2.1 (pvr 004e 1201)
Model name: POWER9, altivec supported
CPU max MHz: 3800.0000
CPU min MHz: 2300.0000
L1d cache: 32K
L1i cache: 32K
L2 cache: 512K
L3 cache: 10240K
NUMA node0 CPU(s): 0-63
NUMA node8 CPU(s): 64-127
NUMA node252 CPU(s):
NUMA node253 CPU(s):
NUMA node254 CPU(s):
NUMA node255 CPU(s):
***CMake***
/autofs/nccs-svm1_sw/summit/.swci/0-core/opt/spack/20180914/linux-rhel7-ppc64le/gcc-4.8.5/cmake-3.15.2-xit2o3iepxvqbyku77lwcugufilztu7t/bin/cmake
cmake version 3.15.2
CMake suite maintained and supported by Kitware (kitware.com/cmake).
***g++***
/sw/summit/gcc/6.4.0/bin/g++
g++ (GCC) 6.4.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
***nvcc***
/sw/summit/cuda/10.1.168/bin/nvcc
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Apr_24_19:12:21_PDT_2019
Cuda compilation tools, release 10.1, V10.1.168
***Python***
/gpfs/alpine/proj-shared/stf011/benjha/rapids_build_0.11/anaconda3/envs/rapids_0.11/bin/python
Python 3.7.5
***Environment Variables***
PATH : /gpfs/alpine/proj-shared/stf011/benjha/rapids_build_0.11/anaconda3/envs/rapids_0.11/bin:/gpfs/alpine/proj-shared/stf011/benjha/rapids_build_0.11/anaconda3/bin:/sw/sources/lsf-tools/2.0/summit/bin:/sw/summit/xalt/1.1.4/bin:/autofs/nccs-svm1_sw/summit/.swci/0-core/opt/spack/20180914/linux-rhel7-ppc64le/gcc-4.8.5/cmake-3.15.2-xit2o3iepxvqbyku77lwcugufilztu7t/bin:/sw/summit/cuda/10.1.168/bin:/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.0.1-20190611-cyaenjgora6now2nusxzkfli4mzjnudx/bin:/sw/summit/gcc/6.4.0/bin:/gpfs/alpine/proj-shared/stf011/benjha/rapids_build_0.11/anaconda3/condabin:/usr/bin:/usr/sbin:/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-4.8.5/darshan-runtime-3.1.7-csygoqyym3m3ysoaperhxlhoiluvpa2u/bin:/sw/sources/hpss/bin:/opt/ibm/spectrumcomputing/lsf/10.1/linux3.10-glibc2.17-ppc64le-csm/etc:/opt/ibm/spectrumcomputing/lsf/10.1/linux3.10-glibc2.17-ppc64le-csm/bin:/opt/ibm/csm/bin:/usr/local/bin:/usr/local/sbin:/opt/ibm/flightlog/bin:/opt/ibutils/bin:/opt/ibm/spectrum_mpi/jsm_pmix/bin:/opt/puppetlabs/bin:/usr/lpp/mmfs/bin
LD_LIBRARY_PATH : /sw/summit/cuda/10.1.168/lib64:/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.0.1-20190611-cyaenjgora6now2nusxzkfli4mzjnudx/lib:/sw/summit/gcc/6.4.0/lib64:/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-4.8.5/darshan-runtime-3.1.7-csygoqyym3m3ysoaperhxlhoiluvpa2u/lib:/opt/ibm/spectrumcomputing/lsf/10.1/linux3.10-glibc2.17-ppc64le-csm/lib
NUMBAPRO_NVVM :
NUMBAPRO_LIBDEVICE :
CONDA_PREFIX : /gpfs/alpine/proj-shared/stf011/benjha/rapids_build_0.11/anaconda3/envs/rapids_0.11
PYTHON_PATH :
***conda packages***
/gpfs/alpine/proj-shared/stf011/benjha/rapids_build_0.11/anaconda3/bin/conda
# packages in environment at /gpfs/alpine/proj-shared/stf011/benjha/rapids_build_0.11/anaconda3/envs/rapids_0.11:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main
bokeh 1.4.0 py37_0
boost 1.67.0 py37_4
bzip2 1.0.8 h6eb9509_2 conda-forge
ca-certificates 2019.11.28 hecc5488_0 conda-forge
certifi 2019.11.28 py37_0 conda-forge
click 7.0 py_0 conda-forge
cloudpickle 1.2.2 py_1 conda-forge
cmake-setuptools 0.1.3 pypi_0 pypi
cudf 0.11.0b0+7.g3498c7e.dirty pypi_0 pypi
cython 0.29.14 py37hb209c28_0 conda-forge
cytoolz 0.10.1 py37h6eb9509_0 conda-forge
dask 2.9.0 py_0 conda-forge
dask-core 2.9.0 py_0 conda-forge
dask-cudf 0.11.0b0+7.g3498c7e.dirty pypi_0 pypi
distributed 2.9.0 py_0 conda-forge
double-conversion 3.1.5 hb209c28_2 conda-forge
flatbuffers 1.11.0 hb209c28_0 conda-forge
freetype 2.10.0 hd9140be_1 conda-forge
fsspec 0.6.2 py_0 conda-forge
heapdict 1.0.1 py_0 conda-forge
icu 58.2 hf484d3e_1000 conda-forge
jinja2 2.10.3 py_0 conda-forge
jpeg 9c h14c3975_1001 conda-forge
libblas 3.8.0 14_openblas conda-forge
libboost 1.67.0 h46d08c1_4
libcblas 3.8.0 14_openblas conda-forge
libedit 3.1.20181209 hc058e9b_0
libffi 3.2.1 hf62a594_5
libgcc-ng 8.2.0 h822a55f_1
libgfortran-ng 8.2.0 h822a55f_2 conda-forge
liblapack 3.8.0 14_openblas conda-forge
libopenblas 0.3.7 ha38281c_6 conda-forge
libpng 1.6.37 h151fe60_0 conda-forge
libprotobuf 3.11.2 hb61f777_0 conda-forge
libstdcxx-ng 8.2.0 h822a55f_1
libtiff 4.1.0 h344c884_1 conda-forge
llvmlite 0.31.0dev0 py37hf484d3e_8 numba
locket 0.2.0 py_2 conda-forge
lz4-c 1.8.3 hb209c28_1001 conda-forge
markupsafe 1.1.1 py37h6eb9509_0 conda-forge
msgpack-python 0.6.2 py37h1bb5118_0 conda-forge
ncurses 6.1 he6710b0_1
numba 0.46.0 py37h962f231_0
numpy 1.17.3 py37hcee8f07_0 conda-forge
nvstrings-cudaunknown 0.0.0.dev0 pypi_0 pypi
olefile 0.46 py_0 conda-forge
openssl 1.1.1d h6eb9509_0 conda-forge
packaging 19.2 py_0 conda-forge
pandas 0.25.3 py37ha9bb74f_0 conda-forge
partd 1.1.0 py_0 conda-forge
pillow 6.2.1 py37h0d2faf8_0
pip 19.3.1 py37_0
protobuf 3.11.2 py37hb209c28_0 conda-forge
psutil 5.6.7 py37h6eb9509_0 conda-forge
py-boost 1.67.0 py37h04863e7_4
pyparsing 2.4.5 py_0 conda-forge
python 3.7.5 h4134adf_0
python-dateutil 2.8.1 py_0 conda-forge
pytz 2019.3 py_0 conda-forge
pyyaml 5.2 py37h6eb9509_0 conda-forge
rapidjson 1.1.0 hb209c28_1002 conda-forge
readline 7.0 h7b6447c_5
rmm 0.11.0b0+80.g2e69ec9.dirty pypi_0 pypi
setuptools 42.0.2 py37_0
six 1.13.0 py37_0 conda-forge
sortedcontainers 2.1.0 py_0 conda-forge
sqlite 3.30.1 h7b6447c_0
tbb 2019.8 hfd86e86_0
tblib 1.6.0 py_0 conda-forge
tk 8.6.8 hbc83047_0
toolz 0.10.0 py_0 conda-forge
tornado 6.0.3 py37h6eb9509_0 conda-forge
wheel 0.33.6 py37_0
xz 5.2.4 h14c3975_4
yaml 0.2.2 h6eb9509_1 conda-forge
zict 1.0.0 py_0 conda-forge
zlib 1.2.11 h7b6447c_3
zstd 1.4.4 h738b7fd_1 conda-forge
</pre></details>
Thanks,
Benjamin
benjha
All 6 comments
Hi @benjha libcumlprims is a conda package required that you can install with:
conda install -c rapidsai-nightly libcumlprims
dantegd
on 24 Dec 2019
Thanks Dante,
I am building cuml for Power architecture (ppc64le), so I need to make sure libcumlprims is being generated with the cuml's build script.
So far, the changes I did in cuml's build.sh were to explicitly set BLAS and LAPACK libraries, and turning ON (then OFF) UCX.
benjha
on 24 Dec 2019
After looking at cuml's make, cmakefiles and so on, I didn't find any command that builds libcumlprims, so likely this part of the code is not being distributed in the regular repository.
Any plans to do this ?
Meanwhile, the workaround was to copy the header files from https://anaconda.org/nvidia/libcumlprims/0.11.0/download/linux-64/libcumlprims-0.11.0-cuda10.1_0.tar.bz2 to my conda env. and remove the libcumlprims dependency in cuml's setup.py
Overall this worked, but after running cuml's python tests I got only the next error:
_______________________________________________ ERROR collecting cuml/test/dask/test_nearest_neighbors.py ________________________________________________
ImportError while importing test module '/gpfs/alpine/stf011/proj-shared/benjha/rapids_build_0.11/build/cuml/python/cuml/test/dask/test_nearest_neighbors.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
cuml/test/dask/test_nearest_neighbors.py:33: in <module>
from cuml.neighbors.nearest_neighbors_mg import \
E ImportError: /gpfs/alpine/stf011/proj-shared/benjha/rapids_build_0.11/build/cuml/python/cuml/neighbors/nearest_neighbors_mg.cpython-37m-powerpc64le-linux-gnu.so: undefined symbol: _ZN8MLCommon6Matrix14PartDescriptorC1EmmRKSt6vectorIPNS0_12RankSizePairESaIS4_EEi
Likely _ZN8MLCommon6Matrix14PartDescriptorC1EmmRKSt6vectorIPNS0_12RankSizePairESaIS4_EEi symbol comes from libcumlprims
benjha
on 4 Jan 2020
@benjha sorry for the delayed response, I got sick and then holiday break got in the way.
As you saw with that issue, that workaround is not going to work for building cuML for ppc. libcumlprims is a precompiled library that contains some multinode multigpu functionality that is not part of the cuML repository.
To compile the python package without that functionality, and avoid those symbol problems you ran into, you can use the --singlegpu flag in the setup.py step, something like:
python setup.py build_ext --singlegpu
That used to be the default build path of cuML 2 or 3 versions ago, since then we build with the multrigpu functionality by default, but that option to build without libcumlprims functionality is still there and should work. I just tried and it worked well, but if you run into any issues please let us know to help troubleshooting.
dantegd
on 5 Jan 2020
Dante is exactly right - the libcumlprims package is a separate library compiled internally to NVIDIA and available via conda for x86-64 only at this time unfortunately.
JohnZed
on 10 Jan 2020
Sorry to comment on a closed issue. But is libcumlprims ever likely to be made available on ppc64le?
adamjm
on 11 Jun 2020
Related issues
aerdem4
路
3Comments
yasmina-altair
路
4Comments
MurrayData
路
6Comments
beckernick
路
8Comments
dantegd
路
8Comments
Most helpful comment
Sorry to comment on a closed issue. But is libcumlprims ever likely to be made available on ppc64le?