Cuml: [BUG] Ridiculously huge effort needed for building cuML from source on a non-docker env!
Hi all,
For the past 3 days (on-and-off), I’ve been trying to build cuML from source, outside of docker, and till today I haven’t succeeded in this effort at all!
So far...
I've tried the following approaches. All of which have miserably failed.
conda + conda-cmake, but explicitly specified gcc
In this approach, I was using environmental modules with gcc v5.4.0, ctk v10.1.105, while cmake was installed via conda itself. This initially failed with FAISS complaining unable to find blas. Was able to fix this by installing openblas via conda itself. However, this fails at linker stage saying undefined reference to google::protobuf* messages. Even gcc v7.3.0 had the same issue.
conda + conda-cmake + conda-forge compilers
Same as above setup, but installed the compilers package from conda-forge channel. This now causes FAISS to error out saying unable to find cublas. When I look at config.log I noticed errors like undefined reference to cublasLt*. I could fix this issue locally via adding -lcublasLt option to the commandline, but couldn't figure out how to pass this information from the cmake of cuML!
Expected behavior
Atleast a guide/doc to setup a conflict-free environment for building cuML from source. A build.md is not enough here. Actually documenting a list of concrete steps would be better, IMO.
Environment details (please complete the following information):
- Environment location: Bare-metal
- Linux Distro/Architecture: CentOS 7
- GPU Model/Driver: V100 and driver 418.87
- CUDA: 10.1
- Method of cuDF install: conda
- Method of cuML install: source at HEAD of branch-0.12
- conda: installed via latest Miniconda script
Tagging @JohnZed @dantegd @cjnolet @datametrician
teju85
👍1
All 16 comments
Here's the script I've been using for this experiment: https://gist.github.com/teju85/51ff12886f5ae9967a4cab6fd0376953
Download and run it as env Root=/path/to/install/conda-and-cuml bash rapids-setup installCuml
teju85
on 19 Dec 2019
Seems like this is not an isolated incident. @nikiforov-sm here also seems to be facing issues while trying to build cuML from source.
teju85
on 19 Dec 2019
Hi Thejaswi,
Did you try creating a conda environment using the cuda_dev_cida10.0.yml file like below?
conda env create -n cuml_dev python=3.7 --file=conda/environments/cuml_dev_cuda10.0.yml
If not, I highly recommend using the yml file. After your conda environment is ready, you can use build.sh file. That mostly works for me.
From: Thejaswi Rao notifications@github.com
Reply-To: rapidsai/cuml reply@reply.github.com
Date: Thursday, December 19, 2019 at 12:12 AM
To: rapidsai/cuml cuml@noreply.github.com
Cc: Subscribed subscribed@noreply.github.com
Subject: [rapidsai/cuml] [BUG] Ridiculously huge effort needed for building cuML from source on a non-docker env! (#1503)
Hi all,
For the past 3 days (on-and-off), I’ve been trying to build cuML from source, outside of docker, and till today I haven’t succeeded in this effort at all!
* So far...*
I've tried the following approaches. All of which have miserably failed.
* conda + conda-cmake, but explicitly specified gcc
In this approach, I was using environmental modules with gcc v5.4.0, ctk v10.1.105, while cmake was installed via conda itself. This initially failed with FAISS complaining unable to find blas. Was able to fix this by installing openblas via conda itself. However, this fails at linker stage saying undefined reference to google::protobuf` messages. Even gcc v7.3.0 had the same issue.
* conda + conda-cmake + conda-forge compilers
Same as above setup, but installed the compilers package from conda-forge channel. This now causes FAISS to error out saying unable to find cublas. When I look at config.log I noticed errors like undefined reference to cublasLt. I could fix this issue locally via adding -lcublasLt option to the commandline, but couldn't figure out how to pass this information from the cmake of cuML!
Expected behavior
Atleast a guide/doc to setup a conflict-free environment for building cuML from source. A build.md is not enough here. Actually documenting a list of concrete steps would be better, IMO.
Environment details (please complete the following information):
- Environment location: Bare-metal
- Linux Distro/Architecture: CentOS 7
- GPU Model/Driver: V100 and driver 418.87
- CUDA: 10.1
- Method of cuDF install: conda
- Method of cuML install: source at HEAD of branch-0.12
Tagging @JohnZedhttps://github.com/JohnZed @dantegdhttps://github.com/dantegd @cjnolethttps://github.com/cjnolet @datametricianhttps://github.com/datametrician
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com/rapidsai/cuml/issues/1503?email_source=notifications&email_token=AINLU4IFJUQHJE2ZDS5R3TDQZL7BTA5CNFSM4J43PBIKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IBQ76HA, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AINLU4MQRNYRX6SEFHZ4ZSTQZL7BTANCNFSM4J43PBIA.
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
oyilmaz-nvidia
on 19 Dec 2019
@oyilmaz-nvidia yes, the script I linked in my 2nd comment above uses the conda-env yaml file itself.
teju85
on 19 Dec 2019
@jakirkham also mentioned trying to use the nvcc_linux-64 conda-forge package. Trying it out now...
teju85
on 19 Dec 2019
@teju85 I work exclusively building cuML on bare metal and build it every day using the instructions in the build document (sometimes with multiple new conda environments in a single day) never using docker at all in my development, and other cuML devs also typically don’t use the Dockerfiles or containers. It seems that the devs that you linked in the issue also were able to build cuML after following Vishal’s advice of creating a new conda environment. On the other hand it would be good to identify the particular issue you’re facing since it could be common with conflicts between a system gcc and conda libraries and document it/ solve it, since it could be more typical outside the common Ubuntu/centOS/conda/etc. that we typically use during development.
The issue that might be hiding here and causing issues could be protobuf missing some abi flags or some additional inheritance configuration which is something to look at ASAP. I use Ubuntu typically, I wonder if @cjnolet has seen the protobuf issues since he uses centOS as well (and I believe no Docker, but not 100% sure).
Could you run the print_env.sh script that is in the cuDF repo? That definitely will help to troubleshoot and to repro the issues
dantegd
on 19 Dec 2019
@dantegd here's the output of print_env.sh
Click here to see environment details
**git***
commit 9ee941716474186dc648af99a805b54ca82f2391 (HEAD, origin/branch-0.12, origin/HEAD, branch-0.12)
Merge: e2d40db 510d703
Author: Dante Gama Dessavre <[email protected]>
Date: Wed Dec 18 16:29:59 2019 -0600
Merge pull request #1494 from teju85/fea-ext-ml-prims-bench
[REVIEW] ml-prims benchmark suite
**git submodules***
bf4f2ea0bd1180b34718ac26eb79b170a4f6290e thirdparty/benchmark (v1.5.0-17-gbf4f2ea)
c3cceac115c072fb63df1836ff46d8c60d9eb304 thirdparty/cub (v1.8.0)
cf0301e00f0825ce46e6c18fb9dda5497c2215a6 thirdparty/cutlass (v1.0.1)
9077ec7efe5b652468ab051e93c67589d5cb8f85 thirdparty/cutlass/tools/external/googletest (release-1.8.0-985-g9077ec7)
656368b5eda4d376177a3355673d217fa95000b6 thirdparty/faiss (v1.5.3-3-g656368b)
6ce9b98f541b8bcd84c5c5b3483f29a933c4aefb thirdparty/googletest (release-1.8.0-1040-g6ce9b98)
600afd55d1fa9bb94fc88fd3a3043cb2d5b20651 thirdparty/treelite (0.32-146-g600afd5)
135ab5cf71ed731fc9fa0653051e7d4884a3652f thirdparty/treelite/3rdparty/fmt (4.1.0)
106ffc04be1abf3ff3399f54ccf149815b287dd9 thirdparty/treelite/3rdparty/protobuf (v3.3.1-623-g106ffc0)
360e66c1c4777c99402cf8cd535aa510fee16573 thirdparty/treelite/3rdparty/protobuf/third_party/benchmark (v1.0.0-49-g360e66c)
4d49691f1a9d944c3b0aa5e63f1db3cad1f941f8 thirdparty/treelite/dmlc-core (v0.3-25-g4d49691)
***OS Information***
CentOS Linux release 7.7.1908 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
CentOS Linux release 7.7.1908 (Core)
CentOS Linux release 7.7.1908 (Core)
Linux hsw215 3.10.0-1062.4.1.el7.x86_64 #1 SMP Fri Oct 18 17:15:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
***GPU Information***
Thu Dec 19 08:15:36 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.00 Driver Version: 418.87.00 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-PCIE... On | 00000000:04:00.0 Off | 0 |
| N/A 34C P0 28W / 250W | 0MiB / 16130MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla V100-PCIE... On | 00000000:05:00.0 Off | 0 |
| N/A 35C P0 26W / 250W | 0MiB / 16130MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 Tesla V100-PCIE... On | 00000000:84:00.0 Off | 0 |
| N/A 32C P0 25W / 250W | 0MiB / 16130MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 Tesla V100-PCIE... On | 00000000:85:00.0 Off | 0 |
| N/A 34C P0 24W / 250W | 0MiB / 16130MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
***CPU***
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 64
On-line CPU(s) list: 0-31
Off-line CPU(s) list: 32-63
Thread(s) per core: 1
Core(s) per socket: 16
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 63
Model name: Intel(R) Xeon(R) CPU E5-2698 v3 @ 2.30GHz
Stepping: 2
CPU MHz: 2301.000
CPU max MHz: 2301.0000
CPU min MHz: 1200.0000
BogoMIPS: 4599.94
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 40960K
NUMA node0 CPU(s): 0-15
NUMA node1 CPU(s): 16-31
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd ibrs ibpb tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear
***CMake***
/home/snanditale/conda/envs/cuml_dev/bin/cmake
cmake version 3.14.5
CMake suite maintained and supported by Kitware (kitware.com/cmake).
***g++***
/home/snanditale/conda/envs/cuml_dev/bin/g++
g++ (crosstool-NG 1.23.0.452-d158) 7.3.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
***nvcc***
/cm/extra/apps/CUDA.linux86-64/10.1.150_418.39/bin/nvcc
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Fri_Feb__8_19:08:17_PST_2019
Cuda compilation tools, release 10.1, V10.1.105
***Python***
/home/snanditale/conda/envs/cuml_dev/bin/python
Python 3.7.3
***Environment Variables***
PATH : /home/snanditale/conda/envs/cuml_dev/bin:/home/snanditale/conda/condabin:/home/snanditale/conda/bin:/cm/extra/apps/CUDA.linux86-64/10.1.150_418.39/nvvm/bin:/cm/extra/apps/CUDA.linux86-64/10.1.150_418.39/bin:/home/snanditale/settings/nvidia/scripts:/home/nv/bin:/home/utils/bin:/home/tools/vcs/vcs_latest:/usr/lib:/etc:/usr/bin/X11:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/lsf/bin:/home/snanditale/settings/scripts:/bin:/sbin:/usr/bin:/home/snanditale/settings/nvidia/scripts:/home/nv/bin:/home/utils/bin:/home/tools/vcs/vcs_latest:/usr/lib:/etc:/usr/bin/X11:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/lsf/bin:/home/snanditale/settings/scripts:/bin:/sbin:/usr/bin:/home/snanditale/settings/nvidia/scripts:/home/nv/bin:/home/utils/bin:/home/tools/vcs/vcs_latest:/usr/lib:/etc:/usr/bin/X11:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/lsf/bin:/home/snanditale/settings/scripts:/bin:/sbin:/usr/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/sbin:/usr/sbin:/cm/extra/apps/Modules/3.2.10/bin:.:/home/gnu/bin:.:/home/gnu/bin:.:/home/gnu/bin
LD_LIBRARY_PATH : /cm/extra/apps/CUDA.linux86-64/10.1.150_418.39/nvvm/lib64:/cm/extra/apps/CUDA.linux86-64/10.1.150_418.39/extras/CUPTI/lib64:/cm/extra/apps/CUDA.linux86-64/10.1.150_418.39/lib64:/usr/lib64
NUMBAPRO_NVVM :
NUMBAPRO_LIBDEVICE :
CONDA_PREFIX : /home/snanditale/conda/envs/cuml_dev
PYTHON_PATH :
***conda packages***
/home/snanditale/conda/condabin/conda
# packages in environment at /home/snanditale/conda/envs/cuml_dev:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main conda-forge
arrow-cpp 0.15.0 py37h090bef1_2 conda-forge
attrs 19.3.0 py_0 conda-forge
binutils-meta 1.0.4 0 conda-forge
binutils_impl_linux-64 2.33.1 he1b5a44_7 conda-forge
binutils_linux-64 2.33.1 h9595d00_15 conda-forge
blas 1.0 openblas anaconda
bokeh 1.4.0 py37_0 conda-forge
boost-cpp 1.70.0 h8e57a91_2 conda-forge
brotli 1.0.7 he1b5a44_1000 conda-forge
bzip2 1.0.8 h516909a_2 conda-forge
c-ares 1.15.0 h516909a_1001 conda-forge
c-compiler 1.0.4 h516909a_0 conda-forge
ca-certificates 2019.11.28 hecc5488_0 conda-forge
certifi 2019.11.28 py37_0 conda-forge
click 7.0 py_0 conda-forge
cloudpickle 1.2.2 py_1 conda-forge
cmake 3.14.5 hf94ab9c_0 conda-forge
compilers 1.0.4 0 conda-forge
cudatoolkit 10.1.243 h6bb024c_0 nvidia
cudf 0.12.0b191219 py37_375 rapidsai-nightly
cudnn 7.6.0 cuda10.1_0 nvidia
cupy 6.6.0 py37ha7c4746_1 conda-forge
curl 7.65.3 hf8cf82a_0 conda-forge
cxx-compiler 1.0.4 hc9558a2_0 conda-forge
cython 0.29.14 py37he1b5a44_0 conda-forge
cytoolz 0.10.1 py37h516909a_0 conda-forge
dask 2.9.0+17.gc7ac3a7 pypi_0 pypi
dask-cuda 0.12.0a191219 py37_36 rapidsai-nightly
dask-cudf 0.12.0b191219 py37_375 rapidsai-nightly
distributed 2.9.0+16.g54efd79 pypi_0 pypi
dlpack 0.2 he1b5a44_1 conda-forge
double-conversion 3.1.5 he1b5a44_2 conda-forge
expat 2.2.5 he1b5a44_1004 conda-forge
fastavro 0.22.8 py37h516909a_0 conda-forge
fastrlock 0.4 py37he1b5a44_1000 conda-forge
fortran-compiler 1.0.4 he991be0_0 conda-forge
freetype 2.10.0 he983fc9_1 conda-forge
fsspec 0.6.2 py_0 conda-forge
gcc_impl_linux-64 7.3.0 habb00fd_2 conda-forge
gcc_linux-64 7.3.0 h553295d_15 conda-forge
gflags 2.2.2 he1b5a44_1002 conda-forge
gfortran_impl_linux-64 7.3.0 hdf63c60_2 conda-forge
gfortran_linux-64 7.3.0 h553295d_15 conda-forge
glog 0.4.0 he1b5a44_1 conda-forge
grpc-cpp 1.23.0 h18db393_0 conda-forge
gxx_impl_linux-64 7.3.0 hdf63c60_2 conda-forge
gxx_linux-64 7.3.0 h553295d_15 conda-forge
heapdict 1.0.1 py_0 conda-forge
icu 64.2 he1b5a44_1 conda-forge
importlib_metadata 1.3.0 py37_0 conda-forge
jinja2 2.10.3 py_0 conda-forge
joblib 0.14.1 py_0 conda-forge
jpeg 9c h14c3975_1001 conda-forge
krb5 1.16.4 h2fd8d38_0 conda-forge
ld_impl_linux-64 2.33.1 h53a641e_7 conda-forge
libclang 8.0.0 h6bb024c_0 rapidsai
libcudf 0.12.0b191219 cuda10.1_375 rapidsai-nightly
libcumlprims 0.12.0a191219 cuda10.1_0 rapidsai-nightly
libcurl 7.65.3 hda55be3_0 conda-forge
libedit 3.1.20170329 hf8c457e_1001 conda-forge
libevent 2.1.10 h72c5cf5_0 conda-forge
libffi 3.2.1 he1b5a44_1006 conda-forge
libgcc-ng 9.2.0 hdf63c60_0 conda-forge
libgfortran-ng 7.3.0 hdf63c60_2 conda-forge
libnvstrings 0.12.0b191219 cuda10.1_375 rapidsai-nightly
libopenblas 0.3.6 h5a2b251_2 anaconda
libpng 1.6.37 hed695b0_0 conda-forge
libprotobuf 3.8.0 h8b12597_0 conda-forge
librmm 0.12.0a191219 cuda10.1_72 rapidsai-nightly
libssh2 1.8.2 h22169c7_2 conda-forge
libstdcxx-ng 9.2.0 hdf63c60_0 conda-forge
libtiff 4.1.0 hfc65ed5_0 conda-forge
libuv 1.34.0 h516909a_0 conda-forge
llvmlite 0.29.0 py37hfd453ef_1 conda-forge
locket 0.2.0 py_2 conda-forge
lz4-c 1.8.3 he1b5a44_1001 conda-forge
markupsafe 1.1.1 py37h516909a_0 conda-forge
more-itertools 8.0.2 py_0 conda-forge
msgpack-python 0.6.2 py37hc9558a2_0 conda-forge
nccl 2.4.6.1 cuda10.1_0 nvidia
ncurses 6.1 hf484d3e_1002 conda-forge
nomkl 3.0 0 anaconda
numba 0.45.1 py37hb3f55d8_0 conda-forge
numpy 1.17.4 py37hd5be1e1_0 anaconda
numpy-base 1.17.4 py37h2f8d375_0 anaconda
nvstrings 0.12.0b191219 py37_375 rapidsai-nightly
olefile 0.46 py_0 conda-forge
openblas 0.3.6 2 anaconda
openblas-devel 0.3.6 2 anaconda
openssl 1.1.1d h516909a_0 conda-forge
packaging 19.2 py_0 conda-forge
pandas 0.24.2 py37hb3f55d8_1 conda-forge
parquet-cpp 1.5.1 2 conda-forge
partd 1.0.0 py_0 conda-forge
patsy 0.5.1 py_0 conda-forge
pillow 6.2.1 py37hd70f55b_1 conda-forge
pip 19.3.1 py37_0 conda-forge
pluggy 0.13.0 py37_0 conda-forge
protobuf 3.8.0 py37he1b5a44_2 conda-forge
psutil 5.6.7 py37h516909a_0 conda-forge
py 1.8.0 py_0 conda-forge
pyarrow 0.15.0 py37h8b68381_1 conda-forge
pynvml 8.0.3 py_0 conda-forge
pyparsing 2.4.5 py_0 conda-forge
pytest 5.3.2 py37_0 conda-forge
python 3.7.3 h357f687_2 conda-forge
python-dateutil 2.8.1 py_0 conda-forge
pytz 2019.3 py_0 conda-forge
pyyaml 5.2 py37h516909a_0 conda-forge
re2 2019.12.01 he1b5a44_0 conda-forge
readline 8.0 hf8c457e_0 conda-forge
rhash 1.3.6 h14c3975_1001 conda-forge
rmm 0.12.0a191219 py37_72 rapidsai-nightly
scikit-learn 0.22 py37h22eb022_0 anaconda
scipy 1.3.2 py37he2b7bc3_0 anaconda
setuptools 42.0.2 py37_0 conda-forge
six 1.13.0 py37_0 conda-forge
snappy 1.1.7 he1b5a44_1002 conda-forge
sortedcontainers 2.1.0 py_0 conda-forge
sqlite 3.30.1 hcee41ef_0 conda-forge
statsmodels 0.10.2 py37hc1659b7_0 conda-forge
tblib 1.6.0 py_0 conda-forge
thrift-cpp 0.12.0 hf3afdfd_1004 conda-forge
tk 8.6.10 hed695b0_0 conda-forge
toolz 0.10.0 py_0 conda-forge
tornado 6.0.3 py37h516909a_0 conda-forge
umap-learn 0.3.10 py37_0 conda-forge
uriparser 0.9.3 he1b5a44_1 conda-forge
wcwidth 0.1.7 py_1 conda-forge
wheel 0.33.6 py37_0 conda-forge
xz 5.2.4 h14c3975_1001 conda-forge
yaml 0.2.2 h516909a_1 conda-forge
zict 1.0.0 py_0 conda-forge
zipp 0.6.0 py_0 conda-forge
zlib 1.2.11 h516909a_1006 conda-forge
zstd 1.4.3 h3b9ef0a_0 conda-forge
👍1
@dantegd @cjnolet I didn't see anything suspicious in the above list. Did you guys find any?
teju85
on 20 Dec 2019
@teju85 there are significant differences from your configuration and the typical setup I use (and tend to see with other devs, of course not universal):
- My compilers:
***g++***
/usr/bin/g++
g++ (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
***nvcc***
/usr/local/cuda-10.1u2/bin/nvcc
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243
yours:
***g++***
/home/snanditale/conda/envs/cuml_dev/bin/g++
g++ (crosstool-NG 1.23.0.452-d158) 7.3.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
***nvcc***
/cm/extra/apps/CUDA.linux86-64/10.1.150_418.39/bin/nvcc
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Fri_Feb__8_19:08:17_PST_2019
Cuda compilation tools, release 10.1, V10.1.105
Generally speaking I would recommend system compilers like the first approach you were doing (at least for now), would there be a chance of getting a print of the printenv script before trying to use the conda compilers?
- The
PATHandLD_LIBRARY_PATHvariables are significantly different. I think the former might have been the source of the issue with protobuf since it should've found it in the conda env, but there might be some adjustment needed in the patch for the protobuf configuration of treelite (i.e. this https://github.com/rapidsai/cuml/blob/branch-0.12/cpp/cmake/treelite_protobuf.patch).
Just for reference:
PATH : /usr/local/cuda-10.1u2/bin:/home/dante/miniconda3/envs/ss1219/bin:/home/dante/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
LD_LIBRARY_PATH : /usr/local/cuda/lib64 (I add the conda lib path here as well, it wasn't in my rc when I logged in to the computer)
Did you change yours before trying to use the conda compilers? Just to be sure of what was causing the protobuf issue.
There are significant differences in the conda environments, but its hard to say if those are significant since the compilers and other extra conda packages might have just brought them in and they might be innocuous.
Additionally I already started the revamped build and developer documentation (with a better/concise set of steps!), and will have a version ready early this coming week, will ping you for a review and to be sure to solve this issue asap. I also leave the printenv result of the workstation I use (that is mostly used for RAPIDS development) where I use no Docker whatsoever just in case you want to compare as well
Click here to see environment details
**git***
commit cb089966a4d0bf6251f25b11455c338004bb49a7 (HEAD -> branch-0.12, origin/branch-0.12, origin/HEAD)
Merge: 724e237f2 5380e1128
Author: Trevor Smith <[email protected]>
Date: Thu Dec 19 08:22:45 2019 -0800
Merge pull request #3641 from trevorsm7/fea-remove-duplicate-macros
[REVIEW] Remove duplicate CUDA_DEVICE_CALLABLE definitions
**git submodules***
b165e1fb11eeea64ccf95053e40f2424312599cc thirdparty/cub (v1.7.1)
63f644be44201467e3938d59ed9d89cc8725c35d thirdparty/jitify (remotes/origin/feature/api_v2_v0.10)
39125e0e476b960c2001f1ec76a3441335ff91b2 thirdparty/libcudacxx (0.8.1-94-g39125e0)
08bc464bd8f4d779e4294305aa7dadebcebcc507 thirdparty/libcudacxx/libcxx (remotes/origin/master-6-g08bc464bd)
***OS Information***
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
Linux gv100-1804 5.0.0-37-generic #40~18.04.1-Ubuntu SMP Thu Nov 14 12:06:39 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
***GPU Information***
Sun Dec 22 11:46:24 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.40 Driver Version: 430.40 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Quadro GV100 Off | 00000000:17:00.0 Off | Off |
| 30% 42C P2 26W / 250W | 1MiB / 32508MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Quadro GV100 Off | 00000000:65:00.0 On | Off |
| 30% 43C P0 28W / 250W | 523MiB / 32505MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 1 1555 G /usr/lib/xorg/Xorg 71MiB |
| 1 2070 G /usr/bin/gnome-shell 62MiB |
| 1 3248 G /usr/lib/xorg/Xorg 197MiB |
| 1 3401 G /usr/bin/gnome-shell 182MiB |
| 1 4412 G gnome-control-center 7MiB |
+-----------------------------------------------------------------------------+
***CPU***
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 12
On-line CPU(s) list: 0-11
Thread(s) per core: 2
Core(s) per socket: 6
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Core(TM) i7-7800X CPU @ 3.50GHz
Stepping: 4
CPU MHz: 1200.127
CPU max MHz: 4000.0000
CPU min MHz: 1200.0000
BogoMIPS: 7000.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 8448K
NUMA node0 CPU(s): 0-11
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req md_clear flush_l1d
***CMake***
/home/dante/miniconda3/envs/ss1219/bin/cmake
cmake version 3.14.5
CMake suite maintained and supported by Kitware (kitware.com/cmake).
***g++***
/usr/bin/g++
g++ (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
***nvcc***
/usr/local/cuda-10.1u2/bin/nvcc
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243
***Python***
/home/dante/miniconda3/envs/ss1219/bin/python
Python 3.7.3
***Environment Variables***
PATH : /usr/local/cuda-10.1u2/bin:/home/dante/miniconda3/envs/ss1219/bin:/home/dante/miniconda3/condabin:/usr/local/cuda-10.1u2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
LD_LIBRARY_PATH : /usr/local/cuda/lib64
NUMBAPRO_NVVM :
NUMBAPRO_LIBDEVICE :
CONDA_PREFIX : /home/dante/miniconda3/envs/ss1219
PYTHON_PATH :
***conda packages***
/home/dante/miniconda3/condabin/conda
# packages in environment at /home/dante/miniconda3/envs/ss1219:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main conda-forge
alabaster 0.7.12 py_0 conda-forge
appdirs 1.4.3 py_1 conda-forge
arrow-cpp 0.15.0 py37h090bef1_2 conda-forge
aspy.yaml 1.3.0 py_0 conda-forge
attrs 19.3.0 py_0 conda-forge
babel 2.7.0 py_0 conda-forge
backcall 0.1.0 py_0 conda-forge
black 19.10b0 py37_0 conda-forge
bleach 3.1.0 py_0 conda-forge
bokeh 1.4.0 py37_0 conda-forge
boost-cpp 1.70.0 h8e57a91_2 conda-forge
brotli 1.0.7 he1b5a44_1000 conda-forge
bzip2 1.0.8 h516909a_2 conda-forge
c-ares 1.15.0 h516909a_1001 conda-forge
ca-certificates 2019.11.28 hecc5488_0 conda-forge
cached-property 1.5.1 py_0 conda-forge
certifi 2019.11.28 py37_0 conda-forge
cffi 1.13.2 py37h8022711_0 conda-forge
cfgv 2.0.1 py_0 conda-forge
chardet 3.0.4 py37_1003 conda-forge
click 7.0 py_0 conda-forge
cloudpickle 1.2.2 py_1 conda-forge
cmake 3.14.5 hf94ab9c_0 conda-forge
cmake_setuptools 0.1.3 py_0 rapidsai-nightly
commonmark 0.9.1 py_0 conda-forge
cryptography 2.8 py37h72c5cf5_1 conda-forge
cudatoolkit 10.1.243 h6bb024c_0 nvidia
cudf 0.12.0b0+379.gcb089966a pypi_0 pypi
cudnn 7.6.0 cuda10.1_0 nvidia
cuml 0.12.0a0+285.g1ed5475a.dirty pypi_0 pypi
cupy 7.0.0 py37ha7c4746_0 conda-forge
curl 7.65.3 hf8cf82a_0 conda-forge
cython 0.29.14 py37he1b5a44_0 conda-forge
cytoolz 0.10.1 py37h516909a_0 conda-forge
dask 2.9.0+19.gc2566eaf pypi_0 pypi
dask-core 2.8.1 py_0 conda-forge
dask-cuda 0.12.0a191219 py37_36 rapidsai-nightly
dask-cudf 0.12.0b0+379.gcb089966a pypi_0 pypi
decorator 4.4.1 py_0 conda-forge
defusedxml 0.6.0 py_0 conda-forge
distributed 2.9.0+16.g54efd79f pypi_0 pypi
dlpack 0.2 he1b5a44_1 conda-forge
docutils 0.15.2 py37_0 conda-forge
double-conversion 3.1.5 he1b5a44_2 conda-forge
editdistance 0.5.3 py37he1b5a44_0 conda-forge
entrypoints 0.3 py37_1000 conda-forge
expat 2.2.5 he1b5a44_1004 conda-forge
fastavro 0.22.8 py37h516909a_0 conda-forge
fastrlock 0.4 py37he1b5a44_1000 conda-forge
flake8 3.7.9 py37_0 conda-forge
flatbuffers 1.11.0 he1b5a44_0 conda-forge
freetype 2.10.0 he983fc9_1 conda-forge
fsspec 0.6.2 py_0 conda-forge
future 0.18.2 py37_0 conda-forge
gflags 2.2.2 he1b5a44_1002 conda-forge
glog 0.4.0 he1b5a44_1 conda-forge
gmp 6.1.2 hf484d3e_1000 conda-forge
grpc-cpp 1.23.0 h18db393_0 conda-forge
heapdict 1.0.1 py_0 conda-forge
hypothesis 4.55.4 py37_0 conda-forge
icu 64.2 he1b5a44_1 conda-forge
identify 1.4.8 py_0 conda-forge
idna 2.8 py37_1000 conda-forge
imagesize 1.1.0 py_0 conda-forge
importlib_metadata 1.3.0 py37_0 conda-forge
ipykernel 5.1.3 py37h5ca1d4c_0 conda-forge
ipython 7.10.1 py37h5ca1d4c_0 conda-forge
ipython_genutils 0.2.0 py_1 conda-forge
isort 4.3.21 py37_0 conda-forge
jedi 0.15.1 py37_0 conda-forge
jinja2 2.10.3 py_0 conda-forge
joblib 0.14.1 py_0 conda-forge
jpeg 9c h14c3975_1001 conda-forge
jsonschema 3.2.0 py37_0 conda-forge
jupyter_client 5.3.3 py37_1 conda-forge
jupyter_core 4.6.1 py37_0 conda-forge
krb5 1.16.4 h2fd8d38_0 conda-forge
ld_impl_linux-64 2.33.1 h53a641e_7 conda-forge
libblas 3.8.0 14_openblas conda-forge
libcblas 3.8.0 14_openblas conda-forge
libclang 8.0.0 h6bb024c_0 rapidsai
libcumlprims 0.12.0a191219 cuda10.1_0 rapidsai-nightly
libcurl 7.65.3 hda55be3_0 conda-forge
libedit 3.1.20170329 hf8c457e_1001 conda-forge
libevent 2.1.10 h72c5cf5_0 conda-forge
libffi 3.2.1 he1b5a44_1006 conda-forge
libgcc-ng 9.2.0 hdf63c60_0 conda-forge
libgfortran-ng 7.3.0 hdf63c60_2 conda-forge
liblapack 3.8.0 14_openblas conda-forge
libllvm8 8.0.1 hc9558a2_0 conda-forge
libopenblas 0.3.7 h5ec1e0e_5 conda-forge
libpng 1.6.37 hed695b0_0 conda-forge
libprotobuf 3.8.0 h8b12597_0 conda-forge
librmm 0.11.0 cuda10.1_0 rapidsai
libsodium 1.0.17 h516909a_0 conda-forge
libssh2 1.8.2 h22169c7_2 conda-forge
libstdcxx-ng 9.2.0 hdf63c60_0 conda-forge
libtiff 4.1.0 hfc65ed5_0 conda-forge
libuv 1.34.0 h516909a_0 conda-forge
llvmlite 0.29.0 py37hfd453ef_1 conda-forge
locket 0.2.0 py_2 conda-forge
lz4-c 1.8.3 he1b5a44_1001 conda-forge
markdown 3.0.1 pypi_0 pypi
markupsafe 1.1.1 py37h516909a_0 conda-forge
mccabe 0.6.1 py_1 conda-forge
mistune 0.8.4 py37h516909a_1000 conda-forge
more-itertools 8.0.2 py_0 conda-forge
msgpack-python 0.6.2 py37hc9558a2_0 conda-forge
mypy_extensions 0.4.3 py37_0 conda-forge
nbconvert 5.6.1 py37_0 conda-forge
nbformat 4.4.0 py_1 conda-forge
nbsphinx 0.5.0 py_0 conda-forge
nccl 2.4.6.1 cuda10.1_0 nvidia
ncurses 6.1 hf484d3e_1002 conda-forge
nodeenv 1.3.3 py_0 conda-forge
notebook 6.0.1 py37_0 conda-forge
numba 0.45.1 py37hb3f55d8_0 conda-forge
numpy 1.17.3 py37h95a1406_0 conda-forge
numpydoc 0.9.1 py_0 conda-forge
nvstrings-cudaunknown 0.0.0.dev0 pypi_0 pypi
olefile 0.46 py_0 conda-forge
openssl 1.1.1d h516909a_0 conda-forge
packaging 19.2 py_0 conda-forge
pandas 0.24.2 py37hb3f55d8_1 conda-forge
pandoc 1.19.2 0 conda-forge
pandocfilters 1.4.2 py_1 conda-forge
parquet-cpp 1.5.1 2 conda-forge
parso 0.5.2 py_0 conda-forge
partd 1.0.0 py_0 conda-forge
pathspec 0.6.0 py_0 conda-forge
patsy 0.5.1 py_0 conda-forge
pexpect 4.7.0 py37_0 conda-forge
pickleshare 0.7.5 py37_1000 conda-forge
pillow 6.2.1 py37hd70f55b_1 conda-forge
pip 19.3.1 py37_0 conda-forge
pluggy 0.13.0 py37_0 conda-forge
pre_commit 1.18.1 py37_0 conda-forge
prometheus_client 0.7.1 py_0 conda-forge
prompt_toolkit 3.0.2 py_0 conda-forge
protobuf 3.8.0 py37he1b5a44_2 conda-forge
psutil 5.6.7 py37h516909a_0 conda-forge
ptyprocess 0.6.0 py_1001 conda-forge
py 1.8.0 py_0 conda-forge
pyarrow 0.15.0 py37h8b68381_1 conda-forge
pycodestyle 2.5.0 py_0 conda-forge
pycparser 2.19 py37_1 conda-forge
pyflakes 2.1.1 py_0 conda-forge
pygments 2.5.2 py_0 conda-forge
pynvml 8.0.3 py_0 conda-forge
pyopenssl 19.1.0 py37_0 conda-forge
pyparsing 2.4.5 py_0 conda-forge
pyrsistent 0.15.6 py37h516909a_0 conda-forge
pysocks 1.7.1 py37_0 conda-forge
pytest 5.3.2 py37_0 conda-forge
python 3.7.3 h357f687_2 conda-forge
python-dateutil 2.8.1 py_0 conda-forge
pytz 2019.3 py_0 conda-forge
pyyaml 5.2 py37h516909a_0 conda-forge
pyzmq 18.1.1 py37h1768529_0 conda-forge
rapidjson 1.1.0 he1b5a44_1002 conda-forge
re2 2019.12.01 he1b5a44_0 conda-forge
readline 8.0 hf8c457e_0 conda-forge
recommonmark 0.6.0 py_0 conda-forge
regex 2019.12.19 py37h516909a_0 conda-forge
requests 2.22.0 py37_1 conda-forge
rhash 1.3.6 h14c3975_1001 conda-forge
rmm 0.11.0 py37_0 rapidsai
scikit-learn 0.22 py37hcdab131_1 conda-forge
scipy 1.4.0 py37h921218d_0 conda-forge
send2trash 1.5.0 py_0 conda-forge
setuptools 42.0.2 py37_0 conda-forge
six 1.13.0 py37_0 conda-forge
snappy 1.1.7 he1b5a44_1002 conda-forge
snowballstemmer 2.0.0 py_0 conda-forge
sortedcontainers 2.1.0 py_0 conda-forge
sphinx 2.3.0 py_0 conda-forge
sphinx-markdown-tables 0.0.10 pypi_0 pypi
sphinx_rtd_theme 0.4.3 py_0 conda-forge
sphinxcontrib-applehelp 1.0.1 py_0 conda-forge
sphinxcontrib-devhelp 1.0.1 py_0 conda-forge
sphinxcontrib-htmlhelp 1.0.2 py_0 conda-forge
sphinxcontrib-jsmath 1.0.1 py_0 conda-forge
sphinxcontrib-qthelp 1.0.2 py_0 conda-forge
sphinxcontrib-serializinghtml 1.1.3 py_0 conda-forge
sphinxcontrib-websupport 1.1.2 py_0 conda-forge
sqlite 3.30.1 hcee41ef_0 conda-forge
statsmodels 0.10.2 py37hc1659b7_0 conda-forge
streamz 0.5.2 pypi_0 pypi
tblib 1.6.0 py_0 conda-forge
terminado 0.8.3 py37_0 conda-forge
testpath 0.4.4 py_0 conda-forge
thrift-cpp 0.12.0 hf3afdfd_1004 conda-forge
tk 8.6.10 hed695b0_0 conda-forge
toml 0.10.0 py_0 conda-forge
toolz 0.10.0 py_0 conda-forge
tornado 6.0.3 py37h516909a_0 conda-forge
traitlets 4.3.3 py37_0 conda-forge
typed-ast 1.4.0 py37h516909a_0 conda-forge
typing_extensions 3.7.4.1 py37_0 conda-forge
umap-learn 0.3.10 py37_0 conda-forge
uriparser 0.9.3 he1b5a44_1 conda-forge
urllib3 1.25.7 py37_0 conda-forge
virtualenv 16.7.5 py_0 conda-forge
wcwidth 0.1.7 py_1 conda-forge
webencodings 0.5.1 py_1 conda-forge
wheel 0.33.6 py37_0 conda-forge
xz 5.2.4 h14c3975_1001 conda-forge
yaml 0.2.2 h516909a_1 conda-forge
zeromq 4.3.2 he1b5a44_2 conda-forge
zict 1.0.0 py_0 conda-forge
zipp 0.6.0 py_0 conda-forge
zlib 1.2.11 h516909a_1006 conda-forge
zstd 1.4.3 h3b9ef0a_0 conda-forge
@dantegd I cannot use the default system-provided compiler on this cluster since it is super-old!
$ which gcc
/usr/bin/gcc
$ gcc --version | grep GCC
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)
$ cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)
However, on this system, there's env-modules with gcc-7.3.0 available. That's something I can use. Let me use this instead of conda-compiler and give you all the details you asked for above.
teju85
on 23 Dec 2019
Curiosity question: @dantegd
If we rely on conda for everything, why don't we recommend picking up compilers from conda itself? Personally, I feel very uneasy installing openblas, cudatoolkit, boost etc via conda, but then use system-gcc to compile the code.
teju85
on 23 Dec 2019
Curiosity question: @dantegd
If we rely on conda for everything, why don't we recommend picking up compilers from conda itself? Personally, I feel very uneasy installing openblas, cudatoolkit, boost etc via conda, but then use system-gcc to compile the code.
@teju85 Just jumping in here. The struggle has been that it's a bit difficult to integrate nvcc into the conda ecosystem and have it use the conda shipped host compilers reliably and easily. @jakirkham has been spearheading a lot of work in this space and I think we recently have come to a solution that is ready to have the tires kicked across the RAPIDS libraries.
kkraus14
on 23 Dec 2019
👍1
I see. Thanks @kkraus14. On specifically the nvcc part, I had a brief chat with @jakirkham on this one and the last time I tried to use nvcc_linux-64 conda package, I think it still expected one to install cuda toolkit (due to licensing restrictions) separately.
teju85
on 23 Dec 2019
oh.. BTW, let me also mention that when I run cmake inside cuML, I always get the following warning from it:
runtime library [libcublas.so.10] in /cm/extra/apps/CUDA.linux86-64/10.1.150_418.39/lib64 may be hidden by files in:
/home/snanditale/conda/envs/cuml_dev/lib
runtime library [libcurand.so.10] in /cm/extra/apps/CUDA.linux86-64/10.1.150_418.39/lib64 may be hidden by files in:
/home/snanditale/conda/envs/cuml_dev/lib
runtime library [libcusolver.so.10] in /cm/extra/apps/CUDA.linux86-64/10.1.150_418.39/lib64 may be hidden by files in:
/home/snanditale/conda/envs/cuml_dev/lib
runtime library [libcudart.so.10.1] in /cm/extra/apps/CUDA.linux86-64/10.1.150_418.39/lib64 may be hidden by files in:
/home/snanditale/conda/envs/cuml_dev/lib
runtime library [libcusparse.so.10] in /cm/extra/apps/CUDA.linux86-64/10.1.150_418.39/lib64 may be hidden by files in:
/home/snanditale/conda/envs/cuml_dev/lib
runtime library [libnvgraph.so.10] in /cm/extra/apps/CUDA.linux86-64/10.1.150_418.39/lib64 may be hidden by files in:
/home/snanditale/conda/envs/cuml_dev/lib
runtime library [libgomp.so.1] in /cm/extra/apps/GCC/7.3.0/GCC-4.8.5_GMP-6.1.2_MPFR-4.0.1_MPC-1.1.0/lib64 may be hidden by files in:
/home/snanditale/conda/envs/cuml_dev/lib
which is probably because of conflict between cudatoolkit in conda and the cuda installation in the system.
teju85
on 23 Dec 2019
@dantegd I still get the protobuf linker errors, even with the above mentioned system-gcc.
Here's my print_env with system-gcc.
Click here to see environment details
**git***
commit 746cf9e8aaee020343159f0129346c3938025a47 (HEAD, origin/branch-0.12, origin/HEAD, branch-0.12)
Merge: 928fdb8 7317379
Author: Corey J. Nolet <[email protected]>
Date: Fri Dec 20 17:17:23 2019 -0500
Merge pull request #1512 from cjnolet/bug-ext-small_benchmark_fix
[REVIEW] Fixing small bug in SpeedupComparisonRunner and adding test
**git submodules***
bf4f2ea0bd1180b34718ac26eb79b170a4f6290e thirdparty/benchmark (v1.5.0-17-gbf4f2ea)
c3cceac115c072fb63df1836ff46d8c60d9eb304 thirdparty/cub (v1.8.0)
cf0301e00f0825ce46e6c18fb9dda5497c2215a6 thirdparty/cutlass (v1.0.1)
9077ec7efe5b652468ab051e93c67589d5cb8f85 thirdparty/cutlass/tools/external/googletest (release-1.8.0-985-g9077ec7)
656368b5eda4d376177a3355673d217fa95000b6 thirdparty/faiss (v1.5.3-3-g656368b)
6ce9b98f541b8bcd84c5c5b3483f29a933c4aefb thirdparty/googletest (release-1.8.0-1040-g6ce9b98)
600afd55d1fa9bb94fc88fd3a3043cb2d5b20651 thirdparty/treelite (0.32-146-g600afd5)
135ab5cf71ed731fc9fa0653051e7d4884a3652f thirdparty/treelite/3rdparty/fmt (4.1.0)
106ffc04be1abf3ff3399f54ccf149815b287dd9 thirdparty/treelite/3rdparty/protobuf (v3.3.1-623-g106ffc0)
360e66c1c4777c99402cf8cd535aa510fee16573 thirdparty/treelite/3rdparty/protobuf/third_party/benchmark (v1.0.0-49-g360e66c)
4d49691f1a9d944c3b0aa5e63f1db3cad1f941f8 thirdparty/treelite/dmlc-core (v0.3-25-g4d49691)
***OS Information***
CentOS Linux release 7.7.1908 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
CentOS Linux release 7.7.1908 (Core)
CentOS Linux release 7.7.1908 (Core)
Linux hsw215 3.10.0-1062.4.1.el7.x86_64 #1 SMP Fri Oct 18 17:15:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
***GPU Information***
Sun Dec 22 21:57:56 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.00 Driver Version: 418.87.00 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-PCIE... On | 00000000:04:00.0 Off | 0 |
| N/A 34C P0 28W / 250W | 0MiB / 16130MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla V100-PCIE... On | 00000000:05:00.0 Off | 0 |
| N/A 35C P0 26W / 250W | 0MiB / 16130MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 Tesla V100-PCIE... On | 00000000:84:00.0 Off | 0 |
| N/A 32C P0 25W / 250W | 0MiB / 16130MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 Tesla V100-PCIE... On | 00000000:85:00.0 Off | 0 |
| N/A 33C P0 24W / 250W | 0MiB / 16130MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
***CPU***
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 64
On-line CPU(s) list: 0-31
Off-line CPU(s) list: 32-63
Thread(s) per core: 1
Core(s) per socket: 16
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 63
Model name: Intel(R) Xeon(R) CPU E5-2698 v3 @ 2.30GHz
Stepping: 2
CPU MHz: 2301.000
CPU max MHz: 2301.0000
CPU min MHz: 1200.0000
BogoMIPS: 4599.94
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 40960K
NUMA node0 CPU(s): 0-15
NUMA node1 CPU(s): 16-31
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd ibrs ibpb tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear
***CMake***
/home/snanditale/conda/envs/cuml_dev/bin/cmake
cmake version 3.14.5
CMake suite maintained and supported by Kitware (kitware.com/cmake).
***g++***
/cm/extra/apps/GCC/7.3.0/GCC-4.8.5_GMP-6.1.2_MPFR-4.0.1_MPC-1.1.0/bin/g++
g++ (GCC) 7.3.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
***nvcc***
/cm/extra/apps/CUDA.linux86-64/10.1.150_418.39/bin/nvcc
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Fri_Feb__8_19:08:17_PST_2019
Cuda compilation tools, release 10.1, V10.1.105
***Python***
/home/snanditale/conda/envs/cuml_dev/bin/python
Python 3.7.3
***Environment Variables***
PATH : /home/snanditale/conda/envs/cuml_dev/bin:/home/snanditale/conda/condabin:/home/snanditale/conda/bin:/cm/extra/apps/CUDA.linux86-64/10.1.150_418.39/nvvm/bin:/cm/extra/apps/CUDA.linux86-64/10.1.150_418.39/bin:/cm/extra/apps/GCC/7.3.0/GCC-4.8.5_GMP-6.1.2_MPFR-4.0.1_MPC-1.1.0/bin:/home/snanditale/settings/nvidia/scripts:/home/nv/bin:/home/utils/bin:/home/tools/vcs/vcs_latest:/usr/lib:/etc:/usr/bin/X11:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/lsf/bin:/home/snanditale/settings/scripts:/bin:/sbin:/usr/bin:/home/snanditale/settings/nvidia/scripts:/home/nv/bin:/home/utils/bin:/home/tools/vcs/vcs_latest:/usr/lib:/etc:/usr/bin/X11:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/lsf/bin:/home/snanditale/settings/scripts:/bin:/sbin:/usr/bin:/home/snanditale/settings/nvidia/scripts:/home/nv/bin:/home/utils/bin:/home/tools/vcs/vcs_latest:/usr/lib:/etc:/usr/bin/X11:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/lsf/bin:/home/snanditale/settings/scripts:/bin:/sbin:/usr/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/sbin:/usr/sbin:/cm/extra/apps/Modules/3.2.10/bin:.:/home/gnu/bin:.:/home/gnu/bin:.:/home/gnu/bin
LD_LIBRARY_PATH : /home/snanditale/conda/envs/cuml_dev/lib:/cm/extra/apps/CUDA.linux86-64/10.1.150_418.39/nvvm/lib64:/cm/extra/apps/CUDA.linux86-64/10.1.150_418.39/extras/CUPTI/lib64:/cm/extra/apps/CUDA.linux86-64/10.1.150_418.39/lib64:/cm/extra/apps/GCC/7.3.0/GCC-4.8.5_GMP-6.1.2_MPFR-4.0.1_MPC-1.1.0/lib64:/cm/extra/apps/GCC/7.3.0/GCC-4.8.5_GMP-6.1.2_MPFR-4.0.1_MPC-1.1.0/lib:/usr/lib64
NUMBAPRO_NVVM :
NUMBAPRO_LIBDEVICE :
CONDA_PREFIX : /home/snanditale/conda/envs/cuml_dev
PYTHON_PATH :
***conda packages***
/home/snanditale/conda/condabin/conda
# packages in environment at /home/snanditale/conda/envs/cuml_dev:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main conda-forge
arrow-cpp 0.15.0 py37h090bef1_2 conda-forge
attrs 19.3.0 py_0 conda-forge
bokeh 1.4.0 py37_0 conda-forge
boost-cpp 1.70.0 h8e57a91_2 conda-forge
brotli 1.0.7 he1b5a44_1000 conda-forge
bzip2 1.0.8 h516909a_2 conda-forge
c-ares 1.15.0 h516909a_1001 conda-forge
ca-certificates 2019.11.28 hecc5488_0 conda-forge
certifi 2019.11.28 py37_0 conda-forge
click 7.0 py_0 conda-forge
cloudpickle 1.2.2 py_1 conda-forge
cmake 3.14.5 hf94ab9c_0 conda-forge
cudatoolkit 10.1.243 h6bb024c_0 nvidia
cudf 0.12.0b191222 py37_501 rapidsai-nightly
cudnn 7.6.0 cuda10.1_0 nvidia
cupy 6.6.0 py37ha7c4746_1 conda-forge
curl 7.65.3 hf8cf82a_0 conda-forge
cython 0.29.14 py37he1b5a44_0 conda-forge
cytoolz 0.10.1 py37h516909a_0 conda-forge
dask 2.9.0+22.gc474e48 pypi_0 pypi
dask-cuda 0.12.0a191222 py37_40 rapidsai-nightly
dask-cudf 0.12.0b191222 py37_501 rapidsai-nightly
distributed 2.9.0+17.g94f0219 pypi_0 pypi
dlpack 0.2 he1b5a44_1 conda-forge
double-conversion 3.1.5 he1b5a44_2 conda-forge
expat 2.2.5 he1b5a44_1004 conda-forge
fastavro 0.22.8 py37h516909a_0 conda-forge
fastrlock 0.4 py37he1b5a44_1000 conda-forge
freetype 2.10.0 he983fc9_1 conda-forge
fsspec 0.6.2 py_0 conda-forge
gflags 2.2.2 he1b5a44_1002 conda-forge
glog 0.4.0 he1b5a44_1 conda-forge
grpc-cpp 1.23.0 h18db393_0 conda-forge
heapdict 1.0.1 py_0 conda-forge
icu 64.2 he1b5a44_1 conda-forge
importlib_metadata 1.3.0 py37_0 conda-forge
jinja2 2.10.3 py_0 conda-forge
joblib 0.14.1 py_0 conda-forge
jpeg 9c h14c3975_1001 conda-forge
krb5 1.16.4 h2fd8d38_0 conda-forge
ld_impl_linux-64 2.33.1 h53a641e_7 conda-forge
libblas 3.8.0 14_openblas conda-forge
libcblas 3.8.0 14_openblas conda-forge
libclang 8.0.0 h6bb024c_0 rapidsai
libcudf 0.12.0b191222 cuda10.1_501 rapidsai-nightly
libcumlprims 0.12.0a191222 cuda10.1_0 rapidsai-nightly
libcurl 7.65.3 hda55be3_0 conda-forge
libedit 3.1.20170329 hf8c457e_1001 conda-forge
libevent 2.1.10 h72c5cf5_0 conda-forge
libffi 3.2.1 he1b5a44_1006 conda-forge
libgcc-ng 9.2.0 hdf63c60_0 conda-forge
libgfortran-ng 7.3.0 hdf63c60_2 conda-forge
liblapack 3.8.0 14_openblas conda-forge
libnvstrings 0.12.0b191222 cuda10.1_501 rapidsai-nightly
libopenblas 0.3.7 h5ec1e0e_6 conda-forge
libpng 1.6.37 hed695b0_0 conda-forge
libprotobuf 3.8.0 h8b12597_0 conda-forge
librmm 0.12.0a191222 cuda10.1_80 rapidsai-nightly
libssh2 1.8.2 h22169c7_2 conda-forge
libstdcxx-ng 9.2.0 hdf63c60_0 conda-forge
libtiff 4.1.0 hfc65ed5_0 conda-forge
libuv 1.34.0 h516909a_0 conda-forge
llvmlite 0.29.0 py37hfd453ef_1 conda-forge
locket 0.2.0 py_2 conda-forge
lz4-c 1.8.3 he1b5a44_1001 conda-forge
markupsafe 1.1.1 py37h516909a_0 conda-forge
more-itertools 8.0.2 py_0 conda-forge
msgpack-python 0.6.2 py37hc9558a2_0 conda-forge
nccl 2.4.6.1 cuda10.1_0 nvidia
ncurses 6.1 hf484d3e_1002 conda-forge
numba 0.45.1 py37hb3f55d8_0 conda-forge
numpy 1.17.3 py37h95a1406_0 conda-forge
nvstrings 0.12.0b191222 py37_501 rapidsai-nightly
olefile 0.46 py_0 conda-forge
openssl 1.1.1d h516909a_0 conda-forge
packaging 19.2 py_0 conda-forge
pandas 0.24.2 py37hb3f55d8_1 conda-forge
parquet-cpp 1.5.1 2 conda-forge
partd 1.1.0 py_0 conda-forge
patsy 0.5.1 py_0 conda-forge
pillow 6.2.1 py37hd70f55b_1 conda-forge
pip 19.3.1 py37_0 conda-forge
pluggy 0.13.0 py37_0 conda-forge
protobuf 3.8.0 py37he1b5a44_2 conda-forge
psutil 5.6.7 py37h516909a_0 conda-forge
py 1.8.0 py_0 conda-forge
pyarrow 0.15.0 py37h8b68381_1 conda-forge
pynvml 8.0.3 py_0 conda-forge
pyparsing 2.4.5 py_0 conda-forge
pytest 5.3.2 py37_0 conda-forge
python 3.7.3 h357f687_2 conda-forge
python-dateutil 2.8.1 py_0 conda-forge
pytz 2019.3 py_0 conda-forge
pyyaml 5.2 py37h516909a_0 conda-forge
re2 2019.12.01 he1b5a44_0 conda-forge
readline 8.0 hf8c457e_0 conda-forge
rhash 1.3.6 h14c3975_1001 conda-forge
rmm 0.12.0a191222 py37_80 rapidsai-nightly
scikit-learn 0.22 py37hcdab131_1 conda-forge
scipy 1.4.0 py37h921218d_0 conda-forge
setuptools 42.0.2 py37_0 conda-forge
six 1.13.0 py37_0 conda-forge
snappy 1.1.7 he1b5a44_1002 conda-forge
sortedcontainers 2.1.0 py_0 conda-forge
sqlite 3.30.1 hcee41ef_0 conda-forge
statsmodels 0.10.2 py37hc1659b7_0 conda-forge
tblib 1.6.0 py_0 conda-forge
thrift-cpp 0.12.0 hf3afdfd_1004 conda-forge
tk 8.6.10 hed695b0_0 conda-forge
toolz 0.10.0 py_0 conda-forge
tornado 6.0.3 py37h516909a_0 conda-forge
umap-learn 0.3.10 py37_0 conda-forge
uriparser 0.9.3 he1b5a44_1 conda-forge
wcwidth 0.1.7 py_1 conda-forge
wheel 0.33.6 py37_0 conda-forge
xz 5.2.4 h14c3975_1001 conda-forge
yaml 0.2.2 h516909a_1 conda-forge
zict 1.0.0 py_0 conda-forge
zipp 0.6.0 py_0 conda-forge
zlib 1.2.11 h516909a_1006 conda-forge
zstd 1.4.3 h3b9ef0a_0 conda-forge
Hi all,
I spent about a week to build cuML without conda, docker, etc... Had a lot of bugs, but all of them were linker missing some libs. To fix protobuf bug mentioned above, I modified CMakeFiles/cuml++.dir/link.txt file and added -lprotobuf at the end.
Another common bug related to python packages is missing declaration of cuda, cudart in setup.py
For example, I modified setup.py this way to fix segfaul when I import cuml in python.
libs = ['cuda', 'cudart', 'cuml++', 'rmm']
library_dirs=[get_python_lib(), libcuml_path, "/usr/local/cuda-10.2/lib64/"]
cataclysmus
on 23 Jun 2020
👍1
Was this page helpful?
0 / 5 - 0 ratings
Related issues
aerdem4
·
3Comments
teju85
·
7Comments
beckernick
·
8Comments
miguelusque
·
8Comments
wphicks
·
8Comments