Dask-cuda: [BUG] Difference between MNMG and SNMG.
Background
We are running into an NCCL error https://github.com/dmlc/xgboost/issues/7019 that's only reproducible by running 2 dask-cuda-worker on 2 different GPUs. These 2 GPUs can be on the same node or on different nodes. In my testing case, it's 1080TI + 1080 attached to the same node.
Basically, launching all GPUs with one single dask-cuda-worker works:
dask-cuda-worker --scheduler-file="sched.json"
Similarly, using LocalCUDACluster also works fine. However, launching 2 GPUs separately causes an NCCL initialization failure:
CUDA_VISIBLE_DEVICES=0 dask-cuda-worker --scheduler-file="sched.json"
CUDA_VISIBLE_DEVICES=1 dask-cuda-worker --scheduler-file="sched.json"
The error message from cuda-memcheck starts with (I used cuda-memcheck to run the worker):
========= Program hit cudaErrorPeerAccessUnsupported (error 217) due to "peer access is not supported between these two devices" on CUDA API call to cudaIpcOpenMemHandle.
The error is not reproducible on some regular clusters like 2x GTX8000 with nvlink nor 2x v100 (tested with both SNMG and MNMG). Maybe @pseudotensor can provide more context here.
Version
dask-cuda version: dask-cuda=0.19.0=py37_0
Test
Since this is reproduced by checking nccl initialization, using XGBoost to run the test seems simple for confirming the error since it already has all the pipelines lined up.
- On a system with 2 GPUs:
dask-scheduler --scheduler-file="sched.json"
CUDA_VISIBLE_DEVICES=0 cuda-memcheck dask-cuda-worker --scheduler-file="sched.json"
CUDA_VISIBLE_DEVICES=1 cuda-memcheck dask-cuda-worker --scheduler-file="sched.json"
You can optionally use cuda-memcheck to see the root CUDA error, without it the error would be nccl complaining not handled system error.
- Then run this script
import xgboost as xgb
from dask_cuda import LocalCUDACluster
from distributed import Client
from dask import dataframe as dd
from sklearn.datasets import load_digits
import dask_cudf
def test_empty_dataset(client):
X_, y_ = load_digits(n_class=2, return_X_y=True)
chunksize = X_.shape[0] // 10
X = dd.from_array(X_, chunksize=chunksize)
y = dd.from_array(y_, chunksize=chunksize)
clf = xgb.dask.DaskXGBClassifier(tree_method="gpu_hist")
clf.client = client
valid_X = dd.from_array(X_, chunksize=chunksize).repartition(npartitions=1)
valid_y = dd.from_array(y_, chunksize=chunksize).repartition(npartitions=1)
X = dask_cudf.from_dask_dataframe(X)
y = dask_cudf.from_dask_dataframe(y)
valid_X = dask_cudf.from_dask_dataframe(valid_X)
valid_y = dask_cudf.from_dask_dataframe(valid_y)
clf.fit(X, y)
def snmg():
with LocalCUDACluster() as cluster:
with Client(cluster) as client:
test_empty_dataset(client)
def mnmg():
with Client(scheduler_file="sched.json") as client:
test_empty_dataset(client)
if __name__ == "__main__":
# snmg()
mnmg()
trivialfis
All 22 comments
We don't test xgboost with dask-cuda currently, so we're possibly limited to what extent we can help.
However, the cudaIpcOpenMemHandle is something we've also encountered with UCX. This is because the same memory handle is attempted to be opened twice. This is something that CUDA 11.2 fixed, and UCX has gone through the work to fix that too, independent of CUDA version. Could you confirm what CUDA version you're running, and try with CUDA 11.2 if you're not doing so already?
pentschev
on 4 Jun 2021
I have tried both CUDA 11.2 and 11.0
We don't test xgboost with dask-cuda currently,
It shouldn't be XGBoost specific, it's an NCCL error and XGBoost happens to use NCCL so I generate this minimal reproducible example.
trivialfis
on 4 Jun 2021
Actually, maybe I'm mistaken, I think I didn't read the whole error, cudaErrorPeerAccessUnsupported is different from what we had in UCX, but trying CUDA 11.2 would be a good exercise anyway.
The actual problem seems to be with NCCL, we also don't test NCCL as with Dask we only use UCX. Last week when discussing other issues with @cjnolet he mentioned some limitations, specifically on having more than one process addressing the same GPU, is that error familiar to you Corey?
pentschev
on 4 Jun 2021
specifically on having more than one process addressing the same GPU
These are 2 different GPUs as illustrated in the CUDA_VISIBLE_DEVICES environment variable.
trivialfis
on 4 Jun 2021
These are 2 different GPUs as illustrated in the
CUDA_VISIBLE_DEVICESenvironment variable.
Yes, but the client and scheduler can both access the GPU as well. The scheduler may need to deserialize data in some very particular cases, which will cause it to create a context. The client will also address GPU 0 if any data is brought back to it. The example you posted above doesn't make it clear to me whether the client will use the GPU or not, could you check if there's a context created on processes other than the two workers for the example above?
pentschev
on 4 Jun 2021
Yes, but the client and scheduler can both access the GPU as well
Ah, got it. Thanks for the explanation! Indeed it's possible that the client process creates a CUDA context. The issue I don't understand is it works on other configurations of GPUs like 2x RTX8000 with nvlink, but fails on the GTX1080TI + GTX1080 case.
I exported the exact same conda environment to test on different configurations just so that I'm not changing the software environment.
trivialfis
on 4 Jun 2021
In the 1080TI case, are you able to run NCCL tests without any RAPIDS code? There are some tests that can be run from the NCCL repository: https://github.com/NVIDIA/nccl . I'm wondering whether this could be a bug in NCCL, it's indeed strange that with the same CUDA version it works with RTX8000 but not 1080TIs. Are you also running the same driver version in both cases?
pentschev
on 4 Jun 2021
In the 1080TI case, are you able to run NCCL tests without any RAPIDS code?
I will try it later. It's irregular since it's 1080TI + 1080 (without the TI).
Are you also running the same driver version in both cases?
Yup, Ubuntu20.04 + NV driver 460.80 .
trivialfis
on 4 Jun 2021
Also, the issue is primarily concerning the difference between using LocalCUDACluster and launching 2 workers separately. Since the former case works fine but the latter fails.
trivialfis
on 4 Jun 2021
Passed all nccl tests.
trivialfis
on 4 Jun 2021
Interesting, if LocalCUDACluster passes, then we can probably discard the issue with multiple processes on the same GPU. It's also funny, because from a process perspective, there isn't much difference of launching dask-cuda-workers or LocalCUDACluster, in all cases each worker is a separate process. Can you reproduce the issue if you launch workers with CUDA_VISIBLE_DEVICES=0,1 dask-cuda-worker --scheduler-file="sched.json"?
pentschev
on 4 Jun 2021
Can you reproduce the issue if you launch workers with
No, it works fine in this case.
trivialfis
on 4 Jun 2021
The only difference in this new test is that both workers have the same parent process. I'm not very familiar with possible NCCL limitations, but I find it hard to believe it could be something in Dask at this point. Would launching the workers the way I mentioned above be an acceptable solution for your problem? I'm out of other ideas now.
pentschev
on 4 Jun 2021
I don't think it would be a solution. The example is just for reproducing the error, not an actual use case (I assume everyone would be happy to use LocalCUDACluster for SNMG). @pseudotensor mentioned that this used to work at some point, so maybe he can provide better context than me.
trivialfis
on 4 Jun 2021
Oh, actually I remember from a very old conversation that CUDA_VISIBLE_DEVICES should include all of the devices for NCCL. For example, when you launch dask-cuda-worker with CUDA_VISIBLE_DEVICES=0,1, then the first device will actually be launched with CUDA_VISIBLE_DEVICES=0,1 and the second with CUDA_VISIBLE_DEVICES=1,0 (see https://github.com/rapidsai/dask-cuda/blob/81bbc6f85575826b13b3fb45894b54135514e668/dask_cuda/utils.py#L473-L493), so they're both "reachable". That's not the case if you specify each one individually. I guess this could be the reason, in which case I would say that the correct way to launch is indeed with CUDA_VISIBLE_DEVICES=0,1, rather than with each device individually.
pentschev
on 4 Jun 2021
The example is just for reproducing the error, not an actual use case (I assume everyone would be happy to use LocalCUDACluster for SNMG).
Specifying CUDA_VISIBLE_DEVICES=0,1 to launch all the workers in a single-node at once doesn't limit it to single-node though. You can still run multi-node with that regardless. In fact, by default dask-cuda-worker by default launches for all CUDA devices, which I believe is the appropriate way for most use cases. IOW, my suggestion is that running dask-cuda-worker --scheduler-file="sched.json" should suffice.
pentschev
on 4 Jun 2021
That's not the case if you specify each one individually.
Thanks for the suggestion. But the example I provided with CUDA_VISIBLE_DEVICES is to simulate the scenario that each worker possesses only 1 GPU so that it looks like they are on different nodes. I think the actual use case would be me launching these 2 workers on different nodes, but I don't have 2 machines that I can tear apart for installing GPUs for testing ... ;-( Sorry for the ambiguity.
I will leave @pseudotensor to follow up on their actual use cases and maybe better examples.
trivialfis
on 4 Jun 2021
I can help submit an issue on NCCL if it's not a Dask issue, but before that, we might need to narrow it down a little bit.
trivialfis
on 4 Jun 2021
What I heard from NCCL developers is that there's no guarantee cudaIpcOpenMemHandle will succeed if the process doesn't know the index of the device on the other end, thus it seems to confirm my suspicion.
Ideas for further testing:
- Run with
NCCL_DEBUG=INFO, we can send the logs to devs so they can check whether something can be done; - Try with
NCCL_P2P_DISABLE=1.
pentschev
on 4 Jun 2021
Disabling P2P workers for me, thanks for the suggestions! Here are the logs:
disable_p2p.txt
enable_p2p.txt
@pseudotensor Could you please try that?
trivialfis
on 5 Jun 2021
No effect for me. I did this repro: https://github.com/dmlc/xgboost/issues/7019#issuecomment-854125114
but I just added NCCL_P2P_DISABLE=1 in front of the 4 commands (scheduler launch, 2 worker launches, and 1 client launch).
This is clearly a major regression in how things used to be for rapids 0.14.
In first trial it gave same NCCL error:
Stack trace:
[bt] (0) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x4f1c29) [0x7f6c69574c29]
[bt] (1) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x50f26f) [0x7f6c6959226f]
[bt] (2) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x19d5e1) [0x7f6c692205e1]
[bt] (3) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x19e279) [0x7f6c69221279]
[bt] (4) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x1d05f9) [0x7f6c692535f9]
[bt] (5) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(XGBoosterUpdateOneIter+0x68) [0x7f6c6913cf18]
[bt] (6) /home/jon/minicondadai_py38/lib/python3.8/lib-dynload/../../libffi.so.7(+0x69dd) [0x7f6daf2b99dd]
[bt] (7) /home/jon/minicondadai_py38/lib/python3.8/lib-dynload/../../libffi.so.7(+0x6067) [0x7f6daf2b9067]
[bt] (8) /home/jon/minicondadai_py38/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so(+0x10da8) [0x7f6daf2cfda8]
And then if I keep same scheduler/worker up, but re-run client, everything then hangs.
node A:
NCCL_DEBUG=INFO NCCL_P2P_DISABLE=1 dask-scheduler --scheduler-file dask_scheduler.json&
NCCL_DEBUG=INFO NCCL_P2P_DISABLE=1 dask-cuda-worker --scheduler-file dask_scheduler.json
node B:
NCCL_DEBUG=INFO NCCL_P2P_DISABLE=1 dask-cuda-worker --scheduler-file dask_scheduler.json
node A:
NCCL_DEBUG=INFO NCCL_P2P_DISABLE=1 python dask_cudf_scitkit_issue7019b.py
gives:
node A:
[1] 13167
(base) jon@mr-dl10:/data/jon/h2oai.fullcondatest3$ distributed.scheduler - INFO - -----------------------------------------------
(base) jon@mr-dl10:/data/jon/h2oai.fullcondatest3$ fgdistributed.scheduler - INFO - -----------------------------------------------
distributed.scheduler - INFO - Clear task state
distributed.scheduler - INFO - Scheduler at: tcp://172.16.2.210:8786
distributed.scheduler - INFO - dashboard at: :8787
NCCL_DEBUG=INFO NCCL_P2P_DISABLE=1 dask-cuda-worker --scheduler-file dask_scheduler.json
distributed.nanny - INFO - Start Nanny at: 'tcp://172.16.2.210:39091'
distributed.nanny - INFO - Start Nanny at: 'tcp://172.16.2.210:43679'
distributed.preloading - INFO - Import preload module: dask_cuda.initialize
distributed.preloading - INFO - Import preload module: dask_cuda.initialize
distributed.preloading - INFO - Run preload setup click command: dask_cuda.initialize
distributed.worker - INFO - Start worker at: tcp://172.16.2.210:32961
distributed.worker - INFO - Listening to: tcp://172.16.2.210:32961
distributed.worker - INFO - dashboard at: 172.16.2.210:36233
distributed.worker - INFO - Waiting to connect to: tcp://172.16.2.210:8786
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 62.89 GiB
distributed.worker - INFO - Local Directory: /data/jon/h2oai.fullcondatest3/dask-worker-space/worker-5pz73yl0
distributed.worker - INFO - Starting Worker plugin <dask_cuda.utils.RMMSetup object at 0x7fbd00eb4610-3ed5712e-e564-4ae7-859c-d0859cc4efe5
distributed.worker - INFO - Starting Worker plugin <dask_cuda.utils.CPUAffinity object at 0x7fbd00eaa-40affab2-2be5-4ff3-8175-c735e3425c78
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Register worker <Worker 'tcp://172.16.2.210:32961', name: tcp://172.16.2.210:32961, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, tcp://172.16.2.210:32961
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://172.16.2.210:8786
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.preloading - INFO - Run preload setup click command: dask_cuda.initialize
distributed.worker - INFO - Start worker at: tcp://172.16.2.210:39087
distributed.worker - INFO - Listening to: tcp://172.16.2.210:39087
distributed.worker - INFO - dashboard at: 172.16.2.210:35701
distributed.worker - INFO - Waiting to connect to: tcp://172.16.2.210:8786
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 62.89 GiB
distributed.worker - INFO - Local Directory: /data/jon/h2oai.fullcondatest3/dask-worker-space/worker-jhwiaxti
distributed.worker - INFO - Starting Worker plugin <dask_cuda.utils.RMMSetup object at 0x7fc25b07e610-a9364ea4-47f5-4ee0-9030-c2b7679d60f8
distributed.worker - INFO - Starting Worker plugin <dask_cuda.utils.CPUAffinity object at 0x7fc25b074-6a1b34ce-2ed7-45bc-96e1-7b0ead2000c4
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Register worker <Worker 'tcp://172.16.2.210:39087', name: tcp://172.16.2.210:39087, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, tcp://172.16.2.210:39087
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://172.16.2.210:8786
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'tcp://172.16.4.17:44225', name: tcp://172.16.4.17:44225, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, tcp://172.16.4.17:44225
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Receive client connection: Client-e2ff201a-c619-11eb-825d-0cc47adb058f
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Run out-of-band function '_start_tracker'
[09:20:11] task [xgboost.dask]:tcp://172.16.2.210:39087 got new rank 0
[09:20:11] task [xgboost.dask]:tcp://172.16.2.210:32961 got new rank 1
[09:20:11] WARNING: /workspace/xgboost/src/learner.cc:572:
Parameters: { "debug_verbose", "early_stopping_rounds", "early_stopping_threshold", "enable_early_stopping_rounds", "monotonicity_constraints", "silent" } might not be used.
This may not be accurate due to some parameters are only used in language bindings but
passed down to XGBoost core. Or some parameters are not used but slip through this
verification. Please open an issue if you find above cases.
[09:20:11] WARNING: /workspace/xgboost/src/learner.cc:572:
Parameters: { "debug_verbose", "early_stopping_rounds", "early_stopping_threshold", "enable_early_stopping_rounds", "monotonicity_constraints", "silent" } might not be used.
This may not be accurate due to some parameters are only used in language bindings but
passed down to XGBoost core. Or some parameters are not used but slip through this
verification. Please open an issue if you find above cases.
mr-dl10:15923:24327 [0] NCCL INFO Bootstrap : Using enp5s0:172.16.2.210<0>
mr-dl10:15923:24327 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
mr-dl10:15923:24327 [0] NCCL INFO NET/IB : No device found.
mr-dl10:15923:24327 [0] NCCL INFO NET/Socket : Using [0]enp5s0:172.16.2.210<0>
mr-dl10:15923:24327 [0] NCCL INFO Using network Socket
NCCL version 2.8.3+cuda11.2
mr-dl10:15920:24250 [0] NCCL INFO Bootstrap : Using enp5s0:172.16.2.210<0>
mr-dl10:15920:24250 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
mr-dl10:15920:24250 [0] NCCL INFO NET/IB : No device found.
mr-dl10:15920:24250 [0] NCCL INFO NET/Socket : Using [0]enp5s0:172.16.2.210<0>
mr-dl10:15920:24250 [0] NCCL INFO Using network Socket
mr-dl10:15923:24327 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC
mr-dl10:15920:24250 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC
mr-dl10:15920:24250 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
mr-dl10:15920:24250 [0] NCCL INFO Setting affinity for GPU 0 to 3ff003ff
mr-dl10:15923:24327 [0] NCCL INFO Channel 00/02 : 0 1 2
mr-dl10:15923:24327 [0] NCCL INFO Channel 01/02 : 0 1 2
mr-dl10:15923:24327 [0] NCCL INFO Trees [0] 1/2/-1->0->-1 [1] 1/-1/-1->0->2
mr-dl10:15923:24327 [0] NCCL INFO Setting affinity for GPU 1 to ff,c00ffc00
mr-dl10:15923:24327 [0] NCCL INFO Channel 00 : 2[2000] -> 0[81000] [receive] via NET/Socket/0
mr-dl10:15923:24327 [0] NCCL INFO Channel 01 : 2[2000] -> 0[81000] [receive] via NET/Socket/0
mr-dl10:15920:24250 [0] NCCL INFO Channel 00 : 1[2000] -> 2[2000] [send] via NET/Socket/0
mr-dl10:15920:24250 [0] NCCL INFO Channel 01 : 1[2000] -> 2[2000] [send] via NET/Socket/0
mr-dl10:15923:24327 [0] NCCL INFO Channel 00 : 0[81000] -> 1[2000] via direct shared memory
mr-dl10:15923:24327 [0] NCCL INFO Channel 01 : 0[81000] -> 1[2000] via direct shared memory
mr-dl10:15920:24250 [0] include/socket.h:406 NCCL WARN Connect to fe80::70bb:c0ff:febd:57b9%3398<46297> failed : Network is unreachable
mr-dl10:15920:24250 [0] NCCL INFO transport/net_socket.cc:315 -> 2
mr-dl10:15920:24250 [0] NCCL INFO include/net.h:21 -> 2
mr-dl10:15920:24250 [0] NCCL INFO transport/net.cc:174 -> 2
mr-dl10:15920:24250 [0] NCCL INFO transport.cc:104 -> 2
mr-dl10:15920:24250 [0] NCCL INFO init.cc:813 -> 2
mr-dl10:15920:24250 [0] NCCL INFO init.cc:878 -> 2
mr-dl10:15920:24250 [0] NCCL INFO init.cc:914 -> 2
mr-dl10:15920:24250 [0] NCCL INFO init.cc:926 -> 2
mr-dl10:15923:24327 [0] NCCL INFO Connected all rings
mr-dl10:15923:24327 [0] NCCL INFO Channel 00 : 0[81000] -> 2[2000] [send] via NET/Socket/0
mr-dl10:15923:24327 [0] NCCL INFO Channel 01 : 0[81000] -> 2[2000] [send] via NET/Socket/0
distributed.worker - WARNING - Compute Failed
Function: dispatched_train
args: ('tcp://172.16.2.210:32961', [b'DMLC_NUM_WORKER=3', b'DMLC_TRACKER_URI=172.16.2.210', b'DMLC_TRACKER_PORT=9091', b'DMLC_TASK_ID=[xgboost.dask]:tcp://172.16.2.210:32961'], {'feature_names': None, 'feature_types': None, 'feature_weights': None, 'meta_names': ['labels'], 'missing': nan, 'parts': [( 0_v1
0 1.491422665e-07
1 2.531357527
2 <NA>
3 1.505901337
4 1.351461172
5 2.331233978
6 1.588958025
7 0.996571183
8 <NA>
9 <NA>
10 3.235293388
11 2.108262539
12 3.203463078
13 <NA>
14 <NA>
15 <NA>
16 <NA>
17 <NA>
18 1.875575781
19 0.338827759
20 1.156249881
21 <NA>
22 2.560552597
23 <NA>
24 <NA>
25 <NA>
26 4.746792316
27 1.899523735
28 2.438667774
29 0.968903363
30 <NA>
31 1.421618938
32 1.60103941
33 2.121357441
34
kwargs: {}
Exception: XGBoostError('[09:20:12] /workspace/xgboost/src/tree/updater_gpu_hist.cu:794: Exception in gpu_hist: NCCL failure :unhandled system error /workspace/xgboost/src/common/device_helpers.cu(67)\n\nStack trace:\n [bt] (0) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x4f1c29) [0x7fbbb9574c29]\n [bt] (1) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x50f26f) [0x7fbbb959226f]\n [bt] (2) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x19d5e1) [0x7fbbb92205e1]\n [bt] (3) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x19e279) [0x7fbbb9221279]\n [bt] (4) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x1d05f9) [0x7fbbb92535f9]\n [bt] (5) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(XGBoosterUpdateOneIter+0x68) [0x7fbbb913cf18]\n [bt] (6) /home/jon/minicondadai_py38/lib/python3.8/lib-dynload/../../libffi.so.7(+0x69dd) [0x7fbd002d69dd]\n [bt] (7) /home/jon/minicondadai_py38/lib/python3.8/lib-dynload/../../libffi.so.7(+0x6067) [0x7fbd002d6067]\n [bt] (8) /home/jon/minicondadai_py38/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so(+0x10da8) [0x7fbd002ecda8]\n\n')
distributed.scheduler - INFO - Remove client Client-e2ff201a-c619-11eb-825d-0cc47adb058f
distributed.scheduler - INFO - Remove client Client-e2ff201a-c619-11eb-825d-0cc47adb058f
distributed.scheduler - INFO - Close client connection: Client-e2ff201a-c619-11eb-825d-0cc47adb058f
mr-dl10:15923:24327 [0] NCCL INFO Call to connect returned Connection refused, retrying
mr-dl10:15923:24327 [0] NCCL INFO Call to connect returned Connection refused, retrying
mr-dl10:15923:24327 [0] NCCL INFO Call to connect returned Connection refused, retrying
mr-dl10:15923:24327 [0] NCCL INFO Call to connect returned Connection refused, retrying
mr-dl10:15923:24327 [0] NCCL INFO Call to connect returned Connection refused, retrying
mr-dl10:15923:24327 [0] NCCL INFO Call to connect returned Connection refused, retrying
mr-dl10:15923:24327 [0] NCCL INFO Call to connect returned Connection refused, retrying
mr-dl10:15923:24327 [0] NCCL INFO Call to connect returned Connection refused, retrying
mr-dl10:15923:24327 [0] NCCL INFO Call to connect returned Connection refused, retrying
mr-dl10:15923:24327 [0] NCCL INFO Call to connect returned Connection refused, retrying
mr-dl10:15923:24327 [0] NCCL INFO Call to connect returned Connection refused, retrying
mr-dl10:15923:24327 [0] NCCL INFO Call to connect returned Connection refused, retrying
mr-dl10:15923:24327 [0] NCCL INFO Call to connect returned Connection refused, retrying
mr-dl10:15923:24327 [0] NCCL INFO Call to connect returned Connection refused, retrying
mr-dl10:15923:24327 [0] NCCL INFO Call to connect returned Connection refused, retrying
mr-dl10:15923:24327 [0] NCCL INFO Call to connect returned Connection refused, retrying
mr-dl10:15923:24327 [0] NCCL INFO Call to connect returned Connection refused, retrying
mr-dl10:15923:24327 [0] NCCL INFO Call to connect returned Connection refused, retrying
mr-dl10:15923:24327 [0] NCCL INFO Call to connect returned Connection refused, retrying
mr-dl10:15923:24327 [0] include/socket.h:406 NCCL WARN Connect to 172.16.2.210<33547> failed : Connection refused
mr-dl10:15923:24327 [0] NCCL INFO bootstrap.cc:417 -> 2
mr-dl10:15923:24327 [0] NCCL INFO transport.cc:95 -> 2
mr-dl10:15923:24327 [0] NCCL INFO init.cc:823 -> 2
mr-dl10:15923:24327 [0] NCCL INFO init.cc:878 -> 2
mr-dl10:15923:24327 [0] NCCL INFO init.cc:914 -> 2
mr-dl10:15923:24327 [0] NCCL INFO init.cc:926 -> 2
Node mr-dl10 IP is 172.16.2.210 and mr-dl17 IP is 172.16.4.17
node B:
distributed.nanny - INFO - Start Nanny at: 'tcp://172.16.4.17:35355'
distributed.preloading - INFO - Import preload module: dask_cuda.initialize
distributed.preloading - INFO - Run preload setup click command: dask_cuda.initialize
distributed.worker - INFO - Start worker at: tcp://172.16.4.17:44225
distributed.worker - INFO - Listening to: tcp://172.16.4.17:44225
distributed.worker - INFO - dashboard at: 172.16.4.17:45785
distributed.worker - INFO - Waiting to connect to: tcp://172.16.2.210:8786
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 125.78 GiB
distributed.worker - INFO - Local Directory: /home/ops/h2oai/dask-worker-space/worker-r5ib81ta
distributed.worker - INFO - Starting Worker plugin <dask_cuda.utils.RMMSetup object at 0x7f2feb2215b0-7b26dd89-a411-4a68-a9eb-9f3e98f3f243
distributed.worker - INFO - Starting Worker plugin <dask_cuda.utils.CPUAffinity object at 0x7f2feb216-afe866d0-41c0-4c8d-9d8b-610b893e3fc2
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Registered to: tcp://172.16.2.210:8786
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
[09:20:11] task [xgboost.dask]:tcp://172.16.4.17:44225 got new rank 2
[09:20:11] WARNING: /workspace/xgboost/src/learner.cc:572:
Parameters: { "debug_verbose", "early_stopping_rounds", "early_stopping_threshold", "enable_early_stopping_rounds", "monotonicity_constraints", "silent" } might not be used.
This may not be accurate due to some parameters are only used in language bindings but
passed down to XGBoost core. Or some parameters are not used but slip through this
verification. Please open an issue if you find above cases.
mr-dl17:20052:20133 [0] NCCL INFO Bootstrap : Using enp4s0:172.16.4.17<0>
mr-dl17:20052:20133 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
mr-dl17:20052:20133 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
mr-dl17:20052:20133 [0] NCCL INFO NET/Socket : Using [0]enp4s0:172.16.4.17<0> [1]veth4cea06a:fe80::70bb:c0ff:febd:57b9%veth4cea06a<0> [2]vethd82dcb3:fe80::94d9:f2ff:feb4:ce85%vethd82dcb3<0>
mr-dl17:20052:20133 [0] NCCL INFO Using network Socket
mr-dl17:20052:20133 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC
mr-dl17:20052:20133 [0] NCCL INFO Trees [0] -1/-1/-1->2->0 [1] 0/-1/-1->2->-1
mr-dl17:20052:20133 [0] NCCL INFO Setting affinity for GPU 0 to 3ff003ff
mr-dl17:20052:20133 [0] NCCL INFO Channel 00 : 1[2000] -> 2[2000] [receive] via NET/Socket/1
mr-dl17:20052:20133 [0] NCCL INFO Channel 01 : 1[2000] -> 2[2000] [receive] via NET/Socket/1
mr-dl17:20052:20133 [0] NCCL INFO Channel 00 : 2[2000] -> 0[81000] [send] via NET/Socket/1
mr-dl17:20052:20133 [0] NCCL INFO Channel 01 : 2[2000] -> 0[81000] [send] via NET/Socket/1
client node A:
<class 'dask_cudf.core.DataFrame'>
<class 'dask_cudf.core.Series'>
Traceback (most recent call last):
File "dask_cudf_scitkit_issue7019b.py", line 28, in <module>
fun()
File "dask_cudf_scitkit_issue7019b.py", line 15, in fun
model.fit(X, y)
File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/dask.py", line 1802, in fit
return self._client_sync(self._fit_async, **args)
File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/dask.py", line 1610, in _client_sync
return self.client.sync(func, **kwargs, asynchronous=asynchronous)
File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/distributed/client.py", line 843, in sync
return sync(
File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/distributed/utils.py", line 353, in sync
raise exc.with_traceback(tb)
File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/distributed/utils.py", line 336, in f
result[0] = yield future
File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
value = future.result()
File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/dask.py", line 1760, in _fit_async
results = await self.client.sync(
File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/dask.py", line 915, in _train_async
results = await client.gather(futures, asynchronous=True)
File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/distributed/client.py", line 1840, in _gather
raise exception.with_traceback(traceback)
File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/dask.py", line 870, in dispatched_train
bst = worker_train(params=local_param,
File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/training.py", line 189, in train
bst = _train_internal(params, dtrain,
File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/training.py", line 81, in _train_internal
bst.update(dtrain, i, obj)
File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/core.py", line 1571, in update
_check_call(_LIB.XGBoosterUpdateOneIter(self.handle,
File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/core.py", line 214, in _check_call
raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [09:20:12] /workspace/xgboost/src/tree/updater_gpu_hist.cu:794: Exception in gpu_hist: NCCL failure :unhandled system error /workspace/xgboost/src/common/device_helpers.cu(67)
Stack trace:
[bt] (0) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x4f1c29) [0x7fbbb9574c29]
[bt] (1) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x50f26f) [0x7fbbb959226f]
[bt] (2) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x19d5e1) [0x7fbbb92205e1]
[bt] (3) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x19e279) [0x7fbbb9221279]
[bt] (4) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x1d05f9) [0x7fbbb92535f9]
[bt] (5) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(XGBoosterUpdateOneIter+0x68) [0x7fbbb913cf18]
[bt] (6) /home/jon/minicondadai_py38/lib/python3.8/lib-dynload/../../libffi.so.7(+0x69dd) [0x7fbd002d69dd]
[bt] (7) /home/jon/minicondadai_py38/lib/python3.8/lib-dynload/../../libffi.so.7(+0x6067) [0x7fbd002d6067]
[bt] (8) /home/jon/minicondadai_py38/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so(+0x10da8) [0x7fbd002ecda8]
pseudotensor
on 5 Jun 2021
Thanks for all the help @pentschev ! I will close this one now since it's not related to dask-cuda.
trivialfis
on 7 Jun 2021
Related issues
beckernick
路
10Comments
beckernick
路
7Comments
pentschev
路
9Comments
gram526
路
3Comments
pradghos
路
11Comments