Dask-cuda: About --enable-nvlink and --enable-infiniband flags.

[benjha]

Hi dask-cuda team,

I am setting up RAPIDS 0.18 scripts for Summit

https://github.com/benjha/nvrapids_olcf/blob/branch-0.18/docs/nvidia-rapids.rst

and I'd like to set UCX as default for dask-cuda communications.

After looking at the documentation, I'd like to make sure dask-cuda-workers can use both --enable-nvlink and --enable-infiniband options; and if so, how dask-cuda knows when to switch between NVLink and Infiniband.

Thanks

@pentschev @jakirkham

[quasiben]

The decision to use NVLink or Infiniband is not controlled by dask-cuda but rather by UCX, which dask-cuda (along with UCX-Py) configures. UCX will determine what the fastest transport is between two endpoints. How it does this is kind of magic and is also somewhat tunable. Dask-CUDA and UCX-Py have been tuned for what we think are best for Dask/RAPIDS workloads.

[quasiben]

Oops, didn't mean to close

[jakirkham]

Though to your point Benjamin, yes both of those can be set together and will be passed into UCX during initialization to make sure UCX enables both. After that we merely feed UCX buffers from host or device and let UCX do its magic 😄

[pentschev]

Hi @benjha , to be a bit more specific on how things will work on Summit, taking into consideration the Summit Node Topology each of the 3 GPUs that share the same NUMA node will transfer over NVLink as their BW is 50GB/s. For GPUs communicating to a GPU on the other NUMA node I believe it will be done over the X-Bus that has a 64GB/s BW, but I'm not totally sure on this one, it may as well go over EDR IB. Finally, for inter-node communications everything must go over EDR IB with 12.5GB/s BW.

[benjha]

There are two bugs related to UCX 1.9.0 on ppc64le (links 1, 2 and link 3) that also affects dask-cuda and the local-send-recv.py benchmark:

The dask-cuda related bug produces the next error (1, 2) when each dask-cuda-worker is initializing:

[h36n04:158119:0:158119] ucp_worker.c:183  Fatal: failed to set active message handler id 1: Invalid parameter

and the local-send-recv.py benchmark produces the next warning after the last iteration (3):

tag_match.c:61   UCX  WARN  unexpected tag-receive descriptor 0x153d35500 was not matched

I am setting the UCX_* vars. recommended in https://dask-cuda.readthedocs.io/en/latest/ucx.html#configuration for the local-send-recv.py and for dask-cuda, I am setting the DASK_UCX__* vars. for the scheduler and the corresponding options for the workers.

@jglaser What UCX version did you use in your BSQL 0.18 deployment ?

BUGS:

1. https://www.mail-archive.com/[email protected]/msg34206.html
2. https://github.com/open-mpi/ompi/issues/8252
3. https://forum.hdfgroup.org/t/parallel-hdf5-test-failures-power9-lustre-openmpi-ucx/7752

[jglaser]

@jglaser What UCX version did you use in your BSQL 0.18 deployment ?

1.8.1 was the most recent version of UCX I was able to get blazingsql and rapids to run with.

[benjha]

Hi @benjha , to be a bit more specific on how things will work on Summit, taking into consideration the Summit Node Topology each of the 3 GPUs that share the same NUMA node will transfer over NVLink as their BW is 50GB/s. For GPUs communicating to a GPU on the other NUMA node I believe it will be done over the X-Bus that has a 64GB/s BW, but I'm not totally sure on this one, it may as well go over EDR IB. Finally, for inter-node communications everything must go over EDR IB with 12.5GB/s BW.

Got it, I did notice better BW when using GPUs from the same socket (about 46.31 GB/s) than when using GPUs from different sockets (about 33 GB/s) with the local-send-recv.py benchmark.

[pentschev]

There are two bugs related to UCX 1.9.0 on ppc64le (links 1, 2 and link 3) that also affects dask-cuda and the local-send-recv.py benchmark:

The dask-cuda related bug produces the next error (1, 2) when each dask-cuda-worker is initializing:

[h36n04:158119:0:158119] ucp_worker.c:183  Fatal: failed to set active message handler id 1: Invalid parameter

This is a strange error because we don't use active messages in UCX-Py (and, by consequence, Dask-CUDA). Does it completely fail after that, or besides the message what is the observable behavior?

and the local-send-recv.py benchmark produces the next warning after the last iteration (3):

tag_match.c:61   UCX  WARN  unexpected tag-receive descriptor 0x153d35500 was not matched

These messages happen during cleanup when processes are shutting down, it's been very hard to find the exact source of them in Dask/UCX-Py, I know it's not great but they are harmless and can be ignored for now.

I am setting the UCX_* vars. recommended in https://dask-cuda.readthedocs.io/en/latest/ucx.html#configuration for the local-send-recv.py and for dask-cuda, I am setting the DASK_UCX__* vars. for the scheduler and the corresponding options for the workers.

@jglaser What UCX version did you use in your BSQL 0.18 deployment ?

BUGS:

1. https://www.mail-archive.com/[email protected]/msg34206.html
2. open-mpi/ompi#8252
3. https://forum.hdfgroup.org/t/parallel-hdf5-test-failures-power9-lustre-openmpi-ucx/7752

With the comments above, I know the warning is fine for now, but the active messages error would be good to know more of what happens after the error is thrown.

[pentschev]

Got it, I did notice better BW when using GPUs from the same socket (about 46.31 GB/s) than when using GPUs from different sockets (about 33 GB/s) with the local-send-recv.py benchmark.

Thanks for confirming, so it looks like transfers between NUMA nodes indeed go over the X-Bus, which is also shared with the rest of the system, thus 33 GB/s seems reasonable depending on the X-Bus load.

[jakirkham]

Is it possible BSQL is using active messages on their own?

[pentschev]

Is it possible BSQL is using active messages on their own?

A quick git grep ucp_am under BSQL doesn't show any entries, but maybe @felipeblazing could confirm that?

[felipeblazing]

@pentschev We use the same context as ucx-py thought we make our own end points. All the initialization code can be found here https://github.com/BlazingDB/blazingsql/blob/16095fbef7c78971b81271dd34f8a0101ebf3625/engine/src/communication/ucx_init.h#L281 . Note that the createUcpContext is only used in testing. During real use it gets the context from ucx-py and shares that context. But you can see how we initialize workers and connect to them in the functions below. Nothing we do should cause it to change the use of Active Messages of which I am woefully ignorant.

[benjha]

For now, I've downgraded to UCX 1.8.1; next is how the UCX related section of the RAPIDS module on Summit looks like:

load("gdrcopy/2.0")
...
setenv("UCX_SOCKADDR_CM_ENABLE","n")
setenv("UCX_MEMTYPE_CACHE","y")
setenv("UCX_RNDV_SCHEME","get_zcopy")
setenv("UCX_NET_DEVICES","mlx5_0:1,mlx5_3:1")
setenv("UCX_MAX_RNDV_RAILS","2")
setenv("UCX_TLS","rc_x,sm,cuda_copy,cuda_ipc,gdr_copy")
...

and this is how I am calling the dask-scheduler and dask-cudar-worker:

...
jsrun --nrs 1 --tasks_per_rs 1 --cpu_per_rs 1 \
      dask-scheduler --interface ib0 --protocol ucx \
                     --scheduler-file $DASK_DIR/my-scheduler.json \
                     --no-dashboard --no-show &
...
jsrun --nrs 12 --rs_per_host 6 --tasks_per_rs 1 --cpu_per_rs 2 --gpu_per_rs 1 --smpiargs='off' \
      dask-cuda-worker --nthreads 1 --memory-limit 82GB --device-memory-limit 16GB --rmm-pool-size=15GB \
                       --enable-nvlink --enable-infiniband  \
                       --death-timeout 60  --interface ib0 --scheduler-file $DASK_DIR/my-scheduler.json --local-directory $DASK_DIR \
                       --no-dashboard &

These set up correctly the dask-cuda cluster

[pentschev]

Thanks for the update @benjha . Just to be sure I understand correctly, you're saying that the only change was downgrading from 1.9.0 to 1.8.1 or were there other changes as well?

[benjha]

Thanks for the update @benjha . Just to be sure I understand correctly, you're saying that the only change was downgrading from 1.9.0 to 1.8.1 or were there other changes as well?

There were also changes with the flags as mentioned above, i.e.:

For 1.9.0 I used those UCX_* and DASK_UCX__* vars. recommended in https://dask-cuda.readthedocs.io/en/latest/ucx.html#configuration, for 1.8.1 I am setting those from my last comment ( based on scripts from the BSQL team).

I'll do further testing once I have more time.

[pentschev]

Thank you @benjha , and yes, the variables are very important and I expect them to work with both UCX 1.8.1 and 1.9.0 (but not with 1.10+ which isn't yet support by UCX-Py/Dask), and in general I think you're fine with either of those versions, but if you happen to test 1.9.0 again, please let us know what results you got.

[pentschev]

It seems that the core of this issue has been resolved, I'm tentatively closing this but feel free to reopen if needed @benjha .