Dask-cuda: Distributed computation support

[vibhatha]

Does dask-cuda support distributed memory computations?

For instance, can this be used to do a distributed join on top of CuDF?

[pentschev]

Does dask-cuda support distributed memory computations?

I'm not exactly sure what you mean be "distributed memory computations", could you elaborate?

For instance, can this be used to do a distributed join on top of CuDF?

Yes, you can use cuDF with Dask-CUDA via Dask-cuDF: https://docs.rapids.ai/api/cudf/stable/10min.html .

[vibhatha]

@pentschev
What I meant was;

On a fixed size dataset, can we distribute it to multiple nodes and do a join distributedly on multi-core multi-node using Dask CuDF?

[jakirkham]

In the example Peter linked, there is a join illustrated.

[vibhatha]

@jakirkham can this join be running on multiple machines. Let's say the data is not fitting a machine ( let's say 8 GPU included machine), so we would have to use multiple machines. So will this work on that kind of a larger dataset?

[pentschev]

Yes, it will work with multiple nodes too. For that the simplest setup is to use a single dask-scheduler process and a dask-cuda-worker process for each worker node node.

[vibhatha]

So internally, it doesn't use any all-to-all communication, but it sends to the master and workers back and forth to do the distributed computation?

[pentschev]

No, data is communicated peer-to-peer, but the scheduler is responsible for creating task graphs and assigning workers to do computation. I suggest referring to Dask Distributed docs for additional details. Note that generally Dask-CUDA is merely a specialization of Dask Distributed workers, and specific features are described in the Dask-CUDA docs.

[vibhatha]

I was also wondering about this since CuDF is on GPUs, isn't it possible to use NCCL underneath to do the communication component?

[pentschev]

No, Dask-CUDA/RAPIDS standard today is to use UCX-Py for communication: https://dask-cuda.readthedocs.io/en/latest/ucx.html .

[pentschev]

I think the original question has been answered, I'm tentatively closing this, @vibhatha please feel free to reopen if there are follow-up questions or open a new issue.

[quasiben]

@vibhatha I think it would be worth reading through the following post on for a more high level understanding of how all these pieces fit together:

• https://medium.com/rapids-ai/high-performance-python-communication-with-ucx-py-221ac9623a6a
• https://medium.com/rapids-ai/no-more-waiting-interactive-big-data-now-32f7b903cf41