Dask-cuda: Worker containers cannot connect to scheduler container

[lmeyerov]

We're stumped on the 'right' way to make a dask-scheduler docker service and then have dask-cuda-workers register with it. We can start them in the same container and have the successfully register, but not as separate ones. The ultimate goal is enabling dask-cuda-workers from the same (and later, different) nodes register with the scheduler, and enable multiple python services instantiate Client()s against the scheduler.

The docker-compose.yml is roughly:

version: "3.5"

networks:
  grph_net:
    name: grph_net
    ipam:
      driver: default
      config:
      - subnet: 172.21.0.0/16
services:
   dask-cuda-scheduler:
      expose: 8786
      cmd: dask-scheduler --port 8786 --interface lo --no-dashboard --no-show
   dask-cuda-worker:
      cmd: dask-cuda-worker --no-dashboard dask-cuda-scheduler:8786
      #note lack of exposed worker/nanny ports..
   some_py_service:
      # => Client('dask-cuda-scheduler:8786')

Notes:

• Blazing docs suggest interface lo for docker, and I didn't see anything in rapids

• With cpu dask-worker, we'd also expose+specify --worker-port 30000:30099 --nanny-port 40000:40099 ... but dask-cuda-worker does not expose an equiv

• scheduler service starts fine:

dask-cuda-scheduler_1  | distributed.scheduler - INFO - -----------------------------------------------
dask-cuda-scheduler_1  | distributed.scheduler - INFO - Clear task state
dask-cuda-scheduler_1  | distributed.scheduler - INFO -   Scheduler at:      tcp://127.0.0.1:8786
dask-cuda-scheduler_1  | distributed.scheduler - INFO -   dashboard at:            127.0.0.1:8787

• worker service fails to connect:

distributed.nanny - INFO -         Start Nanny at: 'tcp://127.0.0.1:35465'
distributed.diskutils - INFO - Found stale lock file and directory '/opt/graphistry/apps/forge/etl-server-python/dask-worker-space/dask-worker-space/worker-fnvt5gci', purging
distributed.preloading - INFO - Import preload module: dask_cuda.initialize
distributed.preloading - INFO - Run preload setup click command: dask_cuda.initialize
distributed.worker - INFO -       Start worker at:      tcp://127.0.0.1:39711
distributed.worker - INFO -          Listening to:      tcp://127.0.0.1:39711
distributed.worker - INFO -          dashboard at:            127.0.0.1:39269
distributed.worker - INFO - Waiting to connect to: tcp://dask-cuda-scheduler:8786
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -                Memory:                    8.19 GB
distributed.worker - INFO -       Local Directory: /opt/graphistry/apps/forge/etl-server-python/dask-worker-space/dask-worker-space/worker-ceve3vfy
distributed.worker - INFO - Starting Worker plugin <dask_cuda.utils.RMMSetup object at 0x7fafe00f3210-13b02e18-7783-4945-9eb1-e5cbe9bb35ae
distributed.worker - INFO - Starting Worker plugin <dask_cuda.utils.CPUAffinity object at 0x7fb01b1fc-7999941b-83eb-429b-b6c9-1c0dc46780b8
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Waiting to connect to: tcp://dask-cuda-scheduler:8786
distributed.worker - INFO - Waiting to connect to: tcp://dask-cuda-scheduler:8786
distributed.worker - INFO - Waiting to connect to: tcp://dask-cuda-scheduler:8786

• The worker does connect if I instead do as localhost within the scheduler container

docker-compose exec dask-cuda-scheduler /bin/bash
> dask-cuda-worker --no-dashboard --interface lo 127.0.0.1:8786 
=> Registered to:       tcp://127.0.0.1:8786

My thinking on next steps are:

• dig into UCX settings for use on nvlink + non-nvlink boxes to see if that gives alternate flags
• see about starting processes via python in case any additional options are exposed that way
• give up and run scheduler + worker in the same process, and move on to ensuring clients can connect

Any ideas welcome! We have LocalCUDACluster working per-docker-python-microservice-process, so looking how to productionize by factoring it out into a shared service.

[pentschev]

Sorry @lmeyerov , there are too many topics in a single issue and I'm not really sure what your exact issue is. It seems that the original issue though is that you can't use another interface other than lo, is that right? Generally dask-scheduler will pick up one interface that has an IP address attached to it, but I'm not really sure how it prioritizes which one to pick. I would suggest checking what are the interfaces available in the container (e.g., eth0) and trying to use that if you haven't done so.

[lmeyerov]

@pentschev Getting there, looks like switching interface to eth0 (thanks!) + using the container name:

Service dask-scheduler::

dask-scheduler --interface eth0 --port 8786 --no-dashboard --no-show

Service dask-cuda-worker:

dask-cuda-worker --no-dashboard --interface eth0 dask-cuda-scheduler:8786

Next step will be:
-- client container in same network connecting to ^^^
-- what needs to be exposed for a remote network client (e.g., http) <-- I'm more skeptical here b/c, afaict, we cannot pin the dask-cuda-worker port range (while dask-worker can)

WIll report back with, hopefully, a working docker-compose.yml :)

[lmeyerov]

Exciting! Confirmed the client works (and mixed sync/async ones, even). We can close the immediate issue.

Working towards a docker-compose.yml to make this more obvious.

[lmeyerov]

Here we go:

docker-compose.yml:

version: "3.5"

x-dask-opts:
  &dask_opts
  image: rapidsai/rapidsai:latest
  environment:
    SCHEDULER_PORT: 8786
  volumes:
    - ./entrypoint.sh:/usr/local/bin/entrypoint.sh:ro
  entrypoint: ["/usr/local/bin/entrypoint.sh"]

services:
  dask-scheduler:
    << : *dask_opts
    command: [ "dask-scheduler", "--port", "8786", "--interface", "eth0", "--no-dashboard", "--no-show" ]
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-Lf", "http://dask-scheduler:8787/health"]
      interval: 10s
      timeout: 10s
      retries: 5
      start_period: 10s
  dask-cuda-worker:
    << : *dask_opts
    restart: unless-stopped
    command: [ "dask-cuda-worker", "--interface", "eth0", "--no-dashboard", "--dashboard-address", "dask-cuda-worker:8787", "dask-scheduler:8786"]
    depends_on:
      - dask-scheduler
  my-app:
    << : *dask_opts
    command: [ "tail", "-f", "/usr/local/bin/entrypoint.sh"]
    entrypoint: ["/usr/local/bin/entrypoint.sh"]

entrypoint.sh (with chmod +x):

#!/bin/bash
set -e
source activate rapids
exec "$@"

Sample usage:

docker-compose up -d
docker-compose logs -f -t -tail=100 # ctlr-c when up
docker-compose run my-app python -c "from dask.distributed import Client; client=Client('dask-scheduler:8786'); print('Connected, shared datasets: ', client.list_datasets())"

[pentschev]

Is there anything more to be done here or can we close this @lmeyerov ?

[lmeyerov]

AFAICT, based on the test 4 days ago, we're good. Looking forward to 0.18, will update my reference docker-compose.yml above with dask-cuda-worker health checking once that's available.

(For folks hitting this via search: even w/ a docker-addressable dask-cuda-worker health endpoint in 0.18, we still must do external checks via client.run(...) to do cudf/bsql/...-specific shared worker resource checks that are inappropriate for vanilla dask-cuda-worker.)