Dask-cuda: FileExistsError: File exists: 'storage'

[drabastomek]

I have been seeing these quite often but the issues seem to be transient in nature -- not all nodes experience this.

I'm running on Azure ML Compute, 20 nodes with 4 V100s with 16GB. Some of the nodes report this:

Starting the daemon thread to refresh tokens in background for process with pid = 224
Entering Run History Context Manager.
Preparing to call script [ start_worker.py ] with arguments: ['--scheduler_ip_port=172.18.0.4:8786', '--use_gpu=True', '--n_gpus_per_node=4']
After variable expansion, calling script [ start_worker.py ] with arguments: ['--scheduler_ip_port=172.18.0.4:8786', '--use_gpu=True', '--n_gpus_per_node=4']

- scheduler is  172.18.0.4:8786
- args:  Namespace(n_gpus_per_node='4', scheduler_ip_port='172.18.0.4:8786', use_gpu='True')
- unparsed:  []
- my rank is  0
- my ip is:  172.18.0.21
- n_gpus_per_node:  4
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.18.0.21:34865'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.18.0.21:37105'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.18.0.21:46857'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.18.0.21:38133'
distributed.preloading - INFO - Import preload module: dask_cuda.initialize
Process Dask Worker process (from Nanny):
Process Dask Worker process (from Nanny):
Traceback (most recent call last):
  File "/opt/conda/envs/rapids/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/opt/conda/envs/rapids/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/process.py", line 191, in _run
    target(*args, **kwargs)
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/nanny.py", line 666, in _run
    worker = Worker(**worker_kwargs)
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/worker.py", line 543, in __init__
    self.data = data[0](**data[1])
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/dask_cuda/device_host_file.py", line 141, in __init__
    self.disk_func = Func(serialize_bytelist, deserialize_bytes, File(path))
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/zict/file.py", line 63, in __init__
    os.mkdir(self.directory)
FileExistsError: [Errno 17] File exists: 'storage'
Traceback (most recent call last):
  File "/opt/conda/envs/rapids/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/opt/conda/envs/rapids/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/process.py", line 191, in _run
    target(*args, **kwargs)
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/nanny.py", line 666, in _run
    worker = Worker(**worker_kwargs)
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/worker.py", line 543, in __init__
    self.data = data[0](**data[1])
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/dask_cuda/device_host_file.py", line 141, in __init__
    self.disk_func = Func(serialize_bytelist, deserialize_bytes, File(path))
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/zict/file.py", line 63, in __init__
    os.mkdir(self.directory)
FileExistsError: [Errno 17] File exists: 'storage'
distributed.preloading - INFO - Import preload module: dask_cuda.initialize
distributed.nanny - INFO - Worker process 248 exited with status 1
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x7fb1ebb6c490>>, <Task finished coro=<Nanny._on_exit() done, defined at /opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/nanny.py:387> exception=TypeError('addresses should be strings or tuples, got None')>)
Traceback (most recent call last):
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/tornado/ioloop.py", line 743, in _run_callback
    ret = callback()
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
    future.result()
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/nanny.py", line 390, in _on_exit
    await self.scheduler.unregister(address=self.worker_address)
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/core.py", line 757, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/core.py", line 556, in send_recv
    raise exc.with_traceback(tb)
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/core.py", line 408, in handle_comm
    result = handler(comm, **msg)
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/scheduler.py", line 2135, in remove_worker
    address = self.coerce_address(address)
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/scheduler.py", line 4844, in coerce_address
    raise TypeError("addresses should be strings or tuples, got %r" % (addr,))
TypeError: addresses should be strings or tuples, got None
distributed.nanny - INFO - Closing Nanny at 'tcp://172.18.0.21:46857'
distributed.worker - INFO -       Start worker at:    tcp://172.18.0.21:43685
distributed.worker - INFO -          Listening to:    tcp://172.18.0.21:43685
distributed.worker - INFO -          dashboard at:          172.18.0.21:34483
distributed.worker - INFO - Waiting to connect to:      tcp://172.18.0.4:8786
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /mnt/batch/tasks/shared/LS_root/jobs/todrabas_testing/azureml/todrabas-dask-benchmarks_1582564385_9b31e947/mounts/workspaceblobstore/azureml/todrabas-dask-benchmarks_1582564385_9b31e947/worker-sfamnyrv
distributed.worker - INFO - Starting Worker plugin <dask_cuda.utils.RMMPool object at 0x7f9a30f10710>-52a534a5-1f63-44f8-883b-29f22e1cc9ec
distributed.worker - INFO - Starting Worker plugin <dask_cuda.utils.CPUAffinity object at 0x7f9a30f89-8abaf39c-62f2-4dc6-94ed-a23d632db081
distributed.worker - INFO - -------------------------------------------------
distributed.nanny - INFO - Worker process 251 exited with status 1
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x7fb1ebb6c490>>, <Task finished coro=<Nanny._on_exit() done, defined at /opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/nanny.py:387> exception=TypeError('addresses should be strings or tuples, got None')>)
Traceback (most recent call last):
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/tornado/ioloop.py", line 743, in _run_callback
    ret = callback()
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
    future.result()
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/nanny.py", line 390, in _on_exit
    await self.scheduler.unregister(address=self.worker_address)
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/core.py", line 757, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/core.py", line 556, in send_recv
    raise exc.with_traceback(tb)
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/core.py", line 408, in handle_comm
    result = handler(comm, **msg)
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/scheduler.py", line 2135, in remove_worker
    address = self.coerce_address(address)
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/scheduler.py", line 4844, in coerce_address
    raise TypeError("addresses should be strings or tuples, got %r" % (addr,))
TypeError: addresses should be strings or tuples, got None
distributed.nanny - INFO - Closing Nanny at 'tcp://172.18.0.21:38133'
distributed.worker - INFO -         Registered to:      tcp://172.18.0.4:8786
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.worker - INFO -       Start worker at:    tcp://172.18.0.21:34807
distributed.worker - INFO -          Listening to:    tcp://172.18.0.21:34807
distributed.worker - INFO -          dashboard at:          172.18.0.21:40565
distributed.worker - INFO - Waiting to connect to:      tcp://172.18.0.4:8786
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /mnt/batch/tasks/shared/LS_root/jobs/todrabas_testing/azureml/todrabas-dask-benchmarks_1582564385_9b31e947/mounts/workspaceblobstore/azureml/todrabas-dask-benchmarks_1582564385_9b31e947/worker-9q2p2egq
distributed.worker - INFO - Starting Worker plugin <dask_cuda.utils.RMMPool object at 0x7f611473a410>-21b3a3b2-d964-42eb-a350-39ba21155f8b
distributed.worker - INFO - Starting Worker plugin <dask_cuda.utils.CPUAffinity object at 0x7f6114736-5dd82753-e3d4-4fc1-8d07-3c55f3b926df
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -         Registered to:      tcp://172.18.0.4:8786
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection

These nodes can only register 2 workers instead of 4. Not sure how to mitigate this... Any suggestions welcome.

[drabastomek]

cc: @mrocklin @jacobtomlinson @kkraus14 @randerzander

[drabastomek]

Code I use to start each worker

# Imports 
import os
import sys
import uuid
import time
import socket
import argparse
import threading
import subprocess

from mpi4py import MPI
from azureml.core import Run
from notebook.notebookapp import list_running_servers

def flush(proc, proc_log):
    while True:
        proc_out = proc.stdout.readline()
        if proc_out == '' and proc.poll() is not None:
            proc_log.close()
            break
        elif proc_out:
            sys.stdout.write(proc_out)
            proc_log.write(proc_out)
            proc_log.flush()

if __name__ == '__main__':
    comm = MPI.COMM_WORLD
    rank = comm.Get_rank()

    ### PARSE ARGUMENTS
    parser = argparse.ArgumentParser()
    parser.add_argument("--scheduler_ip_port", default=None)
    parser.add_argument("--use_gpu",           default=False)
    parser.add_argument("--n_gpus_per_node",   default=0)

    args, unparsed = parser.parse_known_args()

    ### CONFIGURE GPU RUN
    GPU_run = eval(args.use_gpu)

    if GPU_run:
        n_gpus_per_node = eval(args.n_gpus_per_node)

    attempt = 0
    ip = None

    while ip == None:
        try:
            ip = socket.gethostbyname(socket.gethostname())
        except socket.timeout:
            time.sleep(1)
            print(f'attempt: {attempt}, ip: {ip}')
            # pass
        attempt += 1

    print("- scheduler is ", args.scheduler_ip_port)
    print("- args: ", args)
    print("- unparsed: ", unparsed)
    print("- my rank is ", rank)
    print("- my ip is: ", ip)
    print("- n_gpus_per_node: ", n_gpus_per_node)

    if not GPU_run:
        cmd = "dask-worker " + args.scheduler_ip_port 
    else:
        os.environ["CUDA_VISIBLE_DEVICES"] = str(list(range(n_gpus_per_node))).strip("[]")
        cmd = "dask-cuda-worker " + args.scheduler_ip_port + " --memory-limit 0"

    worker_log = open("worker_{rank}_log.txt".format(rank=rank), "w")
    worker_proc = subprocess.Popen(cmd.split(), universal_newlines=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

    flush(worker_proc, worker_log)

[mrocklin]

Can you update zict and see if some of that goes away?

On Mon, Feb 24, 2020 at 9:38 AM Tomek Drabas [e-mail]
wrote:

cc: @mrocklin https://github.com/mrocklin @jacobtomlinson
https://github.com/jacobtomlinson @kkraus14
https://github.com/kkraus14 @randerzander
https://github.com/randerzander

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rapidsai/dask-cuda/issues/243?email_source=notifications&email_token=AACKZTBYI6RES3T2NTKIIUDREQA3BA5CNFSM4K2M5O72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMY2S5A#issuecomment-590457204,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACKZTGF2P7XFZOOKPUEEMLREQA3BANCNFSM4K2M5O7Q
.

[drabastomek]

Can you update zict and see if some of that goes away?

Any specific version should I target, or simply pip install --upgrade zict?

[drabastomek]

Confirming: updated to the latest zict didn't help the situation: still have plenty of File exists: 'storage' errors and the workers don't register with the scheduler.

[drabastomek]

Check here: https://github.com/rapidsai/dask-cuda/issues/243#issuecomment-590458884 :)

[drabastomek]

In general, this is a part of a greater project: https://github.com/drabastomek/dask-cloudprovider/blob/master/dask_cloudprovider/providers/azure/azureml.py

[quasiben]

what version of zict do you have ? perhaps install from Github:

pip install git+git://github.com/dask/zict.git

[drabastomek]

The docker env used in these runs is as follows (https://hub.docker.com/r/todrabas/aml_rapids)

FROM rapidsai/rapidsai-nightly:cuda10.1-runtime-ubuntu18.04-py3.7
MAINTAINER [email protected]

USER root:root

ENV LANG=C.UTF-8 LC_ALL=C.UTF-8
ENV DEBIAN_FRONTEND noninteractive
ENV LD_LIBRARY_PATH "/usr/local/cuda/extras/CUPTI/lib64:${LD_LIBRARY_PATH}"

# Install Common Dependencies
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
    # SSH and RDMA
    libmlx4-1 \
    libmlx5-1 \
    librdmacm1 \
    libibverbs1 \
    libmthca1 \
    libdapl2 \
    dapl2-utils \
    openssh-client \
    openssh-server \
    iproute2 && \
    # Others
    apt-get install -y \
    build-essential \
    bzip2 \
    wget \
    cpio && \
    apt-get clean -y && \
    rm -rf /var/lib/apt/lists/*


# Intel MPI installation
ENV INTEL_MPI_VERSION 2018.3.222
ENV PATH $PATH:/opt/intel/compilers_and_libraries/linux/mpi/bin64
RUN cd /tmp && \
    wget -q "http://registrationcenter-download.intel.com/akdlm/irc_nas/tec/13063/l_mpi_${INTEL_MPI_VERSION}.tgz" && \
    tar zxvf l_mpi_${INTEL_MPI_VERSION}.tgz && \
    sed -i -e 's/^ACCEPT_EULA=decline/ACCEPT_EULA=accept/g' /tmp/l_mpi_${INTEL_MPI_VERSION}/silent.cfg && \
    cd /tmp/l_mpi_${INTEL_MPI_VERSION} && \
    ./install.sh -s silent.cfg --arch=intel64 && \
    cd / && \
    rm -rf /tmp/l_mpi_${INTEL_MPI_VERSION}* && \
    rm -rf /opt/intel/compilers_and_libraries_${INTEL_MPI_VERSION}/linux/mpi/intel64/lib/debug* && \
    echo "source /opt/intel/compilers_and_libraries_${INTEL_MPI_VERSION}/linux/mpi/intel64/bin/mpivars.sh" >> ~/.bashrc

RUN source activate rapids && \
    pip install --upgrade azureml-sdk && \
    conda install -n rapids matplotlib && \
    pip install cugraph-cuda100 && \
    pip install scapy && \
    pip install mpi4py && \
    pip install --upgrade fsspec adlfs && \
    pip install --upgrade pyshp && \
    pip install --upgrade dask-lightgbm && \
    pip install --upgrade xgboost && \
    pip install --upgrade zict

I'll try installing the zict from GitHub and will report back.

[pentschev]

I think the reason this isn't working is that the fix for it was never merged into zict: https://github.com/dask/zict/pull/33. This issue has previously been reported in https://github.com/rapidsai/dask-cuda/issues/232.

I just pinged people in zict to see if we can get it merged asap, in the meantime I would suggest trying the fix in https://github.com/dask/zict/pull/33/files, I think it will allow you to move past this issue @drabastomek .

[pentschev]

The fix was just merged, that said I would suggest trying to install it from github, as per https://github.com/rapidsai/dask-cuda/issues/243#issuecomment-590472837.

[drabastomek]

Getting zict from GitHub solved the issue! Woot! Thanks for such a quick turnaround! Case closed!

[pentschev]

Glad it worked @drabastomek !