Dask-cuda: [bug] dask cuda worker died for race condition

[pradghos]

dask cuda worker sometimes failing for race condition to create its storage space -

(gdf) [pradghos@host  dask_start]$ dask-cuda-worker "tcp://9.3.89.66:8786"
distributed.dashboard.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: pip install jupyter-server-proxy
distributed.nanny - INFO -         Start Nanny at: 'tcp://9.3.89.135:39097'
distributed.nanny - INFO -         Start Nanny at: 'tcp://9.3.89.135:35201'
distributed.nanny - INFO -         Start Nanny at: 'tcp://9.3.89.135:38867'
distributed.nanny - INFO -         Start Nanny at: 'tcp://9.3.89.135:32983'
distributed.dashboard.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: pip install jupyter-server-proxy
distributed.dashboard.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: pip install jupyter-server-proxy
distributed.dashboard.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: pip install jupyter-server-proxy
distributed.dashboard.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: pip install jupyter-server-proxy
Process Dask Worker process (from Nanny):
distributed.preloading - INFO - Import preload module: dask_cuda.initialize
Traceback (most recent call last):
  File "/mnt/pai/home/pradghos/anaconda3/envs/gdf/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/mnt/pai/home/pradghos/anaconda3/envs/gdf/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/mnt/pai/home/pradghos/anaconda3/envs/gdf/lib/python3.6/site-packages/distributed/process.py", line 191, in _run
    target(*args, **kwargs)
  File "/mnt/pai/home/pradghos/anaconda3/envs/gdf/lib/python3.6/site-packages/distributed/nanny.py", line 674, in _run
    worker = Worker(**worker_kwargs)
  File "/mnt/pai/home/pradghos/anaconda3/envs/gdf/lib/python3.6/site-packages/distributed/worker.py", line 542, in __init__
    self.data = data[0](**data[1])
  File "/mnt/pai/home/pradghos/anaconda3/envs/gdf/lib/python3.6/site-packages/dask_cuda/device_host_file.py", line 124, in __init__
    self.disk_func = Func(serialize_bytelist, deserialize_bytes, File(path))
  File "/mnt/pai/home/pradghos/anaconda3/envs/gdf/lib/python3.6/site-packages/zict/file.py", line 63, in __init__
    os.mkdir(self.directory)
FileExistsError: [Errno 17] File exists: 'storage'
distributed.nanny - INFO - Worker process 19311 exited with status 1
distributed.preloading - INFO - Import preload module: dask_cuda.initialize
distributed.preloading - INFO - Import preload module: dask_cuda.initialize
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x7ffd7728a198>>, <Task finished coro=<Nanny._on_exit() done, defined at /mnt/pai/home/pradghos/anaconda3/envs/gdf/lib/python3.6/site-packages/distributed/nanny.py:396> exception=TypeError('addresses should be strings or tuples, got None',)>)

However, other three worker started and serving the request - because of this issue - We might have observed less worker running than requested !

example - below user has started 8 worker but one worker died because of the race condition - and rest are serving the request -

    Scheduler: tcp://9.3.89.66:8786
    Dashboard: http://9.3.89.66:8787/status 
Cluster
    Workers: 7
    Cores: 7
    Memory: 350.00 GB

[pradghos]

Reason behind the race condition -
file - python3.6/site-packages/zict/file.py

    def __init__(self, directory, mode='a'):
        self.directory = directory
        self.mode = mode
        self._keys = set()
        if not os.path.exists(self.directory):
            os.mkdir(self.directory)                  =====> race condition is here. 
        else:
            for n in os.listdir(self.directory):
                self._keys.add(_unsafe_key(n))

    def __str__(self):
        return '<File: %s, mode="%s", %d elements>' % (self.directory, self.mode, len(self))

[pentschev]

Thanks for the report @pradghos . I think this is something we could fix in zict, do you agree with that @mrocklin ?

Maybe a way to handle this would be a try / except FileExistsError instead of the current if / else statement?

[mrocklin]

Sounds good to me. The makedirs solution seems cleanest to me.

On Thu, Jan 30, 2020 at 3:05 AM Peter Andreas Entschev <
[email protected]> wrote:

Thanks for the report @pradghos https://github.com/pradghos . I think
this is something we could fix in zict, do you agree with that @mrocklin
https://github.com/mrocklin ?

Maybe a way to handle this would be a try / except FileExistsError
instead of the current if / else statement?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rapidsai/dask-cuda/issues/232?email_source=notifications&email_token=AACKZTFQS6EJPF2CROKI3ADRAKX7JA5CNFSM4KNTJAJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKKS6RY#issuecomment-580202311,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACKZTGYAIDVOQSWCNIIAKTRAKX7JANCNFSM4KNTJAJA
.

[pentschev]

@pradghos would you be interested in filing a PR to resolve that in zict?

[pentschev]

@mrocklin I was just checking makedirs docs and it seems to me like we would have the same error, unless I misinterpreted what you meant.

[mrocklin]

I was referring to the second option presented in the opening comment with
the exists_ok=True keyword.

I don't mean to apply any sort of constraints here though. It was just an
idle comment. Please do whatever you all think is best.

On Thu, Jan 30, 2020 at 7:10 AM Peter Andreas Entschev <
[email protected]> wrote:

@mrocklin https://github.com/mrocklin I was just checking makedirs docs
https://docs.python.org/2/library/os.html#os.makedirs and it seems to
me like we would have the same error, unless I misinterpreted what you
meant.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rapidsai/dask-cuda/issues/232?email_source=notifications&email_token=AACKZTFTMEA4WO67GFNUNJDRALUVRA5CNFSM4KNTJAJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKLKPLQ#issuecomment-580298670,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACKZTEXVNIT6JBBD3XLGQTRALUVRANCNFSM4KNTJAJA
.

[pentschev]

Ok, you're replying from email and I'm doing that from the GH interface, I didn't see those because they were edited out. @pradghos I agree with @mrocklin that you had good suggestions for a fix, not sure why you edited them out. :)

If you're up to filing a PR to resolve the issue @pradghos , I suggest then you then use your preferred method. If you can't file a PR then I'll do that tomorrow or Monday.

[pradghos]

Ok, you're replying from email and I'm doing that from the GH interface, I didn't see those because they were edited out. @pradghos I agree with @mrocklin that you had good suggestions for a fix, not sure why you edited them out. :)

If you're up to filing a PR to resolve the issue @pradghos , I suggest then you then use your preferred method. If you can't file a PR then I'll do that tomorrow or Monday.

I was not very sure about the fix initially suggested by me :-) ;
.. But really appreciate for looking into it @pentschev @mrocklin - Sure, I am planning to create PR with the fix in zict component on Monday. Please let me know if it is fine. Thank you !

[pentschev]

It sounds great, good to see you contribute @pradghos !

[pradghos]

os.makedirs(self.directory, exist_ok=True) will not work for python 2.7 and CI is failing for the same reason ! I think we have to go another approach by handling the exception - EEXIST using try catch.

[pentschev]

@pradghos your zict PR has been merged, thanks for the work there! I think it should solve the issue here, so I'll tentatively close it here, but feel free to reopen should you encounter any related issues.