Dask-cuda: [BUG] dask-cuda(0.7.0) is broken with latest distributed(2.0.1)

[pradghos]

dask-cuda is not compatible with latest distributed

Log:-

(base) builder@5ab0d442c86f:~$ dask-cuda-worker
Traceback (most recent call last):
  File "/opt/anaconda3/bin/dask-cuda-worker", line 11, in <module>
    load_entry_point('dask-cuda==0.7.0', 'console_scripts', 'dask-cuda-worker')()
  File "/opt/anaconda3/lib/python3.6/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/opt/anaconda3/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2843, in load_entry_point
    return ep.load()
  File "/opt/anaconda3/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2434, in load
    return self.resolve()
  File "/opt/anaconda3/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2440, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/opt/anaconda3/lib/python3.6/site-packages/dask_cuda/dask_cuda_worker.py", line 12, in <module>
    from distributed.worker import _ncores
ImportError: cannot import name '_ncores'
(base) builder@5ab0d442c86f:~$

Conda package :-

(base) builder@5ab0d442c86f:~$ conda list | grep distributed
distributed               2.0.1                      py_0
(base) builder@5ab0d442c86f:~$

(base) builder@5ab0d442c86f:~$ conda list | grep dask
dask                      2.0.0                      py_0
dask-core                 2.0.0                      py_0
dask-cuda                 0.7.0           py36_489.g8bce79e    
dask-cudf                 0.9.0a          py36_493.g2167909    
(base) builder@5ab0d442c86f:~$

[pradghos]

Probably same issue is there in latest dask-cuda also -
https://github.com/rapidsai/dask-cuda/blob/88caf22f5f6373f938fe51c939a9bbc7cbb86710/dask_cuda/dask_cuda_worker.py#L20
https://github.com/rapidsai/dask-cuda/blob/88caf22f5f6373f938fe51c939a9bbc7cbb86710/dask_cuda/dask_cuda_worker.py#L17

[pentschev]

Yes, they are incompatible. You will either need to upgrade to dask-cuda 0.8 or downgrade dask and distributed packages.

[pentschev]

Probably same issue is there in latest dask-cuda also -
https://github.com/rapidsai/dask-cuda/blob/88caf22f5f6373f938fe51c939a9bbc7cbb86710/dask_cuda/dask_cuda_worker.py#L20

https://github.com/rapidsai/dask-cuda/blob/88caf22f5f6373f938fe51c939a9bbc7cbb86710/dask_cuda/dask_cuda_worker.py#L17

It's fixed in branch-0.8, I think those changes aren't yet reflect on 0.9/nightly.

[pradghos]

Thanks for looking @pentschev ! is below check mandatory for dask-cuda 0.8
https://github.com/rapidsai/dask-cuda/blob/88caf22f5f6373f938fe51c939a9bbc7cbb86710/conda/recipes/dask-cuda/meta.yaml#L27

[pentschev]

If you're using CuPy with Dask, yes, which is the major (granted, not the only one) reason for dask-cuda. Are you trying to run on an older NumPy version?

[pradghos]

@pentschev : Thanks for the information . Just trying to understand the reason of the check / dependency.

[pentschev]

It's basically because NumPy 1.16.0 is the first release that implements NEP-18. The functionality here depends heavily on that, even though some minor things may work without it.

[pradghos]

@pentschev : One query I have observed in the latest conda build script is having -
https://github.com/rapidsai/dask-cuda/blob/88caf22f5f6373f938fe51c939a9bbc7cbb86710/conda/recipes/dask-cuda/build.sh#L3
Is there any specific reason to add the above here. I see - conda meta.yaml is already having it -
https://github.com/rapidsai/dask-cuda/blob/88caf22f5f6373f938fe51c939a9bbc7cbb86710/conda/recipes/dask-cuda/meta.yaml#L26

Thank you !

[pentschev]

Sorry, I completely missed your comment here @pradghos, and you're right that the version is incorrect, thanks for reporting it!

I've opened a PR to fix this is 0.8, we should release a conda package for it as well.

[ChethanUK]

PR to fix this is 0.8

Even in 0.8 facing this probelm:

conda list

cudf                      0.8.0                    py37_0    rapidsai
cuml                      0.8.0           cuda10.0_py37_0    rapidsai
dask-core                 2.2.0                      py_0    conda-forge
dask-cuda                 0.8.0                    py37_0    rapidsai
dask-cudf                 0.8.0                    py37_0    rapidsai
dask-cuml                 0.8.0                    py37_0    rapidsai
dask-xgboost              0.2.0.dev28      cuda10.0py37_0    rapidsai/label/xgboost
decorator                 4.4.0                      py_0    conda-forge
dill                      0.3.0                    py37_0    conda-forge
distributed               2.2.0                      py_1    conda-forge
numba                     0.45.1           py37hb3f55d8_0    conda-forge
numpy                     1.17.0           py37h95a1406_0    conda-forge

Error:
cat /var/log/dask-cuda-workers.log

Traceback (most recent call last):
  File "/opt/conda/anaconda/envs/RAPIDS/bin/dask-cuda-worker", line 11, in <module>
    load_entry_point('dask-cudf==0.7.0', 'console_scripts', 'dask-cuda-worker')()
  File "/opt/conda/anaconda/envs/RAPIDS/lib/python3.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/opt/conda/anaconda/envs/RAPIDS/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2843, in load_entry_point
    return ep.load()
  File "/opt/conda/anaconda/envs/RAPIDS/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2434, in load
    return self.resolve()
  File "/opt/conda/anaconda/envs/RAPIDS/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2440, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/opt/conda/anaconda/envs/RAPIDS/lib/python3.7/site-packages/dask_cuda/dask_cuda_worker.py", line 12, in <module>
    from distributed.worker import _ncores
ImportError: cannot import name '_ncores' from 'distributed.worker' (/opt/conda/anaconda/envs/RAPIDS/lib/python3.7/site-packages/distributed/worker.py)

[ChethanUK]

conda package

Does the latest conda package solve the above problem?

But I can see that, it's still distributed 1.28.0 :
https://github.com/rapidsai/dask-cuda/blob/0c5f93dadf76d795d092c81508470841a4e50294/conda/recipes/dask-cuda/meta.yaml#L26

[pentschev]

@ChethanUK can you ensure the machine where the traceback came from is actually running dask-cuda 0.8? I think that is from 0.7, the import _ncores was removed is 0.8. If you're running a cluster, it could be that the node running the worker has a different version than the one where you pasted conda list from.

[ChethanUK]

actually running dask-cuda 0.8?

First I installed dask-cuda 0.7 on all then Created new env with 0.8 in all, but ya let me check out

[ChethanUK]

Solved it, had to delete old env.

new env

Deleted old env in all workers and then launched dask schedulers and workers in new env where dask-cuda 0.8 is installed.

import _ncores was removed is 0.8

Ya, it was from 0.7 only...

[pentschev]

Since this is related to an older version of dask-cuda which isn't supported anymore I'm closing this issue. If anything similar happens to an up-to-date version, please feel free to file a new issue.