Dask-cuda: [BUG] LocalCUDACluster -- NoneType has no attribute 'process'

[drabastomek]

Describe the bug
When starting a LocalCUDACluster it throws tornado.application error.

Full stack:

tornado.application - ERROR - Exception in callback <bound method Nanny.memory_monitor of <Nanny: None, threads: 1>>
Traceback (most recent call last):
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/tornado/ioloop.py", line 907, in _run
    return self.callback()
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/nanny.py", line 414, in memory_monitor
    process = self.process.process
AttributeError: 'NoneType' object has no attribute 'process'

Steps/Code to reproduce bug

from dask_cuda import LocalCUDACluster
cluster = LocalCUDACluster()

Expected behavior
LocalCUDACluster() should return the object without error.

Environment overview (please complete the following information)

• Environment location: Docker
• Method of cuDF install: Docker
  
  
  
  • If method of install is [Docker], provide docker pull & docker run commands used
  
  

docker pull rapidsai/rapidsai-nightly:cuda11.0-runtime-ubuntu18.04
docker run --gpus all --rm -it -p 8888:8888 -p 8787:8787 -p 8786:8786 \
    rapidsai/rapidsai-nightly:cuda11.0-runtime-ubuntu18.04

[drabastomek]

Just checked and the issues is still present even if I use CUDA 10.2.

[kkraus14]

Moving this to the dask_cuda repo.

[quasiben]

Hmm, I just tried this and I'm not able to reproduce:

In [1]: from dask_cuda import LocalCUDACluster
   ...: cluster = LocalCUDACluster()

Were you loading from ipython/notebook/file ? If file, I think we still need to do something like the following:

```python
from dask_cuda import LocalCUDACluster

if __name__ == "__main__":
cluster = LocalCUDACluster()
````

[quasiben]

same with the repro provided:

Digest: sha256:63f8b21ad4c0910bf33b6988a6f00fce4387796bdefe7340c003f1ea9b98a429
Status: Downloaded newer image for rapidsai/rapidsai-nightly:cuda11.0-runtime-ubuntu18.04
docker.io/rapidsai/rapidsai-nightly:cuda11.0-runtime-ubuntu18.04
(20200918) quasiben@DirtyHorseZ:~/Github/nvtx-examples$ docker run --gpus all --rm -it -p 8888:8888 -p 8787:8787 -p 8786:8786 \
>     rapidsai/rapidsai-nightly:cuda11.0-runtime-ubuntu18.04
A JupyterLab server has been started!
To access it, visit http://localhost:8888 on your host machine.
Ensure the following arguments were added to "docker run" to expose the JupyterLab server to your host machine:
      -p 8888:8888 -p 8787:8787 -p 8786:8786
Make local folders visible by bind mounting to /rapids/notebooks/host
rapids@9a5b3f103832:/rapids/notebooks$ ipython
Python 3.7.8 | packaged by conda-forge | (default, Jul 31 2020, 02:25:08)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.15.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from dask_cuda import LocalCUDACluster

In [2]: cluster = LocalCUDACluster()
/opt/conda/envs/rapids/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject
  return f(*args, **kwds)

In [3]: cluster
Out[3]: LocalCUDACluster('tcp://127.0.0.1:34735', workers=2, threads=2, memory=83.78 GB)

In [4]:

[drabastomek]

@quasiben did a bit more debugging. Seems like this issue is related to permissions on host folders mounted within a running container.

I can repro the following:

1. If I create a new notebook (in the root folder of the container e.g. rapids/notebooks, and then do

from dask_cuda import LocalCUDACluster
cluster = LocalCUDACluster()

all works fine.

1. I can open notebook that already exists in a mounted location (e.g. rapids/notebooks/host) but then get a _wall of red_ messages like I pasted at the top.
2. I cannot create new notebooks in the host folder anymore -- get 403 Forbidden error.
3. Moving the notebook from host to the root folder of the container solves the issue.

Can you try reproing the above? Also, do you know if anything had changed between 0.15 and 0.16 in terms of how the containers are being defined? I could run this notebook in the 0.15 and didn't get the 403 error.

[drabastomek]

OK, I think I found the 'culprit' -- I think the default user in the container is now rapids and used to be root. At least all the notebooks and data I was saving back to host were owned by root:root.

[quasiben]

Probably running as rapids make more sense -- glad you found the issue. If it was/is a perms issue you could also set local_directory to /tmp:

from dask_cuda import LocalCUDACluster
cluster = LocalCUDACluster(local_directory='/tmp')

[drabastomek]

A 'less-than-ideal' workaround I have now is to chmod 0777 to the host folder and that solves the issue. But definitely not sure, moving forward, how that's going to play.

[quasiben]

I tried with both notebook which come pre-baked in the image and new ones and I'm still not able to reproduce. But it is disconcerting that the notebooks are all owned by root rather than RAPIDS. @ajschmidt8 is this something we could change ?

@drabastomek for reference the images are built from this repo: https://github.com/rapidsai/docker

[drabastomek]

@quasiben Did you mount a folder within the container with the -v switch? And then try to create a notebook in the mounted host folder and/or run an existing notebook from the host folder that starts a LocalCUDACluster?

What (for now) worked for me was to give 0777 permissions to the host folder or copy the notebook to the container file system.

AFAIK docker defaults to root and I think one of the reasons for that choice is this. I may be wrong though.

EDIT: Looks like these changes were introduced with this commit https://github.com/rapidsai/docker/commit/ca28f05a3b1b428f73d75f92580829617b1d5322#diff-5c18ef72771564b7f43c497dc507aeab

[ajschmidt8]

I tried with both notebook which come pre-baked in the image and new ones and I'm still not able to reproduce. But it is disconcerting that the notebooks are all owned by root rather than RAPIDS. @ajschmidt8 is this something we could change ?

@drabastomek for reference the images are built from this repo: https://github.com/rapidsai/docker

@quasiben, @drabastomek, it does seem like the rapids user is causing some issues here. Ultimately, we added the rapids user along with the -e HOST_USER_ID=$(id -u $USER) runtime flag so that users could use volume mounts without files on the host machine being owned by root. However, it seems that the new user has been causing some permissions errors for users that we didn't anticipate. I opened up a rapidsai/docker repo issue here https://github.com/rapidsai/docker/issues/184 and will be looking into it over the next few days. Thanks for bringing this to my attention.

[drabastomek]

@ajschmidt8 Thanks! I think we may close this bug now as it's not really a LocalCUDACluster issue but rather the permissions problem.

[quasiben]

Thanks all!

[ajschmidt8]

@drabastomek, for awareness, we've removed the rapids user in the PR below. There were a lot of small issues that were piling up with the rapids user that we didn't feel we could reliably address in time for next week's 0.16 release. We may bring back the rapids user in the future, but for now we'll go back to using the default root user. Our new nightlies with the root user should be published by the end of today.

https://github.com/rapidsai/docker/pull/186