Dask-cuda: CI Failures

[pentschev]

There are currently 2 tests failing in PRs, see https://gpuci.gpuopenanalytics.com/job/rapidsai/job/gpuci-v0.20/job/dask-cuda/job/prb/job/dask-cuda-gpu-test/CUDA=11.0,GPU_LABEL=gpu-a100,OS=ubuntu16.04,PYTHON=3.7/1/ . This is failing for both branches 0.19 (https://github.com/rapidsai/dask-cuda/pull/562, https://github.com/rapidsai/dask-cuda/pull/563) and 0.20 (https://github.com/rapidsai/dask-cuda/pull/523).

I've been trying to reproduce the issues locally but haven't been successful so far.

[pentschev]

The first error in test_dask_cuda_worker.py::test_cuda_visible_devices_and_memory_limit_and_nthreads is:

tornado.application - ERROR - Exception in callback <bound method Nanny.memory_monitor of <Nanny: None, threads: 4>>
Traceback (most recent call last):
  File "/datasets/pentschev/miniconda3/envs/rn-101-0.19.210408/lib/python3.8/site-packages/tornado/ioloop.py", line 905, in _run
    return self.callback()
  File "/datasets/pentschev/miniconda3/envs/rn-101-0.19.210408/lib/python3.8/site-packages/distributed/nanny.py", line 417, in memory_monitor
    process = self.process.process
AttributeError: 'NoneType' object has no attribute 'process'

This goes away if I revert https://github.com/dask/distributed/pull/4661 , could you check that @charlesbluca since you're the author?

The second error I still haven't been able to reproduce.

[charlesbluca]

From that error, it looks like the worker is still seen as running even after the WorkerProcess is closed; a shot in the dark, but could this have something to do with how dask/distributed#4661 calls nvml.real_time() directly, rather than offloading the function like what is done for the WorkerState?

https://github.com/dask/distributed/blob/d70314fb8c542ad3c46de724b93271dd98844a4b/distributed/worker.py#L3786-L3791

[charlesbluca]

@quasiben pointed out that this error could be occurring because dask/distributed#4661 makes it so Distributed initializes the CUDA context before dask-cuda, so I made some changes to prevent this:

• https://github.com/charlesbluca/distributed/tree/fix-dask-cuda-ci

I'm going to open a PR, but @pentschev could you verify that this branch fixes the test? It works locally for me but I'd just like to make sure 🙂

[charlesbluca]

I'm not able to reproduce the other error either, but it's definitely related to dask/distributed#4661, as the stack trace shows the problem comes from the attempt at initializing CUDA context in the system monitor:

...
/opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/core.py:160: in __init__
    self.monitor = SystemMonitor()
/opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/system_monitor.py:43: in __init__
    gpu_extra = nvml.one_time()
/opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/diagnostics/nvml.py:45: in one_time
    h = _pynvml_handles()
/opt/conda/envs/rapids/lib/python3.7/site-packages/distributed/diagnostics/nvml.py:31: in _pynvml_handles
    return pynvml.nvmlDeviceGetHandleByIndex(gpu_idx)
/opt/conda/envs/rapids/lib/python3.7/site-packages/pynvml/nvml.py:922: in nvmlDeviceGetHandleByIndex
    check_return(ret)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

ret = 2

    def check_return(ret):
        if (ret != NVML_SUCCESS):
>           raise NVMLError(ret)
E           pynvml.nvml.NVMLError_InvalidArgument: Invalid Argument

/opt/conda/envs/rapids/lib/python3.7/site-packages/pynvml/nvml.py:366: NVMLError_InvalidArgument

Hoping that this might be another side effect of the CUDA context being initialized in the wrong place, but if not, we can follow up on fixing this either here or in dask/distributed#4688.

[pentschev]

It does seem to fix locally for me as well, thanks @charlesbluca .

However, I also see the Invalid Argument error without https://github.com/dask/distributed/pull/4688 , which by looking at the _pynvml_handles code indicates it's when the parsing of CUDA_VISIBLE_DEVICES. That means it has nothing to do with the context (which makes sense, as NVML doesn't depend/initialize the CUDA context), but it's happening CUDA_VISIBLE_DEVICES is properly set in Dask-CUDA.

The context vs CUDA_VISIBLE_DEVICES comment above isn't super important now that the issue seems to be resolved, it's more of an informative nature.

[charlesbluca]

Thanks for the clarification! I had assumed that nvml.init_once() was initializing the CUDA context - do you know what this is actually accomplishing?

[pentschev]

It's initializing NVML itself if it hasn't been initialized yet (i.e., it is a singleton), but NVML doesn't initialize a CUDA context. Under the hood it calls nvmlInit_v2, which as stated doesn't initialize a GPU (i.e., CUDA context).

[quasiben]

Sorry I steered you incorrectly @charlesbluca . I thought nvml initialized a CUDA context

[charlesbluca]

No worries @quasiben! I renamed the PR to better reflect this.

[charlesbluca]

I'm rerunning tests on my PRs now - if these failures are gone, then we should be able to close this.

[pentschev]

All tests are passing now, we can close this. Thanks @charlesbluca for fixing the issue.

[jakirkham]

Thanks for confirming Peter! 😄