Dask-cuda: Spill over after libcudf++ merge is causing CUDA_ERROR_OUT_OF_MEMORY issues

Created on 9 Jan 2020 · 18Comments · Source: rapidsai/dask-cuda

Spill over after libcudf++ merge is causing CUDA_ERROR_OUT_OF_MEMORY issues

After the libcudf++ merge, the spill over mechanism might be failing.

The current hypothesis is, in dask-cuda it looks like if it spills to disk when moving back to the GPU it will allocate via numba instead of RMM.

Relevant code lines are:

From Dask-cuda:

https://github.com/rapidsai/dask-cuda/blob/db07453b130e7fea082279f8bc234f6227718b5b/dask_cuda/device_host_file.py#L90

From distributed (example of how it should be handled ):

https://github.com/dask/distributed/blob/4a8a4f3bce378406e83a099e8a12fc9bc12ef25c/distributed/comm/ucx.py#L45-L63

CC: @jakirkham @pentschev @kkraus14 .

Code to recreate the issue:

https://gist.github.com/VibhuJawa/dbf2573954db86fb193b687022a20f46

Note:
I have not run the cleaned up code again on exp01 but the issue should still be there. (Exp-01 was busy)

Stack Trace

ERROR Call to cuMemAlloc results in CUDA_ERROR_OUT_OF_MEMORY
ERROR Call to cuMemAlloc results in CUDA_ERROR_OUT_OF_MEMORY
distributed.worker - ERROR - [2] Call to cuMemAlloc results in CUDA_ERROR_OUT_OF_MEMORY
Traceback (most recent call last):
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 744, in _attempt_allocation
    allocator()
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 759, in allocator
    driver.cuMemAlloc(byref(ptr), bytesize)
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 294, in safe_cuda_api_call
    self._check_error(fname, retcode)
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 329, in _check_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [2] Call to cuMemAlloc results in CUDA_ERROR_OUT_OF_MEMORY

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/distributed/worker.py", line 2455, in execute
    data[k] = self.data[k]
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/dask_cuda/device_host_file.py", line 152, in __getitem__
    return self.device_buffer[key]
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/zict/buffer.py", line 70, in __getitem__
    return self.slow_to_fast(key)
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/zict/buffer.py", line 57, in slow_to_fast
    value = self.slow[key]
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/zict/func.py", line 39, in __getitem__
    return self.load(self.d[key])
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/dask_cuda/device_host_file.py", line 90, in host_to_device
    frames = [cuda.to_device(f) if ic else f for ic, f in zip(s.is_cuda, s.parts)]
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/dask_cuda/device_host_file.py", line 90, in <listcomp>
    frames = [cuda.to_device(f) if ic else f for ic, f in zip(s.is_cuda, s.parts)]
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/devices.py", line 225, in _require_cuda_context
    return fn(*args, **kws)
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/api.py", line 111, in to_device
    to, new = devicearray.auto_device(obj, stream=stream, copy=copy)
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/devicearray.py", line 704, in auto_device
    devobj = from_array_like(obj, stream=stream)
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/devicearray.py", line 642, in from_array_like
    writeback=ary, stream=stream, gpu_data=gpu_data)
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/devicearray.py", line 103, in __init__
    gpu_data = devices.get_context().memalloc(self.alloc_size)
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 761, in memalloc
    self._attempt_allocation(allocator)
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 751, in _attempt_allocation
    allocator()
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 759, in allocator
    driver.cuMemAlloc(byref(ptr), bytesize)
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 294, in safe_cuda_api_call
    self._check_error(fname, retcode)
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 329, in _check_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [2] Call to cuMemAlloc results in CUDA_ERROR_OUT_OF_MEMORY
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x7fb3dd96c410>>, <Task finished coro=<Worker.execute() done, defined at /raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/distributed/worker.py:2438> exception=CudaAPIError(2, 'Call to cuMemAlloc results in CUDA_ERROR_OUT_OF_MEMORY')>)
Traceback (most recent call last):
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 744, in _attempt_allocation
    allocator()
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 759, in allocator
    driver.cuMemAlloc(byref(ptr), bytesize)
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 294, in safe_cuda_api_call
    self._check_error(fname, retcode)
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 329, in _check_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [2] Call to cuMemAlloc results in CUDA_ERROR_OUT_OF_MEMORY

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/tornado/ioloop.py", line 743, in _run_callback
    ret = callback()
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
    future.result()
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/distributed/worker.py", line 2455, in execute
    data[k] = self.data[k]
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/dask_cuda/device_host_file.py", line 152, in __getitem__
    return self.device_buffer[key]
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/zict/buffer.py", line 70, in __getitem__
    return self.slow_to_fast(key)
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/zict/buffer.py", line 57, in slow_to_fast
    value = self.slow[key]
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/zict/func.py", line 39, in __getitem__
    return self.load(self.d[key])
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/dask_cuda/device_host_file.py", line 90, in host_to_device
    frames = [cuda.to_device(f) if ic else f for ic, f in zip(s.is_cuda, s.parts)]
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/dask_cuda/device_host_file.py", line 90, in <listcomp>
    frames = [cuda.to_device(f) if ic else f for ic, f in zip(s.is_cuda, s.parts)]
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/devices.py", line 225, in _require_cuda_context
    return fn(*args, **kws)
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/api.py", line 111, in to_device
    to, new = devicearray.auto_device(obj, stream=stream, copy=copy)
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/devicearray.py", line 704, in auto_device
    devobj = from_array_like(obj, stream=stream)
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/devicearray.py", line 642, in from_array_like
    writeback=ary, stream=stream, gpu_data=gpu_data)
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/devicearray.py", line 103, in __init__
    gpu_data = devices.get_context().memalloc(self.alloc_size)
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 761, in memalloc
    self._attempt_allocation(allocator)
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 751, in _attempt_allocation
    allocator()
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 759, in allocator
    driver.cuMemAlloc(byref(ptr), bytesize)
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 294, in safe_cuda_api_call
    self._check_error(fname, retcode)
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 329, in _check_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [2] Call to cuMemAlloc results in CUDA_ERROR_OUT_OF_MEMORY
ERROR Call to cuMemAlloc results in CUDA_ERROR_OUT_OF_MEMORY
ERROR Call to cuMemAlloc results in CUDA_ERROR_OUT_OF_MEMORY
distributed.worker - ERROR - [2] Call to cuMemAlloc results in CUDA_ERROR_OUT_OF_MEMORY
Traceback (most recent call last):
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 744, in _attempt_allocation
    allocator()
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 759, in allocator
    driver.cuMemAlloc(byref(ptr), bytesize)
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 294, in safe_cuda_api_call
    self._check_error(fname, retcode)
  File "/raid/vjawa/conda_install/conda_env/envs/cudf_12_8_jan/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 329, in _check_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [2] Call to cuMemAlloc results in CUDA_ERROR_OUT_OF_MEMORY

Source

VibhuJawa

Most helpful comment

Thanks a lot @VibhuJawa for testing this. I'll make sure this is merged for 0.12, will leave this issue open until we merge it there.

pentschev on 18 Jan 2020

🎉2 👍2

All 18 comments

cudf                      0.12.0b200108          py37_839    rapidsai-nightly
dask-cudf                 0.12.0b200108          py37_839    rapidsai-nightly
libcudf                   0.12.0b200108      cuda10.1_839    rapidsai-nightly

dask                      2.9.1                      py_0    conda-forge
dask-core                 2.9.1                      py_0    conda-forge
dask-cuda                 0.12.0a200108           py37_47    rapidsai-nightly
dask-cudf                 0.12.0b200108          py37_839    rapidsai-nightly

VibhuJawa on 9 Jan 2020

Thinking about solutions here, it would be useful to allocate a buffer with a particular allocator and then fill that buffer with data as opposed to letting Numba just allocate one. Fortunately numba.cuda.to_device does support an optional to parameter. This expects a Numba DeviceNDArray object. Though we can construct a view onto our buffer using numba.cuda.as_cuda_array. Using these features we could do something like this:

import numpy
import numba
import rmm

a = a = np.arange(5, dtype="u1")
b = rmm.DeviceBuffer(size=a.nbytes)

numba.cuda.to_device(a, to=numba.cuda.as_cuda_array(b))

jakirkham on 9 Jan 2020

Distributed allocates GPU buffers using RMM or Numba based on what it can find. The allocations happen using this code, which could be useful to just add here or expose in Distributed somehow.

jakirkham on 9 Jan 2020

Could you also share more details on the platform you got this error @VibhuJawa ? I was trying to reproduce on my side in a DGX-2 and I couldn't get it to fail on 1, 2, 8 or 16 GPUs. Below are the versions of packages I used during the test:

cudf                      0.12.0b200109         py37_1016    rapidsai-nightly
dask                      2.9.1                      py_0    conda-forge
dask-core                 2.9.1                      py_0    conda-forge
dask-cuda                 0.12.0a200109           py37_47    rapidsai-nightly
dask-cudf                 0.12.0b200109         py37_1016    rapidsai-nightly
libcudf                   0.12.0b200109     cuda10.0_1016    rapidsai-nightly

pentschev on 9 Jan 2020

@pentschev ,
It was not erring previously because of a mistake in the passing spilt_row_groups argument.
I have updated the gist , it should now hopefully show the error.

The only change was that, i missed an s in the spilt_row_groups argument which caused a discrepancy in the partition_sizes.
.

My bad, sorry for the mistake.

Could you also share more details on the platform you got this error @VibhuJawa ? I was trying to reproduce on my side in a DGX-2 and I couldn't get it to fail on 1, 2, 8 or 16 GPUs. Below are the versions of packages I used during the test:

I was doing this on all 16 GPUS.

Also tested on todays nightly.

cudf                      0.12.0b200110         py37_1062    rapidsai-nightly
dask-cudf                 0.12.0b200110         py37_1062    rapidsai-nightly
libcudf                   0.12.0b200110      cuda9.2_1062    rapidsai-nightly

dask                      2.9.1                      py_0    conda-forge
dask-core                 2.9.1                      py_0    conda-forge
dask-cuda                 0.12.0a200110           py37_47    rapidsai-nightly
dask-cudf                 0.12.0b200110         py37_1062    rapidsai-nightly

VibhuJawa on 10 Jan 2020

Thanks @VibhuJawa , I can reproduce that now, I'll take a better look at how this can be solved early next week.

pentschev on 10 Jan 2020

@VibhuJawa could you test #227?

pentschev on 15 Jan 2020

@pentschev , I tested #227 and it works now successfully. Thanks a lot for your work on this and sorry for the delay in testing.

Tested on below Cudf versions (for record keeping) :

cudf                      0.12.0b200116         py37_1452    rapidsai-nightly
dask-cudf                 0.12.0b200116         py37_1452    rapidsai-nightly
libcudf                   0.12.0b200116     cuda10.1_1422    rapidsai-nightly

VibhuJawa on 17 Jan 2020

🎉1

Did you also test without PR ( https://github.com/rapidsai/dask-cuda/pull/227 ) using the same cuDF versions? Just curious if something in cuDF also affected it.

jakirkham on 17 Jan 2020

Did you also test without PR ( #227 ) using the same cuDF versions? Just curious if something in cuDF also affected it.

Yup. Tested on the same above versions .

VibhuJawa on 17 Jan 2020

Did you also test without PR ( #227 ) using the same cuDF versions? Just curious if something in cuDF also affected it.

Yup. Tested on the same above versions .

Just for clarification, @VibhuJawa does that mean it did not work in pure cuDF with the same version (i.e., this PR definitively caused the fix)?

beckernick on 17 Jan 2020

👍1

Did you also test without PR ( #227 ) using the same cuDF versions? Just curious if something in cuDF also affected it.

Yup. Tested on the same above versions .

Just for clarification, @VibhuJawa does that mean it did not work in pure cuDF with the same version (i.e., this PR definitively caused the fix)?

Yup, I believe so.

I tested it on the same environment by just doing a source install of dask-cuda (branch 277).

I.E, It works on below :

# packages in environment at /raid/vjawa/conda_install/conda_env/envs/cudf_12_16_jan:
cudf                      0.12.0b200116         py37_1452    rapidsai-nightly
dask-cudf                 0.12.0b200116         py37_1452    rapidsai-nightly
libcudf                   0.12.0b200116     cuda10.1_1422    rapidsai-nightly
dask-cuda                 0.6.0.dev0+191.g59e1f14          pypi_0    pypi ### source install on this branch

And Fails on below:

# packages in environment at /raid/vjawa/conda_install/conda_env/envs/cudf_12_16_jan:
cudf                      0.12.0b200116         py37_1452    rapidsai-nightly
dask-cudf                 0.12.0b200116         py37_1452    rapidsai-nightly
libcudf                   0.12.0b200116     cuda10.1_1422    rapidsai-nightly
dask-cuda                 0.12.0a200117           py37_47    rapidsai-nightly

VibhuJawa on 17 Jan 2020

👍2

Great, thanks for checking Vibhu! 😄

jakirkham on 17 Jan 2020

Thanks a lot @VibhuJawa for testing this. I'll make sure this is merged for 0.12, will leave this issue open until we merge it there.

pentschev on 18 Jan 2020

🎉2 👍2

This patch should be in the latest nightlies. @VibhuJawa, would you be able to try them and let us know if they are working?

jakirkham on 21 Jan 2020

This patch should be in the latest nightlies. @VibhuJawa, would you be able to try them and let us know if they are working?

@jakirkham, Sure will update here once i get the time .

VibhuJawa on 21 Jan 2020

👍1

For others following this thread, Peter's fix solved the issue. Though if anyone finds that is incorrect, please don't hesitate to correct me 🙂

jakirkham on 30 Jan 2020

@jakirkham , Yup, The issue no longer seems to be present as the workflow works now. Thanks for closing.

VibhuJawa on 30 Jan 2020

👍1

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Test Pack/Unpack w/ cuDF Merge

quasiben · 7Comments

[BUG] Error during launch of dask-cuda-worker

galipremsagar · 3Comments

Worker containers cannot connect to scheduler container

lmeyerov · 6Comments

Shuffle benchmark fails while getting transfer logs w/ ucx 1.11

ayushdg · 10Comments

Extend memory spilling to multiple storage media

pentschev · 9Comments