Cudf: [BUG] MemoryError: std::bad_alloc: CUDA error at: ../include/rmm/mr/device/cuda_memory_resource.hpp:68: cudaErrorMemoryAllocation out of memory

[stromal]

it only comes up if I have 2 kernels running and I am using cudf in both

---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
<ipython-input-3-628af068fea6> in <module>

----> 9 data = cudf.read_csv('data.csv', dtype = col_types)


~/anaconda3/envs/rapids/lib/python3.7/contextlib.py in inner(*args, **kwds)
     72         def inner(*args, **kwds):
     73             with self._recreate_cm():
---> 74                 return func(*args, **kwds)
     75         return inner
     76 

~/anaconda3/envs/rapids/lib/python3.7/site-packages/cudf/io/csv.py in read_csv(filepath_or_buffer, lineterminator, quotechar, quoting, doublequote, header, mangle_dupe_cols, usecols, sep, delimiter, delim_whitespace, skipinitialspace, names, dtype, skipfooter, skiprows, dayfirst, compression, thousands, decimal, true_values, false_values, nrows, byte_range, skip_blank_lines, parse_dates, comment, na_values, keep_default_na, na_filter, prefix, index_col, **kwargs)
     84         na_filter=na_filter,
     85         prefix=prefix,
---> 86         index_col=index_col,
     87     )
     88 

cudf/_lib/csv.pyx in cudf._lib.csv.read_csv()

MemoryError: std::bad_alloc: CUDA error at: ../include/rmm/mr/device/cuda_memory_resource.hpp:68: cudaErrorMemoryAllocation out of memory

it can produce error later on during training but I cant reproduce that error message. It says there is x allocated memory space and I try to fit in double of that. (so I am using it looks at the other kernel where I use the same dataset under the same variable, but even if I change the variable names where I load in the data it still gives the same error)

[kkraus14]

It looks like you're just running out of memory: cudaErrorMemoryAllocation out of memory

If you're running multiple IPython / Jupyter kernels it's likely that variables are being kept around which is preventing GPU memory from being freed. I'd suggest reusing variables more aggressively / using del or setting variables to None to allow Python garbage collection to free GPU memory.

[harrism]

If RMM pool allocation is being used in both Jupyter kernels, then you would easily run out of memory, since both kernels would have separate pools.