Dlib: Python : Cannot recover from CUDA out of memory

[rgrenonlkr]

Expected Behavior

When trying to run cnn_face_detection_model_v1 on a big image dlib raises a RuntimeError (Cuda OOM).
After that, we should be able to make other calls to dlib.
Side note : is there a way to raise an OutOfMemoryException instead of the quite generic RuntimeError ?

Current Behavior

After a RuntimeError caused by CUDA OOM the lib become useless for the rest of the process lifetime.

Steps to Reproduce

import dlib
import GPUtil
import cv2
from os import path

cnn_face_detector = dlib.cnn_face_detection_model_v1(
    path.expanduser("~/CODE/MODELS/mmod_human_face_detector.dat"))

img = cv2.imread('./samples/continue/A/009.jpg')
img_ok = cv2.resize(img, (5000,5000))
img_ko = cv2.resize(img, (9000,9000))

gpu = GPUtil.getGPUs()[0]
print("Dlib {}, GPU-memfree: {:.0f}".format(dlib.__version__, gpu.memoryFree))

try:
    faces = cnn_face_detector(img_ok)
    print('Detection OK for img_ok')
    faces = cnn_face_detector(img_ko)
    print('Detection OK for img_ko')
except RuntimeError as e:
    print(e)
    gpu = GPUtil.getGPUs()[0]
    print("Detection failed > img.size: {:.1f} | GPU-memfree: {:.0f}".format(img_ko.size / 1000000, gpu.memoryFree))

try:
    faces = cnn_face_detector(img_ok)
    print('Detection OK for img_ok')
except RuntimeError as e:
    print(e)
    gpu = GPUtil.getGPUs()[0]
    print("Detection failed > img.size: {:.1f} | GPU-memfree: {:.0f}".format(img_ok.size / 1000000, gpu.memoryFree))

Output :

Dlib 19.17.0, GPU-memfree: 6170
Detection OK for img_ok
Error while calling cudaMalloc(&data, new_size*sizeof(float)) in file /tmp/pip-install-w9uyl3ed/dlib/dlib/cuda/gpu_data.cpp:218. code: 2, reason: out of memory
Detection failed > img.size: 243.0 | GPU-memfree: 2928
Error while calling cudaGetLastError() in file /tmp/pip-install-w9uyl3ed/dlib/dlib/cuda/gpu_data.cpp:114. code: 2, reason: out of memory
Detection failed > img.size: 75.0 | GPU-memfree: 1892

• Version: 19.17.0
• Where did you get dlib: pip install dlib
• Platform: Ubuntu 18.04, CUDA 10.0, cuDNN 7.4.2
• Compiler: gcc 7.3.0
• Hardware: RTX 2070 8GB

[davisking]

You should structure your code so you do not run out of memory. It is not
possible in general to do much after you run out ram.

[rgrenonlkr]

Thanks for your answer.

I tend to agree with you and our production is designed in order to favor constant GPU-RAM usage. In such case we kill the slave/worker and expect our scheduler/manager program to handle this.

However, more generally speaking, I'm not familiar with GPU coding but it sounds legitimate for me that after an unsuccessful malloc (??) you could wait for a little time and try again hoping that RAM as been freed meanwhile.

[davisking]

You shouldn't do that. Unless you free ram elsewhere it's not going to
suddenly become available. You need to think about how to structure your
software so that out-of-ram conditions don't happen. Moreover, some
platforms don't even tell you when you run out of RAM, the OS just kills
your program. Like with CPU RAM in Linux, the default behavior is to claim
that all memory allocations succeed, even ones far in excess of the
available amount of RAM. What happens then is that when your program gets
around to touching a block of memory not really available it's abruptly
terminated by Linux. The general philosophy is that you don't catch OOM
conditions. OOM means you need to rethink your software's structure or
what you are asking it to do and set things up so OOM can't happen.

[rgrenonlkr]

I totally agree in a CPU RAM only world (https://www.quora.com/Why-is-it-not-a-good-idea-to-catch-java-lang-OutOfMemoryError)
However _if_ one considers GPU as a shared ressource it seems reasonable to me not to kill the process when the resource is temporary unavailable.
Also I understand that it is design decision and not a bug so I'll close this issue.

Thanks again for your time !

[isConic]

@davisking
Coming from the experience of installing DLIB as a submodule of a python face_recognition library. DLIB seems to be the only of the 4 deep learning models that trigger this RAM issue. after only a few passes of face-extraction on a 1000X1200(roughly) nd-array crashes the entire system.

I am willing to consider restructuring my code, however the responsibility of memory management of 3rd party CUDA integration is a bit beyond me and what I expected from this standalone module.

I do not believe that I'm sending too many frames to process at once. We're talking at most 10 a second which shouldn't more than 30 Mb on an 8gb RTX 2070 Super.

What do you mean by changing software design? Ideas on what can be done?

[davisking]

That image is probably getting upsampled (or not, I don't know what you are doing), maybe you also have it set to batch a bunch of them together, which uses more RAM, then it goes through a CNN to find faces, which creates a whole bunch of complicated image features in the process, which consumes a lot more RAM than the input image. This is just how CNNs work. You can't go running this kind of model on big images and expect a small amount of RAM usage. If you want something that uses less RAM use the HOG face detector, or something similar, or send fewer images at a time to the GPU.

[flydragon2018]

somewhere else says that you can configure the percentage of whole gpu ram, e.g 80%, so that you won't run out memory. I forget where though.
it seems to me that DLIB keeps sending request of cudaMalloc without proper memory check（ perhaps ???), even though I change the batch size to 1. the cudaMalloc() call quickly trigger runtime error of out of memory.

[davisking]

The amount of memory allocated is a strict function of the amount of data you load or what you are doing. It's not up to dlib. e.g. if you load a really huge image, it's going to take a lot of ram. There isn't anything I can do to stop you from running out of ram.