Face_recognition: CUDA error with DLIB and cnn.

[IWriteThings]

• face_recognition version: 1.0.0
• Python version: python3
• Operating System: Manjaro

Description

The application fails to run and reports an error in reference to my CUDA runtime and driver.

What I Did

I updated my system and am now getting the following errors in face_recognition. I am running the latest version of dlib and cuda.

Traceback (most recent call last):
  File "./example.py", line 2, in <module>
    from ml_face import *
  File "/home/aaron/Development/bulk-image-facial-recognition/ml_face.py", line 1, in <module>
    import face_recognition
  File "/usr/lib/python3.6/site-packages/face_recognition/__init__.py", line 7, in <module>
    from .api import load_image_file, face_locations, batch_face_locations, face_landmarks, face_encodings, compare_faces, face_distance
  File "/usr/lib/python3.6/site-packages/face_recognition/api.py", line 21, in <module>
    cnn_face_detector = dlib.cnn_face_detection_model_v1(cnn_face_detection_model)
RuntimeError: Error while calling cudaGetDevice(&the_device_id) in file /home/aaron/Development/tmp/dlib/dlib/dnn/gpu_data.cpp:178. code: 35, reason: CUDA driver version is insufficient for CUDA runtime version

[ageitgey]

This means the CUDA library being loaded is too new for your Nvidia drivers. In other words, you only half-upgraded CUDA and things are out of sync.

You can check what versions are loaded with the nvidia-smi command.

To fix it, try updating your Nvidia drivers to the latest version manually.

[IWriteThings]

┌─<aaron @ alien-alpha in ~/D/bulk-image-facial-recognition>
└─< (master)* >──» nvidia-smi                                                                                                                   0 < 14:11:18
Sat Dec 23 14:11:20 2017       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.98                 Driver Version: 384.98                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GPU         Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   42C    P8    N/A /  N/A |     11MiB /  1998MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0       338      G   /usr/lib/xorg-server/Xorg                      8MiB |
+-----------------------------------------------------------------------------+

This all appears to be latest.

[conanwsz]

Just reboot your computer

[Anveena]

Thu Jul 19 00:47:57 2018

| NVIDIA-SMI 398.26                 Driver Version: 398.26                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1080   WDDM  | 00000000:01:00.0  On |                  N/A |
| 37%   46C    P0    62W / 300W |   1146MiB /  8192MiB |      8%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      3776    C+G   ...common\wallpaper_engine\wallpaper64.exe N/A      |
|    0     10260    C+G   ...017\Professional\Common7\IDE\devenv.exe N/A      |
|    0     12108    C+G   ...6)\Google\Chrome\Application\chrome.exe N/A      |
|    0     13628    C+G   C:\Windows\System32\WWAHost.exe            N/A      |
|    0     13916    C+G   ...hell.Experiences.TextInput.InputApp.exe N/A      |
|    0     13944    C+G   C:\Windows\explorer.exe                    N/A      |
|    0     14128    C+G   ...dows.Cortana_cw5n1h2txyewy\SearchUI.exe N/A      |
|    0     15860    C+G   ...t_cw5n1h2txyewy\ShellExperienceHost.exe N/A      |
|    0     16272    C+G   C:\Windows\System32\dwm.exe                N/A      |
|    0     17420    C+G   ...iles (x86)\Tencent\WeChat\WeChatWeb.exe N/A      |
+-----------------------------------------------------------------------------+

the same case @win10 , imageSize 31204160 & faceSize 10001000
here is the error info:

 face_recognition.face_locations(frFile,model="cnn")
  File "C:\Program Files\Python37\lib\site-packages\face_recognition\api.py", line 116, in face_locations
    return [_trim_css_to_bounds(_rect_to_css(face.rect), img.shape) for face in _raw_face_locations(img, number_of_times_to_upsample, "cnn")]
  File "C:\Program Files\Python37\lib\site-packages\face_recognition\api.py", line 100, in _raw_face_locations
    return cnn_face_detector(img, number_of_times_to_upsample)
RuntimeError: Error while calling cudaMalloc(&data, new_size*sizeof(float)) in file e:\code\dlib-19.15\dlib\cuda\gpu_data.cpp:195. code: 2, reason: out of memory

[ageitgey]

@Anveena That just means your GPU ran out of memory. It looks like lots of other programs are also using the GPU and are already using 1GB, leaving 7GB. Then depending on the size of the image you are processing, it might just be too big to fit into that remaining memory.

Try reducing the image size and seeing if that works.

[Anveena]

@ageitgey thank you,I'm a beginner at face recognition,i know resize img can run well,i join this issue for it's amazing that a 3120X4160 img can use so much memory.so i worry about that if sth was wrong with my code or building

[chanduthedev]

Hi @ageitgey,

I am running FR on Windows 10 GPU machine. I also also faced out of memory(OOM) error. After resizing image, OOM issue got resolved.

I would like to know is there any impact on accuracy if resizing image to 1/2 or 1/3.

Thanks in advance.

[KishoreKonakanti]

@chanduthedev If you are doing object detection after resizing, it depends on the object you are trying to detect. For example, accuracy may not drop much if you are detecting a person while the same may drop substantially if you are detecting smaller objects like spectacles.