Dlib: MemoryError: bad allocation Training an object detector with 700+ images.

[adrian1906]

Expected Behavior

Finish object detection training on 732 images (1200x1800) using dlib.train_simple_object_detector without errors. The images were all tagged using imglab.

Current Behavior

After about 5 hours of training (with GPU) I get the following error:
MemoryError: bad allocation

The code works if I used 100 images however the performance is poor. I tried reducing the sizes of the images to 480x720 and then used imglab again to tag the object. When I ran the training session on the new images, I received errors that some boxes were too small. I then returned to trying to figure how to work with the bigger files.

I'm running Windows 10 i7 64bit 12GB RAM and NVIDIA GeForce GT 705 with virtual memory set to 17.868 GB

I have option.ve_verbose = True but it did not return any information while training so I do not know how many images were processed before the error. I plan to do trial and error by incrementing by 100 images. This will be time-consuming and I was hoping I can get some guidance as to how to estimate the memory requirements a priori.

Steps to Reproduce

Run my python code from the command line (not within Spyder). Spyder does not return option.be_verbose information until the very end.
It was verified that the GPU is being used.

• Version: 19.13
• Where did you get dlib: GitHub
• Platform: Windows 10 64 Bit
• Compiler: Visual Studio 14 2015

[davisking]

You just need more RAM. That's what that error means. Also, this part of dlib doesn't use the GPU.

[adrian1906]

Thank you.

I gathered that much. But how much is needed? The max on my machine is 16GB. Will that be enough? Should I expect to be able to compensate by increasing the pagefile size? Is it possible to train in smaller batches and then combined the results into a single detector? Is it possible to get information on the average size of the feature vector for each image. If so, I could use that value to plan ahead.

Also, I'm surprised that the simple object detection does not use the GPU?

[davisking]

No, you either need to use smaller images, fewer images, or get more RAM in
your machine.

This code doesn't use the GPU. It's not a DNN or anything like that.

[brankalakic]

Leaving this for future visitors of this issue. I had a similar issue even though I had 16G RAM. I compiled the code as 32 bit at first, which limits the memory that can be used to 2G (as the CMakeLists says). After compiling it as 64 bit, I had no problems any more.

[davisking]

I assume @brankalakic is not using Python. When using Python the setup.py file will compile in 64bit or 32bit to match the version of pyhton you are using. There isn't anything you do to set this when you build dlib. But you should certainly be using 64bit Python. But that isn't anything the dlib installer has control over.

To install dlib, everyone should be following the instructions here: http://dlib.net/compile.html. If you are doing anything different then you are probably doing it wrong. In particular, there is a lot of nonsense on the internet about how to install dlib. Follow the official instructions: http://dlib.net/compile.html

Maybe the issue here was using 32bit python. Or maybe it was the images were too big. It is hard to know. It should be noted that the simple python training API automatically upsamples images when small objects are in the training dataset. So that can increase RAM usage as well. In general, you don't want to be training on images that have an obscene number of non-target pixels in them. Not because it's bad for the algorithm, but if only 0.00000001% of the pixels in your images are the objects you want to train on then you are going to run out of RAM before you get a lot of training objects into memory.