Dlib: GPU training fails with - out of memory

Created on 6 Dec 2017 · 6Comments · Source: davisking/dlib

I am running dlib Release build on Windows, made from the latest source using VS 2015. Trying to run the dnn_mmod_train_find_cars_ex.exe model on some custom data (around 200 jpegs), tagged properly and xml files formed as expected.

CPU training runs fine for days, but GPU build crashes right away. I get the exit:

code: 2, reason: out of memory

Log file is attached, as well as a printout of nvidia-smi taken during the process loading. I tried this on 2 machines, one has a GTX 1080, the other is a laptop with Nvidia 950M, both produce the same result.

log.txt

Any help much appreciated.

inactive

Source

timnev

👍1

Most helpful comment

Change the code to use less RAM. For instance, make the batch sizes
smaller or make the crops smaller.

davisking on 6 Dec 2017

👍3

All 6 comments

Change the code to use less RAM. For instance, make the batch sizes
smaller or make the crops smaller.

davisking on 6 Dec 2017

👍3

I am struggling to find where these parameters are defined. Could you please point me in the right direction?

timnev on 19 Dec 2017

Read the introductory dnn example programs. I've written detailed tutorials that explain all this stuff. They explain everything you need to know.

davisking on 19 Dec 2017

@timnev I had the same problem. and My computer memory 16GB. Not enough, Can you tell me how you handled it?(cpu)

jackweiwang on 16 Mar 2018

Warning: this issue has been inactive for 172 days and will be automatically closed on 2018-09-07 if there is no further activity.

If you are waiting for a response but haven't received one it's likely your question is somehow inappropriate. E.g. you didn't follow the issue submission instructions, or your question is easily answerable by reading the FAQ, dlib's documentation, or a Google search.