Keras: Out of memory issues (GPU)

Created on 29 Jan 2016 · 14Comments · Source: keras-team/keras

I'm training a model with Theano/CUDA, and if I attempt to specify a large batch_size (1024 in my case), it reports an out of memory error, which is understandable. However, if I change it back to a size that previously worked (I'm doing it in a notebook), it will still be out of memory, as if it didn't attempt to free whatever it allocated for the previous attempt, so I'm forced to restart the Python process (and reload all data/recompile models).

I can provide model code if needed, its on a laptop that does not have internet access currently.

stale

Source

d07RiV

Most helpful comment

Hitting this as well. I've been playing with a trivial model in jupyter and I've observed that while iterating in jupyter my GPU used memory (as reported by nvidia-settings) steadily increases in ~200mb increments until finally theano generates an out of memory error.

Calling del model, gc.collect() seems to have no effect on used GPU memory. Restarting the jupyter kernel resolves it, but is undesirable.

moof2k on 8 Jun 2016

👍10

All 14 comments

It sounds like a problem with IPython Notebook, maybe it doesn't release allocated resources (memory). It happened to me before.
I suggest you develop your model in a text editor or IDE, and run it through console.

jocicmarko on 29 Jan 2016

It is possible that cuda driver don't like this and stay in a bad state. It
isn't able to recover from all type of error. I'm not sure which one it
recover correctly and which one it don't.

Calling a_theano_function.free() could help free the memory, but as it was
said, running it outside ipython notebook would fix those problems.

On Fri, Jan 29, 2016 at 7:17 AM, Marko Jocić [email protected]
wrote:

It sounds like a problem with IPython Notebook, maybe it doesn't release
allocated resources (memory). It happened to me before.
I suggest you develop your model in a text editor or IDE, and run it
through console.

—
Reply to this email directly or view it on GitHub
https://github.com/fchollet/keras/issues/1587#issuecomment-176726753.

nouiz on 29 Jan 2016

If I run it as a complete python program then it obviously wouldn't be a problem because the exception would terminate the python process, but that defeats the purpose of being able to keep training the model and checking the results without having to recompile it every time.

I'm pretty sure running it in python shell would be the same as using the notebook, though.

It seems that it starts allocating large amounts of memory, but when it runs out it throws an exception and doesn't free the memory. I don't know if forcing garbage collection would help, but that theano free function looks like it would help, thanks.

d07RiV on 29 Jan 2016

Calling del model, gc.collect() seems to have no effect on used GPU memory. Restarting the jupyter kernel resolves it, but is undesirable.

moof2k on 8 Jun 2016

👍10

Has anyone managed to find a way around this (i.e. without restarting the kernel) ?

MannyKayy on 18 Feb 2017

@MannyKayy deleting all IPython references to the model and calling the garbage collector works for me (using theano):

%xdel model
import gc
for i in range(3): gc.collect()

slavivanov on 21 Feb 2017

having this issue myself with tensorflow, above^ method doesn't solve issue in notebook.

Goddard on 17 Mar 2017

👍4

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.