Keras: clear_session() doesn't clear memory from GPU

[nkumar15]

Apologies If I am not able to understand the obvious solution mentioned in other issues opened/closed for same problem,
However after reading the issues, I used clear_session and reset_default_graph function, but still its doesn't clear the memory.

Below is the code I am testing and at prompt Break2, I was expecting GPU memory to be released, but still it doesn't clear the memory.

Can somebody please shed some light, what wrong I am doing here?

Version tested on:
Tensorflow 1.5.0
CUDA 9.0
Keras 2.1.3

from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras.layers import Embedding
from keras.layers import LSTM
from keras import backend as be

import tensorflow as tf

config = tf.ConfigProto()

config.gpu_options.allow_growth = True
config.gpu_options.per_process_gpu_memory_fraction = 0.5 
be.tensorflow_backend.set_session(tf.Session(config=config))

model = Sequential()
model.add(Embedding(1000, output_dim=256))
model.add(LSTM(128))
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy',
              optimizer='rmsprop',
              metrics=['accuracy'])

x = input("Break 1")
be.clear_session()
tf.reset_default_graph()
x = input("Break 2")

[pinkeshbadjatiya]

I also faced similar problem a few days back, Apparently, you cannot forcefully clear the GPU memory as it might cause fragmentation. Either the GPU clears it automatically, or if you want to do it, then you can kill the process and GPU handles the remains of the process.

[nkumar15]

Okay, I got your point, but it is strange that api is claiming it will clear the gpu memory.
Is there anything we misunderstood from clear_session() functionality?

[phobrain]

What if you reset the graph before clear_session?

[ldm314]

I have the same issue. TF 1.12.0, Keras 2.2.4.

Every iteration of the loop I save the weights to disk, clear the session, create new models and load the weights. After a dozen or so iterations, OOM errors.

    def cleanup_memory(self):
        K.clear_session()
        del self.combined
        del self.discriminator
        del self.generator
        gc.collect()

[phobrain]

    del model

    K.clear_session()

On Thursday, January 17, 2019, 5:20:59 PM PST, Brian Moore <[email protected]> wrote:

I have the same issue. TF 1.12.0, Keras 2.2.4.

Every iteration of the loop I save the weights to disk, clear the session, create new models and load the weights. After a dozen or so iterations, OOM errors.
def cleanup_memory(self):
K.clear_session()
del self.combined
del self.discriminator
del self.generator
gc.collect()

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[chg0901]

I have a answer from https://stackoverflow.com/a/52354943/7194271
Releasing RAM memory
For releasing the RAM memory, just do del Variables as suggested by @nuric in the comment.

Releasing GPU memory
This is a little bit trickier than releasing the RAM memory. Some people will suggest you the following code (Assuming you are using keras)

from keras import backend as K
K.clear_session()
However, the above code doesn't work for all people. (Even when you try del Models, it is still not going to work)

If the above method doesn't work for you, then try the following (You need to install the numba library first):

from numba import cuda
cuda.select_device(0)
cuda.close()
The reason behind it is: Tensorflow is just allocating memory to the GPU, while CUDA is responsible for managing the GPU memory.

If CUDA somehow refuses to release the GPU memory after you have cleared all the graph with K.clear_session(), then you can use the cuda library to have a direct control on CUDA to clear up GPU memory.

[Darkhunter9]

Cuda solution is very effective, except it will thoroughly clear everything in GPU, works for me.

[freepi]

@Darkhunter9 or anyone, how are you resetting Keras after cuda.close() so that you can continue using Keras in the same process? I tried setting a new session but that just gave error "failed to memset memory: CUDA_ERROR_INVALID_VALUE"

[Darkhunter9]

@Darkhunter9 or anyone, how are you resetting Keras after cuda.close() so that you can continue using Keras in the same process? I tried setting a new session but that just gave error "failed to memset memory: CUDA_ERROR_INVALID_VALUE"

In my project, I don't need to use Keras after cuda.close(). After running cuda.close(), I run some OpenCL code and that works fine.
I suppose you need to open a new TF session.
The ultimate solution may be running the whole TF model in a separate process (using Process in Python) and kill it after returning the output back to the main process. This works when I have multiple OpenCL sessions running simultaneously in individual processes.