Keras: out of memory when using model.predict()

[qinhaifangpku]

hi~ I am now using keras to build my network. the training is normal. but when I use the model.predict() to predict the results, it happend to a error :
out of memory.
It is weired because it can train normally but can not test.
Please help me, thank you very much!

Please make sure that the boxes below are checked before you submit your issue. If your issue is an implementation question, please ask your question on StackOverflow or join the Keras Slack channel and ask there instead of filing a GitHub issue.

Thank you!

• [x] Check that you are up-to-date with the master branch of Keras. You can update with:
  pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps

• [x] If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

• [x] If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
  pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps

• [x] Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

[unrealwill]

Hello,
I agree with you that it seems strange, because model usually use less memory to predict than to train. It happened to me once or twice when I was near the max of the memory. Graphs in train phase and in predict phase are usually different, so they can result in a different memory allocation resulting in different memory segmentation and different memory usage.
If it is indeed an out of memory bug.
You can try to set a small batch_size in predict.
You can try to monitor the memory usage using nvidia-smi.
You can also save the model weights after training. Close python. Relaunch it, Load model weights from disk and try a model.predict from fresh.
You can try to use CNMeM.
If this is not an out of memory bug but something more subtle, You will have to investigate more and provide more info.

[qinhaifangpku]

@unrealwill thank you for your advice, but I have decrease the batch size. it did not work. and relaunch the python ,it did not work either.
I use the K.function() can get the middle layer output but when i try to get the last layer to get the predict results. it failed. out of memory....

[unrealwill]

https://keras.io/getting-started/faq/
"How can I obtain the output of an intermediate layer?"

Have you tried the first method instead of K.function() (I'm not used to K.function() so can't advice you)

Try monitoring the memory usage (print summary and nvidia-smi) or try running it in CPU.
If you get it to work with more memory then you are just on the limit and this is not a bug. If you don't get it to work with more memory then you probably have a bug somewhere in your code but you don't provide enough information.

[qinhaifangpku]

@unrealwill hi~ very appreciate for your advice! I am now can use the model.predict() function to predict results. it is out of memory because i did not set the batch size to 1 or 2. So it may convert all the images to the network so it happened to out of memory error.

[sarath777]

Resize the image
since its fully connected layer and you are giving large image for prediction it will take whole the memory, so you need to resize the image before prediction

[f-huang]

python==3.7.3
tf.__version__==2.0.0-alpha0
tf.keras.__version__==2.2.4-tf

I also ran into this MemoryError when trying to predict an array with shape (62000, 190000).

I used DennyBritz text-cnn-model. Fitting on hundreds of thousands lines works fine. (training's input shape approximately equals (500000, 190000) with batch_size=128)

I tried predicting using model.predict() with batch_size at 1, 2, 8, 16, 32, etc. it still doesn't work.
When I predict on less lines (30000), it works though.

I tried looping through the predictions but still got the Error.
How can I solve this ?

---

Current predict:
Y_pred = self._trained_model.predict(X, batch_size=32, verbose=1, workers=self.workers, use_multiprocessing=True)

---

Predictions loop:

Y_pred = []
i, chunksize = 0, 10000`
for idx in range(0, len(X), chunksize):
    Y_pred += list(self._trained_model.predict(X[idx:(i+1)*chunksize],
                verbose=1, batch_size=1))
    i += 1
Y_pred = np.array(Y_pred)

EDIT: here's the exception

  File "C:\Users\me\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\keras\engine\training.py", line 1167, in predict
    callbacks=callbacks)
  File "C:\Users\me\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\keras\engine\training_arrays.py", line 358, in model_iteration
    aggregator.create(batch_outs)
  File "C:\Users\me\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\keras\engine\training_utils.py", line 130, in create
    self.results.append(np.zeros(shape, dtype=batch_out.dtype))
MemoryError

[csiki]

I have the same problem: both my model and one batch takes around 10% of the GPU memory. When I use .predict(...) or run the model in a loop using the call function (model(...)), I run out of memory after a while - no matter the batch size. In your implementation it seems the data is kept on the GPU, even after it is returned to the host device - which is unquestionably a bug, there's no extra computation that one could do with the previous batch remaining in the GPU memory.

[brandonbell11]

Same issue. I am using model.predict() in a loop and appending that scalar to an array. It appears the results from model.predict() are kept around somewhere and it is ballooning my memory and causing OOM errors after some time.

[petamind]

Same issue. I used model.predict() and it blew 35GB TPU

[marnixkoops]

Same issue here, calling model.predict on a (210944, 4) dimensional array running on a Tesla P100-16GB

[M00NSH0T]

if anyone else encounters this, try:

from tensorflow.compat.v1.keras.backend import set_session
config = tf.compat.v1.ConfigProto()
config.gpu_options.allow_growth = True  # dynamically grow the memory used on the GPU
config.log_device_placement = True  # to log device placement (on which device the operation ran)
sess = tf.compat.v1.Session(config=config)
set_session(sess)

I found the code in a Medium post, but it fixes a lot of memory related issues I've encountered with tensorflow 2.0+.

Here's the Medium post