Keras: Limit the resource usage for tensorflow backend

[fabiofumarola]

When I use tensorflow as backend I got an high memory usage on my GPUs. I check that is possible to limit memory usage by using tf.ConfigProto

config = tf.ConfigProto(
    gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.5),
    device_count = {'GPU': 1}
)

sess_1 = tf.Session(graph=graph_1, config=config)
sess_2 = tf.Session(graph=graph_2, config=config)

would it be useful to add this option in keras? if yes where is the best place to do it without corrupt keras design principles?

[towr]

You can configure and pass a tensorflow session to Keras

import tensorflow as tf
from keras.backend.tensorflow_backend import set_session
config = tf.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.3
set_session(tf.Session(config=config))

On the other hand I can't find it in the documentation, so I don't know if that's a stable feature. (I just finally thought of looking for ConfigProto in the source to see if I could "fix" this problem for myself.)

[zxweed]

How can I put this limit in keras.json? I want to make this limit default to any keras model...

[tstandley]

I wish it could be set on the command line like CUDA_VISIBLE_DEVICES

[justttry]

does the factor be configured dynamically?
here is the factor:
config.gpu_options.per_process_gpu_memory_fraction = 0.3

[zhangkao]

You can configure and pass a tensorflow session to Keras

import tensorflow as tf
from keras.backend.tensorflow_backend import set_session
config = tf.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.3
config.gpu_options.visible_device_list = "0"
set_session(tf.Session(config=config))

[caugusta]

This may be a bit silly, but what do you mean by 'pass a tensorflow session to Keras'? Do you mean 'write this at the beginning of the same program that uses Keras'?

[towr]

Yes, basically.
(Though, technically it doesn't need to be at the start, just before you make Keras do anything.)

[farahanams]

it does not work for me.

[kikacaty]

@farahanams It does not work for me neither. Any updated solutions to it?

[lucasvinhtran]

The solution above does not work for me as well. I tried different solutions but it seems that Keras does not support memory allocation at this moment. Any idea?

[milkyklim]

I suspect that something went wrong with the current Keras version 2.0.8. I upgraded today from version 2.0.6 and now Tensorflow allocates all the memory on both of my GPU's _before_ execution of any cells in the Jupyter notebook. Previously, I could use CUDA_VISIBLE_DEVICES and tf.ConfigProto to set the memory limits.

I am using keras 2.0.8 and tensorflow 1.3.0 on ubuntu 16.04 LTS.

UPD: CUDA_VISIBLE_DEVICES="-1" KERAS_BACKEND=tensorflow jupyter notebook works for limiting the number of GPU's.

[Xcat24]

@milkyklim try to use tf code to set GPU before loading any module of the keras such as

-------------------------- set gpu using tf ---------------------------
import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.Session(config=config)
-------------------  start importing keras module ---------------------
import keras.backend.tensorflow_backend as K
import keras......

it worked for me on keras 2.0.8 and 2.0.9

[Parkinson-Z]

from keras import backend as K

if 'tensorflow' == K.backend():
import tensorflow as tf
from keras.backend.tensorflow_backend import set_session
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
config.gpu_options.visible_device_list = "0"
#session = tf.Session(config=config)
set_session(tf.Session(config=config))

it worked for me on keras 2.0.4 , tensorflow 1.4 ubuntu 16.04.03 LTS. (as 'towr' said ' just before you make Keras do anything')

[tylertroy]

Don't forget to reinstall the module after you make changes to the code.

/path/to/Mask_RCNN$ python3 setup.py install

[Flock1]

The code specially worked for me when I complied it just before I loaded the model during the inference. So not just when it's training but during inference also this will work.

[sebma]

@zhangkao Hi,
In your comment, is this config.gpu_options.visible_device_list = "0" necessary ?

[dillondavis]

@Parkinson-Z
do you have to manually run sess.run() at some point before training a keras model?
Im doing this while trying to train a Keras model but training does not begin

[loretoparisi]

@towr I have the same issue while on CPU and using Tensorflow JAVA library. The log is the same of python:

2018-09-06 13:10:59.126759: W tensorflow/core/framework/allocator.cc:101] Allocation of 662520000 exceeds 10% of system memory.
2018-09-06 13:11:08.928374: W tensorflow/core/framework/allocator.cc:101] Allocation of 662520000 exceeds 10% of system memory.
2018-09-06 13:11:19.279909: W tensorflow/core/framework/allocator.cc:101] Allocation of 662520000 exceeds 10% of system memory.

I have 16GB, and I want to limit TF to reserve a max amount of memory (like 30% of the java heap, etc.).