Keras: Using allow_growth on keras with tensorflow

[oxrider]

I'm using Keras with TensorFlow to train a large number of tiny networks (~4 layers, less than 30 nodes in each layer). Currently TF allocates all GPU memory to a single process and therefore prevents me from opening more learning processes in parallel. I found on TF document that I can use

config.gpu_options.allow_growth = True
session = tf.Session(config=config, ...)

to do this. However, I wasn't able to integrate that into keras. Does someone know the way to initialize a tf session on keras? Thank you very much!

[kudkudak]

Keras uses global TF session. You should be able to set it to your custom session using this function: https://github.com/fchollet/keras/blob/master/keras/backend/tensorflow_backend.py#L86

[oxrider]

Thank you very much. This perfectly solved my problem.

[vijaycd]

I am sorry, I am new to KEras. How do I use it in my .py file. I need to have the equivalent of

config = tf.ConfigProto()
config.gpu_options.allow_growth = True
config.log_device_placement=True

sess = tf.Session(config=config)  #With the two options defined above

with tf.Session() as sess:
...

Thanks.

[zoltan-fedor]

@vijaycd , if you are still looking for an actual code you can copy-paste into your Keras code to have Tensorflow dynamically allocate the GPU memory:

import tensorflow as tf
from keras.backend.tensorflow_backend import set_session
config = tf.ConfigProto()
config.gpu_options.allow_growth = True  # dynamically grow the memory used on the GPU
config.log_device_placement = True  # to log device placement (on which device the operation ran)
                                    # (nothing gets printed in Jupyter, only if you run it standalone)
sess = tf.Session(config=config)
set_session(sess)  # set this TensorFlow session as the default session for Keras

[vijaycd]

Thank you, will file for later use. I was wondering whether my earlier
request had entered cybervoid. Guess not, someone is active!
Best,
Vijay

https://mailtrack.io/ Sent with Mailtrack
https://chrome.google.com/webstore/detail/mailtrack-for-gmail-inbox/ndnaehgpjlnokgebbaldlmgkapkpjkkb?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality

On Thu, Feb 15, 2018 at 11:14 AM, Zoltan Fedor [e-mail]
wrote:

@vijaycd https://github.com/vijaycd , if you are still looking for an
actual code you can copy-paste into your Keras code to have Tensorflow
dynamically allocate the GPU memory:

import tensorflow as tf
from keras.backend.tensorflow_backend import set_session
config = tf.ConfigProto()
config.gpu_options.allow_growth = True # dynamically grow the memory used on the GPU
config.log_device_placement = True # to log device placement (on which device the operation ran)
# (nothing gets printed in Jupyter, only if you run it standalone)
sess = tf.Session(config=config)
set_session(sess) # set this TensorFlow session as the default session for Keras

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/keras-team/keras/issues/4161#issuecomment-366031228,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AZBBQ0Irp-8g0bil1YHlv0iMzC-1nOPBks5tVIIhgaJpZM4KeQR2
.

[krikru]

@kudkudak Which function were you referring to (providing a link to a specific row in a source file is only guaranteed to work until the file is edited)?

[kudkudak]

I think I was referring to set_session :)

[krikru]

Thank you!

[henningsway]

I am currently take advantage of dynamic GPU memory allocation, for a scikit-learn GridSearchCV build on the KerasClassifier using the functional API.

I set the config via set_session, but somehow neither dynamic nor fixed allocation seems to make a difference, when I start the gridsearch with n_jobs = 2.

I frequently run into:

UserWarning: A worker stopped while some jobs were given to the executor.
This can be caused by a too short worker timeout or by a memory leak.

Also in nvidia-smi I do not see, that the processes take up less memory, than before.

Any thoughts on this?

[josh24990]

Hi,

I have tried implementing the code suggested by @zoltan-fedor which seems to have worked for so many others, however with no luck for me. I receive the following error message:

UnknownError: 2 root error(s) found.
  (0) Unknown: Fail to find the dnn implementation.
     [[{{node bidirectional_1/CudnnRNN_1}}]]
     [[loss/mul/_197]]
  (1) Unknown: Fail to find the dnn implementation.
     [[{{node bidirectional_1/CudnnRNN_1}}]]
0 successful operations.
0 derived errors ignored.

For reference, I'm writing in Keras within a Google Colab notebook, and I have GPU access.

I'm a relative newbie to the field, so any ideas on what's going on here and how to fix it would be greatly appreciated. Let me know if you need any more info too. Thanks.

[rinpoche-peregrine]

Keras uses global TF session. You should be able to set it to your custom session using this function: https://github.com/fchollet/keras/blob/master/keras/backend/tensorflow_backend.py#L86

This link doesn't link to the right code anymore. The lines referred to was this:

def set_session(session):
    '''Sets the global TF session.
    '''
    global _SESSION
    _SESSION = session

[rinpoche-peregrine]

Hi,

I have tried implementing the code suggested by @zoltan-fedor which seems to have worked for so many others, however with no luck for me. I receive the following error message:

UnknownError: 2 root error(s) found.
  (0) Unknown: Fail to find the dnn implementation.
   [[{{node bidirectional_1/CudnnRNN_1}}]]
   [[loss/mul/_197]]
  (1) Unknown: Fail to find the dnn implementation.
   [[{{node bidirectional_1/CudnnRNN_1}}]]
0 successful operations.
0 derived errors ignored.

For reference, I'm writing in Keras within a Google Colab notebook, and I have GPU access.

I'm a relative newbie to the field, so any ideas on what's going on here and how to fix it would be greatly appreciated. Let me know if you need any more info too. Thanks.

Reinstall CUDA and cuDNN