Keras: K.get_session() leads to RuntimeError: The Session graph is empty on Keras 2.0.9

[datumbox]

Calling K.get_session() using Keras 2.0.9 (backend TF 1.4) leads to following exception:

$ python
Python 2.7.6 (default, Oct 26 2016, 20:30:19) 
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from keras import backend as K
Using TensorFlow backend.
2017-11-03 13:44:40.499249: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2017-11-03 13:44:40.725248: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties: 
name: Quadro K2200 major: 5 minor: 0 memoryClockRate(GHz): 1.124
pciBusID: 0000:03:00.0
totalMemory: 3.95GiB freeMemory: 3.32GiB
2017-11-03 13:44:40.908348: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 1 with properties: 
name: Quadro K2200 major: 5 minor: 0 memoryClockRate(GHz): 1.124
pciBusID: 0000:81:00.0
totalMemory: 3.95GiB freeMemory: 3.91GiB
2017-11-03 13:44:40.908450: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Device peer to peer matrix
2017-11-03 13:44:40.908500: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1051] DMA: 0 1 
2017-11-03 13:44:40.908525: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1061] 0:   Y N 
2017-11-03 13:44:40.908536: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1061] 1:   N Y 
2017-11-03 13:44:40.908560: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Quadro K2200, pci bus id: 0000:03:00.0, compute capability: 5.0)
2017-11-03 13:44:40.908574: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:1) -> (device: 1, name: Quadro K2200, pci bus id: 0000:81:00.0, compute capability: 5.0)
>>> K.get_session()
2017-11-03 13:44:46.409150: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Quadro K2200, pci bus id: 0000:03:00.0, compute capability: 5.0)
2017-11-03 13:44:46.409197: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:1) -> (device: 1, name: Quadro K2200, pci bus id: 0000:81:00.0, compute capability: 5.0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 180, in get_session
    [tf.is_variable_initialized(v) for v in candidate_vars])
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 889, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1049, in _run
    raise RuntimeError('The Session graph is empty.  Add operations to the '
RuntimeError: The Session graph is empty.  Add operations to the graph before calling run().

On Keras 2.0.8 there is no problem:

$ python
Python 2.7.6 (default, Oct 26 2016, 20:30:19) 
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from keras import backend as K
Using TensorFlow backend.
>>> K.get_session()
2017-11-03 13:43:41.642325: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2017-11-03 13:43:41.878929: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties: 
name: Quadro K2200 major: 5 minor: 0 memoryClockRate(GHz): 1.124
pciBusID: 0000:03:00.0
totalMemory: 3.95GiB freeMemory: 3.32GiB
2017-11-03 13:43:42.073808: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 1 with properties: 
name: Quadro K2200 major: 5 minor: 0 memoryClockRate(GHz): 1.124
pciBusID: 0000:81:00.0
totalMemory: 3.95GiB freeMemory: 3.91GiB
2017-11-03 13:43:42.073906: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Device peer to peer matrix
2017-11-03 13:43:42.073954: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1051] DMA: 0 1 
2017-11-03 13:43:42.073977: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1061] 0:   Y N 
2017-11-03 13:43:42.073988: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1061] 1:   N Y 
2017-11-03 13:43:42.074011: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Quadro K2200, pci bus id: 0000:03:00.0, compute capability: 5.0)
2017-11-03 13:43:42.074024: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:1) -> (device: 1, name: Quadro K2200, pci bus id: 0000:81:00.0, compute capability: 5.0)
<tensorflow.python.client.session.Session object at 0x7f583528d9d0>
>>> 

@fchollet It seems to be caused by this refactoring: https://github.com/fchollet/keras/commit/9166733c3c144739868fe0c30d57b861b4947b44

• [x] Check that you are up-to-date with the master branch of Keras. You can update with:
  pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps

• [x] If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

• [x] If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
  pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps

• [x] Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

[demcoderseinnachbar]

Moving sess = K.get_session() to the latest possible location solved the problem in my code.

[datumbox]

Patched on PR #8377

[paragon00]

I'm getting this issue too .. will this patch get merged into the main branch or sth?

also, downgrading from keras 2.0.9 to 2.0.8 fixes it, as u noted

[obichkin]

i had the same issue. now downgraded to keras 2.0.8 and it helped.

[Cxinfeng]

if we upgrade keras 2.0.8,it will occur an other problem.keras 2.0.8 has no multi_gpu_model