Keras: Loading multiple models

[kaltu]

Environment:

>>> import sys
>>> print(sys.version)
3.5.3 (v3.5.3:1880cb95a742, Jan 16 2017, 16:02:32) [MSC v.1900 64 bit (AMD64)]
>>> import tensorflow
>>> tensorflow.__version__
'1.4.0'
>>> import keras
Using TensorFlow backend.
>>> keras.__version__
'2.0.9'

I can run this code:

from tensorflow import Graph
from keras.models import model_from_json
print('loading first model...')
graph1 = Graph()
with graph1.as_default():
    with open('model1_arch.json') as arch_file:
        model1 = model_from_json(arch_file.read())
print('loading second model...')
graph2 = Graph()
with graph2.as_default():
    with open('model2_arch.json') as arch_file:
        model2 = model_from_json(arch_file.read())

It prints:

Using TensorFlow backend.
loading first model...
loading second model...

looks fine.
And this code:

from tensorflow import Graph
from keras.models import model_from_json
print('loading first model...')
graph1 = Graph()
with graph1.as_default():
    with open('model1_arch.json') as arch_file:
        model1 = model_from_json(arch_file.read())
    model1.load_weights('model1_weights.h5')   # <--- add this line
print('loading second model...')
graph2 = Graph()
with graph2.as_default():
    with open('model2_arch.json') as arch_file:
        model2 = model_from_json(arch_file.read())

It prints:

Using TensorFlow backend.
loading first model...
2017-11-19 17:00:05.742339: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2017-11-19 17:00:06.276240: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:1030] Found device 0 with properties: 
name: GeForce GTX 950M major: 5 minor: 0 memoryClockRate(GHz): 0.928
pciBusID: 0000:01:00.0
totalMemory: 4.00GiB freeMemory: 3.35GiB
2017-11-19 17:00:06.276809: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 950M, pci bus id: 0000:01:00.0, compute capability: 5.0)
loading second model...

So far so good.

However, I run this code:

from tensorflow import Graph
from keras.models import model_from_json
print('loading first model...')
graph1 = Graph()
with graph1.as_default():
    with open('model1_arch.json') as arch_file:
        model1 = model_from_json(arch_file.read())
    model1.load_weights('model1_weights.h5')
print('loading second model...')
graph2 = Graph()
with graph2.as_default():
    with open('model2_arch.json') as arch_file:
        model2 = model_from_json(arch_file.read())
    model2.load_weights('model2_weights.h5')   # <--- add this line

It doesn't work.

Using TensorFlow backend.
loading first model...
2017-11-19 17:05:23.982670: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2017-11-19 17:05:24.522351: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:1030] Found device 0 with properties: 
name: GeForce GTX 950M major: 5 minor: 0 memoryClockRate(GHz): 0.928
pciBusID: 0000:01:00.0
totalMemory: 4.00GiB freeMemory: 3.35GiB
2017-11-19 17:05:24.522861: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 950M, pci bus id: 0000:01:00.0, compute capability: 5.0)
loading second model...
Traceback (most recent call last):
  File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\client\session.py", line 1064, in _run
    allow_operation=False)
  File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\framework\ops.py", line 3035, in as_graph_element
    return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
  File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\framework\ops.py", line 3114, in _as_graph_element_locked
    raise ValueError("Tensor %s is not an element of this graph." % obj)
ValueError: Tensor Tensor("Placeholder:0", shape=(40980, 50), dtype=float32) is not an element of this graph.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:/university/django-and-bootstrap-template/mysite/rate/models/test/test.py", line 15, in <module>
    model2.load_weights('model2_weights.h5')
  File "C:\Program Files\Python35\lib\site-packages\keras\models.py", line 733, in load_weights
    topology.load_weights_from_hdf5_group(f, layers)
  File "C:\Program Files\Python35\lib\site-packages\keras\engine\topology.py", line 3142, in load_weights_from_hdf5_group
    K.batch_set_value(weight_value_tuples)
  File "C:\Program Files\Python35\lib\site-packages\keras\backend\tensorflow_backend.py", line 2247, in batch_set_value
    get_session().run(assign_ops, feed_dict=feed_dict)
  File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\client\session.py", line 889, in run
    run_metadata_ptr)
  File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\client\session.py", line 1067, in _run
    + e.args[0])
TypeError: Cannot interpret feed_dict key as Tensor: Tensor Tensor("Placeholder:0", shape=(40980, 50), dtype=float32) is not an element of this graph.

I'm trying to duplicate the solution found here.
These two models are good if and only if they're not in each other's way.

[emilk]

I have the exact same issue.

Interestingly, if I remove the Graph scopes:

from tensorflow import Graph
from keras.models import model_from_json
print('loading first model...')
with open('model1_arch.json') as arch_file:
    model1 = model_from_json(arch_file.read())
model1.load_weights('model1_weights.h5')
print('loading second model...')
with open('model2_arch.json') as arch_file:
    model2 = model_from_json(arch_file.read())
model2.load_weights('model2_weights.h5')

then the code runs for me – until I try run model2.predict in which case I get an output that is wrong just because I loaded model1. So loading the first model in the same graph messes with the other.

[mark86092]

This is because the keras share a global session if no default tf session provided

When the model1 created, it is on graph1
When the model1 loads weight, the weight is on a keras global session which is associated with graph1

When the model2 created, it is on graph2
When the model2 loads weight, the global session does not know the graph2

A solution below may help,

graph1 = Graph()
with graph1.as_default():
    session1 = Session()
    with session1.as_default():
        with open('model1_arch.json') as arch_file:
            model1 = model_from_json(arch_file.read())
        model1.load_weights('model1_weights.h5')
        # K.get_session() is session1

# do the same for graph2, session2, model2

[emilk]

That worked splendidly – thanks a lot @mark86092 !

[sakinaljana]

Hi @mark86092 @emilk,

I also encounter this problem, the difference is i'm only use one model of .h5 that already include arch and weight. Already follow your solution above but still not successful. Should i export the weight and arch separately?

This actually not an error, rather the prediction take forever (never finished and you need to cancel manually).

and also importantly, which one we use for prediction phase? session or graph? Thanks in advance!

SOLVED GUYS, please look at my solution on this: https://github.com/keras-team/keras/issues/2397#issuecomment-385317242

[Luonic]

@mark86092 thank you! Very helpful
To anyone writing eval in same file as train in different graphs this snippet would be helpful:

train_graph = tf.Graph()
train_sess = tf.Session(graph=train_graph, config=tf.ConfigProto(use_per_session_threads=False))
with train_graph.as_default(), train_sess.as_default():
    # Build your train graph here

eval_graph = tf.Graph()
eval_sess = tf.Session(graph=eval_graph, config=tf.ConfigProto(use_per_session_threads=False))
with eval_graph.as_default(), eval_sess.as_default():
    # Buld your eval graph here

I'm not shure config=tf.ConfigProto(use_per_session_threads=False) is really needed, you can try without it

[AbhinavJhanwar]

Hi @dyngts ,

I think your solution would be something like this:

```
Solution 1:
from keras.models import load_model
from tensorflow import Graph, Session
graph1 = Graph()
with graph1.as_default():
session1 = Session()
with session1.as_default():
# load model
model1 = load_model("model.h5")
output1 = model1.predict(testData)

do the same for graph2, session2, model2, output2

Solution 2:
You can try by creating a seperate method to load and predict the output. Something like I have done is here:
def getOutput():
from keras.models import load_model
from tensorflow import Graph, Session
graph = Graph()
with graph.as_default():
session = Session()
with session.as_default():
# load model
model = load_model("model.h5")
output = model.predict(testData)
return output

[momonala]

If you want to use multiple models across modules, this is a solution that worked for me. I created a new Model class with its own tf graph session instance, and then loaded the model inside a static method. This way whenever a model loads its weights, it knows which graph session to use (the one of its instance).

# psuedocode

class Model:
    @staticmethod
    def loadmodel(path):
        return loadmodel(path)

    def ___init__(self, path):
       self.model = self.loadmodel(path)
       self.graph = tf.get_default_graph()

    def predict(self, X):
        with self.graph.as_default():
            return self.model.predict(X)

model1 = Model('model1.h5')
model1.predict(test_data)

model2 = Model('model2.h5')
model2.predict(test_data)

[HuiQiu1982]

@mark86092 ，hi~, what‘s means of "if" in “This is because the keras share a global session if no default tf session provided” ? Can i use mutil-thread by creat several keras sessios? i confirmed tf support multi-thread，by found keras has only one global session which disable the multi-thread through it.。。。

If possible, please help me in create multi-thread task through keras+tf backend, thank you very much!!

[menon92]

Hello @momonala,
I am trying to implement your code like bellow,

class MModel:    
    @staticmethod
    def loadmodel(path):
        model = Model(input_img, output_img)
        model.load_weights(path)
        return model

    def __init__(self, path):
       self.model = self.loadmodel(path)
       self.graph = tf.get_default_graph()

    def predict(self, X):
        with self.graph.as_default():
            return self.model.predict(X)

model1 = MModel(model_path_1)
model2 = MModel(model_path_2)

but it loads last mode 'model2' . What could be the reason

My Tensorflow as Keras verison,

>>> tf.__version__
'1.8.0'
>>> keras.__version__
'2.2.4'
>>>

Thanks

[momonala]

Do you mind elaborating a bit on "but it loads last mode 'model2' ." Not exactly sure what you mean here. Thanks

[menon92]

@momonala thanks for your replay,
"but it loads last mode 'model2' ." means when I predict using model1.predict(test_data) command, it's output like model2.predict(test_data)

I am basically try to do,

model1 = load_weights('model1.h5')
model2 = load_weights('model2.h5')

models_list = [model1, model2]

for model in models_list:
    prediction = model.predict(test_data)
    # store the prediction of model1 and model2 

How can I do this in keras

Thanks

[momonala]

That code should work, with the change of how you load the model. Are you getting the same predictions for both models?

model1 = MModel(model_path_1)
model2 = MModel(model_path_2)

models_list = [model1, model2]
predictions = []
for model in models_list:
    prediction = model.predict(test_data)
    predictions.append(prediction)

[menon92]

@momonala,
Yes, I am getting the same predictions for both models. But when I load my models one by one then I get different prediction. What could be the cause.

Thanks

[momonala]

Hmm its hard to tell without seeing the inputs to the model and the full code (not just psuedocode), weights of the models, etc.. Perhaps this is better a question for Stackoverflow.

[menon92]

@momonala,
My network architecture for both model1 & model2 is same. But I trained this network with two completely different datasets.

Network code,

input_img = Input(shape=(input_size,))
hidden_1 = Dense(hidden_size, activation='relu')(input_img)
code = Dense(code_size, activation='relu')(hidden_1)
hidden_2 = Dense(hidden_size, activation='relu')(code)
output_img = Dense(input_size, activation='sigmoid')(hidden_2)

I trained this network and saved my models using model.save_weights(model_name.h5) command.
after saving both of my models I use this script to test my models,

class MModel:    
    @staticmethod
    def loadmodel(path):
        model = Model(input_img, output_img)
        model.load_weights(path)
        return model
    def __init__(self, path):
       self.model = self.loadmodel(path)
       self.graph = tf.get_default_graph()

    def predict(self, X):
        with self.graph.as_default():
            return self.model.predict(X)

model1 = MModel('model1.h5')
model2 = MModel('model2.h5')

models_list = [model1, model2]
for model in models_list:
    prediction = model.predict(test_data)
    print(prediction)

Now problem is prediction for both model1 & model2 gives the same prediction for test_data. But when I load each model individually then model1 & model2 gives different predictions.

@mark86092 , @emilk, @kaltu Could anyone help me. I'm stuck on this problem. How can I solve this issue.

Thanks

[wangyexiang]

Have you solved your issue？I meet a similar question. I appreciate if you could point me to a solution. @HuiQiu1982

[mktal]

I am having the same issue as @menon92. Any solution?

[HuiQiu1982]

@mktal @wangyexiang ， try like this：

in main thread ：
sessionArray[i] = tf.Session(tf.Graph())

in thead_n:
session = sessionArray[n]
with session.as_default():
with session.graph.as_default()
do something。。。

[salihkaragoz]

Same issue.
One model, two different weights.
I want to make predictions at the same time.
Any solution? Thx

[wangyexiang]

A compromise method: Simply change the backend to mxnet. @salihkaragoz

[Anisha007]

@mark86092
My requirement is to build a web app using flask where I am loading 2 keras models for 2 different sub applications.

Your solution will load the models in the memory every time a request comes thus increasing the response time. Is there any way by which we can load both the models only once in memory using the init method?

Any leads from any one will be highly appreciated...

[zolekode]

Any body with this error after implementing @mark86092 solution ?
Attempting to capture an EagerTensor without building a function