Tfjs: Failed to load Sequential model created containing a Model layer

[yoongkang]

To get help from the community, check out our Google group.

TensorFlow.js version

0.4.0 (This is the Python library)
0.11.1 (For JavaScript library)

Browser version

I'm using the Python library, but also tried on Google Chrome Version 66.0.3359.139

Describe the problem or feature request

I have a Keras model trained using transfer learning on top of ResNet50. Initially, I trained a separate model using bottleneck features from ResNet50. Later, I attempted to combine these two models. The model summary is as below:

In [8]: model.summary()
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
resnet50 (Model)             (None, None, None, 2048)  23587712
_________________________________________________________________
reshape_2 (Reshape)          (None, 1, 1, 2048)        0
_________________________________________________________________
sequential_4 (Sequential)    (None, 133)               272517
=================================================================
Total params: 23,860,229
Trainable params: 23,807,109
Non-trainable params: 53,120
_________________________________________________________________

I was successful in using this combined model to perform predictions without generating bottleneck features -- despite a warning saying I have an uncompiled model.

I then attempted to use tensorflowjs to save this keras model, and read it again. This is the error I received:

In [9]: tfjs.converters.save_keras_model(model, './letstry3')


In [10]: tfjs.converters.load_keras_model('./letstry3/model.json')
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-10-08bba8228c6f> in <module>()
----> 1 tfjs.converters.load_keras_model('./letstry3/model.json')

~/udacity/udacity-dev/lib/python3.6/site-packages/tensorflowjs/converters/keras_tfjs_loader.py in load_keras_model(config_json_path, weights_path_prefix, weights_data_buffers, load_weights, use_unique_name_scope)
    206   return _deserialize_keras_model(config_json['modelTopology'],
    207                                   weight_entries=weight_entries,
--> 208                                   use_unique_name_scope=use_unique_name_scope)

~/udacity/udacity-dev/lib/python3.6/site-packages/tensorflowjs/converters/keras_tfjs_loader.py in _deserialize_keras_model(model_topology_json, weight_entries, use_unique_name_scope)
     72     weights_list = []
     73     for weight_name in weight_names:
---> 74       weights_list.append(weights_dict[weight_name])
     75     model.set_weights(weights_list)
     76

KeyError: 'conv1_2/kernel'

I also attempted to load this model via your JavaScript library, and I got this error:

Uncaught (in promise) Error: Could not find weights in manifest with names: conv1/kernel, conv1/bias, bn_conv1/gamma, ..(truncated)

I am pretty new to Keras and Tensorflow so it is entirely possible I am doing something completely out of the intended use case. In the event that is true, I would really love some advice on what to do instead.

Thanks!

Code to reproduce the bug / link to feature request

Please see above.

[bileschi]

Hi yoongkang@,
Sorry you are having trouble. This seems like a use-case that should work. Can you try something? Can you test whether you can save and load the model using the standard Keras h5 format? I.e., does the following work?

model.save('my_model.h5')  # creates a HDF5 file 'my_model.h5'
del model  # deletes the existing model

# returns a compiled model
# identical to the previous one
model = load_model('my_model.h5')

[yoongkang]

Hi @bileschi, thanks for your response!

My model was previously saved with the hdf5 extension. I just popped open the shell and did this, which worked.

In [1]: import keras
/Users/yoongkang/udacity/udacity-dev/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.


In [2]: model_from_weights = keras.models.load_model('dog_model2.weights.hdf5')
2018-05-26 11:34:03.193812: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
/Users/yoongkang/udacity/udacity-dev/lib/python3.6/site-packages/keras/models.py:255: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.
  warnings.warn('No training configuration found in save file: '


In [3]: model_from_weights.save('my_model.h5')


In [4]: new_model = keras.models.load_model('my_model.h5')
/Users/yoongkang/udacity/udacity-dev/lib/python3.6/site-packages/keras/models.py:255: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.
  warnings.warn('No training configuration found in save file: '


In [5]: new_model.summary()
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
resnet50 (Model)             (None, None, None, 2048)  23587712
_________________________________________________________________
reshape_2 (Reshape)          (None, 1, 1, 2048)        0
_________________________________________________________________
sequential_4 (Sequential)    (None, 133)               272517
=================================================================
Total params: 23,860,229
Trainable params: 23,807,109
Non-trainable params: 53,120
_________________________________________________________________

As you can see, there are some warnings there about my CPU containing instructions the Tensorflow binary was not compiled to use, not sure if that's relevant to the issue at hand.

EDIT: Would it be helpful for me to upload the .h5 file somewhere?

[jessetrana]

I am running into a similar issue but with Mobilenet loaded as a model. I use Mobilenet as a block inside a more complex model and then slice out part of the network. If I keep the Model as a block it fails, and if extract the input and output layer tensors of just the Mobilenet part it also fails.

[bileschi]

@yoongkang That would be really helpful, if you could post your .h5 file :). It sounds like we might have a bug with nested models?

[yoongkang]

@bileschi No worries, I've uploaded this to an S3 bucket:

https://s3-ap-southeast-2.amazonaws.com/yoongkang-machine-learning/my_model.h5

[bileschi]

Alright, I'm able to reproduce the error locally. Looking to find the root cause.

[bileschi]

Ok, the root cause of the naming mismatch is that loading the same model multiple times in the same scope leads to models with different weight names (to prevent collisions). You can reproduce this with the following code:

with tf.name_scope("my_scope"):
  model_1 = keras.models.model_from_json(json.dumps(model_topology_json))
  model_2 = keras.models.model_from_json(json.dumps(model_topology_json))
  model_3 = keras.models.model_from_json(json.dumps(model_topology_json))
print(model_1.weights[0])
print(model_2.weights[0])
print(model_3.weights[0])

yields

<tf.Variable 'conv1/kernel:0' shape=(7, 7, 3, 64) dtype=float32_ref>
<tf.Variable 'conv1_1/kernel:0' shape=(7, 7, 3, 64) dtype=float32_ref>
<tf.Variable 'conv1_2/kernel:0' shape=(7, 7, 3, 64) dtype=float32_ref>

Investigating what is the desired behavior here / workaround. There may be a way around this using the following pattern

with tf.Graph().as_default(), tf.Session():
   # your code here

or alternatively

model_prime = tfjs.converters.load_keras_model('./letstry3/model.json', use_unique_name_scope=True) 

But now i'm getting a different error
https://gist.github.com/bileschi/2c065740282eba1d9a05c87fe5d73ee4

[bileschi]

FYI: as a temporary work around, the command line seems to work here:

tensorflowjs_converter --input_format keras ./my_model.h5 ./my_model_as_tfjs 

Still investigating on the code side.

[caisq]

Part of this issue is related to an issue in TensorFlow Keras
https://github.com/tensorflow/tensorflow/issues/19836

[caisq]

FYI, regarding https://github.com/tensorflow/tensorflow/issues/19836, the issue has been fixed in tensorflow. But for tensorflowjs to resolve this issue, we need to wait for the next release of tensorflow (1.9.0). RC0 of 1.9.0 just came out this week. The final release of 1.9.0 is on the way.

[nsthorat]

@caisq any updates on the TensorFlow side of things?

[caisq]

Fixed now.