Coremltools: ValueError: Unknown layer: ReLU Converting a MobileNet pre-trained model

[aminecherif94]

I am converting a MobileNet pre-trained model that I trained using Keras. I am getting this error with Keras 2.1.6
Traceback (most recent call last):
File "keras_to_coreml_converter.py", line 18, in
class_labels=output_labels, image_input_names='image')
File "/home/francoiszerhouni/anaconda3/lib/python3.6/site-packages/coremltools/converters/keras/_keras_converter.py", line 752, in convert
custom_conversion_functions=custom_conversion_functions)
File "/home/francoiszerhouni/anaconda3/lib/python3.6/site-packages/coremltools/converters/keras/_keras_converter.py", line 550, in convertToSpec
custom_objects=custom_objects)
File "/home/francoiszerhouni/anaconda3/lib/python3.6/site-packages/coremltools/converters/keras/_keras2_converter.py", line 192, in _convert
model = _keras.models.load_model(model, custom_objects = custom_objects)
File "/home/francoiszerhouni/anaconda3/lib/python3.6/site-packages/keras/models.py", line 270, in load_model
model = model_from_config(model_config, custom_objects=custom_objects)
File "/home/francoiszerhouni/anaconda3/lib/python3.6/site-packages/keras/models.py", line 347, in model_from_config
return layer_module.deserialize(config, custom_objects=custom_objects)
File "/home/francoiszerhouni/anaconda3/lib/python3.6/site-packages/keras/layers/__init__.py", line 55, in deserialize
printable_module_name='layer')
File "/home/francoiszerhouni/anaconda3/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 144, in deserialize_keras_object
list(custom_objects.items())))
File "/home/francoiszerhouni/anaconda3/lib/python3.6/site-packages/keras/engine/topology.py", line 2525, in from_config
process_layer(layer_data)
File "/home/francoiszerhouni/anaconda3/lib/python3.6/site-packages/keras/engine/topology.py", line 2511, in process_layer
custom_objects=custom_objects)
File "/home/francoiszerhouni/anaconda3/lib/python3.6/site-packages/keras/layers/__init__.py", line 55, in deserialize
printable_module_name='layer')
File "/home/francoiszerhouni/anaconda3/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 138, in deserialize_keras_object
': ' + class_name)
ValueError: Unknown layer: ReLU

[aseemw]

This doesn't seem like a converter error.
It looks like there is an error in loading the Keras model itself at this line:
model = _keras.models.load_model(model, custom_objects = custom_objects)

To debug, for the moment, first just try to load the Keras model and see if that works without errors:

from keras.models import load_model
model = load_model('my_model.h5')
print(model.summary())

[aminecherif94]

Thank you for you reply. I changed it to keras 2.2.2 and he model loaded successfully. But I got this
Traceback (most recent call last):
File "keras_to_coreml_converter.py", line 20, in
class_labels=output_labels, image_input_names='image')
File "/home/francoiszerhouni/anaconda3/lib/python3.6/site-packages/coremltools/converters/keras/_keras_converter.
py", line 752, in convert
custom_conversion_functions=custom_conversion_functions)
File "/home/francoiszerhouni/anaconda3/lib/python3.6/site-packages/coremltools/converters/keras/_keras_converter.
py", line 534, in convertToSpec
from . import _keras2_converter
File "/home/francoiszerhouni/anaconda3/lib/python3.6/site-packages/coremltools/converters/keras/_keras2_converter
.py", line 70, in
_keras.applications.mobilenet.DepthwiseConv2D:_layers2.convert_convolution,
AttributeError: module 'keras.applications.mobilenet' has no attribute 'DepthwiseConv2D'

[aminecherif94]

Ok I fixed this but now I am getting this error:

Traceback (most recent call last):
File "keras_to_coreml_converter.py", line 20, in
class_labels=output_labels, image_input_names='image')
File "/home/francoiszerhouni/anaconda3/lib/python3.6/site-packages/coremltools/converters/keras/_keras_converter.py", line 752, in convert
custom_conversion_functions=custom_conversion_functions)
File "/home/francoiszerhouni/anaconda3/lib/python3.6/site-packages/coremltools/converters/keras/_keras_converter.py", line 550, in convertToSpec
custom_objects=custom_objects)
File "/home/francoiszerhouni/anaconda3/lib/python3.6/site-packages/coremltools/converters/keras/_keras2_converter.py", line 199, in _convert
_check_unsupported_layers(model, add_custom_layers)
File "/home/francoiszerhouni/anaconda3/lib/python3.6/site-packages/coremltools/converters/keras/_keras2_converter.py", line 105, in _check_unsupported_layers
raise ValueError("Keras layer '%s' not supported. " % str(type(layer)))
ValueError: Keras layer '' not supported.
francoiszerhouni@instance-1:~/ai-toolkit-iot-edge/Skin cancer detection/skin_cancer_coreml_model$

[bourdakos1]

@aminecherif94 I was having a similar issue, try this:

from keras.utils.generic_utils import CustomObjectScope

with CustomObjectScope({'relu6': keras.applications.mobilenet.relu6, 'DepthwiseConv2D': keras.applications.mobilenet.DepthwiseConv2D}):
    # coreml conversion code

[mmalekzadeh]

I've got the same error. Coremltools needs an upgrade.

[aminecherif94]

Yeah, I had to rewrote some of the script to make it work. DepthwiseConv2D and RELU6 have to be added. Also, lambda layers are not handled by CoreML.

[aseemw]

You should be able to use the tip of master.
A change recently went it (PR #278) that resolves the issue with the way Keras changed its way of exposing ReLU6 layer between versions 2.2.0 and 2.2.1

[mmalekzadeh]

Thanks, @aseemw. I installed from the GitHub repo instead of pip repository:
pip install git+https://github.com/apple/coremltools.git
And it's fine now.

However, I have other issues from graph.build() in _keras2_converter.py which seems it comes from my Model that is trained with the latest Keras version.

[aminecherif94]

@mmalekzadeh Please write down the full error message.

[mmalekzadeh]

@aminecherif94 Here's the error:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-21-26e8d3454ec5> in <module>()
      1 coreml_model = coremltools.converters.keras.convert(
----> 2     model=mymodel)
      3 coreml_model.save('mymodel.mlmodel')

/anaconda/envs/py35/lib/python3.5/site-packages/coremltools/converters/keras/_keras_converter.py in convert(model, input_names, output_names, image_input_names, input_name_shape_dict, is_bgr, red_bias, green_bias, blue_bias, gray_bias, image_scale, class_labels, predicted_feature_name, model_precision, predicted_probabilities_output, add_custom_layers, custom_conversion_functions)
    758                       predicted_probabilities_output,
    759                       add_custom_layers,
--> 760                       custom_conversion_functions=custom_conversion_functions)
    761 
    762     return _MLModel(spec)

/anaconda/envs/py35/lib/python3.5/site-packages/coremltools/converters/keras/_keras_converter.py in convertToSpec(model, input_names, output_names, image_input_names, input_name_shape_dict, is_bgr, red_bias, green_bias, blue_bias, gray_bias, image_scale, class_labels, predicted_feature_name, model_precision, predicted_probabilities_output, add_custom_layers, custom_conversion_functions, custom_objects)
    554                                            add_custom_layers=add_custom_layers,
    555                                            custom_conversion_functions=custom_conversion_functions,
--> 556                                            custom_objects=custom_objects)
    557     else:
    558         raise RuntimeError(

/anaconda/envs/py35/lib/python3.5/site-packages/coremltools/converters/keras/_keras2_converter.py in _convert(model, input_names, output_names, image_input_names, input_name_shape_dict, is_bgr, red_bias, green_bias, blue_bias, gray_bias, image_scale, class_labels, predicted_feature_name, predicted_probabilities_output, add_custom_layers, custom_conversion_functions, custom_objects)
    207     # Build network graph to represent Keras model
    208     graph = _topology2.NetGraph(model)
--> 209     graph.build()
    210 
    211     # The graph should be finalized before executing this

/anaconda/envs/py35/lib/python3.5/site-packages/coremltools/converters/keras/_topology2.py in build(self, is_top_level)
    676             keras_layer = self.keras_layer_map[layer]
    677             predecessors = self.reverse_edge_map[layer]
--> 678             successors = self.edge_map[layer]
    679             new_layers = [layer+'_'+str(i) for i in range(len(predecessors))]
    680             self.layer_list[idx:idx+1] = new_layers

KeyError: 'Decoder'

[aminecherif94]

It seems like there is a problem with you Keras model as CoreML is not able to get the successor or next layers of your Keras model. Please do this to visualize the layers of your model and copy paste the output here so that I can see what is the problem with your model :

print(mymodel.summary())
Do it before the line that calls CoreML.

[mmalekzadeh]

Thanks, @aminecherif94. But I don't think so,
Here is the Model summary:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_7 (InputLayer)         (None, 2, 128, 1)         0         
_________________________________________________________________
conv2d_20 (Conv2D)           (None, 2, 128, 100)       400       
_________________________________________________________________
batch_normalization_9 (Batch (None, 2, 128, 100)       400       
_________________________________________________________________
conv2d_21 (Conv2D)           (None, 2, 128, 100)       30100     
_________________________________________________________________
batch_normalization_10 (Batc (None, 2, 128, 100)       400       
_________________________________________________________________
max_pooling2d_13 (MaxPooling (None, 2, 64, 100)        0         
_________________________________________________________________
conv2d_22 (Conv2D)           (None, 2, 64, 100)        30100     
_________________________________________________________________
batch_normalization_11 (Batc (None, 2, 64, 100)        400       
_________________________________________________________________
max_pooling2d_14 (MaxPooling (None, 2, 32, 100)        0         
_________________________________________________________________
conv2d_23 (Conv2D)           (None, 2, 32, 100)        30100     
_________________________________________________________________
batch_normalization_12 (Batc (None, 2, 32, 100)        400       
_________________________________________________________________
Encoder (Conv2D)             (None, 2, 32, 1)          301       
_________________________________________________________________
batch_normalization_13 (Batc (None, 2, 32, 1)          4         
_________________________________________________________________
conv2d_transpose_5 (Conv2DTr (None, 2, 32, 100)        400       
_________________________________________________________________
batch_normalization_14 (Batc (None, 2, 32, 100)        400       
_________________________________________________________________
conv2d_transpose_6 (Conv2DTr (None, 2, 64, 100)        30100     
_________________________________________________________________
batch_normalization_15 (Batc (None, 2, 64, 100)        400       
_________________________________________________________________
conv2d_transpose_7 (Conv2DTr (None, 2, 128, 100)       30100     
_________________________________________________________________
batch_normalization_16 (Batc (None, 2, 128, 100)       400       
_________________________________________________________________
conv2d_transpose_8 (Conv2DTr (None, 2, 128, 100)       30100     
_________________________________________________________________
Decoder (Conv2D)             (None, 2, 128, 1)         301       
=================================================================
Total params: 184,806
Trainable params: 183,404
Non-trainable params: 1,402
_________________________________________________________________

[aminecherif94]

Python raises a KeyError whenever a dict() object is requested (using the format a = adict[key]) and the key is not in the dictionary. In your case, KeyError: 'Decoder', which means it is not able to get the successor of layer 'Decoder'. But Decoder layer logically can't have a successor as it is the final layer. So the problem is with the output layer. To test what I am saying just import any pertained Keras model and convert it to test if CoreML can convert it. But If I had to take a wild guess, when you implemented the Decoder layer, you did not specify it as being the output layer. Like this:

outputs = Conv2D(...)(base_model.output)

[aminecherif94]

Here is the code where you are getting the error:

        # Duplicate models for weight sharing
        idx = self._get_first_shared_layer()
        while idx >= 0:
            layer = self.layer_list[idx]
            keras_layer = self.keras_layer_map[layer]
            predecessors = self.reverse_edge_map[layer]
            successors = self.edge_map[layer]
            new_layers = [layer+'_'+str(i) for i in range(len(predecessors))]
            self.layer_list[idx:idx+1] = new_layers
            for i, new_layer in enumerate(new_layers):
                self.edge_map[new_layer] = []
                self.reverse_edge_map[new_layer] = []
                self.keras_layer_map[new_layer] = keras_layer
                pred = predecessors[i]
                self._add_edge(pred, new_layer)
                for succ in successors:
                    self._add_edge(new_layer, succ)
            self._remove_old_edges(layer)
            self.keras_layer_map.pop(layer)
            idx = self._get_first_shared_layer()

It calls the function get_first_shared_layer() which returns the index of the shared layer. The outer while loop should stop when idx<0 but it is entering the loop when more time because idx is not less than 0 and it looks for the successor of Decoder layer which does not exist and raises KeyError: 'Decoder'

[mmalekzadeh]

But Decoder is the last layer, why it looks for the successor of Decoder layer?

[aminecherif94]

That is what I am saying, when you implemented the last layer Decoder you did not specify it as final layer or output layer.

[mmalekzadeh]

Thanks, @aminecherif94, for the hints. I've added a Reshape((2,128,1)) layer which is actually doing nothing and now it works.

[aminecherif94]

It was my pleasure @mmalekzadeh :)

[relax94]

For Keras 2.2.4 and Tensorflow 1.12.0 I found a solution.

Save model weights & architecture like:

model_json = model.to_json()
open('architecture.json', 'w').write(model_json)
model.save_weights('weights.h5', overwrite=True)

And for converting a model to CoreML .mlmodel I use:

import coremltools

from keras.layers import DepthwiseConv2D, ReLU
from pathlib import Path
from keras.models import model_from_json
from tensorflow.python.keras.utils.generic_utils import CustomObjectScope

model_architecture = './Networks/architecture.json'
model_weights = './Networks/weights.h5'

model_structure = Path(model_architecture).read_text()

with CustomObjectScope({'relu6': ReLU ,'DepthwiseConv2D': DepthwiseConv2D}):
    model = model_from_json(model_structure)
    model.load_weights(model_weights)

    output_labels = ['0', '1', '2', '3', '4', '5', '6']
    coreml_model = coremltools.converters.keras.convert(
        model, input_names=['image'], output_names=['output'],
        class_labels=output_labels, image_input_names='image')

    coreml_model.save('ModelX.mlmodel')