Coremltools: "RuntimeError: failed to convert from IR to Core ML." error when converting tf.keras to coreml in tensorflow 2.0

[michaelarfreed]

🐞Describe the bug

• A clear and brief description of what the bug is. I am unable to convert a tf.keras model to coreml in tensorflow 2.0 because of a RuntimeError
• Is this a converter issue? Yes. If yes, please specify which converter (sci-kit, keras, xgboost etc.). tensorflow converter, using tf.keras in tensorflow 2.0

Trace

AssertionError                            Traceback (most recent call last)
~/anaconda3/envs/myenv/lib/python3.7/site-packages/coremltools/converters/tensorflow/_tf_converter.py in convert(filename, inputs, outputs, image_input_names, is_bgr, red_bias, green_bias, blue_bias, gray_bias, image_scale, class_labels, predicted_feature_name, predicted_probabilities_output, add_custom_layers, custom_conversion_functions, custom_shape_functions, **kwargs)
     95                                  custom_shape_functions=custom_shape_functions,
---> 96                                  optional_inputs=optional_inputs)
     97     except ImportError as e:

~/anaconda3/envs/myenv/lib/python3.7/site-packages/coremltools/converters/nnssa/coreml/ssa_converter.py in ssa_convert(ssa, top_func, inputs, outputs, image_input_names, is_bgr, red_bias, green_bias, blue_bias, gray_bias, image_scale, class_labels, predicted_feature_name, predicted_probabilities_output, add_custom_layers, custom_conversion_functions, custom_shape_functions, optional_inputs)
    132     for f in list(ssa.functions.values()):
--> 133         check_connections(f.graph)
    134 

~/anaconda3/envs/myenv/lib/python3.7/site-packages/coremltools/converters/nnssa/commons/basic_graph_ops.py in check_connections(gd)
    151         for i in v.control_outputs:
--> 152             assert (k in gd[i].control_inputs)
    153 

AssertionError: 

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
<ipython-input-4-d56b125b2b47> in <module>
      4                  output_feature_names= ['output'],
      5                  input_name_shape_dict= {'input': [1,450]},
----> 6                  class_labels=['A','B','C','D','E','F','G']) )
      7
      8

~/anaconda3/envs/myenv/lib/python3.7/site-packages/coremltools/converters/tensorflow/_tf_converter.py in convert(filename, inputs, outputs, image_input_names, is_bgr, red_bias, green_bias, blue_bias, gray_bias, image_scale, class_labels, predicted_feature_name, predicted_probabilities_output, add_custom_layers, custom_conversion_functions, custom_shape_functions, **kwargs)
     98         raise ImportError('backend converter not found. {}'.format(e))
     99     except Exception as e:
--> 100         raise RuntimeError('failed to convert from IR to Core ML. {}'.format(e))
    101 
    102     return MLModel(model_spec, useCPUOnly=use_cpu_only)

RuntimeError: failed to convert from IR to Core ML. 

To Reproduce

• If a python script can reproduce the error, please paste the code snippet

import tensorflow as tf
mymodel = tf.keras.Sequential([ 
    tf.keras.layers.Reshape((75,6),input_shape=(6*75,)), 
    tf.keras.layers.Dense(500,activation='relu'), 
    tf.keras.layers.Flatten(), 
    tf.keras.layers.Dense(7,activation='softmax')])  
mymodel.compile(optimizer='adam',loss='categorical_crossentropy')
mymodel.save('/path/to/mymodel.h5',save_format='h5')
coremltools.converters.tensorflow.convert('/path/to/mymodel.h5',  
                  mlmodel_path='my_model.mlmodel',  
                  output_feature_names= ['output'],   
                  input_name_shape_dict= {'input': [1,450]}, 
                  class_labels=['A','B','C','D','E','F','G']))     

• If applicable, please attach the source model
  
  
  
  • If the model cannot be shared publicly, please attach it via filing a bug report at https://developer.apple.com/bug-reporting/ and provide the reference number here
    
    I am not attaching a source model because I can replicate it with any model I train in Keras.
  
  
• If it is a model conversion issue and the conversion succeeds, however, if there is a numerical mismatch between the original and the coreml model, please paste script used for comparison.

System environment (please complete the following information):

• coremltools version (e.g., 3.0b5): 3.1
• OS (e.g., MacOS, Linux): MacOS
• macOS version (if applicable): 10.14.16
• XCode version (if applicable): N/A
• How you install python (anaconda, virtualenv, system): anaconda
• python version (e.g. 3.7): 3.7
• any other relevant information:
  - tensorflow version: 2.0

Additional context

I was using coremltools converter before. I was training a model in tf.keras (version tf 1.13.1) and then using the keras converter, and it caused no errors. I am migrating to tf 2.0 and this is the last step where I am encountering errors. I have confirmed that my model trains, saves, loads, and validates well, and that the error is coming from the coreml converter step.

---

Update

I did some more digging... I think the Reshape layer is causing the conversion error. When I tried with the following model, I got a different error

mymodel = tf.keras.Sequential([ 
    tf.keras.layers.Dense(500,activation='relu',input_shape=(6*75,)), 
    tf.keras.layers.Flatten(), 
    tf.keras.layers.Dense(7,activation='softmax')])  ])

The error I got was:

~/anaconda3/envs/myenv/lib/python3.7/site-packages/coremltools/models/model.py:111: RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was: Error compiling model: "Error reading protobuf spec. validator error: The .mlmodel supplied is of version 4, intended for a newer version of Xcode. This version of Xcode supports model version 3 or earlier.".

[aseemw]

@michaelarfreed the code snippet you provided should work correctly, if you make a slight change to it. That is, provide the correct output name to the convert function. Instead of 'output', it can be obtained via keras_model.output[0].name

model = tf.keras.Sequential()
model.add(tf.keras.layers.Reshape((75, 6), input_shape=(6 * 75,)))
model.add(tf.keras.layers.Dense(500, activation='relu'))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(7, activation='softmax'))
model.save("/tmp/keras_model.h5")

# convert this model to Core ML format
input_name = model.inputs[0].name.split(':')[0]
keras_out_name = model.outputs[0].name.split(':')[0]
graph_output_name = keras_out_name.split('/')[-1]
mlmodel = tfcoreml.convert(tf_model_path="/tmp/keras_model.h5",
                         input_name_shape_dict={input_name: (1, 6*75)},
                         output_feature_names=[graph_output_name],
                         minimum_ios_deployment_target='13')
mlmodel.save('/tmp/keras_model.mlmodel')

[aseemw]

The converter should raise a more informative error message though. So will keep this issue open to track that change.

[aseemw]

(Have at least updated the documentation in PR #546 )

[michaelarfreed]

Thank you so much for the response! This fixed that issue in converter, and I am no longer getting this error in this example code or in my repo. In my less simplified repo, I am now getting an error AttributeError: module 'tensorflow' has no attribute 'reset_default_graph', but I will open a new issue for this.

EDIT: Nevermind, I forgot to include minimum_ios_deployment_target='13', so now it is working.

[michaelarfreed]

I have a follow up question: calling tfcoreml.convert (the code you provided) works great, but I would prefer to use coremltools.converters.tensorflow.convert. However, I am still getting the same RuntimeError when I change the call to

coreml_model = coremltools.converters.tensorflow.convert(os.path.join(hparams.model_dir,yml_out['model_filename']),  
                                                            input_name_shape_dict={input_name: (1,6*75)},
                                                            output_feature_names=[graph_out_name],
                                                            class_labels = gestL, 
                                                            minimum_ios_deployment_target='13')

Is the new convention to use tfcoreml instead of coremltools? Or should I still be able to call from coremltools with the same arguments?

[1duo]

This bug fix should be included in coremltools 3.2 release. Please upgrade your coremltools(pip install --upgrade coremltools) to verify. Feel free to re-open if you still encountering this issue. Thanks!

[rodyt]

@1duo can confirm that after updating to coremltools 3.2, the error no longer arises.

However, another issue came up: I cannot convert a model with 2 outputs in coremltools: https://github.com/tf-coreml/tf-coreml/issues/374

Any ideas? @aseemw