Coremltools: Can't convert TF2 Zoo model

[fotiDim]

🐞Description the bug

• Tensorflow model zoo was recently updated to TF2. I attempted to convert one of their sample models (efficientdet_d5_coco17_tpu-32) but got a model format error.

Trace

---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
 in 
      4 
      5 # Convert to CoreML
----> 6 mlmodel = ct.convert(detect_fn, source='tensorflow')

~/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/coremltools/converters/_converters_entry.py in convert(model, source, inputs, outputs, classifier_config, minimum_deployment_target, **kwargs)
    256             outputs=outputs,
    257             classifier_config=classifier_config,
--> 258             **kwargs
    259         )
    260 

~/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/coremltools/converters/mil/converter.py in _convert(model, convert_from, convert_to, converter_registry, **kwargs)
    118     backend_converter = backend_converter_type()
    119 
--> 120     prog = frontend_converter(model, **kwargs)
    121     common_pass(prog)
    122     out = backend_converter(prog, **kwargs)

~/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/coremltools/converters/mil/converter.py in __call__(self, *args, **kwargs)
     50 
     51         tf2_loader = TF2Loader(*args, **kwargs)
---> 52         return tf2_loader.load()
     53 
     54 

~/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/coremltools/converters/mil/frontend/tensorflow/load.py in load(self)
     56         logging.info("Loading TensorFlow model '{}'".format(self.model))
     57         outputs = self.kwargs.get("outputs", None)
---> 58         self._graph_def = self._graph_def_from_model(outputs)
     59 
     60         if self._graph_def is not None and len(self._graph_def.node) == 0:

~/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/coremltools/converters/mil/frontend/tensorflow2/load.py in _graph_def_from_model(self, outputs)
    108             return self.extract_sub_graph(graph_def, outputs)
    109         else:
--> 110             raise NotImplementedError(msg.format(self.model))
    111 
    112     def _tf_ssa_from_graph_def(self, fn_name="main"):

NotImplementedError: Expected model format: [SavedModel | [concrete_function] | tf.keras.Model | .h5], got ._UserObject object at 0x1852730d0>

To Reproduce

Use this notebook and add this code a last step:

!pip install coremltools==4.0b2
import coremltools as ct

# Convert to CoreML
mlmodel = ct.convert(detect_fn, source='tensorflow')

System environment (please complete the following information):

• coremltools version 4.0b2
• macOS version 10.15.5 (19F101)
• XCode version (if applicable): Version 11.6 (11E708)
• How you install python (anaconda, virtualenv, system): Pipenv
  This is my pipfile:
  ```
  [[source]]
  name = "pypi"
  url = "https://pypi.org/simple"
  verify_ssl = true

[dev-packages]

[packages]
coremltools = "==4.0b2"
tensorflow = ""
jupyter = ""
matplotlib = "*"
scipy = "==1.4.1"

[requires]
python_version = "3.7"

[siddhantsomani]

Can you try passing in the path to the SavedModel folder directly?

mlmodel = ct.convert(model="models/research/object_detection/test_data/efficientdet_d5_coco17_tpu-32/saved_model/", source="tensorflow")

[fotiDim]

@siddhantsomani in that case I am getting this trace:

Running TensorFlow Graph Passes:   0%|          | 0/5 [00:00<?, ? passes/s]ERROR:root:Constant Propagation pass failed: Fetch argument 'StatefulPartitionedCall/MultiscaleGridAnchorGenerator/GridAnchorGenerator/assert_equal_1/Assert/Assert:0' cannot be interpreted as a Tensor. ("The name 'StatefulPartitionedCall/MultiscaleGridAnchorGenerator/GridAnchorGenerator/assert_equal_1/Assert/Assert:0' refers to a Tensor which does not exist. The operation, 'StatefulPartitionedCall/MultiscaleGridAnchorGenerator/GridAnchorGenerator/assert_equal_1/Assert/Assert', exists but only has 0 outputs.")
Traceback (most recent call last):
  File "/Users/fotidim/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3556, in _as_graph_element_locked
    return op.outputs[out_n]
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/fotidim/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 305, in __init__
    fetch, allow_tensor=True, allow_operation=True))
  File "/Users/fotidim/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3512, in as_graph_element
    return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
  File "/Users/fotidim/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3561, in _as_graph_element_locked
    (repr(name), repr(op_name), len(op.outputs)))
KeyError: "The name 'StatefulPartitionedCall/MultiscaleGridAnchorGenerator/GridAnchorGenerator/assert_equal_1/Assert/Assert:0' refers to a Tensor which does not exist. The operation, 'StatefulPartitionedCall/MultiscaleGridAnchorGenerator/GridAnchorGenerator/assert_equal_1/Assert/Assert', exists but only has 0 outputs."

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/fotidim/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/coremltools/converters/mil/frontend/tensorflow/tf_graph_pass/constant_propagation.py", line 96, in _constant_propagation
    result_list = sess.run(query_list)
  File "/Users/fotidim/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 958, in run
    run_metadata_ptr)
  File "/Users/fotidim/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1166, in _run
    self._graph, fetches, feed_dict_tensor, feed_handles=feed_handles)
  File "/Users/fotidim/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 477, in __init__
    self._fetch_mapper = _FetchMapper.for_fetch(fetches)
  File "/Users/fotidim/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 266, in for_fetch
    return _ListFetchMapper(fetch)
  File "/Users/fotidim/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 378, in __init__
    self._mappers = [_FetchMapper.for_fetch(fetch) for fetch in fetches]
  File "/Users/fotidim/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 378, in <listcomp>
    self._mappers = [_FetchMapper.for_fetch(fetch) for fetch in fetches]
  File "/Users/fotidim/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 276, in for_fetch
    return _ElementFetchMapper(fetches, contraction_fn)
  File "/Users/fotidim/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 315, in __init__
    'Tensor. (%s)' % (fetch, str(e)))
ValueError: Fetch argument 'StatefulPartitionedCall/MultiscaleGridAnchorGenerator/GridAnchorGenerator/assert_equal_1/Assert/Assert:0' cannot be interpreted as a Tensor. ("The name 'StatefulPartitionedCall/MultiscaleGridAnchorGenerator/GridAnchorGenerator/assert_equal_1/Assert/Assert:0' refers to a Tensor which does not exist. The operation, 'StatefulPartitionedCall/MultiscaleGridAnchorGenerator/GridAnchorGenerator/assert_equal_1/Assert/Assert', exists but only has 0 outputs.")
Running TensorFlow Graph Passes:  20%|██        | 1/5 [00:05<00:23,  5.94s/ passes]

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-12-caac80a09913> in <module>
      5 # Convert to CoreML
      6 # mlmodel = ct.convert(detect_fn, source='tensorflow')
----> 7 mlmodel = ct.convert(model="models/research/object_detection/test_data/efficientdet_d5_coco17_tpu-32/saved_model/", source="tensorflow")

~/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/coremltools/converters/_converters_entry.py in convert(model, source, inputs, outputs, classifier_config, minimum_deployment_target, **kwargs)
    256             outputs=outputs,
    257             classifier_config=classifier_config,
--> 258             **kwargs
    259         )
    260 

~/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/coremltools/converters/mil/converter.py in _convert(model, convert_from, convert_to, converter_registry, **kwargs)
    118     backend_converter = backend_converter_type()
    119 
--> 120     prog = frontend_converter(model, **kwargs)
    121     common_pass(prog)
    122     out = backend_converter(prog, **kwargs)

~/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/coremltools/converters/mil/converter.py in __call__(self, *args, **kwargs)
     50 
     51         tf2_loader = TF2Loader(*args, **kwargs)
---> 52         return tf2_loader.load()
     53 
     54 

~/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/coremltools/converters/mil/frontend/tensorflow/load.py in load(self)
     77             )
     78 
---> 79         program = self._program_from_tf_ssa()
     80         logging.debug("program:\n{}".format(program))
     81         return program

~/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/coremltools/converters/mil/frontend/tensorflow2/load.py in _program_from_tf_ssa(self)
    166                 tf_passes, desc="Running TensorFlow Graph Passes", unit=" passes"
    167             ):
--> 168                 tf_pass(self._tf_ssa)
    169 
    170         if self.debug:

~/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/coremltools/converters/mil/frontend/tensorflow2/tf_graph_pass/rewrite_control_flow_functions.py in rewrite_control_flow_functions(tf_ssa)
    543 def rewrite_control_flow_functions(tf_ssa):
    544     for fn_name, fn in tf_ssa.functions.items():
--> 545         _rewrite_cond_functions(tf_ssa, fn)
    546     for fn_name, fn in tf_ssa.functions.items():
    547         _eliminate_loop_cond_nodes(tf_ssa, fn)

~/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/coremltools/converters/mil/frontend/tensorflow2/tf_graph_pass/rewrite_control_flow_functions.py in _rewrite_cond_functions(tf_ssa, fn)
    258                     idx = then_fn.outputs.index(mapped_name)
    259                 else:  # in else_fn.outputs
--> 260                     idx = else_fn.outputs.index(mapped_name)
    261             else:
    262                 idx = i

ValueError: '' is not in list

[siddhantsomani]

I can reproduce this. Any ideas if this is related to the 'delete_asserts control flow' bug @1duo ?

[akhilraja-p]

I have the same error. Unable to convert any tf2.0 models

[akhilraja-p]

++ @1duo for more visibility
@jakesabathia2 This is similar to Issue

[advaza]

@fotiDim upgrading coremltools to 4.0b4 solved the issue to me.
I'm trying to convert the same efficientdet model, but I'm now getting a different error:

Running TensorFlow Graph Passes: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:01<00:00,  3.46 passes/s]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-473b9a27a6e0> in <module>
----> 1 mlmodel = ct.convert(pb_path, source='tensorflow')

~/anaconda/lib/python3.6/site-packages/coremltools/converters/_converters_entry.py in convert(model, source, inputs, outputs, classifier_config, minimum_deployment_target, **kwargs)
    263             outputs=outputs,
    264             classifier_config=classifier_config,
--> 265             **kwargs
    266         )
    267

~/anaconda/lib/python3.6/site-packages/coremltools/converters/mil/converter.py in _convert(model, convert_from, convert_to, converter_registry, **kwargs)
    132     frontend_converter = frontend_converter_type()
    133
--> 134     prog = frontend_converter(model, **kwargs)
    135     common_pass(prog)
    136

~/anaconda/lib/python3.6/site-packages/coremltools/converters/mil/converter.py in __call__(self, *args, **kwargs)
     72
     73         tf2_loader = TF2Loader(*args, **kwargs)
---> 74         return tf2_loader.load()
     75
     76

~/anaconda/lib/python3.6/site-packages/coremltools/converters/mil/frontend/tensorflow/load.py in load(self)
     78             )
     79
---> 80         program = self._program_from_tf_ssa()
     81         logging.debug("program:\n{}".format(program))
     82         return program

~/anaconda/lib/python3.6/site-packages/coremltools/converters/mil/frontend/tensorflow2/load.py in _program_from_tf_ssa(self)
    176
    177         converter = TF2Converter(self._tf_ssa, **self.kwargs)
--> 178         return converter.convert()
    179
    180     def _populate_sub_graph_input_shapes(self, graph, graph_fns):

~/anaconda/lib/python3.6/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py in convert(self)
    403         for g_name in self.graph_stack[1:]:
    404             self.context.add_graph(g_name, self.tfssa.functions[g_name].graph)
--> 405         self.convert_main_graph(prog, graph)
    406
    407         # Apply TF frontend passes on Program. These passes are different

~/anaconda/lib/python3.6/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py in convert_main_graph(self, prog, graph)
    332             for name in func_inputs.keys():
    333                 self.context.add(name, ssa_func.inputs[name])
--> 334             outputs = convert_graph(self.context, graph, self.outputs)
    335             ssa_func.set_outputs(outputs)
    336             prog.add_function("main", ssa_func)

~/anaconda/lib/python3.6/site-packages/coremltools/converters/mil/frontend/tensorflow/convert_utils.py in convert_graph(context, graph, outputs)
    144     """
    145     connect_global_initializer(graph)
--> 146     nodes = topsort(graph)
    147
    148     if outputs is None:

~/anaconda/lib/python3.6/site-packages/coremltools/converters/mil/frontend/tensorflow/basic_graph_ops.py in topsort(graph)
    332         nextboundary = []
    333     if len(ret) != len(graph):
--> 334         raise ValueError("Graph is not a DAG!")
    335     return ret
    336

ValueError: Graph is not a DAG!

Any idea if that could be solved?

[akhilraja-p]

Same error for F-RCNN graphs as well.
However for the SSD I get this error

Running TensorFlow Graph Passes: 100%|██████████| 5/5 [00:03<00:00, 1.47 passes/s]

Converting Frontend ==> MIL Ops: 63%|██████▎ | 2847/4491 [00:05<00:03, 504.93 ops/s]

`NotImplementedError Traceback (most recent call last)
in
17 get_ipython().system('ls')
18
---> 19 mlmodel = ct.convert("ssd/saved_model")
20 # mlmodel = ct.convert(tf_keras_model,source="tensorflow")
21

~/opt/anaconda3/envs/MLG/lib/python3.7/site-packages/coremltools/converters/_converters_entry.py in convert(model, source, inputs, outputs, classifier_config, minimum_deployment_target, *kwargs)
263 outputs=outputs,
264 classifier_config=classifier_config,
--> 265 *kwargs
266 )
267

~/opt/anaconda3/envs/MLG/lib/python3.7/site-packages/coremltools/converters/mil/converter.py in _convert(model, convert_from, convert_to, converter_registry, *kwargs)
132 frontend_converter = frontend_converter_type()
133
--> 134 prog = frontend_converter(model, *kwargs)
135 common_pass(prog)
136

~/opt/anaconda3/envs/MLG/lib/python3.7/site-packages/coremltools/converters/mil/converter.py in __call__(self, args, *kwargs)
72
73 tf2_loader = TF2Loader(args, *kwargs)
---> 74 return tf2_loader.load()
75
76

~/opt/anaconda3/envs/MLG/lib/python3.7/site-packages/coremltools/converters/mil/frontend/tensorflow/load.py in load(self)
78 )
79
---> 80 program = self._program_from_tf_ssa()
81 logging.debug("program:\n{}".format(program))
82 return program

~/opt/anaconda3/envs/MLG/lib/python3.7/site-packages/coremltools/converters/mil/frontend/tensorflow2/load.py in _program_from_tf_ssa(self)
176
177 converter = TF2Converter(self._tf_ssa, **self.kwargs)
--> 178 return converter.convert()
179
180 def _populate_sub_graph_input_shapes(self, graph, graph_fns):

~/opt/anaconda3/envs/MLG/lib/python3.7/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py in convert(self)
403 for g_name in self.graph_stack[1:]:
404 self.context.add_graph(g_name, self.tfssa.functions[g_name].graph)
--> 405 self.convert_main_graph(prog, graph)
406
407 # Apply TF frontend passes on Program. These passes are different

~/opt/anaconda3/envs/MLG/lib/python3.7/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py in convert_main_graph(self, prog, graph)
332 for name in func_inputs.keys():
333 self.context.add(name, ssa_func.inputs[name])
--> 334 outputs = convert_graph(self.context, graph, self.outputs)
335 ssa_func.set_outputs(outputs)
336 prog.add_function("main", ssa_func)

~/opt/anaconda3/envs/MLG/lib/python3.7/site-packages/coremltools/converters/mil/frontend/tensorflow/convert_utils.py in convert_graph(context, graph, outputs)
178 node.op, node.original_node
179 )
--> 180 raise NotImplementedError(msg)
181 _add_op(context, node)
182

NotImplementedError: Conversion for TF op 'NonMaxSuppressionV5' not implemented.

name: "StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores/NonMaxSuppressionV5"
op: "NonMaxSuppressionV5"
input: "StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/unstack"
input: "StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/Reshape"
input: "StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/Minimum"
input: "StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores/iou_threshold"
input: "StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores/score_threshold"
input: "StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores/soft_nms_sigma"
attr {
key: "T"
value {
type: DT_FLOAT
}
}
attr {
key: "pad_to_max_output_size"
value {
b: false
}
}`

[hlloydt]

I am also seeing both these errors when converting models for the TensorFlow2 Zoo:

• Converting RetinaNet: "NotImplementedError: Conversion for TF op 'NonMaxSuppressionV5' not implemented."
• Converting EfficientDet: "ValueError: Graph is not a DAG!"

I would love to see these get fixed and the op 'NonMaxSuppressionV5' get implemented, please.