Coremltools: ValueError: list.remove(x): x not in list

[danvargg]

🐞Describe the bug

I'm training a multi-input CNN (tf.keras) with TF2.0.0. I keep getting ValueError: list.remove(x): x not in list, but I don't understand what's the list, what's x, do I need to rename nodes?

I can see that the converter starts, but the error comes after some conversion. I tried tfcoreml and converting from saved weights with no success.
Is there something I'm missing?

This is the process I follow for conversion:

tf.keras.backend.clear_session()
tf.keras.backend.set_learning_phase(0)

# model is a .h5 file produced from a model.fit callback (modelcheckpoint)
# I have a custom dense layer that has __init__, build, call and config implemented
model = tf.keras.models.load_model(model, custom_objects = {'DenseLayer': DenseLayer})

tf.saved_model.save(model, export_dir = pb_model) # save the model as pb for conversion

# my target ios is 12
coremltools_model = coremltools.converters.tensorflow.convert(
        _PB_MODEL,
        input_name_shape_dict={
            'B': [1, 20, 40, 1],
            'G': [1, 20, 40, 1],
            'R': [1, 20, 40, 1],
            'X': [1, 2],
            'Y': [1, 4]},
        output_feature_names=['dense_layer_18/Identity'],
        minimum_ios_deployment_target='12'
    )

coremltools_model.save(ml_model)

Trace

Converting pb model to .mlmodel......
0 assert nodes deleted
872 nodes deleted
36 nodes deleted
0 nodes deleted
[Op Fusion] fuse_bias_add() deleted 24 nodes.
77 identity nodes deleted
Traceback (most recent call last):
  File "tf2coreml.py", line 198, in <module>
convertModel(model=_TRAIN_MODEL, ml_model=_ML_MODEL, pb_model=_PB_MODEL)
  File "tf2coreml.py", line 174, in convertModel
minimum_ios_deployment_target='12'
  File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/coremltools/converters/tensorflow/_tf_converter.py", line 193, in convert
optional_inputs=optional_inputs)
  File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/coremltools/converters/nnssa/coreml/ssa_converter.py", line 130, in ssa_convert
p(ssa)
  File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/coremltools/converters/nnssa/coreml/graph_pass/op_removals.py", line 20, in
remove_no_ops_and_shift_control_dependencies
    f.graph[each_control_input].control_outputs.remove(node.name)
ValueError: list.remove(x): x not in list

System environment (please complete the following information):

• coremltools version: 3.2
• OS: Ubuntu 16.04 LTS (AWS EC2)
• Python version: 3.6.5
• any other relevant information:
  
  
  
  • Tensorflow: 2.0.0
  
  

[jakesabathia2]

@danvargg seems like a bug, we are looking into it :).

[jakesabathia2]

@danvargg I think your bug can be fixed in this PR https://github.com/apple/coremltools/pull/662.
You can first clone the branch and see what happens,
if it still not work please provide your model to us to reproduce the error on your model.

[danvargg]

@jakesabathia2 , thanks a lot! Will give it a try it in a few days.

[jakesabathia2]

@danvargg please reopen the issue if the fix doesn't work for you.

[danvargg]

Will do, thanks a lot!

[danvargg]

Hi @jakesabathia2, I just got back to these models.
I see the change was merged into master. I upgraded coremltools and ran the conversion, but it threw out the below error. Should I open a separate issue? It does convert a big chunk of the ops.

Traceback (most recent call last):
  File "tf2coreml.py", line 103, in <module>
    convertModel(model=_TRAIN_MODEL, ml_model=_ML_MODEL)
  File "tf2coreml.py", line 94, in convertModel
    minimum_ios_deployment_target='13')
  File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/coremltools/converters/tensorflow/_tf_converter.py", line 193, in convert
    optional_inputs=optional_inputs)
  File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/coremltools/converters/nnssa/coreml/ssa_converter.py", line 154, in ssa_convert
    converter.convert()
  File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/coremltools/converters/nnssa/coreml/ssa_converter.py", line 581, in convert
    convert_func(node)
  File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/coremltools/converters/nnssa/coreml/ssa_converter.py", line 801, in _convert_slice
    begin_masks = [True if i in node.attr['begin_masks'] else False for i in range(rank)]
  File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/coremltools/converters/nnssa/coreml/ssa_converter.py", line 801, in <listcomp>
    begin_masks = [True if i in node.attr['begin_masks'] else False for i in range(rank)]
KeyError: 'begin_masks'