Coremltools: ValueError: '' is not in list - converter tensorflow coremltools==4.0b4

[nikitaame]

🐞Describe the bug

Trying to convert the model from here but getting: ValueError: '' is not in list error being thrown.

Trace

 File "fastpunct.py", line 187, in <module>
    fastpunct = FastPunct()
  File "fastpunct.py", line 175, in __init__
    mlmodel = coremltools.convert(self.model)
  File "/usr/local/lib/python3.8/site-packages/coremltools/converters/_converters_entry.py", line 258, in convert
    proto_spec = _convert(
  File "/usr/local/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 134, in _convert
    prog = frontend_converter(model, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 74, in __call__
    return tf2_loader.load()
  File "/usr/local/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/load.py", line 80, in load
    program = self._program_from_tf_ssa()
  File "/usr/local/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow2/load.py", line 167, in _program_from_tf_ssa
    tf_pass(self._tf_ssa)
  File "/usr/local/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow2/tf_graph_pass/rewrite_control_flow_functions.py", line 548, in rewrite_control_flow_functions
    _rewrite_while_loop_functions(tf_ssa, fn)
  File "/usr/local/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow2/tf_graph_pass/rewrite_control_flow_functions.py", line 495, in _rewrite_while_loop_functions
    idx = body_fn.outputs.index(mapped_name)
ValueError: '' is not in list

To Reproduce

import os
import pickle
import pydload

import numpy as np
import coremltools
import tensorflow as tf

from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, LSTM, Dense, TimeDistributed, Activation, dot, concatenate, Bidirectional
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.utils import to_categorical

def get_text_encodings(texts, parameters):

    enc_seq = parameters["enc_token"].texts_to_sequences(texts)
    pad_seq = pad_sequences(enc_seq, maxlen=parameters["max_encoder_seq_length"],
                            padding='post')
    pad_seq = to_categorical(pad_seq, num_classes=parameters["enc_vocab_size"])
    return pad_seq


def get_extra_chars(parameters):
    allowed_extras = []
    for d_c, d_i in parameters["dec_token"].word_index.items():
        if d_c.lower() not in parameters["enc_token"].word_index:
            allowed_extras.append(d_i)
    return allowed_extras

def get_model_instance(parameters):

    encoder_inputs = Input(shape=(None, parameters["enc_vocab_size"],))
    encoder = Bidirectional(LSTM(128, return_sequences=True, return_state=True),
                            merge_mode='concat')
    encoder_outputs, forward_h, forward_c, backward_h, backward_c = encoder(encoder_inputs)

    encoder_h = concatenate([forward_h, backward_h])
    encoder_c = concatenate([forward_c, backward_c])

    decoder_inputs = Input(shape=(None, parameters["dec_vocab_size"],))
    decoder_lstm = LSTM(256, return_sequences=True)
    decoder_outputs = decoder_lstm(decoder_inputs, initial_state=[encoder_h, encoder_c])

    attention = dot([decoder_outputs, encoder_outputs], axes=(2, 2))
    attention = Activation('softmax', name='attention')(attention)
    context = dot([attention, encoder_outputs], axes=(2, 1))
    decoder_combined_context = concatenate([context, decoder_outputs])

    output = TimeDistributed(Dense(128, activation="relu"))(decoder_combined_context)
    output = TimeDistributed(Dense(parameters["dec_vocab_size"], activation="softmax"))(output)

    model = Model([encoder_inputs, decoder_inputs], [output])
    model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

    return model


def decode(model, parameters, input_texts, allowed_extras, batch_size):
    input_texts_c = input_texts.copy()
    out_dict = {}
    input_sequences = get_text_encodings(input_texts, parameters)

    parameters["reverse_dec_dict"][0] = "\n"
    outputs = [""]*len(input_sequences)

    target_text = "\t"
    target_seq = parameters["dec_token"].texts_to_sequences([target_text]*len(input_sequences))
    target_seq = pad_sequences(target_seq, maxlen=parameters["max_decoder_seq_length"],
                               padding="post")
    target_seq_hot = to_categorical(target_seq, num_classes=parameters["dec_vocab_size"])

    extra_char_count = [0]*len(input_texts)
    prev_char_index = [0]*len(input_texts)
    i = 0
    while len(input_texts) != 0:
        curr_char_index  = [i - extra_char_count[j] for j in range(len(input_texts))]
        input_encodings = np.argmax(input_sequences, axis=2)

        cur_inp_list = [input_encodings[_][curr_char_index[_]] if curr_char_index[_] < len(input_texts[_]) else 0 for _ in range(len(input_texts))]
        output_tokens = model.predict([input_sequences, target_seq_hot], batch_size=batch_size)
        sampled_possible_indices = np.argsort(output_tokens[:, i, :])[:, ::-1].tolist()
        sampled_token_indices = []
        for j, per_char_list in enumerate(sampled_possible_indices):
            for index in per_char_list:
                if index in allowed_extras:
                    if parameters["reverse_dec_dict"][index] == '\n' and cur_inp_list[j] != 0:
                        continue
                    elif parameters["reverse_dec_dict"][index] != '\n' and prev_char_index[j] in allowed_extras:
                        continue
                    sampled_token_indices.append(index)
                    extra_char_count[j] += 1
                    break
                elif parameters["enc_token"].word_index[parameters["reverse_dec_dict"][index].lower()] == cur_inp_list[j]:
                    sampled_token_indices.append(index)
                    break

        sampled_chars = [parameters["reverse_dec_dict"][index] for index in sampled_token_indices]

        outputs = [outputs[j] + sampled_chars[j] for j, output in enumerate(outputs)]
        end_indices = sorted([index for index, char  in enumerate(sampled_chars) if char == '\n'], reverse=True)
        for index in end_indices:
            out_dict[input_texts[index]] = outputs[index].strip()
            del  outputs[index]
            del input_texts[index]
            del extra_char_count[index]
            del sampled_token_indices[index]
            input_sequences = np.delete(input_sequences, index, axis=0)
            target_seq = np.delete(target_seq, index, axis=0)
        if i == parameters["max_decoder_seq_length"]-1 or len(input_texts) == 0:
            break
        target_seq[:,i+1] = sampled_token_indices
        target_seq_hot = to_categorical(target_seq, num_classes=parameters["dec_vocab_size"])
        prev_char_index = sampled_token_indices
        i += 1
    outputs = [out_dict[text] for text in input_texts_c]
    return outputs


model_links = {
            'en': {
                    'checkpoint': 'https://github.com/notAI-tech/fastPunct/releases/download/checkpoint-release/fastpunct_eng_weights.h5',
                    'params': 'https://github.com/notAI-tech/fastPunct/releases/download/checkpoint-release/parameter_dict.pkl'
                },

            }

lang_code_mapping = {
    'english': 'en',
    'french': 'fr',
    'italian': 'it'
}

class FastPunct():
    model = None
    parameters = None
    def __init__(self, lang_code="en", weights_path=None, params_path=None):
        if lang_code not in model_links and lang_code in lang_code_mapping:
            lang_code = lang_code_mapping[lang_code]

        if lang_code not in model_links:
            print("fastPunct doesn't support '" + lang_code + "' yet.")
            print("Please raise a issue at https://github.com/notai-tech/fastPunct/ to add this language into future checklist.")
            return None

        home = os.path.expanduser("~")
        lang_path = os.path.join(home, '.fastPunct_' + lang_code)
        weights_path = os.path.join(lang_path, 'checkpoint.h5')
        params_path = os.path.join(lang_path, 'params.pkl')

        if not os.path.exists(lang_path):
            os.mkdir(lang_path)

        if not os.path.exists(weights_path):
            print('Downloading checkpoint', model_links[lang_code]['checkpoint'], 'to', weights_path)
            pydload.dload(url=model_links[lang_code]['checkpoint'], save_to_path=weights_path, max_time=None)

        if not os.path.exists(params_path):
            print('Downloading model params', model_links[lang_code]['params'], 'to', params_path)
            pydload.dload(url=model_links[lang_code]['params'], save_to_path=params_path, max_time=None)


        with open(params_path, "rb") as file:
            self.parameters = pickle.load(file)
        self.parameters["reverse_enc_dict"] = {i:c for c, i in self.parameters["enc_token"].word_index.items()}
        self.model = get_model_instance(self.parameters)
        self.model.load_weights(weights_path)
        self.allowed_extras = get_extra_chars(self.parameters)

        mlmodel = coremltools.convert(self.model)
        mlmodel.save('punctuation.mlmodel')

    def punct(self, input_texts, batch_size=32):
        input_texts = [text.lower() for text in input_texts]
        return decode(self.model, self.parameters, input_texts, self.allowed_extras, batch_size)

    def fastpunct(self, input_texts, batch_size=32):
        # To be implemented
        return None

if __name__ == "__main__":
    fastpunct = FastPunct()
    print(fastpunct.punct(["oh i thought you were here", "in theory everyone knows what a comma is", "hey how are you doing", "my name is sheela i am in love with hrithik"]))

System environment (please complete the following information):

• coremltools version: 4.0b4
• OS: macOS
• macOS version (if applicable): 10.15.5 Beta (19F83c)
• How you install python: system
• python version (e.g. 3.7): 3.8.5
• TensorFlow version: 2.3.0
• Keras version: 2.4.3

[1duo]

Hi @nikitaame, thanks for opening a separate issue. And thanks for providing such detailed reproduce scripts! Looks like you might have an older version of coremltools. I can run your script and convert this model without any error. Can you make sure you have a clean install of 4.0b4 and try again?

Some detailed logs if that helps:

$ pip show coremltools
Name: coremltools
Version: 4.0b4
Summary: Community Tools for Core ML
Home-page: https://github.com/apple/coremltools
Author: Apple Inc.
Author-email: [email protected]
License: BSD
Location: /Volumes/data/testOsVenvPy3.7/lib/python3.7/site-packages
Requires: numpy, attrs, sympy, attr, scipy, six, packaging, tqdm, protobuf
Required-by: coremltools-internal, pymilane

Conversion logs:

Downloading checkpoint https://github.com/notAI-tech/fastPunct/releases/download/checkpoint-release/fastpunct_eng_weights.h5 to /Users/yuduo/.fastPunct_en/checkpoint.h5
MB100% (3.0 of 3.0) |#######################################################################################################################################################################################################| Elapsed Time: 0:00:00 Time:  0:00:00
Succefully Downloaded to: /Users/yuduo/.fastPunct_en/checkpoint.h5
Downloading model params https://github.com/notAI-tech/fastPunct/releases/download/checkpoint-release/parameter_dict.pkl to /Users/yuduo/.fastPunct_en/params.pkl
MB100% (1.0 of 1.0) |#######################################################################################################################################################################################################| Elapsed Time: 0:00:00 Time:  0:00:00
Succefully Downloaded to: /Users/yuduo/.fastPunct_en/params.pkl
Running TensorFlow Graph Passes: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00,  7.22 passes/s]
Converting Frontend ==> MIL Ops: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 13/13 [00:00<00:00, 30393.51 ops/s]
Converting Frontend ==> MIL Ops:   0%|                                                                                                                                                                                                    | 0/13 [00:00<?, ? ops/sWARNING:root:Input ls elem type unknown. Override with <class 'coremltools.converters.mil.mil.types.type_tensor.tensor.<locals>.tensor'>                                                                                                   | 0/40 [00:00<?, ? ops/s]
Converting Frontend ==> MIL Ops: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 40/40 [00:00<00:00, 6004.95 ops/s]
Converting Frontend ==> MIL Ops: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 13/13 [00:00<00:00, 39887.31 ops/s]
Converting Frontend ==> MIL Ops:   0%|                                                                                                                                                                                                    | 0/13 [00:00<?, ? ops/sWARNING:root:Input ls elem type unknown. Override with <class 'coremltools.converters.mil.mil.types.type_tensor.tensor.<locals>.tensor'>                                                                                                   | 0/40 [00:00<?, ? ops/s]
Converting Frontend ==> MIL Ops: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 40/40 [00:00<00:00, 5361.33 ops/s]
Converting Frontend ==> MIL Ops: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 13/13 [00:00<00:00, 40751.83 ops/s]
Converting Frontend ==> MIL Ops:   0%|                                                                                                                                                                                                    | 0/13 [00:00<?, ? ops/sWARNING:root:Input ls elem type unknown. Override with <class 'coremltools.converters.mil.mil.types.type_tensor.tensor.<locals>.tensor'>                                                                                                   | 0/40 [00:00<?, ? ops/s]
Converting Frontend ==> MIL Ops: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 40/40 [00:00<00:00, 6175.81 ops/s]
Converting Frontend ==> MIL Ops: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 13/13 [00:00<00:00, 40359.70 ops/s]
Converting Frontend ==> MIL Ops:   0%|                                                                                                                                                                                                    | 0/13 [00:00<?, ? ops/sWARNING:root:Input ls elem type unknown. Override with <class 'coremltools.converters.mil.mil.types.type_tensor.tensor.<locals>.tensor'>                                                                                                   | 0/40 [00:00<?, ? ops/s]
Converting Frontend ==> MIL Ops: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 40/40 [00:00<00:00, 6286.19 ops/s]
Converting Frontend ==> MIL Ops: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 13/13 [00:00<00:00, 42072.49 ops/s]
Converting Frontend ==> MIL Ops:   0%|                                                                                                                                                                                                    | 0/13 [00:00<?, ? ops/sWARNING:root:Input ls elem type unknown. Override with <class 'coremltools.converters.mil.mil.types.type_tensor.tensor.<locals>.tensor'>                                                                                                   | 0/40 [00:00<?, ? ops/s]
Converting Frontend ==> MIL Ops: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 40/40 [00:00<00:00, 5869.24 ops/s]
Converting Frontend ==> MIL Ops: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 13/13 [00:00<00:00, 42498.79 ops/s]
Converting Frontend ==> MIL Ops:   0%|                                                                                                                                                                                                    | 0/13 [00:00<?, ? ops/sWARNING:root:Input ls elem type unknown. Override with <class 'coremltools.converters.mil.mil.types.type_tensor.tensor.<locals>.tensor'>                                                                                                   | 0/40 [00:00<?, ? ops/s]
Converting Frontend ==> MIL Ops: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 40/40 [00:00<00:00, 5664.92 ops/s]
Converting Frontend ==> MIL Ops: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 163/163 [00:00<00:00, 350.10 ops/s]
Running MIL optimization passes: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 17/17 [00:00<00:00, 2360.09 passes/s]
Translating MIL ==> MLModel Ops: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 18893.26 ops/s]
Translating MIL ==> MLModel Ops: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24/24 [00:00<00:00, 2010.73 ops/s]
Translating MIL ==> MLModel Ops: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 19599.55 ops/s]
Translating MIL ==> MLModel Ops: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24/24 [00:00<00:00, 2155.16 ops/s]
Translating MIL ==> MLModel Ops: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 19972.88 ops/s]
Translating MIL ==> MLModel Ops: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24/24 [00:00<00:00, 493.24 ops/s]
Translating MIL ==> MLModel Ops: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 231/231 [00:00<00:00, 2590.67 ops/s]
['Oh, I thought you were here.', 'In theory, everyone knows what a comma is.', 'Hey, how are you doing?', 'My name is Sheela, I am in love with Hrithik.']

[nikitaame]

@1duo really appreciate you checking. I didn't have 4.03b properly set up. Error not showing up anymore! Thank you and sorry for that 😦

[nikitaame]

@1duo could you share what version of Xcode you're using? I am though unfortunately getting:

RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was: Error compiling model: "Error reading protobuf spec. validator error: The .mlmodel supplied is of version 5, intended for a newer version of Xcode. This version of Xcode supports model version 4 or earlier.".

I'm on Xcode 12.0.1

[1duo]

Hi @nikitaame, no worries, and Xcode 12+ should be sufficient.

[aseemw]

You would also need a newer OS (macOS 11, iOS 14 etc), since this model, spec 5, uses features that are available only in the latest Core ML framework.

[nikitaame]

@aseemw you mean to run on device? So I can compile with Xcode 12 on macOS 10.15 onto an iOS14 device fine? Or I need the macOS to be on it too?

[aseemw]

Right, you can compile the model with Xcode 12.
But when you are running the model, you can only do so, on the latest OS, since its a spec 5 model.