Coremltools: Tensorflor 2.0 tf.keras CNN to updatable Core ML

[knutnordin]

❓Question

Is there currently support for making a Core ML model converted from tf.keras with Tensorflow 2.0 updatable? Existing documentation and examples mainly seem to refer to standalone keras.

Context

After converting a tf.keras CNN, saved in .h5 format, using coremltools.converters.tensorflow.convert, and trying to make the model updatable, we run into the following issues:

1. Keras Dense layers have been converted to type batchedMatmul in Core ML. When trying to apply make_updatable we get an error stating that "Only ['innerProduct', 'convolution'] layers are supported to be marked updatable."

2. The output layer is converted to type softmaxND. Supplying this layer as input to set_categorical_cross_entropy_loss gives the error "Categorical Cross Entropy loss layer input (Identity) must be a softmax layer output."

System Information

• Tensorflow 2.0
• coremltools 3.1

[anilkatti]

@knutnordin can you paste how you are calling convert method here?

Unfortunately, we don't have support for updating batchedMatmul at this point but, I might be able to provide some workarounds.

[knutnordin]

Thanks for the reply!
Here is a snippet with the conversion call:

# CNN Layers
cnn_input = layers.Input(shape=(height, width, channels), name='cnn_input')
cnn_conv1 = layers.Conv2D(32, kernel_size, activation='relu')(cnn_input)
cnn_pool1 = layers.MaxPooling2D(pool_size)(cnn_conv1)
cnn_drop1 = layers.Dropout(0.2)(cnn_pool1)
cnn_conv2 = layers.Conv2D(64, kernel_size, activation='relu')(cnn_drop1)
cnn_pool2 = layers.MaxPooling2D(pool_size)(cnn_conv2)
cnn_drop2 = layers.Dropout(0.2)(cnn_pool2)
cnn_conv3 = layers.Conv2D(128, kernel_size, activation='relu')(cnn_drop2)
cnn_drop3 = layers.Dropout(0.2)(cnn_conv3)
cnn_flat = layers.Flatten()(cnn_drop3)

# Classification layers
class_dense1 = layers.Dense(256, activation='relu')(cnn_flat)
class_drop1 = layers.Dropout(0.5)(class_dense1)
class_output = layers.Dense(labels, activation='softmax', name='output')(class_drop1)

# Model
model = models.Model(inputs=cnn_input, outputs=class_output)

# Train and save
model.compile(optimizer='adam',
             loss='categorical_crossentropy',
             metrics=['accuracy'],
             weighted_metrics=['accuracy'])

history = model.fit(x=X_train, y=Y_train, validation_data=(X_test, Y_test), class_weight=weights, epochs=num_epochs)

model.save('./tf2_model.h5')

# Convert to Core ML
coreml_model = coremltools.converters.tensorflow.convert('./tf2_model.h5', 
                                input_name_shape_dict={
                                    'cnn_input': [1, height, width, channels]},
                                output_feature_names=['Identity'],
                                minimum_ios_deployment_target='13',
                                respect_trainable=True)

[anilkatti]

@knutnordin tensorflow-coreml converter does not support respect_trainable. Only keras converter supports that feature.

But, that's fine. You can still make the model updatable by explicitly calling make_updatable like you mentioned in your comment.

In this case, could you try changing the minimum_ios_deployment_target to 12? If that works, Dense layer should get translated to InnerProduct thus allowing you to mark it as updatable. Let me know if that works for you.

[knutnordin]

@anilkatti Thanks for the suggestions! I gave this a try, but ended up with an identical model as before, i.e. batchedMatmul and softmaxND layers are still present.

coreml_model = coremltools.converters.tensorflow.convert('./tf2_model.h5', 
                                                        input_name_shape_dict={
                                                            'cnn_input': [1, height, width, channels]},
                                                        output_feature_names=['Identity'],
                                                        minimum_ios_deployment_target='12')

I also tried going through the tfcoreml tool, as I understand this has a different code path for pre iOS 13 versions, but it seems to cause compatibility issues, supposedly with Tensorflow 2.0.

coreml_model = tfcoreml.convert('./tf2_model.h5', 
                                input_name_shape_dict={
                                    'cnn_input': [1, height, width, channels]},
                                output_feature_names=['Identity'],
                                minimum_ios_deployment_target='12')

AttributeError: module 'tensorflow' has no attribute 'reset_default_graph'

[apmason]

Has anyone found a solution for this?

[apmason]

I reverted to using keras instead of tf.keras and got it to work.

[jankaltoun]

@anilkatti do you know if anything has changed about support of batchedMatmul?

I'm still getting the same error with TF 2.0 and Coremltools 3.3.
I'm unable to find a way for coremltools.converters.tensorflow.convert to use innerProduct instead of batchedMatmul.

ValueError: Layer sequential/dense_1/MatMul is not supported to be marked as updatable. Only ['innerProduct', 'convolution'] layers are supported to be marked updatable.

[anilkatti]

coremltools.converters.tensorflow.convert does use batchedMatmul instead of innerProduct and unfortunately, updating a batchedMatmul is not supported in the current version. Using keras convertor is definitely a workaround but, I am not sure if that suits your use case. We will explore some more options and get back to you.

[mrfarhadi]

@jankaltoun If you cannot use Keras converter instead of tensorflow converter, one workaround is to manually replace the batchedMatmul with innerProduct. If you share your model, I can provide you a script to do that using coremltools.

[jankaltoun]

Wow, thank you for such a quick response @anilkatti and @mrfarhadi !
A script would be amazing both for making this work and as a learning material :slightly_smiling_face: .

My model can be downloaded here.
It's a simple MNIST, similar to this tutorial.

I would like the dense and dense_1 layers to be innerProduct instead of batchedMatmul so that I can make them updatable.

[Id: 10], Name: Identity (Type: softmaxND)
          Updatable: False
          Input blobs: ['sequential/dense_1/MatMul']
          Output blobs: ['Identity']
[Id: 9], Name: sequential/dense_1/MatMul (Type: batchedMatmul)
          Updatable: False
          Input blobs: ['sequential/dense/Relu']
          Output blobs: ['sequential/dense_1/MatMul']
[Id: 8], Name: sequential/dense/Relu (Type: activation)
          Updatable: False
          Input blobs: ['sequential/dense/MatMul']
          Output blobs: ['sequential/dense/Relu']
[Id: 7], Name: sequential/dense/MatMul (Type: batchedMatmul)
          Updatable: False
          Input blobs: ['sequential/flatten/Reshape']
          Output blobs: ['sequential/dense/MatMul']

Thank you so much for your help!

[cpesarchick]

I would be interested in your script also.

Also the original question had another problem with set_categorical_cross_entropy_loss (builder.py). The method is looking for layer_type == 'softmax'. but after conversion it is 'softmaxND'. Should the code also check the layer_type == 'softmaxND'?

[mrfarhadi]

@cpesarchick That is not supported yet.
One workaround for now is to use Keras converter to convert your model instead of coremltools.converters.tensorflow.convert. Like what is shown on this tutorial: https://github.com/apple/coremltools/blob/master/examples/updatable_models/updatable_mnist.ipynb

[mrfarhadi]

Just so you know, although the current simple workaround is to use Keras converter (which uses innerProduct layer instead of batchedMatmul), we are still working on the workaround for the models with batchedMatmul layer. Thanks for your patience.

[mrfarhadi]

@jankaltoun
Here is a model identical to yours. It has InnerProduct and Convolution layers which can be updatable. Please find it inside the zip file. You can making it updatable by calling make_updatable method on the layers, attach loss, optimizer and epochs. Please let me know if it works for you.

And here is the script I used to create it:

from keras.models import Sequential
from keras.layers import Dense, Flatten, Conv2D

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=(28, 28, 1)))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dense(10, activation='softmax'))

from coremltools.converters import keras as keras_converter
class_labels = ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']
mlmodel = keras_converter.convert(model, input_names=['image'],
                                  output_names=['digitProbabilities'],
                                  class_labels=class_labels,
                                  predicted_feature_name='digit')

mlmodel.save('MNIST_withInnerProduct.mlmodel')

MNIST_withInnerProduct.mlmodel.zip

[jankaltoun]

@mrfarhadi thank you so much for your help and the time you spent on this!

If I understand it correctly, there's basically no way to achieve this using tf.keras and the only way is to use keras outside TF, right?

[mrfarhadi]

@jankaltoun Correct. CoreML converter creates batchedMatmul and softmaxND layers out of tensorflow converter. These layers are not supported for training. The best way for now is to use Keras converter or create the identical model with coremltools NN builder API.

[vsatpathy]

Any idea if there has been progress in coreml4.0 where it tackles batchedMatmul and softmaxND layers to make it updatable?

[williew17]

@mrfarhadi when using the script you provided I get the error AttributeError: module 'tensorflow' has no attribute 'get_default_graph' which I thought was happening because I needed to use tf.keras instead of keras. Is this a version issue?

[vsatpathy]

@mrfarhadi

tf = 2.x
keras = 2.2.4
coreml = 3.0

Models don't compile

tf = 2.x
keras = 2.2.4
coreml = 4.0b2

All models compile and convolution layers are only updatable not dense layers

tf = 1.14
keras = 2.2.4
coreml = 3.0

Everything works