Edgetpu: ERROR: Didn't find op for builtin opcode 'TRANSPOSE_CONV' version '3'

[fjp]

Hi, I am trying to compile a model that uses tensorflow.keras.layers.Conv2DTranspos and get the following error:

$ edgetpu_compiler model_quant.tflite
Edge TPU Compiler version 2.1.302470888
ERROR: Didn't find op for builtin opcode 'TRANSPOSE_CONV' version '3'

ERROR: Registration failed.

Invalid model: model_quant.tflite
Model could not be parsed

I followed the Retrain a classification model using post-training quantization notebook. I've installed tf-nightly:

$ pip list | grep tf
tf-estimator-nightly           2.3.0.dev2020061401
tf-nightly                     2.3.0.dev20200614
tflite-runtime                 2.1.0.post1

Create a TensorFlow Lite model with TFLiteConverter.
The following creates a basic (un-quantized) TensorFlow Lite model:

converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_path)
tflite_model = converter.convert()
with open('model.tflite', 'wb') as f:
    f.write(tflite_model)

Convert the model again with post-training quantization:

dataset_dir = "path/to/dataset"

# A generator that provides a representative dataset
def representative_data_gen():
    dataset_list = tf.data.Dataset.list_files(dataset_dir + '*.jpg')
    dataset_list = dataset_list.take(110)
    for i in range(100):
        image = next(iter(dataset_list))
        image = tf.io.read_file(image)
        image = tf.io.decode_jpeg(image, channels=3)
        image = tf.image.resize(image, [IMAGE_SIZE, IMAGE_SIZE])
        image = tf.cast(image / 255., tf.float32)
        image = tf.expand_dims(image, 0)
        yield [image]

# This enables quantization
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_types = [tf.int8]
# This ensures that if any ops can't be quantized, the converter throws an error
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
# These set the input and output tensors to uint8 (added in r2.3)
converter.inference_input_type = tf.uint8    # also tried with tf.int8
converter.inference_output_type = tf.uint8 # also tried with tf.int8
# And this sets the representative dataset so we can quantize the activations
converter.representative_dataset = representative_data_gen
tflite_model = converter.convert()

with open('model_quant.tflite', 'wb') as f:
    f.write(tflite_model)

Finally tried to compile the model_quant.tflite with the edgetpu-compiler (version 2.1.302470888) as mentioned above.

According to the documentation Transpose_CONV should be one of the supported operations.

Any idea how to fix this error? Thank you.

[Namburger]

@fjp Thanks for reporting this issue.

FYI We've been getting a few of these: https://stackoverflow.com/questions/62321919/edge-tpu-compiler-error-didnt-find-op-for-builtin-opcode-resize-nearest-neigh

My best guess is that the newer tensorflow version has changed the transpose_conv to something the compiler hasn't supported. Most likely can be workaround by not using the tf-nightly and downgrade to an older version of tensorflow. I'll try to reproduce this issue today and see and check with our team.

[Namburger]

@fjp Actually, could you clarify what you've changed from the retrain script?
I'm guessing that you changed tf.keras.layers.Conv2D(filters=32, kernel_size=3, activation='relu'), to tf.keras.layers.Conv2DTranspose?

[fjp]

@Namburger thanks looking into this and the provided link. I've also tried using tensorflow 2.2 which results in the following error:

$ edgetpu_compiler model_quant.tflite
Edge TPU Compiler version 2.1.302470888

Internal compiler error. Aborting!

I am not using the model from the retrain script but my own. It contains:

from tensorflow.keras.layers import Conv2DTranspose
from tensorflow.keras.layers import LeakyReLU

...

x = Conv2DTranspose(f, (3, 3), strides=2, padding="same")(x)
x = LeakyReLU(alpha=0.2)(x)

...

I guess the problem is that LeakyReLU is not supported? However, the error mentions: TRANSPOSE_CONV is not supported (when using tf-nightly 2.3).

I'll try to use ReLU instead of LeakyReLU and test with both tensorflow versions 2.2 and 2.3 (nightly).

[Namburger]

@fjp I have an updates:
The sweet spot is with tensorflow2.2 for our released compiler right now if you want to downgrade to that. FYI, you may need to uninstall tf-nightly completely and install tf2.2 :/

 » python3 -m pip install tensorflow==2.2
 » python3 -c 'print(__import__("tensorflow").__version__)'
2.2.0

Here is an example for making this model with Conv2DTranspose (with tf2.1* the tflite converter fails because it wasn't supported yet and with tf2.3* edgetpu_compiler fails):

import tensorflow as tf
import numpy as np

# Create the base model from the pre-trained MobileNet V2
base_model = tf.keras.applications.MobileNetV2(input_shape=(224, 224, 3),
                                              include_top=False,
                                              weights='imagenet')

base_model.trainable = False
model = tf.keras.Sequential([
  base_model,
  tf.keras.layers.Conv2DTranspose(filters=32, kernel_size=3, activation='relu'),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.GlobalAveragePooling2D(),
  tf.keras.layers.Dense(units=5, activation='softmax')
])
model.compile(optimizer='adam',
              loss='categorical_crossentropy',
              metrics=['accuracy'])
model.summary()
print('Number of trainable weights = {}'.format(len(model.trainable_weights)))

converter = tf.lite.TFLiteConverter.from_keras_model(model)
# This enables quantization
converter.optimizations = [tf.lite.Optimize.DEFAULT]
# This ensures that if any ops can't be quantized, the converter throws an error
converter.target_spec.supported_types = [tf.int8]
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
def representative_data_gen():
   for _ in range(100):
       yield [np.random.random((1, 224, 224, 3)).astype(np.float32)]
converter.representative_dataset = representative_data_gen
tflite_model = converter.convert()

with open('model_quant.tflite', 'wb') as f:
    f.write(tflite_model)

Note that these options are only available for tensorflow2.3* and up

converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8

Therefore, your IO tensors will still be float.

and compile:

edgetpu_compiler -s model_quant.tflite                                       vunam@penguin
Edge TPU Compiler version 2.1.302470888

Model compiled successfully in 545 ms.

Input model: model_quant.tflite
Input size: 3.09MiB
Output model: model_quant_edgetpu.tflite
Output size: 3.11MiB
On-chip memory used for caching model parameters: 3.33MiB
On-chip memory remaining for caching model parameters: 4.39MiB
Off-chip memory used for streaming uncached model parameters: 0.00B
Number of Edge TPU subgraphs: 1
Total number of operations: 74
Operation log: model_quant_edgetpu.log

Model successfully compiled but not all operations are supported by the Edge TPU. A percentage of the model will instead run on the CPU, which is slower. If possible, consider updating your model to use only operations supported by the Edge TPU. For details, visit g.co/coral/model-reqs.
Number of operations that will run on Edge TPU: 72
Number of operations that will run on CPU: 2

Operator                       Count      Status

DEQUANTIZE                     1          Operation is working on an unsupported data type
MEAN                           1          Mapped to Edge TPU
FULLY_CONNECTED                1          Mapped to Edge TPU
SOFTMAX                        1          Mapped to Edge TPU
ADD                            10         Mapped to Edge TPU
PAD                            5          Mapped to Edge TPU
QUANTIZE                       1          Operation is otherwise supported, but not mapped due to some unspecified limitation
CONV_2D                        35         Mapped to Edge TPU
TRANSPOSE_CONV                 1          Mapped to Edge TPU
RELU                           1          Mapped to Edge TPU
DEPTHWISE_CONV_2D              17         Mapped to Edge TPU

Hope this helps :)

[Namburger]

@fjp FYI, LeakyRely will fails, although I guess the compiler never got to it since it failed on Conv2DTranspose first :)

[fjp]

@Namburger when using tensorflow 2.2.0 and even keeping the options converter.inference_input_type and converter.inference_output_type (I guess they just have no effect in 2.2?) it is possible to compile the model (sometimes):

$ edgetpu_compiler -s model_quant.tflite
Edge TPU Compiler version 2.1.302470888

Model compiled successfully in 47 ms.

Input model: model_quant.tflite
Input size: 5.59MiB
Output model: model_quant_edgetpu.tflite
Output size: 5.62MiB
On-chip memory used for caching model parameters: 3.00KiB
On-chip memory remaining for caching model parameters: 8.03MiB
Off-chip memory used for streaming uncached model parameters: 0.00B
Number of Edge TPU subgraphs: 1
Total number of operations: 27
Operation log: model_quant_edgetpu.log

Model successfully compiled but not all operations are supported by the Edge TPU. A percentage of the model will instead run on the CPU, which is slower. If possible, consider updating your model to use only operations supported by the Edge TPU. For details, visit g.co/coral/model-reqs.
Number of operations that will run on Edge TPU: 1
Number of operations that will run on CPU: 26

Operator                       Count      Status

MUL                            4          More than one subgraph is not supported
QUANTIZE                       1          Operation is otherwise supported, but not mapped due to some unspecified limitation
LEAKY_RELU                     4          Operation not supported
TRANSPOSE_CONV                 3          More than one subgraph is not supported
CONV_2D                        1          Mapped to Edge TPU
CONV_2D                        1          More than one subgraph is not supported
DEQUANTIZE                     1          Operation is working on an unsupported data type
RESHAPE                        2          Operation is otherwise supported, but not mapped due to some unspecified limitation
FULLY_CONNECTED                2          More than one subgraph is not supported
LOGISTIC                       1          More than one subgraph is not supported
ADD                            7          More than one subgraph is not supported

According to this output the model currently isn't really optimized for the Edge TPU. Probably because of the unused converter options (converter.inference_input_type and converter.inference_output_type) and for this compilation run I still used LeakyReLU instead of just ReLU. I see lots of More than one subgraph is not supported and Operation is otherwise supported, but not mapped due to some unspecified limitation. Do you have suggestions to improve compilation result of this model? Otherwise I'll try to follow the requirements mentioned in the docs

And as mentioned, the model compiles only sometimes. Re-running the compilation sometime fails:

Edge TPU Compiler version 2.1.302470888

Internal compiler error. Aborting!

Is it possible to inspect why this is happening? Thank you.

[Namburger]

@fjp Would you give PRELU a go instead, I'm not sure if LeakyRelu is in our roadmap at this time

[fjp]

@Namburger yes I'll try using PReLU. Any ideas when it will be supported? Or is it already in tf 2.3? With tf 2.2 I am getting: RuntimeError: Quantization not yet supported for op: PRELU.

And here is the current compiler output results using ReLU:

$ edgetpu_compiler -s model_relu_quant.tflite
Edge TPU Compiler version 2.1.302470888

Model compiled successfully in 917 ms.

Input model: model_relu_quant.tflite
Input size: 5.58MiB
Output model: model_relu_quant_edgetpu.tflite
Output size: 5.63MiB
On-chip memory used for caching model parameters: 21.75KiB
On-chip memory remaining for caching model parameters: 7.45MiB
Off-chip memory used for streaming uncached model parameters: 3.00KiB
Number of Edge TPU subgraphs: 1
Total number of operations: 23
Operation log: model_relu_quant_edgetpu.log

Model successfully compiled but not all operations are supported by the Edge TPU. A percentage of the model will instead run on the CPU, which is slower. If possible, consider updating your model to use only operations supported by the Edge TPU. For details, visit g.co/coral/model-reqs.
Number of operations that will run on Edge TPU: 6
Number of operations that will run on CPU: 17

Operator                       Count      Status

RESHAPE                        2          Operation is otherwise supported, but not mapped due to some unspecified limitation
DEQUANTIZE                     1          Operation is working on an unsupported data type
FULLY_CONNECTED                2          More than one subgraph is not supported
LOGISTIC                       1          More than one subgraph is not supported
ADD                            2          Mapped to Edge TPU
ADD                            5          More than one subgraph is not supported
MUL                            2          Mapped to Edge TPU
MUL                            2          More than one subgraph is not supported
QUANTIZE                       1          Operation is otherwise supported, but not mapped due to some unspecified limitation
TRANSPOSE_CONV                 3          More than one subgraph is not supported
CONV_2D                        2          Mapped to Edge TPU

Still a lot of More than one subgraph is not supported, Operation is otherwise supported, but not mapped due to some unspecified limitation and Operation is working on an unsupported data type. Any hints on how to get more operations run on the Edge TPU?

[Namburger]

@fjp I see, is it okay to submit your model here for me to take a look?

[fjp]

Sure, here is the link to the SavedModel.

[Namburger]

Sorry about that, I mean the tflite model, I wanted to run it through our internal compiler(with extra logs) to see where the compiler breaks the model into different subgraph and find out what can be done on the user side

[fjp]

Sorry for sharing the wrong model. I've uploaded the (quantized) tflite models (LeakyReLU, ReLU, PReLU) including the compiled ones. You find them at the same link. Thank you for taking the time.

[Namburger]

@fjp Okay, took a look, I found a bug in our compiler and it is quite messy now...
1) The relu model, the compiler broke off at the very first reshape op, which I believe due to large tensor size. I wanted to enable some internal features to get more logs and found a bug :( I'm guessing you are still using compiler version 2.0.291256449? Even the newer released (2.1.302470888) did not work. I'll file an internal bug to get this fix hopefully before we release the next version.
2) LeakyRelu model is expected since it is not supported yet
3) For Prelu, I suspect you should be able to do full quantization with tf-nightly?

Apologies for all these issues

[fjp]

Thank you @Namburger for the investigation.

1. The relu model, the compiler broke off at the very first reshape op, which I believe due to large tensor size.

I am also wondering why there are two Reshape ops. To build the model I use just one x = Reshape((volumeSize[1], volumeSize[2], volumeSize[3]))(x) with an input tensor of shape TensorShape([None, 160000]) and output of TensorShape([None, 50, 50, 64]). Do you know if these tensor sizes are too large and what would be a suitable size?

I wanted to enable some internal features to get more logs and found a bug :( I'm guessing you are still using compiler version 2.0.291256449? Even the newer released (2.1.302470888) did not work. I'll file an internal bug to get this fix hopefully before we release the next version.

Thanks reporting this. As far as I can tell from the compiler output I am using the latest release (Edge TPU Compiler version 2.1.302470888).

1. LeakyRelu model is expected since it is not supported yet

Any updates if this will be supported in the future?

1. For Prelu, I suspect you should be able to do full quantization with tf-nightly?

I'll give it a try, although the reason I opened this issue was because of tf-nighlty (2.3). Here is the error mentioned in the first post:

$ edgetpu_compiler model_quant.tflite
Edge TPU Compiler version 2.1.302470888
ERROR: Didn't find op for builtin opcode 'TRANSPOSE_CONV' version '3'

ERROR: Registration failed.

Invalid model: model_quant.tflite
Model could not be parsed

However, for this compilation run I used LeakyReLU. I'll try PreLU with tf-nighly.

Thanks again for your help and updates.