Edgetpu: Internal Compiler Error: Aborting - A Small Custom Model

[pirazor]

Hello,

I would like to kindly get your help in compiling this int8 quantized model. This is ERFNet, a segmentation model that has all the supported operations that should be compiled on Edge Tpu. I am using the latest Edge TPU Compiler version 15.0.340273435.

Quantized tflite model

Thank you.

[Namburger]

@usbguru @Naveen-Dodda could you guys take a look at this model?

[pirazor]

Thanks for very fast response @Namburger !

[PINTO0309]

I also quantized ERFNet and compiled it into EdgeTPU for a test run.

Edge TPU Compiler version 15.0.340273435
TensorFlow tf-nightly==2.5.0-dev20201204
Dataset Cityscapes
The model's citation repository https://github.com/Eromera/erfnet_pytorch.git

• 256x512, 384x768, 512x1024
  https://github.com/PINTO0309/PINTO_model_zoo/tree/master/075_ERFNet

Strangely enough, 512x1024 was aborted by the edgetpu_compiler, but the other resolutions compiled successfully.

Model structure diagram of ERFNet Full Integer Quant tflite

[pirazor]

Thank you @PINTO0309, amazing effort ! Did you use ONNX converter to a saved model, then implement the post-quantization ?

[PINTO0309]

@pirazor
I converted while optimizing the model with the following flow.
PyTorch -> ONNX -> ONNX Simplifier -> OpenVINO -> openvino2tensorflow

openvino2tensorflow automatically generates TensorFlow Lite and EdgeTPU Compiler-optimized NHWC models for 4D input models in NCHW format. Also, openvino2tensorflow can automatically generate a Post-Training quantization model.

Procedure

1. Generate ONNX files from https://github.com/Eromera/erfnet_pytorch.git
2. Optimizing the ONNX model with https://github.com/daquexian/onnx-simplifier.git
3. https://docs.openvinotoolkit.org/latest/openvino_docs_install_guides_installing_openvino_linux.html while converting the ONNX model to IR format using OpenVINO Optimization (Model Optimizer)
4. https://github.com/PINTO0309/openvino2tensorflow.git openvino2tesnsorflow is used to generate a more optimized state model. It also automatically solves the problems of the TFLite Converter or EdgeTPU compiler.

Problems encountered by the TFLite Converter or EdgeTPU compiler

1. issue : Resize Nearest Neighbor does not convert #187 - google-coral/edgetpu
2. issue : Yolov4-Tiny EdgeTPU Missing Leaky Relu #51 - PINTO0309/PINTO_model_zoo
3. issue : RESIZE_NEAREST_NEIGHBOR Operation version not supported #42024 - tensorflow/tensorflow

Problems when converting a model from NCHW to NHWC

[English] Converting PyTorch, ONNX, Caffe, and OpenVINO (NCHW) models to Tensorflow / TensorflowLite (NHWC) in a snap

[pirazor]

@PINTO0309 oh wow, that's awesome ! The struggles we are going through just to be able to serve a relatively simple model... I will give it a try. @Namburger don't get me wrong we love our Edge TPU, I just wish it would be as easy as using Jetson for inference :)

[pirazor]

@PINTO0309, I spent my entire day to obtain the same result on my ERFNet model. Your work in the openvino2tensorflow was very impressive, yet I still have no luck with successful EdgeTPU compilation.

If it is not too much trouble for you, would you mind giving a try using your method ? I provided here the model files (onnx, simplified-onnx, small calibration data, and a conversion script for post-quantization)

Thank you !

[PINTO0309]

@pirazor

In processing with NHWC, the following four Transpose OPs are unnecessary and must be removed. I will return home in nine hours to work on it and let you know the result.

Incidentally, one of the major advantages of using OpenVINO IR is that you can edit the .xml file to manually remove unwanted OPs and change the axis of the transposition. There is no need to modify the Model program at all.

• Example of manually removing Transpose OP

[PINTO0309]

@pirazor
Transpose before and after Softmax is very tricky. Is it possible to remove Transpose and change Softmax axis=3 to axis=1? OpenVINO optimization is unstable.

[pirazor]

@PINTO0309, apologies for my delayed response. Would you mind trying converting the attached onnx model to edgetpu model. I removed the softmax and the uncessary transpose is now disappeared. But, I am now struggling to find a suitable Tensorflow version :(
ERFNet_Lane
Happy New Year btw.