Mask_rcnn: Mask RCNN on TensorFlow Lite

[JaviBonilla]

Has anyone used Mask RCNN on TensorFlow Lite? I managed to create a tflite file from my network, but there are some TensorFlow Lite unsupported operators used in Mask RCNN: ResizeNearestNeighbor, Stack, and TensorFlowShape.

Some of the operators in the model are not supported by the standard TensorFlow Lite runtime. If you have a custom implementation for them you can disable this error with --allow_custom_ops. Here is a list of operators for which you will need custom implementations: ResizeNearestNeighbor, Stack, TensorFlowShape.

Has anyone managed to implement such operators? There is some information over here about how to do it, but I wanted to ask in case that someone already implemented them. https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/lite/g3doc/custom_operators.md.

[ghost]

@JaviBonilla did you solve this?

[JaviBonilla]

Hi @CoolJack11,

No, I'm still trying to figure it out. I am testing with the Mask RCNN Inception v2 COCO from TensoFlow detection model zoo, but there are still some issues. Have a look at https://github.com/tensorflow/tensorflow/issues/19293

[JonathanLehner]

any updates on this?

[JaviBonilla]

@JonathanLehner No luck yet, but the TensorFlow guys are looking at the Mask RCNN TensorFlow Lite conversion failure at https://github.com/tensorflow/tensorflow/issues/19293

[jps892]

@JaviBonilla any update on this? please if you have any let us know

[JaviBonilla]

@jps892 No updates sorry. Sure, I will let you know if there are any news.

[jinfagang]

Pls make some speed test after u migrate pb model into tflite. We'd like to run it on a edge device.

[PINTO0309]

I tried to generate tflite. It includes custom operations. Mask RCNN InceptionV2
https://github.com/PINTO0309/PINTO_model_zoo

[xavier-faromatics]

Any updates or alternative models?

[bmabir17]

This is how i was able to convert it
https://gist.github.com/bmabir17/754a6e0450ec4fd5e25e462af949cde6

did any of you tried to run inference on it using python interpreter or android?

[ValentinRicher]

This is how i was able to convert it
https://gist.github.com/bmabir17/754a6e0450ec4fd5e25e462af949cde6

did any of you tried to run inference on it using python interpreter or android?

Hello, I used your code to generate the .tflite model and it worked, thank you very much.
However, when I tried to used the lite model with the Python interpreter I had a "segmentation fault" problem.

Here is my code :

import numpy as np
import tensorflow as tf

# Load TFLite model and allocate tensors.
interpreter = tf.lite.Interpreter(model_path="mask_rcnn_coco.tflite")
interpreter.allocate_tensors()

# Get input and output tensors.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

# Test model on random input data.
input_shape = input_details[0]['shape']
input_data = np.array(np.random.random_sample(input_shape), dtype=np.float32)
interpreter.set_tensor(input_details[0]['index'], input_data)

interpreter.invoke()

# The function `get_tensor()` returns a copy of the tensor data.
# Use `tensor()` in order to get a pointer to the tensor.
output_data = interpreter.get_tensor(output_details[0]['index'])
print(output_data)

The error is "Segmentation fault: 11" occurring at interpreter.invoke()
I don't know if you faced the same issue when trying to use it with python, android or iOS

[bmabir17]

It seems the tflite python interpreter does not support the select_tf_ops. This is under active development. So i am now trying to infer the model using Android interpreter.

[sayakpaul]

I am now getting the following:

RuntimeError: tensorflow/lite/kernels/gather.cc:80 0 <= axis && axis < NumDimensions(input) was not true.Node number 222 (GATHER) failed to prepare.

I am using tf-nightly (version: 2.3.0-dev20200528). Used the above code snippet and the TF Lite model converted using your awesome gist @bmabir17.

[bmabir17]

@sayakpaul I have faced the same error. And have created an issue in tensorflow repo, but still have not found any solution for it

[sayakpaul]

Thanks for letting me know :)

[ulhaqi12]

Hi,
Has anyone succeeded?
I have tried converting in using code on this https://gist.github.com/bmabir17/754a6e0450ec4fd5e25e462af949cde6. I obtained .tflite file but it is not working with tflite interpreter in python. giving some errors.
Anyone, who able to create a working tflite model of Mask RCNN?
Best,
Ikram

[bmabir17]

I was going through the converted model and found that tflite converted model has some difference compared with the original model.pb(saved before conversion to tflite).
It seems the range op conversion has some problem. And this seems to be the input for the expand dims.
_Tensorflow Model(original)_

_Tflite Model(converted)_

Can anyone help why is this happening?
P.S the model conversion showed no error.
My conversion code

[bmabir17]

@sayakpaul It seems if you convert the model using toco converter from tensorflow 1.15.0 the problem goes away. But now i get the a different set of errors, because of the problem explained on my previous comment

[oholimoli]

Is now possible to convert RNNs with masks to tflite?