Mediapipe: Why use custom op Convolution2DTransposeBias in palm_detection.tflite?

[apivovarov]

I'm not sure why mediapipe uses custom op Convolution2DTransposeBias in palm_detection.tflite.
Can it be replaced with built-In ops TRANSPOSE_CONV and ADD? TRANSPOSE_CONV op was added to TFLite in version 1.9

I tried to create Keras model with Conv2DTranspose layer and I set use_bias=True and bias_initializer='random_uniform'.

https://keras.io/layers/convolutional/#conv2dtranspose

data = keras.layers.Input(shape=(32, 32, 3))
x = keras.layers.Conv2D(filters=10, kernel_size=(3, 3), strides=(2, 2), padding='same')(data)
x2 = keras.layers.Conv2DTranspose(filters=10, kernel_size=(3, 3), strides=(2, 2), padding='same', 
        use_bias=True,  bias_initializer='random_uniform')(x)

That Keras model can be converted to TFLite model. Keras Conv2DTranspose op was mapped to two built-in TFLite operators TRANSPOSE_CONV and ADD

Resulting TFLite model visualization: (download html and open it): https://www.dropbox.com/s/au39pvmw3fquhm1/conv2d_transpose_bias.tflite.htm?dl=0

[scm-ns]

@apivovarov I see their CPU implmentation. (Done in cpu_op_resolver) But I cannot find their GPU implementation. If you look at https://github.com/google/mediapipe/blob/731d2b95363d58f12acb29a6f8435ec33fe548d9/mediapipe/util/tflite/op_resolver.cc#L48, no functions are actually being registered to the custom op.

[mcclanahoochie]

The GPU implementation is here
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/delegates/gpu/gl/kernels/transpose_conv.cc
and gets registered in the graph here https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/delegates/gpu/common/model_builder.cc#L887

A fused op such as this is more efficient on GPU than running two ops sequentially, and has identical result as the split version (tconv + add). The models are built with GPU use case as priority, and are back-ported to support CPU via a custom op.

[scm-ns]

@mcclanahoochie Thank you very much. That makes a lot of sense. Just a question. That seems like open gl implementation and not metal. So will that work on iOS?

Another question if you can help me with this. I am trying to build a C++ static framework for iOS, so that I can add the custom ops. I am not using the C API, so cannot use the TensorflowLiteC/Swift/Objc pods. There is a TensorflowLite pod, but that is version 1.3, not 2. I can't find the bazel build file which generates the TensorflowLite pod, which has the C++ headers in them.

[apivovarov]

We converted palm_detection.tflite to use built-in TRANSPOSE_CONV + ADD ops instead of custom op transpose_conv_bias.
The outputs are absolutely the same. But CPU performance is 2.5x faster for the model which uses built-it ops palm_detection_builtin.tflite.

Model visualization palm_detection_builtin.tflite.html

[sebastijan94]

We converted palm_detection.tflite to use built-in TRANSPOSE_CONV + ADD ops instead of custom op transpose_conv_bias.
The outputs are absolutely the same. But CPU performance is 2.5x faster for the model which uses built-it ops palm_detection_builtin.tflite.

Model visualization palm_detection_builtin.tflite.html

Can this model be used for multi hand detection?

[eknight7]

With release https://github.com/google/mediapipe/releases/tag/v0.6.4, see multi-hand tracking:
https://github.com/google/mediapipe/blob/master/mediapipe/docs/multi_hand_tracking_mobile_gpu.md

[jiuqiant]

We are closing this issue for now due to lack of activity.

[xuchengggg]

We converted palm_detection.tflite to use built-in TRANSPOSE_CONV + ADD ops instead of custom op transpose_conv_bias.
The outputs are absolutely the same. But CPU performance is 2.5x faster for the model which uses built-it ops palm_detection_builtin.tflite.

Model visualization palm_detection_builtin.tflite.html

Hi, how could you change ops in tflite model?

[apivovarov]

download schema

wget https://raw.githubusercontent.com/tensorflow/tensorflow/r1.15/tensorflow/lite/schema/schema.fbs

convert to json

flatc -t schema.fbs  -- mobilenet_v1_0.75_224.tflite

edit json

vi mobilenet_v1_0.75_224.json

I recommend to visualize tflite model to get tensor ids and buffer ids

python3 tensorflow/lite/tools/visualize.py \
$MODELS_DIR/mobilenet_v1_0.75_224.tflite \
$MODELS_DIR/mobilenet_v1_0.75_224.html

Convert edited json to tflite

flatc -b schema.fbs  mobilenet_v1_0.75_224_fixed.json

[xuchengggg]

Thank you!!