Mediapipe: Running hand detection model through Python

[DmitryRyumin]

When I run the model for determining hands through Python, I encounter slow work in the process of determining hands! More precisely, it is
interpreter.invoke ()
Frame rate drops sharply from 40 to 4!
How can I increase the frame rate to 20 or 25?

[jiuqiant]

I recommend reaching out to the TensorFlow community for such questions. MediaPipe pipelines integrate with TF Lite via its C++ APIs (for both CPU and GPU backends), and running the open sourced model without MediaPipe is beyond the scope of this project.

[nagarjun-ananth]

@DmitryRyumin Could you please let me know how you loaded the tflite model on python. I've an unsupported custom operation for its interpreter.

[wolterlw]

@nagarjun-ananth
you have to build tensorflow from source and add custom operations in the Python wrapper.

You can do it directly in tensorflow/lite/python/interpreter_wrapper/
Here is the relevant part of interpreter_wrapper.cc function CreateInterpreterWrapper

```
auto resolver = absl::make_unique();
resolver->AddCustom("MaxPoolingWithArgmax2D",
tflite_operations::RegisterMaxPoolingWithArgmax2D());
resolver->AddCustom("MaxUnpooling2D",
tflite_operations::RegisterMaxUnpooling2D());
resolver->AddCustom("Convolution2DTransposeBias",
tflite_operations::RegisterConvolution2DTransposeBias());

auto interpreter = CreateInterpreter(model.get(), *resolver);
`` Then you include and link CutomOp implementations frommediapipe/util/tflite/operations` and you're good

[wolterlw]

@DmitryRyumin how do you crop the input image for joint detection?
I've read somewhere they use landmarks 0 and 2 to find hand rotation, but I'm not sure how they resize the palm bounding box to fit the fingers in.

[fanzhanggoogle]

It's a simple heuristic. Just enlarge the rectangle and move the palm part lower. Here's the actual parameters we use to transform palm detection rectangle for later landmark detection.

@DmitryRyumin how do you crop the input image for joint detection?
I've read somewhere they use landmarks 0 and 2 to find hand rotation, but I'm not sure how they resize the palm bounding box to fit the fingers in.

[mgyong]

@DmitryRyumin
Another user has created a minimal python interface to the Google MediaPipe Handtracking pipeline
https://github.com/wolterlw/hand_tracking
It has custom ops supported using tflite mode in python

[sebastijan94]

When I run the model for determining hands through Python, I encounter slow work in the process of determining hands! More precisely, it is
interpreter.invoke ()
Frame rate drops sharply from 40 to 4!
How can I increase the frame rate to 20 or 25?

Did you manage to increase the FPS?