Mediapipe: OpenCV Put Text

[sne21star]

Hello,

I am having some issues writing words on top of the image. I am using the Hand Gesture recognition calculator to get an output stream of words.

constexpr char aslLettersTag[] = "ASL";

and then I added this line in ::mediapipe::Status HandGestureRecognitionCalculator::GetContract
if (cc->Outputs().HasTag("ASL")) {
cc->Outputs().Tag(aslLettersTag).Set();
}

and this line in ::mediapipe::Status HandGestureRecognitionCalculator::Process(
CalculatorContext *cc)
cc->Outputs()
.Tag(aslLettersTag)
.Add("Hello!!!!!!!!!", cc->InputTimestamp());
return ::mediapipe::OkStatus();

I then send this output stream as an input stream for the Open CV put text calculator.

However I get this error:
I0320 15:27:42.768482 205182400 demo_run_graph_main.cc:58] Initialize the calculator graph.
I0320 15:27:42.771833 205182400 demo_run_graph_main.cc:62] Initialize the camera or load the video.
I0320 15:27:44.733304 205182400 demo_run_graph_main.cc:83] Start running the calculator graph.
I0320 15:27:44.733888 205182400 demo_run_graph_main.cc:88] Start grabbing and processing frames.
I0320 15:27:44.977525 224604160 hand-gesture-recognition-calculator.cc:102] No Hand Detected
I0320 15:27:44.977602 227287040 hand-gesture-recognition-calculator.cc:102] No Hand Detected
I0320 15:27:45.244771 227287040 hand-gesture-recognition-calculator.cc:273] Finger States: 01101
I0320 15:27:45.244791 227287040 hand-gesture-recognition-calculator.cc:274] ___
I0320 15:27:45.244801 228896768 hand-gesture-recognition-calculator.cc:273] Finger States: 11101
I0320 15:27:45.244812 228896768 hand-gesture-recognition-calculator.cc:274] ___
F0320 15:27:45.244814 228896768 collection.h:53] Failed to get tag "ASL" index 0
hand_tracking_cpu(54746,0x70000d8c2000) malloc: error for object 0x109c8fec2: pointer being freed was not allocated
hand_tracking_cpu(54746,0x70000d8c2000) malloc: set a breakpoint in malloc_error_break to debug
Abort trap: 6

Anyone know how to fix this issue?

[jiuqiant]

Seems like the graph config has some problem. You need to specify an output stream with tag "ASL" in the node of HandGestureRecognitionCalculator in your graph config. For example,

node {
  calculator:HandGestureRecognitionCalculator
  input_stream: ...
  output_stream: ...
  output_stream: "ASL:asl_stream"
}

The current situation seems to be

• The GetContract() function doesn't set up the "ASL" output stream for you because you ask it to do this only if the output stream with tag "ASL" is specified in the graph.
• The process() function assumes that output stream with tag "ASL" always exists and it outputs something to that stream.

[sne21star]

I believe I did that in render_cpu.pbtxt

I am trying to use this calculator: handgesture
and get the strings from here and input it in this calculator opencvputtext
I fixed some other issues however now I am getting this error

F0320 21:04:00.196940 17125376 collection.h:53] Failed to get tag "ASL" index 0
* Check failure stack trace: *
@ 0x10c5c58ff google::LogMessageFatal::~LogMessageFatal()
@ 0x10c5c2019 google::LogMessageFatal::~LogMessageFatal()
@ 0x10c07ac82 mediapipe::internal::CollectionErrorHandlerFatal<>::GetFallback()
@ 0x10c3810bc mediapipe::HandGestureRecognitionCalculator::Process()
@ 0x10c3c0e07 mediapipe::CalculatorNode::ProcessNode()
@ 0x10c3b679b mediapipe::internal::SchedulerQueue::RunCalculatorNode()
@ 0x10c3b6262 mediapipe::internal::SchedulerQueue::RunNextTask()
@ 0x10c3ba149 mediapipe::ThreadPool::RunWorker()
@ 0x10c3b9b45 mediapipe::ThreadPool::WorkerThread::ThreadBody()
@ 0x7fff753f72eb _pthread_body
@ 0x7fff753fa249 _pthread_start
Abort trap: 6

Still seems like the same one but it has a different error message.

[sne21star]

I think I realized that problem is that the render_cpu.pbtxt is not recognizing the new nodes I have created while it runs. However I am still unsure...

[eknight7]

It's because there is a commented line for the output_stream:"ASL:asl_letters" in the modified hand_landmark_cpu.pbtxt graph.

To fix this, either uncomment that line if appropriate, or add the following to your calculator's Process() function:

if (cc->Outputs().HasTag("ASL")) {
   cc->Outputs().Tag("ASL").AddPacket(
          MakePacket<std::string>(asl_letter)
              .At(cc->InputTimestamp()));

}

That builds the demo for me. Nice work! :)

[sne21star]

Thank you. Now it works however the text still doesn't come on to the screen. I realized it is because the nodes I have created:

node {
  calculator: "HandGestureRecognitionCalculator"
   input_stream: "NORM_LANDMARKS:landmarks"
   input_stream: "NORM_RECT:rect"
   output_stream: "ASL:text_to_put"
 }

node {
   calculator: "OpenCvPutTextCalculator"
   input_stream: "text_to_put"
   output_stream: "out_image_frames"
 }

# Draws annotations and overlays them on top of the input images.
node {
  calculator: "AnnotationOverlayCalculator"
  input_stream: "INPUT_FRAME:input_image"
  input_stream: "detection_render_data"
  input_stream: "landmark_render_data"
  input_stream: "rect_render_data"
  output_stream: "OUTPUT_FRAME:output_image"
}

The output stream from OpenCvPutTextCalculator should be the input stream for AnnotationOverlayCalculator. However, if I try to pass it in, the program breaks. Is there any way to change the output stream to render data type and add it as an input stream for AnnotationOverlayCalculator or is there an easier way?

[eknight7]

Yes, instead of using an OpenCvPutTextCalculator, use a simple StringToRenderDataCalculator that generates a new RenderData annotation object for an std::string.

I wrote a quick StringToRenderDataCalculator that you can try out. Copy it to the hand-gesture-recognition directory, and add the following changes to the renderer_cpu.pbtxt:

1. Comment out/Remove the OpenCvPutTextCalculator node.
2. Add the following node:
   node { calculator: "StringToRenderDataCalculator" input_stream: "TEXT:text_to_put" output_stream: "RENDER_DATA:text_render_data" }
3. Add the text_render_data as an input stream to the AnnotationOverlayCalculator node:
   node { calculator: "AnnotationOverlayCalculator" input_stream: "INPUT_FRAME:input_image" input_stream: "detection_render_data" input_stream: "landmark_render_data" input_stream: "rect_render_data" input_stream: "text_render_data" output_stream: "OUTPUT_FRAME:output_image" }

Change the output of the hand-gesture-recognition-calculator.cc to output a string containing one of the detected letters instead of the string "Hello!".

Rebuild the app via

bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 \
    mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu

and run it using

GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_cpu \
    --calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt

Here is a screenshot of one output frame I got using your graph:

[sne21star]

Hi,

So I tried implementing your changes however I get this error:
GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_cpu --calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtx

E0321 14:32:09.723388 122648000 demo_run_graph_main.cc:163] Failed to run the graph: ; Can't find file: mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtx

Any ideas how to fix this?

[eknight7]

There was a typo, fixed it, it should end with pbtxt...

[sne21star]

Awesome thanks it works! Should this also work on mobile applications as well?

[sne21star]

Actually I tried building the android application however it keeps crashing when I add these changes to the gpu files. Is there something extra I have to add?

[sne21star]

Actually nevermind I made a mistake it works! Thank you very much !!

[eknight7]

Great! Closing the issue. Feel free to open if you have more questions.