Mediapipe: Unable to attach observer to output stream "output_detections" because it doesn't exist.

[TanmoyPanigrahi]

Hi, I'm customizing the desktop example to get the raw score/confidence value for each detection. Most examples used output_detections but using

constexpr char kConfidence[] = "output_detections"; just gives me the error message

I had previously made enough progress to print the score using by adding -

constexpr char kConfidence[] = "plam_detections" but the program used to freeze when I actually put my palm in front of the camera and the program reached the poller. Didn't even print that it broke; the video just froze.

...
ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller poller_confi,
graph.AddOutputStreamPoller(kConfidence));
MP_RETURN_IF_ERROR(graph.StartRun({}));

...
...
mediapipe::Packet confi;
std::cout<<"checkpoint 1"<<"\n";
if (!poller_confi.Next(&confi)) {
std::cout<<"broke"<<"\n";
break;
}
auto& confival = confi.Get>();
for (const ::mediapipe::Detection& detection : confival) {
std::cout << detection.DebugString();
std::cout<<"detected stuff"<<"\n";
}
...
I would appreciate it if someone could share why output detections isn't being accepted or why the program probably freezes?

[mgyong]

Can you provide link to your code repo for us to reproduce the issue?

[TanmoyPanigrahi]

@mgyong @jiuqiant
https://github.com/TanmoyPanigrahi/MPIssue

this is the link to my code. It currently crashes on the first detection of the palm. I've spent a lot of time trying to come up with an explanation but can't really come up with one that seems logical to me. Really appreciate the help!

[eknight7]

Which graph are you passing to the demo at runtime? Please share the command you are using to run the demo?

[TanmoyPanigrahi]

Compile

bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu

Run

GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_cpu --calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt

@eknight7 I run the hand tracking graph on it. And i'm using the palm_detections stream for input.

I've figured that it won't have output_detections so it doesn't find it but mediapipe seems to be freezing on the first detection which shouldn't be happening.

[eknight7]

Okay, the reason the program freezes is because of this line:

 if (!poller_confi.Next(&confi)) {

poller_confi.Next(&confi) waits for a packet in the kConfidence output stream to show up. But there will be no output after a palm is detected because according to the graph logic, the palm_detection branch of the graph runs only if detection failed in the previous iteration.

This is logic is implemented using the GateCalculator (https://github.com/google/mediapipe/blob/master/mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt#L27-L41)

# Drops the incoming image if HandLandmarkSubgraph was able to identify hand
# presence in the previous image. Otherwise, passes the incoming image through
# to trigger a new round of hand detection in HandDetectionSubgraph.
node {
  calculator: "GateCalculator"
  input_stream: "input_video"
  input_stream: "DISALLOW:prev_hand_presence"
  output_stream: "hand_detection_input_video"

  node_options: {
    [type.googleapis.com/mediapipe.GateCalculatorOptions] {
      empty_packets_as_allow: true
    }
  }
}

So the program should not expect a Detection output packet at every frame.
This answers the question of why the program is getting stuck at run time.

[TanmoyPanigrahi]

@eknight7 is there any way to get a detection output for every frame? I am trying to get the confidence score for each hand detection in the video feed.

[eknight7]

Palm detection is an expensive step. As described in https://mediapipe.readthedocs.io/en/latest/hand_tracking_mobile_gpu.html, it is used only to localize the hand in an image, then the landmark subgraph runs on the localized patch to extract hand landmarks.

If you want to only get confidence scores of the palm detections, you could use https://mediapipe.readthedocs.io/en/latest/hand_detection_mobile_gpu.html, i.e. the hand_detection graph that runs detection on each frame.

But, if you also want the landmarks, the hand tracking solution optimizes performance by doing away with the need for running palm detection on every frame, when landmarks were visible in the previous frame. You can also use the hand_presence_score output of thehand_landmark subgraph https://github.com/google/mediapipe/blob/master/mediapipe/graphs/hand_tracking/subgraphs/hand_landmark_gpu.pbtxt#L100-L106.

[TanmoyPanigrahi]

@eknight7 I got it done!! THANK YOU SO MUCH!!