Mediapipe: Stuck at first frame during FeshMesh detection without face.

[noirmist]

Based on this code, I do the same things at face_mesh demo.

And I realize the poller is stopped when the face is not showing.
Does anyone how to fix this issue?
Here are the brief code and debugging.

Add debugging print at demo_run_graph_main.cc

    mediapipe::Packet packet;
    if (!poller.Next(&packet)) break;
    // Use packet.Get to recover values from packet
    auto& output_frame = packet.Get<mediapipe::ImageFrame>();

    LOG(INFO) << "before polling landmark frame: " << k;

    //Polling the poller to get landmark packet
    mediapipe::Packet landmark_packet;
    LOG(INFO) << "landmark packet " << k;

    if (!poller_landmark.Next(&landmark_packet)){
        LOG(INFO) << "polling down.";
        break;
    }   
    else
        LOG(INFO) << "next showed.";

    LOG(INFO) << "after polling landmark frame: " << k;

    auto& output_landmarks = landmark_packet.Get<std::vector<::mediapipe::NormalizedLandmarkList>>();

    LOG(INFO) << "after landmark frame: " << k;
    ++k;

Add debugging print at demo_run_graph_main.cc

/ The public interface of output stream poller.
class OutputStreamPoller {
 public:
  OutputStreamPoller(const OutputStreamPoller&) = delete;
  OutputStreamPoller& operator=(const OutputStreamPoller&) = delete;
  OutputStreamPoller(OutputStreamPoller&&) = default;
  // Move assignment needs to be explicitly defaulted to allow ASSIGN_OR_RETURN
  // on `StatusOr<OutputStreamPoller>`.
  OutputStreamPoller& operator=(OutputStreamPoller&&) = default;

  // Resets OutputStramPollerImpl and cleans the internal packet queue.
  void Reset() {
    auto poller = internal_poller_impl_.lock();
    CHECK(poller) << "OutputStreamPollerImpl is already destroyed.";
    poller->Reset();
  }

  // Gets the next packet (block until it is available or the stream is
  // done).  Returns true if successful.
  ABSL_MUST_USE_RESULT bool Next(Packet* packet) {
    LOG(INFO) << "next 1" ;
    auto poller = internal_poller_impl_.lock();
    LOG(INFO) << "next locked" ;
    if (!poller) {
        LOG(INFO) << "next failed" ;
      return false;
    }   
    LOG(INFO) << "next succeed" << packet ;

    return poller->Next(packet);
  }

Output result

The code is stuck at last line. The only way to quit is ctrl+c.

The demo works fine when the face mesh is showing, this issue only happen when the face is not detected.
So I test it with covered camera.

[dom607]

I think you need to create a custom calculator and modify the graph to use it.

The Next() method works synchronously, and if there are no packets in the corresponding output stream, it is designed to wait until the packet comes in. The graph used in the face_mesh demo is supposed to output nothing to the landmark output stream if no face is found. This is why your program has stuck. And as far as I know, Graph or Poller do not provide a way to check this.

I solved it by attaching a custom calculator that always outputs the number of faces found. Then polled the landmarks only if there were faces. To make the calculator's Process() method always called, you need an input stream to trigger. The input image or BATCH_END stream may be used for this purpose.

[mgyong]

@dom607 It would be great if you could share the graph and custom calculator you have created for always outputting the number of faces found. We can incorporate this for the next version of the face mesh release. Thanks

[dom607]

@mgyong https://github.com/google/mediapipe/pull/867 It took a while to try to make it into a generally usable form.

[lackhole]

@dom607 Can you share your graph and custom calculator? It's ok even if it's not refined

[dom607]

@lackhole see PR #867

[lackhole]

@dom607 thanks a lot 👍

[homuler]

I also encountered this problem, and PacketPresenceCalculator looks helpful.
I tried it with demo_run_graph_main_gpu.cc and it worked.

```face_mesh_desktop_live_gpu.pbtxt

face_mesh_desktop_live_gpu.pbtxt

input_stream: "input_video"

output_stream: "output_video"
output_stream: "multi_face_landmarks"

output_stream: "landmark_presence"

...
node {
calculator: "FaceRendererGpu"
input_stream: "IMAGE:throttled_input_video"
input_stream: "LANDMARKS:multi_face_landmarks"
input_stream: "NORM_RECTS:face_rects_from_landmarks"
input_stream: "DETECTIONS:face_detections"
output_stream: "IMAGE:output_video"
}

add PacketPresenceCalculator

node {
calculator: "PacketPresenceCalculator"
input_stream: "PACKET:multi_face_landmarks"
output_stream: "PRESENCE:landmark_presence"
}

```demo_run_graph_main_gpu.cc
...
    ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller presence_poller,
                   graph.AddOutputStreamPoller("landmark_presence"));
    ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller landmark_poller,
                   graph.AddOutputStreamPoller("multi_face_landmarks"));

...
    // Get the graph result packet, or stop if that fails.
    mediapipe::Packet packet;
    mediapipe::Packet presence_packet;
    mediapipe::Packet landmark_packet;
    if (!poller.Next(&packet)) break;

    // check whether the packet exists
    if (!presence_poller.Next(&presence_packet)) break;
    auto is_landmark_present = presence_packet.Get<bool>();

    if (is_landmark_present) {
      // fetch landmarks only when they exist
      if (landmark_poller.Next(&landmark_packet)) {
        // do something
      }
    }
...

[bryn-m-hughes]

It is worth being careful with the packet presence calculator approach when using a flow limiter calculator. In my experience the packet presence calculator will mark any packets past the first non-present packet as not present when a flow limiter is upstream. I haven't really looked into the cause of this, but it is worth noting.
I also believe the calculator in the pull request had some strange interactions with flow limiters because of the affects of flow limiters on timestamps, but again, I haven't looked into it thoroughly.