Mediapipe: How To Extract Detections from a Graph?

[SamuelJakes]

I'm trying to modify the hand detection gpu example app. I can see that the detections are outputted as Normalised Rects in the second last node ("DetectionsToRectsCalculator") from here:
https://github.com/google/mediapipe/blob/master/mediapipe/graphs/hand_tracking/subgraphs/hand_detection_gpu.pbtxt
What would be the correct way to get the coordinates for those normalised rects, so that they can be used in the MainActivity.java file?

[eknight7]

The DetectionsToRectsCalculator presently converts the first Detection result from DETECTIONS:palm_detections input stream to a NormalizedRect object in the NORM_RECT:palm_rect output stream.

To get the coordinates in Java, use code similar to reading a proto object with NormalizedLandmark as shown in handtrackinggpu/MainActivity.java.

Try the following code changes:

Add to rect.proto:

option java_package = "com.google.mediapipe.formats.proto";
option java_outer_classname = "RectProto";

Change the following lines in formats/BUILD to

java_lite_proto_library(
    name = "rect_java_proto_lite",
    strict_deps = 0,
    visibility = [
        "//mediapipe:__subpackages__",
    ],
    deps = [":rect_proto"],
)

Try adding the following code snippet to handdetectiongpu/MainActivity.java:

import com.google.mediapipe.formats.proto.RectProto.NormalizedRect;

@Override
  protected void onCreate(Bundle savedInstanceState) {
    ...
    processor.getVideoSurfaceOutput().setFlipY(FLIP_FRAMES_VERTICALLY);

    processor.addPacketCallback(
        "palm_rect",
        (packet) -> {
          byte[] palmRectRaw = PacketGetter.getProtoBytes(packet);
          try {
            NormalizedRect palmRect = NormalizedRect.parseFrom(palmRectRaw);
            if (palmRect == null) {
              Log.d(TAG, "[TS:" + packet.getTimestamp() + "] No hand rect.");
              return;
            }
            Log.d(TAG, "[TS:" + packet.getTimestamp() + "] Rect: " 
                  + getRectDebugString(palmRect));
          } catch (InvalidProtocolBufferException e) {
            Log.e(TAG, "Couldn't Exception received - " + e);
            return;
          }
        });

    PermissionHelper.checkAndRequestCameraPermissions(this);
  }

  private static String getRectDebugString(NormalizedRect rect) {
    // TODO: Please double-check the names of these getter functions
    // for fields in NormalizedRect.
    String rectString = "x_center: " + rect.getXCenter()
                        + ", y_center: " + rect.getYCenter()
                        + ", height: " + rect.getHeight()
                        + ", width: " + rect.getWidth();
    return rectString;
  }

And add the following dependency to handdetectiongpu/BUILD:

android_library(
    name = "mediapipe_lib",
    srcs = glob(["*.java"]),
    assets = [
        ":binary_graph",
        "//mediapipe/models:palm_detection.tflite",
        "//mediapipe/models:palm_detection_labelmap.txt",
    ],
    assets_dir = "",
    manifest = "AndroidManifest.xml",
    resource_files = glob(["res/**"]),
    deps = [
        ":mediapipe_jni_lib",
        "//mediapipe/framework/formats:rect_java_proto_lite",
        # Include everything else as before
    ],
)

Let us know if it doesn't work, we will add this code to the app example with this code for future reference.

[SamuelJakes]

Thanks so much for the incredibly detailed reply. A few things I found when trying this code:

The following additional imports in MainActivity.java were also needed to get it to compile:
import com.google.mediapipe.framework.PacketGetter;
import com.google.protobuf.InvalidProtocolBufferException;
import android.util.Log;

Those are the only changes that I made to the code you posted, and it currently compiles but the app crashes upon being launched. That's where I'm currently at, not sure yet why the app is crashing.

[SamuelJakes]

Here's the crash log caused by this code as it stands, I'm not sure what the best way to create the "corresponding output stream" would be to fix this:
----- beginning of crash 2020-02-19 09:07:39.252 8232-8250/? E/AndroidRuntime: FATAL EXCEPTION: ExternalTextureConverter Process: com.google.mediapipe.apps.handdetectiongpu, PID: 8232 com.google.mediapipe.framework.MediaPipeException: unknown: ; Input Stream "palm_rect" for node with sorted index 1 does not have a corresponding output stream. at com.google.mediapipe.framework.Graph.nativeStartRunningGraph(Native Method) at com.google.mediapipe.framework.Graph.startRunningGraph(Graph.java:289) at com.google.mediapipe.components.FrameProcessor.startGraph(FrameProcessor.java:306) at com.google.mediapipe.components.FrameProcessor.maybeAcceptNewFrame(FrameProcessor.java:229) at com.google.mediapipe.components.FrameProcessor.onNewFrame(FrameProcessor.java:244) at com.google.mediapipe.components.ExternalTextureConverter$RenderThread.renderNext(ExternalTextureConverter.java:261) at com.google.mediapipe.components.ExternalTextureConverter$RenderThread.lambda$onFrameAvailable$0$ExternalTextureConverter$RenderThread(ExternalTextureConverter.java:211) at com.google.mediapipe.components.ExternalTextureConverter$RenderThread$$Lambda$0.run(Unknown Source:4) at android.os.Handler.handleCallback(Handler.java:873) at android.os.Handler.dispatchMessage(Handler.java:99) at android.os.Looper.loop(Looper.java:193) at com.google.mediapipe.glutil.GlThread.run(GlThread.java:134)

[eknight7]

Are you able to run the app without the amended code?

[SamuelJakes]

Yes I am - I'm using bazel within Android Studio and it normally compiles and runs fine on the phone.

[SamuelJakes]

Notably, the code compiles and runs if I just comment out the processor.addPacketCallback method inserted into the MainActivity.java file.

[eknight7]

Okay, the issue seems to be with the palm_rect stream missing from the graph according to the error. Use hand_rect_form_palm_detections instead of palm_rect in the processor.addPacketCallback

[SamuelJakes]

It runs! Note that there was a typo and I got it to work by using hand_rect_from_palm_detections instead of hand_rect_ form_palm_detections.

One final question - is there a simple way to extend this for getting the normalised rects of all palms on the screen, not just the first one detected?

Thanks again, you've been a really great help.

[eknight7]

You could modify your app to use the multi_hand_detection graph which outputs a list of NORM_RECTS and extract them in Java.

[brianm-sra]

Following this thread. I'm interested in getting the rectangles for Android handtrackinggpu and 3D box (cube) for Android objectdetection3d (shoes, chairs, etc.) in Java friendly format. I will try to follow the instructions from eknight7 above

[brianm-sra]

@eknight7 Can you please clarify if a change is still needed in mediapipe/framework/formats/BUILD
and how to get the RectProto.NormalizedRect to be included in the AAR generated for Android Studio?

I added "//mediapipe/framework/formats:rect_java_proto_lite" to the deps for the android_binary section in mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu/BUILD

If I include both of these in framework/formats/BUILD

198 mediapipe_proto_library(
199     name = "rect_proto",
200     srcs = ["rect.proto"],
201     visibility = ["//visibility:public"],
202     deps = ["//mediapipe/framework/formats:location_data_proto"],
203 )
204 
205 java_lite_proto_library(
206     name = "rect_java_proto_lite",
207     strict_deps = 0,
208     visibility = [
209         "//mediapipe:__subpackages__",
210     ],
211     deps = [":rect_proto"],
212 )

Then I get an error at build time

$ bazel build -c opt --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --fat_apk_cpu=arm64-v8a,armeabi-v7a //mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example:mp_hand_tracking_aar
ERROR: /my/directory/mediapipe/mediapipe/framework/formats/BUILD:205:1: java_lite_proto_library rule 'rect_java_proto_lite' in package 'mediapipe/framework/formats' conflicts with existing java_lite_proto_library rule, defined at /my/directory/mediapipe/mediapipe/framework/formats/BUILD:198:1

[bobbyngwu]

Hi,

To get this working for me, I had to add rect_java_proto_lite here

android_library(
name = "demo_lib",
srcs = glob([".java"]),
manifest = "AndroidManifest.xml",
resource_files = glob(["res/"]),
deps = [
*"//mediapipe/framework/formats:rect_java_proto_lite",
],
)

However, now I am stuck trying to figure out how to include this in the aar build as I cannot make reference to it in my use case since it is missing. Glad if you can help me. This is what my build file for aar looks like:

load("//mediapipe/java/com/google/mediapipe:mediapipe_aar.bzl", "mediapipe_aar")

mediapipe_aar(
name = "mp_hand_detection_aar",
calculators = [
"//mediapipe/graphs/hand_tracking:detection_mobile_calculators",
]
)

f you try to add deps, src, it will throw an error syaing got unexpected keyword arguments
Glad if you can help @eknight7 , thanks

After some bit of investigation, turn out I need to add that as part of the proto in mediapipe_aar.bzl to access the RectProto. Leaving comment incase a newbie like me stumbles unto this :)