Mediapipe: Extracting keypoints from face detection

[vertify-mldev1]

I am using the facedetectiongpu example on android (packed to an aar file) and am currently trying to modify it such that the actual detections are output, not only the output video stream. In #454 it is described that you can use DetectionsToRectsCalculator to get the bounding boxes of the detections. How can I extract keypoints like eyes, nose, etc.?
I already tried to read the detection proto object in Java (like in the handtracking example), but got a crash at line:
byte[] detectionsRaw = PacketGetter.getProtoBytes(packet);

Error log:

F/libc    ( 6390): Fatal signal 6 (SIGABRT), code -6 (SI_TKILL) in tid 7000 (Thread-10), pid 6390 (.camera_test)
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Build fingerprint: 'samsung/a40eea/a40:9/PPR1.180610.011/A405FNXXU3ATA4:user/release-keys'
Revision: '4'
ABI: 'arm64'
pid: 6390, tid: 7000, name: Thread-10  >>> com.exampleapp.camera_test <<<
signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
    x0  0000000000000000  x1  0000000000001b58  x2  0000000000000006  x3  0000000000000008
    x4  63207469202c273e  x5  63207469202c273e  x6  63207469202c273e  x7  656220746f6e6e61
    x8  0000000000000083  x9  a268f4b2a654d1d3  x10 0000000000000000  x11 fffffffc7ffffbdf
    x12 0000000000000001  x13 747265766e6f6320  x14 0000000000000000  x15 ffffffffffffffff
    x16 0000007adf0712a8  x17 0000007adefb0740  x18 0000000000000001  x19 00000000000018f6
    x20 0000000000001b58  x21 0000007a3f53c4c4  x22 0000007a3f52d418  x23 0000007a429c55b9
    x24 0000000000000008  x25 0000007a0d0d8588  x26 0000007a0d0d8588  x27 0000000000000000
    x28 0000007a0d0d6780  x29 0000007a0d0d6510
    sp  0000007a0d0d64d0  lr  0000007adefa3f44  pc  0000007adefa3f6c
backtrace:
    #00 pc 0000000000021f6c  /system/lib64/libc.so (abort+116)
    #01 pc 0000000000804d9c  /data/app/com.exampleapp.camera_test-T-kXeE4HrDGxR4SzEuTTxA==/lib/arm64/libmediapipe_jni.so (google::logging_fail()+8)
    #02 pc 0000000000804214  /data/app/com.exampleapp.camera_test-T-kXeE4HrDGxR4SzEuTTxA==/lib/arm64/libmediapipe_jni.so (google::LogMessage::SendToLog()+920)
    #03 pc 0000000000804788  /data/app/com.exampleapp.camera_test-T-kXeE4HrDGxR4SzEuTTxA==/lib/arm64/libmediapipe_jni.so (google::LogMessage::Flush()+220)
    #04 pc 0000000000807e78  /data/app/com.exampleapp.camera_test-T-kXeE4HrDGxR4SzEuTTxA==/lib/arm64/libmediapipe_jni.so (google::LogMessageFatal::~LogMessageFatal()+16)
    #05 pc 0000000000675ba8  /data/app/com.exampleapp.camera_test-T-kXeE4HrDGxR4SzEuTTxA==/lib/arm64/libmediapipe_jni.so (mediapipe::Packet::GetProtoMessageLite() const+300)
    #06 pc 0000000000325a44  /data/app/com.exampleapp.camera_test-T-kXeE4HrDGxR4SzEuTTxA==/lib/arm64/libmediapipe_jni.so (Java_com_google_mediapipe_framework_PacketGetter_nativeGetProtoBytes+52)
    #07 pc 00000000005633e0  /system/lib64/libart.so (art_quick_generic_jni_trampoline+144)
    #08 pc 000000000055a64c  /system/lib64/libart.so (art_quick_invoke_static_stub+604)
    #09 pc 00000000000d00e8  /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+232)
    #10 pc 00000000002834c0  /system/lib64/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+344)
    #11 pc 000000000027d4c8  /system/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+968)
    #12 pc 000000000052afe8  /system/lib64/libart.so (MterpInvokeStatic+204)
    #13 pc 000000000054cc14  /system/lib64/libart.so (ExecuteMterpImpl+14612)
    #14 pc 000000000032f958  /dev/ashmem/dalvik-classes9.dex extracted in memory from /data/app/com.exampleapp.camera_test-T-kXeE4HrDGxR4SzEuTTxA==/base.apk!classes9.dex_6390_6390 (deleted) (com.google.mediapipe.framework.PacketGetter.getProtoBytes+8)
    #15 pc 00000000002571cc  /system/lib64/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadERKNS_20CodeItemDataAccessorERNS_11ShadowFrameENS_6JValueEb.llvm.2026799985+488)
    #16 pc 000000000025ccc0  /system/lib64/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*, art::JValue*)+216)
    #17 pc 000000000027d4ac  /system/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+940)
    #18 pc 000000000052afe8  /system/lib64/libart.so (MterpInvokeStatic+204)
    #19 pc 000000000054cc14  /system/lib64/libart.so (ExecuteMterpImpl+14612)
    #20 pc 000000000001a1d2  /dev/ashmem/dalvik-classes12.dex extracted in memory from /data/app/com.exampleapp.camera_test-T-kXeE4HrDGxR4SzEuTTxA==/base.apk!classes12.dex_6390_6390 (deleted) (com.exampleapp.camera_test.MainActivity.lambda$initMediapipe$0+14)
    #21 pc 00000000002571cc  /system/lib64/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadERKNS_20CodeItemDataAccessorERNS_11ShadowFrameENS_6JValueEb.llvm.2026799985+488)
    #22 pc 000000000025ccc0  /system/lib64/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*, art::JValue*)+216)
    #23 pc 000000000027d4ac  /system/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+940)
    #24 pc 000000000052afe8  /system/lib64/libart.so (MterpInvokeStatic+204)
    #25 pc 000000000054cc14  /system/lib64/libart.so (ExecuteMterpImpl+14612)
    #26 pc 0000000000019a94  /dev/ashmem/dalvik-classes12.dex extracted in memory from /data/app/com.exampleapp.camera_test-T-kXeE4HrDGxR4SzEuTTxA==/base.apk!classes12.dex_6390_6390 (deleted) (com.exampleapp.camera_test.-$$Lambda$MainActivity$PlN-6XToOze6Zhi-jsLdA4LJWYo.process)
    #27 pc 00000000002571cc  /system/lib64/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadERKNS_20CodeItemDataAccessorERNS_11ShadowFrameENS_6JValueEb.llvm.2026799985+488)
    #28 pc 000000000051a378  /system/lib64/libart.so (artQuickToInterpreterBridge+1020)
    #29 pc 00000000005634fc  /system/lib64/libart.so (art_quick_to_interpreter_bridge+92)
    #30 pc 000000000055a388  /system/lib64/libart.so (art_quick_invoke_stub+584)
    #31 pc 00000000000d00c8  /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+200)
    #32 pc 00000000004614b0  /system/lib64/libart.so (art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*)+104)
    #33 pc 0000000000462844  /system/lib64/libart.so (art::InvokeVirtualOrInterfaceWithVarArgs(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, std::__va_list)+440)
    #34 pc 0000000000343218  /system/lib64/libart.so (art::JNI::CallVoidMethodV(_JNIEnv*, _jobject*, _jmethodID*, std::__va_list)+656)
    #35 pc 0000000000100760  /system/lib64/libart.so (art::(anonymous namespace)::CheckJNI::CallMethodV(char const*, _JNIEnv*, _jobject*, _jclass*, _jmethodID*, std::__va_list, art::Primitive::Type, art::InvokeType)+2592)
    #36 pc 00000000000edb64  /system/lib64/libart.so (art::(anonymous namespace)::CheckJNI::CallVoidMethodV(_JNIEnv*, _jobject*, _jmethodID*, std::__va_list)+92)
    #37 pc 00000000003139f4  /data/app/com.exampleapp.camera_test-T-kXeE4HrDGxR4SzEuTTxA==/lib/arm64/libmediapipe_jni.so (_JNIEnv::CallVoidMethod(_jobject*, _jmethodID*, ...)+116)
    #38 pc 000000000031119c  /data/app/com.exampleapp.camera_test-T-kXeE4HrDGxR4SzEuTTxA==/lib/arm64/libmediapipe_jni.so (mediapipe::android::Graph::CallbackToJava(_JNIEnv*, _jobject*, mediapipe::Packet const&)+576)
    #39 pc 00000000005ccf70  /data/app/com.exampleapp.camera_test-T-kXeE4HrDGxR4SzEuTTxA==/lib/arm64/libmediapipe_jni.so (mediapipe::tool::CallbackCalculator::Process(mediapipe::CalculatorContext*)+108)
    #40 pc 000000000060854c  /data/app/com.exampleapp.camera_test-T-kXeE4HrDGxR4SzEuTTxA==/lib/arm64/libmediapipe_jni.so (mediapipe::CalculatorNode::ProcessNode(mediapipe::CalculatorContext*)+812)
    #41 pc 00000000005f7ae0  /data/app/com.exampleapp.camera_test-T-kXeE4HrDGxR4SzEuTTxA==/lib/arm64/libmediapipe_jni.so (mediapipe::internal::SchedulerQueue::RunCalculatorNode(mediapipe::CalculatorNode*, mediapipe::CalculatorContext*)+312)
    #42 pc 00000000005f7534  /data/app/com.exampleapp.camera_test-T-kXeE4HrDGxR4SzEuTTxA==/lib/arm64/libmediapipe_jni.so (mediapipe::internal::SchedulerQueue::RunNextTask()+232)
    #43 pc 000000000062713c  /data/app/com.exampleapp.camera_test-T-kXeE4HrDGxR4SzEuTTxA==/lib/arm64/libmediapipe_jni.so (mediapipe::ThreadPool::RunWorker()+392)
    #44 pc 0000000000626d7c  /data/app/com.exampleapp.camera_test-T-kXeE4HrDGxR4SzEuTTxA==/lib/arm64/libmediapipe_jni.so (mediapipe::ThreadPool::WorkerThread::ThreadBody(void*)+1388)
    #45 pc 00000000000836c8  /system/lib64/libc.so (__pthread_start(void*)+196)
    #46 pc 0000000000023bac  /system/lib64/libc.so (__start_thread+68)

Thank you.

[vertify-mldev1]

I just tried the same procedure with the rect proto object as output, that works. But this way I can't receive the keypoints

[eknight7]

Hi! Can you confirm you have added appropriate BUILD file dependencies as mentioned in https://github.com/google/mediapipe/issues/454#issuecomment-585537487?

It would be helpful to see the BUILD file and the MainActivity.java of your project if you can post it here. Thanks!

[vertify-mldev1]

Hi, thanks for your help!

I think i added the appropriate dependencies, as follows:

formats/BUILD:

java_lite_proto_library(
    name = "location_data_java_proto_lite",
    strict_deps = 0,
    visibility = ["//mediapipe:__subpackages__"],
    deps = [":location_data_proto"],
)

formats/annotation/BUILD:

java_lite_proto_library(
    name = "rasterization_java_proto_lite",
    strict_deps = 0,
    visibility = ["//mediapipe:__subpackages__"],
    deps = [":rasterization_proto"],
)
# Expose the proto source files for building mediapipe AAR.
filegroup(
    name = "protos_src",
    srcs = glob(["*.proto"]),
    visibility = ["//mediapipe:__subpackages__"],
)

formats/location_data.proto

option java_package = "com.google.mediapipe.formats.proto";
option java_outer_classname = "LocationDataProto";

formats/annotation/rasterization.proto

option java_package = "com.google.mediapipe.formats.annotation.proto";
option java_outer_classname = "RasterizationProto";

java/com/google/mediapipe/mediapipe_aar.bzl

_proto_java_src_generator(
    name = "rasterization_proto",
    proto_src = "mediapipe/framework/formats/annotation/rasterization.proto",
    java_lite_out = "com/google/mediapipe/formats/annotation/proto/RasterizationProto.java",
    srcs = ["//mediapipe/framework/formats/annotation:protos_src"],
)

_proto_java_src_generator(
    name = "location_data_proto",
    proto_src = "mediapipe/framework/formats/location_data.proto",
    java_lite_out = "com/google/mediapipe/formats/proto/LocationDataProto.java",
    srcs = ["//mediapipe/framework/formats/annotation:protos_src", "//mediapipe/framework/formats:protos_src"],
)

_proto_java_src_generator(
    name = "detection_proto",
    proto_src = "mediapipe/framework/formats/detection.proto",
    java_lite_out = "com/google/mediapipe/formats/proto/DetectionProto.java",
    srcs = ["//mediapipe/framework/formats/annotation:protos_src", "//mediapipe/framework/formats:protos_src"],
)

android_library(
    name = name + "_android_lib",
    srcs = [
        ...
        "com/google/mediapipe/formats/annotation/proto/RasterizationProto.java",
        "com/google/mediapipe/formats/proto/LocationDataProto.java",
        "com/google/mediapipe/formats/proto/DetectionProto.java",
        "com/google/mediapipe/formats/proto/RectProto.java",
    ],
    manifest = "AndroidManifest.xml",
    proguard_specs = ["//mediapipe/java/com/google/mediapipe/framework:proguard.pgcfg"],
    deps = [
        ":" + name + "_mediapipe_jni_lib",
        "//mediapipe/framework:calculator_java_proto_lite",
        "//mediapipe/framework:calculator_profile_java_proto_lite",
        "//mediapipe/framework/tool:calculator_graph_template_java_proto_lite",
        "//mediapipe/framework/formats/annotation:rasterization_java_proto_lite",
        "//mediapipe/framework/formats:location_data_java_proto_lite",
        "//mediapipe/framework/formats:detection_java_proto_lite",
        ...
    ],
)

The MainActivity is kind of big, In short I'm displaying a surface with the video stream output from the graph (so i can see that the face detection is working). Additionally the raw outputs should be retrieved with the callback (crashing)
The important parts:

processor =
        new FrameProcessor(
                getApplicationContext(),
                eglManager.getNativeContext(),
                BINARY_GRAPH_NAME,
                INPUT_VIDEO_STREAM_NAME,
                OUTPUT_VIDEO_STREAM_NAME);
processor.getVideoSurfaceOutput().setFlipY(FLIP_FRAMES_VERTICALLY);

processor.addPacketCallback(
        OUTPUT_DETECTIONS_STREAM_NAME,
        packet -> {
            byte[] detectionsRaw = PacketGetter.getProtoBytes(packet);
            try {
                DetectionList detections = DetectionList.parseFrom(detectionsRaw);
                if (detections == null) {
                    Log.d(TAG, "[TS:" + packet.getTimestamp() + "] No detections.");
                    return;
                }
                Log.d(TAG,"[TS:"
                                + packet.getTimestamp()
                                + "] #Detections: "
                                + detections.getDetectionCount());
                Log.d(TAG, getDetectionsDebugString(detections));
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
);

converter = new ExternalTextureConverter(eglManager.getContext());
converter.setFlipY(FLIP_FRAMES_VERTICALLY);
converter.setConsumer(processor);

converter.setSurfaceTextureAndAttachToGLContext(
        mpCameraSurfaceTexture,
        previewWidth,
        previewWidth
);

[bhack]

It could be nice to have a general example in the repo that access to the output bounding rects and keypoints/landmarks for manipulation.
Cause It Is common to many task/models like face detectio+landmarks, Human detection+joints/pose , 3d cuboid object detection+object landmarks/keypoints and so on..

[russelldj]

I agree with @bhack 's comment. I'm trying to run hand tracking on a couple of videos and output the results as a text file. I was hoping I could just modify a renderer or something like that but haven't figured out how to yet. A demo for something like this would be really cool or suggestions for possible solutions. Thanks!

[vertify-mldev1]

In release v0.7.1 they added support for detection proto in MediaPipe AAR

List detections = PacketGetter.getProtoVector(packet, Detection.parser());

now works for me.

Thank you!

[bhack]

@vertify-mldev1 Is it java/Android only?

[vertify-mldev1]

I am not sure, I only tried it using Java on Android.

[sufiaji]

Hi @vertify-mldev1
Could you please explain on how to get the face landmarks (eyes, mouth, nose) from Detection objects returned from getProtoVector? Because if I check in MainActivity.java of objectdetectiongpu, Detection object only has 2 attributes: Label and Score.
Thanks

[sufiaji]

Hi @vertify-mldev1,

Please help. I encountered same error with you upon calling getProtoVector
JNI DETECTED ERROR IN APPLICATION: JNI GetObjectClass called with pending exception java.lang.NoClassDefFoundError: com.google.mediapipe.formats.proto.DetectionProto$Detection
I follow all your steps on changing the file but no luck.
Please kindly your suggestion
Thanks

[vertify-mldev1]

Hello, did you upgrade to v0.7.1? Is the aar file included in the project? I did not have to edit anything with v0.7.1.

[sufiaji]

Hi @vertify-mldev1 ,

Yes, the problem was on google protobuf google. I was using lite version, now I am using full version of protobuf and problem disappeared.
Thanks.

[brianm-sra]

EDIT: I solved this by rebuilding with gradlew on command-line, removing .idea directory, re-launching Android Studio, re-opening the gradle project.

_How do I upgrade to release v0.7.1 ? I pulled master branch of code from mediapipe github repo and built aar and copied it to my app's libs directory.
But Android Studio is not able to see com.google.mediapipe.formats.proto.DetectionProto
as something available to import_