Mediapipe: Access hand landmarks position in iOS

[HebaMohamed]

Hello team,

How to access hand fingers landmarks position in iOS, in order to add ARKit to them ?

The iOS hand tracking project only reflect a stream not positions

[mgyong]

@HebaMohamed see related issue #200 We are working on providing obj C & java APIs to access result protos @eknight7
Will release something shortly

[HebaMohamed]

Thank you so much @mgyong , but the last release v0.6.4 did not provide the landmarks access yet,
There is any information about the release date please ?

[eknight7]

For iOS, you can use C++ since the code is in Objective-C++. Can you try this out?

Firstly, similar to output_stream, you need to add a listener for hand_landmarks stream in the (single) hand tracking project. Add the following line before the return newGraph statement in (MPPGraph*)loadGraphFromResource:(NSString*)resource method in https://github.com/google/mediapipe/blob/master/mediapipe/examples/ios/handtrackinggpu/ViewController.mm:

[newGraph addFrameOutputStream:"hand_landmarks" outputPacketType:MPPPacketTypeRaw];

Then, we need to add a listener for packets in this output stream. Implement the following method, again in https://github.com/google/mediapipe/blob/master/mediapipe/examples/ios/handtrackinggpu/ViewController.mm:

- (void)mediapipeGraph:(MPPGraph*)graph
       didOutputPacket:(const mediapipe::Packet&)packet
            fromStream:(const std::string&)streamName {
  if (streamName == "hand_landmarks") {
    if (packet.IsEmpty()) {
      NSLog(@"At timestamp: %lld, no hand landmarks available.", packet.Timestamp().Value());
      return;
    }
    const auto& landmarks = packet.Get<std::vector<mediapipe::NormalizedLandmark>();
    NSLog(@"At timestamp: %lld, number of landmarks on hand = %d", packet.Timestamp().Value(), landmarks.size());
  }
}

Note that you need a dependency on

#include "mediapipe/framework/formats/landmark.pb.h"

in the file you are modifying (and modify the corresponding rule in BUILD file as well).

[mgyong]

@HebaMohamed fixed in v0.6.6 Pls check it out and let us know

[HebaMohamed]

@mgyong Thank you so much for the update! I confirm that it working now 👍 but the rotation values are missing (Euler angles), any idea where to get it please ?