Mediapipe: How to convert mediapipe coordinates into Unity world space coordinates?

[BoneFlat]

Hi MediaPipe Team, i'm trying to convert 21 landmarks into unity 3d world space.
I follow these step to do that:
Step 1: I follow @eknight7 instruction from #454 to get NormalizedRect in image coordinates.
Step 2: Because "NormalizedLandmark" is landmark location in HandRect coordinates, so i can convert 21 point into image coordinates.
Step 3: I'm using "ScreenToWorldSpace" method in Unity to convert 21 point in image coordinate to 3d world space with a specifi depth (depth = 3 in my case).

But the final result in Unity has a little discrepancies , you can see in image below.
Can you give me some advise to convert the landmarks. And i also want to know is there a solution to get the real depth?

[CalebGreer]

I have been trying to use mediapipe with Unity as well and was wondering do you have mediapipe running within Unity or are you getting the landmarks somewhere else and then sending it to Unity?

[BoneFlat]

@CalebGreer, i'm building aar (android archive library) and use it as plugin for Unity

[CalebGreer]

@BoneFlat Oh, I had tried that but couldn't get it to work. Would you mind sharing your solution or how you got the aar to work within Unity?

[BoneFlat]

@CalebGreer, i'm following "the left" solution at https://github.com/google/mediapipe/issues/36#issuecomment-524477282.
For plugin, I'm using 2 aar plugin , one is mediapipe aar we have build, another is wrote to hold some method like "InitGraph, StartGraph, SetInputBitmap, GetLandMark ...".
In Unity, i get texture from webcamtexture, convert it to byte array and use byte array for plugin's input

[BoneFlat]

Now, i'm ok with x and y in image coordinates but with the depth, i still have no idea

[SwatiModi]

Hey @BoneFlat,
Can you tell more about the another AAR you used with InitGraph, StartGraph functions? I checked out FrameProcessor.java but unable to understand how to use it with the byte array input to mediapipe from unity.

[justinduynguyen]

i'm considering that how to stream a byte array from unity webcam to mediapipe ? Any solution? Thank you.

[brianm-sra]

@BoneFlat did you make any progress on this?

[BoneFlat]

Sorry all for the late answer, i made a simple project that show how to use mediapipe with Unity, i also made a unity package, you can use this for another project. You can found all here: https://gitlab.com/thangnh.sas/mediapipe-unity-hand-tracking

[JackieNek]

I made a solution to solve the depth and made a simple game base on that here: https://gitlab.com/JackieNek/single-hand-tracking . You can see demo here: https://youtu.be/paZ3sXtKTq8

[brianm-sra]

@BoneFlat : in your ARHand.cs file I found this code:
for (int i = 0; i < 21; i++) {
float xScreen = Screen.width * ((arr[i * 3 + 1] - 0.5f * (1 - c)) / c);
float yScreen = Screen.height * (arr[i * 3]);
this.landmarks[i] = cam.ScreenToWorldPoint(new Vector3(xScreen, yScreen, arr[i * 3 + 2] / 80 + 0.4f));
}

Would you be willing to share how you determined that those formulas were correct?
In particular for X and Z

X - 0.5f * (1 - c)) / c
Z / 80 + 0.4f

Were these specific to camera calibration on your particular camera device ?

[justinduynguyen]

Thank you in advance guys.

[TheBricktop]

is it possible to build analogically build the media pipe as a
plugin for unity desktop?

[tutu5110]

All very cool! @BoneFlat

[sachinjambhulkar]

@BoneFlat Thank you ! The package is working like a charm. The only drag i have seen in FPS... i feel this has to be taken up to run over the GPU for better performance.

[danieljtrujillo]

@sachinjambhulkar could you describe in detail how you implemented it?

[sachinjambhulkar]

@danieljtrujillo clone the repo: https://gitlab.com/thangnh.sas/mediapipe-unity-hand-tracking. navigate to the base directory of this project, right click over the file "MediapipeUnityPackage.unitypackage" and open with the Unity.

You can observe the complete project with all assets getting imported to unity Editor. Open the SampleScene.unity and you can make it run on your preferred platform / end device target.

[zhouwei5113]

@BoneFlat : in your ARHand.cs file I found this code:
for (int i = 0; i < 21; i++) {
float xScreen = Screen.width * ((arr[i * 3 + 1] - 0.5f * (1 - c)) / c);
float yScreen = Screen.height * (arr[i * 3]);
this.landmarks[i] = cam.ScreenToWorldPoint(new Vector3(xScreen, yScreen, arr[i * 3 + 2] / 80 + 0.4f));
}

Would you be willing to share how you determined that those formulas were correct?
In particular for X and Z

X - 0.5f * (1 - c)) / c
Z / 80 + 0.4f

Were these specific to camera calibration on your particular camera device ?

@brianm-sra Do you have any ideas about the formula？