Arfoundation-samples: Map Camera Video onto 3D Face Geometry

[jessekirbs]

Hello! I'm trying to figure out is it's possible to utilize the camera feed and map that as a texture onto 3D face geometry using AR Foundation - like at this link. Are the TrueDepth and LIDAR camera feeds available from the iPhone?

I can't find any documentation on how to do this in anything other than Swift. Any guidance would be greatly appreciated. Thank you!

[tdmowrer]

That sample uses Scene Kit, which has a utility for extracting the face geometry and material. ARFoundation lets you get the face geometry & uv texture coordinates and the camera image, but there is no utility to combine them.

[DavidMohrhardt]

Offering up a possible solution, you could use a grab pass shader to grab the background where the face mesh is about to be rendered.

[sam598]

Instead of a grab pass shader I would take the CPU camera image like @tdmowrer suggested. ARFoundationBackgroundRenderer crops the image and projection matrix to the phone screen, so this will get you the full image from ARKit. You can create a camera projection matrix for that image using the camera intrinsics.

The UV coordinates of the face mesh will be the screen space position of the face geometry vertices, calculated using that projection matrix. You should be able to calculate those vertices on the CPU using MultiplyPoint, however it would be fastest to pass the projection matrix to a custom vertex shader and calculate the UVs there.

Keep in mind this real time method does not account for occlusion. If a face is captured from the side, the hidden part of the mesh will be a mirror image.

[jessekirbs]

Thanks for the suggestions! Will test all this out and report back.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.