Mediapipe: How to render different face effect

[ashrvstv]

In face effect module I can 3d data as
glasses.pbtxt
facepaint.pngblob
glasses.pngblob.

I am trying to add few more models to experiment but i couldnt find any documenation or information of pngblob data.
It seems like that using glasses.pbtxt 3d model is generated at runtime and glasses.pngblob is getting used as texture. Can you please clear that is it right and how is it happening

Ques 1-
Can you please provide any documentation of pngblob datatype.
How can i create new 3d model (pngblob / binarypb ) to render on face.
Most common format of 3d model data are OBJ, FBX, etc. Is there any way to convert these format of 3d data to binarypb / pngblob?

Ques 2-
it is mentioned in gl_animation_overlay_calculator.cc that .obj.uuu can be created using the mentioned SimpleObjEncryptor but I couldn't find that. can you please specify where to find that ?

ANIMATION_ASSET (String, required):
//     Path of animation file to load and render. Should be generated by
//     //java/com/google/android/apps/motionstills/SimpleObjEncryptor with
//     --compressed_mode=true.  See comments and documentation there for more
//     information on custom .obj.uuu file format.

[kostyaby]

Hey @ashrvstv,

Thanks for your interest!

PNGblob

In fact, it is a regular PNG format; .pngblob is used instead of .png to prevent iOS build from preprocessing the image. OpenCV is unable to read a PNG file preprocessed by the iOS build. So you can use any filename.png for your effect, just rename it filename.pngblob

3D Model

Ultimately, the model .pbbinary file must follow the mediapipe.face_geometry.Mesh3d binary protobuf format in order to be rendered. Here's a path for turning an OBJ 3D model into that format step-by-step:

• Convert an OBJ model into mediapipe.face_geometry.Mesh3d text protobuf format. We do not provide an official OBJ -> Mesh3d pbtxt converter, but I can share a non-official one that I wrote to convert the glasses OBJ asset: link
• Use encode_binary_proto Bazel rule to convert model.pbtxt into model.pbbinary: link

If you have your 3D models in a format other than OBJ, then please try to convert those into OBJ first. Please note, that your model must not have more than 65535 (2 ^ 16 - 1) vertices due to the effect renderer limitations.

The gl_animation_overlay_calculator.cc question

I believe that converter is not available in this repository, sorry about that! You can understand what format is expected by reading its parsing logic: link. Then, you might implement your own converter. I believe, there's nothing we can easily share with you right now to ease your workload :/

[at-duymai]

Hello. @kostyaby
I have a question. Usually 3d softwares often export OBJ file with MTL file. How can I create a PNG file of glasses like in the demo Faceeffect?

[kostyaby]

Hey @at-duymai,

One of the reasons I decided against supporting OBJ format directly was exactly the fact that we'd have to support MTL material shading and it felt too complex for a basic example. Therefore, we do support OBJ topology format via converting into mediapipe.face_geometry.Mesh3d via the non-official converter; however, we only support the simplest color texture shading model, so there's no need for MTL file as all those diffuse / reflectivity / other parameters don't make sense for the color texture shading model.

To create an asset for our face effect renderer, you first need to model it in some 3D asset authoring tool (Blender / Maya / Cinema 4D). After the asset is modeled, you need to export its topology as OBJ and then convert it into mediapipe.face_geometry.Mesh3d using the tool; as for the asset material part, you just need to export its color texture (remember, our shading model only support that component) as PNG and hook it up into the renderer

[breathim]

@kostyaby how to load the images like facepaint.pngblob dynamically from a file storage(google drive) ?
or is there any provision to change the color of the texture dynamically which is marked in the .pngblob file.

[kostyaby]

Hey @breathim,

I believe that dynamic asset management is an orthogonal functionality which is well beyond our face effect example, as it lies in the realm of general mobile/desktop/web platform app development. At this point, MediaPipe doesn't provide any built-in solution for that and is instead strictly about running image / video processing pipelines

As long as you are able to download an asset and pass the correct path to the renderer, it'll have no problem reading / interpreting those instead of app-bundled assets.

[ashrvstv]

@kostyaby what do I need to modify in case I want to add more complex 3D models with lighting (e.g. diamond). In that case color texture won't be enough. So how can I add support for mtl files?

[kostyaby]

Hey @ashrvstv,

This is an example renderer, that showcases some basic techniques of face AR. At the current moment, we do not intend to make it as comprehensive as what you might expect from other production-quality AR renderers like Snapchat / Facebook has in their user-facing products

If you are interested in mobile platforms, I'd recommend you to take a lot at https://github.com/google/filament. It shouldn't be hard to drive the Filament renderer using mediapipe.face_geometry.FaceGeometry to create effects of a much better visual quality.

[GamerWael]

Hey @kostyaby, is there any example or documentation for using the effect renderer for face effects with the tensorflowJS model?

[konovalov-k]

@ashrvstv You must create your own file-calculator which implement filament as a renderer. I work at this, but it's a little bit complicated)

[kostyaby]

@GamerWael, there is none; I'm not even sure MediaPipe has a TensorflowJS inference calculator as TensorflowJS itself is not a native library

You could theoretically make a calculator like that only for Web platform and use Emscripten bindings to pass data between native WASM binary and TensorflowJS. At this point, I'm not sure it worth it as Tensorflow Lite inference calculator is being beautifully compiled into WASM and has Web-specific accelerations via XNNPACK

@konovalov-k, I've been looking into that a few months ago; if you are interested, I'll share my progress notes here:

• There are 2 ways Filament can be integrated with MediaPipe: 1) as a calculator and 2) as a separate mobile app component, that passes signals from MediaPipe graph into the Filament renderer
• The challenging part about completing 1) is that MediaPipe and Filament have pretty different ways to get GPU initialized; Filament needs a native surface, MediaPipe doesn't provide that explicitly.
• In the end, I was able to only make the OpenGL backend work by providing my own backend::OpenGLPlatform implementation, which is essentially no-op and thus allows Mediapipe to handle GPU initialization (PlatformWebGL is what I used as a basis for my custom platform). Other backends (Metal on iOS, Vulkan on modern Android) wasn't possible to hook up with the way Filament is currently implemented; I chatted with the framework team a bit and generally they were not super happy that I was meddling with custom GPU initialization and warned me that there might be potential future issues as they do not expect to provide support at such low-level of the API.
• TLDR on 1): it is possible to integrate Filament as calculator, but it's challenging, fragile and only gives you access to the old OpenGL ES 3.0 graphics backend.
• This brings me to the 2) solution. You set up MediaPipe graph & Filament renderer separately in your app; then, set up a mechanism for passing face tracking results from the MP graph into the renderer (if you have apps for Android + iOS, then you'd have to write this logic twice OR extract common pieces into a native C++ library of your own). I haven't tried it myself, but it sounds way easier as it mostly about plumbing code that doesn't meddle with GPU

Hopefully this was helpful!

[jianinz]

Hi @kostyaby, i followed your suggestions with steps to convert another 3D glass OBJ into mediapipe.face_geometry.Mesh3d text protobuf format, but somehow when load it in Android, the glass was partially seen (half of the glass) and it does not attach to the the eye area, that makes me wonder maybe i am missing some critical configuration to define the boundary or using landmarks to define where the glass should go to?

[kostyaby]

Hey jianinz@,

It sounds to me like your 3D glasses model wasn't aligned with the canonical face mesh 3D model (mentioned here under the Bridges static and runtime spaces point; model OBJ)

Can you please confirm, that when both the new glasses model & the canonical face model are imported into the same 3D scene, the glasses are "aligned" with the face? You can check that by, for example, importing both models into some 3D software tool (Blender, Maya etc)

[jianinz]

Hi @kostyaby, you are correct, they are not aligned, the new glass model is way too small compare to the canonical face 3D model. Thanks!

[jianinz]

Hi @kostyaby , given the example of face effect is to use GateCalculator to decide whether it's face paint or glass effect to be rendered. I wonder that is it possible to mux different face effects (glass, earrings, necklace etc) at the same time? I have tried to use multiple _FaceGeometryEffectRendererCalculator_ within each having its own effect texture and mesh_3d, in the end using _ImmediateMuxCalculator_ to mux them, unfortunately it does not work as expected. Do you know if i missed something important here? Thanks

[kostyaby]

Hey @jianinz,

Currently in the example, there are 2 effects guarded by a single is_facepaint_effect_selected flag. If it's true, then the Facepaint effect will be rendered; if it's false, then the Glasses effect will be rendered. When you switch from 2 effects to 3+ effects, this single boolean flag approach no longer works

For N effects, I'd introduce N graph input streams with boolean flags. Ex: is_facepaint_effect_selected, is_glasses_effect_selected, is_necklace_effect_selected etc. With every input frame, you must then sent all these N boolean flags into the graph and exactly one of them must be true

Alternatively, you can implement a new calculator in C++ that picks the effect based on the index - this way, you'll be able to keep the number of additional graph input streams to zero by essentially replacing the is_facepaint_effect_selected boolean flag packet with selected_effect_idx integer packet.

[jianinz]

Thanks @kostyaby for your suggestions. Will look into that!

[Nvelu048]

Hi @kostyaby i followed your below reply and generated model.pbtxt

basically i used this by converting glb to obj

Hey @at-duymai,

One of the reasons I decided against supporting OBJ format directly was exactly the fact that we'd have to support MTL material shading and it felt too complex for a basic example. Therefore, we do support OBJ topology format via converting into mediapipe.face_geometry.Mesh3d via the non-official converter; however, we only support the simplest color texture shading model, so there's no need for MTL file as all those diffuse / reflectivity / other parameters don't make sense for the color texture shading model.

To create an asset for our face effect renderer, you first need to model it in some 3D asset authoring tool (Blender / Maya / Cinema 4D). After the asset is modeled, you need to export its topology as OBJ and then convert it into mediapipe.face_geometry.Mesh3d using the tool; as for the asset material part, you just need to export its color texture (remember, our shading model only support that component) as PNG and hook it up into the renderer

Now, how to export color texture as PNG.

Or where to find color texture for 3d model

Need your guidance

[kostyaby]

Hi @Nvelu048,

Given that Glb is a format provided alongside with glTF, I'd be looking for a corresponding texture converter somewhere around tooling provided by Khronos.

For that Deep Cowboy Medical Face Mask model in particular, you can alternatively choose to download it as Original / Updated glTF from the Poly website and use the provided JPG texture as-is (our renderer should supports both PNG & JPG)

[Nvelu048]

Hi @kostyaby

as per your guidance, i did as follows:

1. Downloaded Deep Cowboy Medical Face Mask and used it's jpg texture
2. For pbtxt file, i have imported glTF file in Blender and aligned it in accordance with canonical_face_mesh and exported it as obj file
3. Then, used the provided script to generate pbtxt file.
4. Then updated the graph and started the app.
   In log i could find assets were successfully loaded.

But still effect doesn't gets rendered

I don't know where i am going wrong

Drive Link having resources i used

[Ravipuniya]

HI @kostyaby i want to export the color texture in png format using blender but enable to figure out how to get from blender. Could anyone please help me. I am able to get .mtl file from blender while exporting .obj from blender.

[kostyaby]

Hey @Ravipuniya,

Have you tried googling? Something along these lines seems relevant: link.

There's of course a possibility, that there are no underlying textures for a given material, but it's rather represented through parameters. In this case, you are looking for a way to "bake" that material into a texture. Something like this seems relevant for this case: link 1, link 2

---

Hey @Nvelu048,

I think the alignment step might be incorrect. I imported both the mask_aligned.obj and canonical_face_mesh.obj into Blender (v 2.79) and they are not aligned. Please check out the following screenshots:

Can you please confirm that importing both *.obj models into your Blender environment produces the same result? If so, then you might want to re-align the 3D model and then re-export.

[brucechou1983]

I'm also trying to figure out how to add new face paint effects to the example app.

It seems to that facepaint.pngblob is aligned with canonical_face_model_uv_visualization.png. For new face paint designs, do we need to manually perform this process? I guess the flow is like:

1. manually label some key points like eyes, mouth, nose and edge of the mask
2. scale and warp the mask to align with the canonical face uv map

Do I miss anything or is there better way to do this? Thanks.

[Nvelu048]

Hi @kostyaby

When i imported mask_aligned.obj file in blender-2.91.0

It was aligned as below in layout mode

I used canonical_face_mesh.fbx, which i got from ARCore Project.

Am i referring the wrong canonical_face_mesh. If so, Kindly share the obj format

[kostyaby]

Hey @brucechou1983,

You are right that the face paint texture should be aligned with UV coords of the canonical face mesh (visualization of which is available in canonical_face_model_uv_visualization.png).

The procedure for generating new face paint texture described by you looks fine to me. Maybe it will simplify the first step if I share this psd file from AR Core repo. AR Core runs a similar backbone for its Augmented Faces API as MediaPipe Face Geometry API; that texture in particular is perfectly aligned with our UVs

---

Hey @Nvelu048,

You are right that AR Core canonical face model is likely positioned differently. According to this, they 1) use 0.01 scale along all axis - which explains why on my screenshots the face mask is so tiny - and 2) use left-handed 3D space (the one with +Z axis instead of -Z axis) - which explains why the face mask is rotated along Y axis.

Besides that psd file that I shared in my response to @brucechou1983, please avoid mixing and matching assets between MediaPipe Face Geometry and AR Core Augmented Faces libraries as they do not necessarily match; You can find assets for the Face Geometry API here

[Nvelu048]

Hi @kostyaby
Thanks a lot. Able to render 3d effects as expected.