Mediapipe: Face Filters using Mediapipe Face Mesh

[Gauraviiitian]

Currently I'm trying to implement a Facial filter (snapchat like) using mediapipe facemesh. I thought of doing it in two steps -

1. Get the coordinates of concerned landmark for the specific filter (e.g. to Augment a crown we'll need forehead's landmarks).
2. Write a calculator to superimpose the filter using the coordinates from previous step.

Now I could see the coordinates of landmarks, facemesh example's subgraph FaceRendererGpu and trying to modify it for superimposing the filter, But it's getting quite hard to implement.

Is there a better way of implementing these kinds of filters? How can I divide the second step in terms of coding, since I'm not expert in C++?

[mgyong]

We will be releasing in end Sept a 3D face system to Face Mesh that will help with (2). Pls just wait for that

[Gauraviiitian]

Thanks @mgyong I'll be looking forward to that.

[mgyong]

MediaPipe Face Transform https://github.com/google/mediapipe/releases/tag/v0.7.11

[rakoo04]

Great release indeed @mgyong . But I want to make a similar solution for web. is there any plans to release for this?

Also, how do I add custom 3d models into this. As I can see here, 3d file format is binarypb and .uuu elsewhere.
effect_mesh_3d_path: "mediapipe/graphs/face_effect/data/glasses.binarypb

How can I add more generic formats here like glb/obj etc. Or is there any way to convert files into the supported formats?

[Gauraviiitian]

Nice one @mgyong , I'm right on it, although I thought I got pretty much control over the keypoint's coordinates. I'll try it and hopefully get it done. Thanks a lot!

[Gauraviiitian]

Hey @mgyong the Face Effect is rendered for a single face, Is there a quick modification to make it work for multiple faces?

[MyChocer]

@Gauraviiitian Do you find some solutions for rendering multiple faces in the Face Effect application?

[Gauraviiitian]

I didn't use their face effect application for my use case, you can take face mesh for multiple faces and then implement your calculator for rendering face effects... I did the same.

[MyChocer]

@Gauraviiitian Would you mind sharing the graph and calculator? I would like to do a quick test on it.

[Gauraviiitian]

Hii @MyChocer, I cannot share the calculator due to confidentiality policy with my company since the project is affiliated for them. Although I can share some key notes for the same -
Break down the project into two parts -

1. Finding the landmark points (for face detection or facemesh) ie the absolute coordiantes.
2. Now rendering any image or filter with the help of landmarks detected.

First one is quite simple using calculations since we already have normalized landmarks, you just need to know which landmark index corresponds to which point on facemesh or face landmark (for this you can google search).
Second one is a bit tricky as we need to render the filter using landmark points and there are two ways of doing this -

1. Using OpenGL which uses GPU buffer Image form, I didn't use it since it was quite complex.
2. Using OpenCV which uses CPU Image form, I used this one effectively for my project.

To use opencv first convert input stream to CPU using GpuBufferToImageFrameCalculator and then you can easily load and use opencv for rendering and performing any kind of filter.
you can also traverse through all of the pixels and set values as per your filter (the naive method).

for more assistance around calculators and graphs you can search for @swatimodi article around mediapipe.

Hope this helps! I'm still to learn a lot things about mediapipe framework.