Mediapipe: How to reduce the jittering of face landmarks?

[Mad-Thanos]

Hi Mediapipe cool guys,
I read the blog about the Face Mesh feature here:https://ai.googleblog.com/2019/03/real-time-ar-self-expression-with.html?m=1

The realtime mesh is so attractive to me, it's incredibly fast and dense. But when I tried the Mediapipe Face Mesh prebuilt demo apk(downloaded from this page:https://google.github.io/mediapipe/solutions/face_mesh ).
I found that everything is good but the dense landmarks are jittering(i.e. shaking) even when I keep the phone and my head static.
Is there any option or method to reduce or eliminate the jittering? What could be the cause of the landmark jittering?

2 Android phones were tested for this demo apk, one is powered by Snapdragon 845 and another is Kirin 980.

Thanks!

[fanzhanggoogle]

Hi, we do expect some jittering as the model released in MediaPipe is extremely light (lighter than the model announced in the blogpost). When deploying the model in production, we apply a low pass filter to get temporally stable prediction . You could give it a try.

[Mad-Thanos]

hi, @fanzhanggoogle Thank you!

Is the high definition model publicly available?

I'm not sure whether it is enough to use a simple filter, such as a sliding window average filter, is it enough to reduce the mesh point noise by averaging the point coordinates on the most recent 3 or more meshes(in a sliding window) ?

[Mad-Thanos]

I will try the sliding window filter first, and then some more complex filter if possible.

[mgyong]

@Mad-Thanos The high definition model is not publicly available and we do not have plans to release.
We plan to provide a new update of the Face mesh model in July/Aug if you have a simple filter solution. Please contribute back to the community. Thanks

[timkabot]

@Mad-Thanos Did you succeed?

[hananbeer]

I tried running lerp() (linear interpolation) for 10 time steps between each model prediction. not perfect, so I also tried updating a keypoint only if the distance of the new keypoint is above certain threshold. not sure which method is better.

the problem is I only work in 2D points, I completely ignore the Z position (depth), but the depth seem to be the most jittery.
sadly even still images provide jittery output.

[Mad-Thanos]

@Mad-Thanos Did you succeed?

Hi @timkabot and @hananbeer , I didn't work on this topic anymore. But I think google mediapipe team had already implied some clues about this topic in the original paper(pls googling mediapipe facemesh paper). They said they used the one-euro filter to smooth out the jittering of each coordinate(x,y,z) of the 3D points. There are already some one-euro filter implementations on github, for example:https://github.com/jaantollander/OneEuroFilter

Looking forward to you guys success.