Tfjs: Feature request to include face rotation in facemesh

[n0noob]

To get help from the community, we encourage using Stack Overflow and the tensorflow.js tag.

TensorFlow.js version

NA

Browser version

NA

Describe the problem or feature request

Currently facemesh detection results include only 468 landmark points. Wouldn't it be better to include face rotation as well in the detection results ?

Code to reproduce the bug / link to feature request

If you would like to get help from the community, we encourage using Stack Overflow and the tensorflow.js tag.

GitHub issues for this repository are tracked in the tfjs union repository.

Please file your issue there, following the guidance in that issue template.

[dikatok]

@n0noob you mean like tilting your head some degree to the left/right? If yes, you can calculate it by yourself. Something like this.

const predictions = await model.estimateFaces(video);

if (predictions.length > 0) {
  const { annotations } = predictions[0];

  const [topX, topY] = annotations['midwayBetweenEyes'][0];

  const [rightX, rightY] = annotations['rightCheek'][0];
  const [leftX, leftY] = annotations['leftCheek'][0];
  const bottomX = (rightX + leftX) / 2;
  const bottomY = (rightY + leftY) / 2;

  const degree = Math.atan((topY - bottomY) / (topX - bottomX));

  console.log(degree);
}

Disclaimer: I didn't test the code above but you can fiddle around with which annotations to use and more precise way to calculate the degree of rotation.

[DamascenoRafael]

I think what he meant was to include the pitch, yaw and roll angles - Aircraft_principal_axes.

It would be an interesting addition and would bring more possibilities for using the model.