Face-api.js: FaceLandmarks68: left and right eyes/brows are reversed.

[a-bronx]

Hello,

FaceLandmarks68.getLeftEye() returns points corresponding to the right eye, and FaceLandmarks68.getRightEye() returns points corresponding to the left eye.
Same issue for the FaceLandmarks68.getLeftEyeBrow() and FaceLandmarks68.getRightEyeBrow().

[justadudewhohacks]

Hi, thanks for reporting. I would highly appreciate PRs for fixing this. :)

[JefferyHus]

@a-bronx would you like to share your case so we can help?

[a-bronx]

Here is a quick test that every point of the right eye/brow has a correct horizontal position relative to the left eye/brow, given that "right" is closer to the origin than "left":

const result = await faceLandmark68Net.detectLandmarks(imgElRect) as FaceLandmarks68
expect(
    Math.max(...result .getRightEyeBrow().map(_=>_.x)) < 
    Math.min(...result .getLeftEyeBrow().map(_=>_.x))
).toBe(true);
expect(
    Math.max(...result .getRightEye().map(_=>_.x)) < 
    Math.min(...result .getLeftEye().map(_=>_.x))
).toBe(true);

To fix it just rename FaceLandmarks68.getLeftXXX() to FaceLandmarks68.getRightXXX() and vice versa, it is just a labeling issue, all geometry is correct.
NOTE: chances a low, but it still may be a breaking change for someone who do calculations based on landmark points.

Sending you PR, but I cannot build the library and run unit tests on my Windows box without Python (and I have no interest to install it), so it will be a "blind" fix and it is up to you to verify and probably correct it.

[JefferyHus]

That was a reason I asked to share your case because we can't swap names for one single case. There are others that were successful getting the right pointer results.

[a-bronx]

My case is a face liveness check based on certain statistic of relative movements of landmark points in a video stream. Before making the check, I validate the face geometry to ensure it is not distorted due to fast movement, glasses etc (which sometime happens), so these garbled "Picasso-like" frames do not affect the statistics. As part of this sanity check I verify that the right eye is correctly positioned relative to the left eye. The assumption is that in a normal (non-mirrored) photo/video frame the subjects' right eye will be closer to the left border of the frame than the left one. But the FaceLandmarks68 does not support this assumption, my code was not working and I had to debug to find out that method names are misleading.

[JefferyHus]

So I think based on your example I see your solution is based on a non-mirrored image. Maybe to have an option or parameter passed will solve problem. Will try, test and let you know.

[a-bronx]

By default source images and source video streams come non-mirrored, so it should be a baseline. If I need to flip photo/video, I use CSS on the whole canvas (e.g. transform: scaleX(-1);) instead of processing the source frame, which is much cheaper, especially for a web camera video stream.

But here is a catch: if I mirror the video in a browser using CSS , then I cannot use the drawDetection() anymore because score text under the bounded box becomes mirrored too and not readable. It is a non-issue for me as I draw a custom face boundary anyway, so just letting you know :) But if you decide to make drawDetection() support drawing over a mirrored representation and flip text labels accordingly, then you need a parameter here.

[BhavsarVipul]

Yes, Flip of the stream (mirroring) is something important (must have) when doing face recognition using a webcam. Hope we can get its support soon.

[herbzhao]

By default source images and source video streams come non-mirrored, so it should be a baseline. If I need to flip photo/video, I use CSS on the whole canvas (e.g. transform: scaleX(-1);) instead of processing the source frame, which is much cheaper, especially for a web camera video stream.

But here is a catch: if I mirror the video in a browser using CSS , then I cannot use the drawDetection() anymore because score text under the bounded box becomes mirrored too and not readable. It is a non-issue for me as I draw a custom face boundary anyway, so just letting you know :) But if you decide to make drawDetection() support drawing over a mirrored representation and flip text labels accordingly, then you need a parameter here.

After playing around a bit, I found a work around. First you need to create a second canvas that is not flipped. And then you need to use the customised draw function: drawBox.draw and drawText.draw. You need to find the correct X coordinate, which is canvas_new.width - box.x - box.width

[AyazRahman]

Yes, Flip of the stream (mirroring) is something important (must have) when doing face recognition using a webcam. Hope we can get its support soon.

Just wondering if this option was ever added to the api