Face-api.js: Suggestions on emotion detection (I'm guessing using landmarks?)

[salamanders]

Google Vision API has a nice "joyLikelihood" Has anyone done the same with landmarks, or is it an entirely separate model that usually works on expressions?

I'm trying to "find the best smile" out of a series of images, didn't want to reinvent wheels.

[justadudewhohacks]

I am currently training a model for facial expression/emotion recognition, landmarks are not sufficient for that task maybe they are sufficient for simply detecting smiling gestures.

[salamanders]

Yay! Hope it trains well! Really rough "this is an open source project and nothing is guaranteed because everyone is doing it out of the kindness of their hearts" best-case ETA?

Tangent: I wonder if landmarks + the face descriptor would result in anything good. I imagine the face descriptor to be a "baseline" of the face's underlying structure, then the landmarks are what the face is currently doing, which kinda-sorta-maybe represents expression. And orientation, so that is tough. So a landmarks.orientationNormalized() - faceDescriptor = expressionData sorta thing.

[justadudewhohacks]

this is an open source project and nothing is guaranteed because everyone is doing it out of the kindness of their hearts

And out of passion :)

Still trying out different architectures and improving the dataset. The models I trained so far didn't achieve more than 70 - 80% accuracy on the test set so I am still trying. Seems it's not an easy task to distinguish between certain expressions such as fear and surprise or sadness and anger. But atleast the accuracy for happiness/smiling usually is about 95%.

best-case ETA

I hope by the end of next week I will have published something or atleast pushed some demos somewhere to play around with.

I imagine the face descriptor to be a "baseline" of the face's underlying structure

Hmm, that's a vague statement, I am not sure whether you can draw reasonable conclusions about facial expressions from the face descriptor. The face descriptor is simply a feature vector and the net that produces those has been trained to output descriptors such, that they have minimal distance to descriptors of faces of the same subjects and maximal distance to faces of different subjects.

But I could imagine that a combination of the probability for happiness from the upcoming face expression predictor + the landmark positions of the mouth region would do the trick for your usecase.