Dlib: Snapchat-like effects

[cfoch]

Hello. I am a member of the GNOME Foundation. I wonder if dlib is suitable for implementing Snapchat-like effects. I would like to implement these kind of effects in Cheese. I have tried dlib's landmark detection and sometimes it fails to detect the landmark for example when I rotate my face (in profile, for example). I want multiple face tracking, I think that Snapchat allows to detect a single face only. So to reorder faces I am using the Hungarian Algorithm. That's what I have done.

Now, I would like that this could be more accurate, because when the landmark detector fails it would give a blinking effect.

Kalman Filter
I would like this detector could be the more "accurate" possible to almost avoid that effect of blinking. I was thinking that I could use the Kalman Filter for that, but I have tried to implemented and I think I didn't get look or that maybe I just need to adjust some parameters. But searching a little bit on GitHub, I found a project that already uses the Kalman Filter and after modifying it (because there were some errors to compile it) I found out that the Kalman Filter actually follows the face very slow, if I move my face faster the behavior of the Kalman Filter is worst. I am not sure if this would be a problem of the Kalman Filter or just some parameters I need to adjust again. I would like you can answer this if possible.

Other tackers
I was watching on YouTube how OpenCV's trackers behave and I have the doubt now if actually what I need to avoid that blinking effect of the landmarks when the detector fails should be solved with a tracker.

To avoid that blinking effect do you think it is just about training the shape predictor? Or do I need a more sophisticated method like tracking or whatever else? What else should I look at?

Thank you so much in advance for your answer.

[davisking]

There are a lot of solutions to the problem you are asking about. A properly configured Kalman filter can help, or retraining whatever part of the model is causing the "blinking". I'm not sure what you mean by blinking. If you mean the detections just vanish that's because the face detector isn't finding a face, which has nothing to do with the landmark model.

[ubicgothg]

I don't really fully understand the question but maybe take a look at this project https://github.com/ageitgey/face_recognition. You can detect several faces and add "ugly" makeup.

[cfoch]

@davisking what I called "blinking" is because the face detector is not finding a face. But this is just a matter of training it? Or there are some additional methods you recommend I should use?

[davisking]

You could train a new face detector to fix it, yes. There are a lot of
things you can do to make this kind of thing smoother. You could train a
shape predictor with a wider range of face bounding boxes and use the shape
predictor to find the location of the face when the detector fails. You
could do this by guessing where the face is based on its location in the
previous frame and then run the shape predictor on that location. I would
retrain the shape predictor to make it more robust at dealing with that
kind of "guessed box" though.

[pandacriss]

i think you could use haar cascades to find faces and landmarks for find fixd points on the face
i found this series of tutorial really interesting https://www.pyimagesearch.com/2017/10/23/raspberry-pi-facial-landmarks-drowsiness-detection-with-opencv-and-dlib/
i have run it on raspberry pi with a few optimizations, its really efficient, on my laptop it dont even over heat my cpu.
you could set a min and max size for the face detector so it use a lot less cpu

[abhiTronix]

@cfoch checkout working real-time working prototype here: Live! Raspberry Pi OpenCV & Dlib Multi-Snapchat Filters Python Implementation [Robust and Fastest] on Raspberry Pi 3 with _Dlib_ and _OpenCV_. ;)

[cfoch]

I was working on this as part of my university dissertation project which I successfully presented in my university . I successfully integrated this con Cheese. It works for multiple faces by using detection every N frames and tracking in between. If someone wants to help, it would be really nice. I will not have too much time to do it for the next months because I will be somewhat busy. But it would be really nice to have the project officially in Cheese.

I haven't had too much time to work to publish it officially or polish my code. I need to do some improvements like loading image files asynchronously, I also was thinking on using a cache system (say, keep at most 32 MB of images in RAM and if more images are needed do LRU). Also because most of the time my implementation tracks faces, the bounding box is different to the one that dlib HOG detector, sometimes the landmark are not detected properly. Probably it would be necessary to train against the bounding boxes generated by the tracker.

source code of GStreamer plugin: https://github.com/cfoch/gst-plugins-cheese
preview in Cheese: https://www.youtube.com/watch?v=RAszwErHQ4Y

[dlib-issue-bot]

Warning: this issue has been inactive for 20 days and will be automatically closed on 2018-09-21 if there is no further activity.

If you are waiting for a response but haven't received one it's likely your question is somehow inappropriate. E.g. you didn't follow the issue submission instructions, or your question is easily answerable by reading the FAQ, dlib's documentation, or a Google search.

[dlib-issue-bot]

Warning: this issue has been inactive for 27 days and will be automatically closed on 2018-09-21 if there is no further activity.

If you are waiting for a response but haven't received one it's likely your question is somehow inappropriate. E.g. you didn't follow the issue submission instructions, or your question is easily answerable by reading the FAQ, dlib's documentation, or a Google search.

[dlib-issue-bot]

Warning: this issue has been inactive for 35 days and will be automatically closed on 2018-09-21 if there is no further activity.

If you are waiting for a response but haven't received one it's likely your question is somehow inappropriate. E.g. you didn't follow the issue submission instructions, or your question is easily answerable by reading the FAQ, dlib's documentation, or a Google search.

[dlib-issue-bot]

Notice: this issue has been closed because it has been inactive for 37 days. You may reopen this issue if it has been closed in error.