Jitsi-meet: Virtual backgrounds using BodyPix

[fpesari]

Is your feature request related to a problem you are facing?
In professional videoconferences, virtual backgrounds are often used (eg. interviewing someone from NY with a Central Park background).

Describe the solution you'd like
This article shows how to use the BodyPix TensorFlow model (already used for background blurring, see #4166) to show virtual backgrounds.

Describe alternatives you've considered
Chroma key is the obvious alternative but it's not very practical.

[ghost]

Backgrounds would be great

[jallamsetty1]

This is definitely a nice to have feature, PRs are most welcome :)

[jilljenn]

:+1: because I want this: http://www.ghibli.jp/info/013251/

[Woodworker-Life]

+1 iLike!

[nuzzar]

This would be great. I ended up using zoom because Jitsi doesn't have this feature.

[iongraphix]

Here is a proof of concept. Source code available
http://bodypix.easyjitsi.com/

[JeffCarpenter]

cf. https://github.com/allo-/virtual_webcam_background

[shishir1979]

Hi @jallamsetty1

By any chance, Is this feature on priority list of product team ?

Is the main issue that the final video can't be 30 fps if using bodypix and canvas ?

Trying to understand this so we can plan our own product better as we use Jitsi as the foundation for conferencing.

Thanks & Cheers,
Shishir

[gabrc52]

Well, this is an open source project. Anyone with the time and skill can submit a pull request with the feature, so it doesn't have to be prioritized by the "product team".

[shishir1979]

Agree @gabrc52

Actually we did try. It just turns out that the video created is around 12 fps , however video transmitted to other participants 4 fps. This makes this bodypix + webrtc just a toy idea.

It seems the challenge with adding virtual background in video conferencing over webrtc is so tough that even Zoom and Webex can't do it. They are presently trying to use WASM and SIMD enfacements to solve this problem for WebRTC, but that is experimental.

So I was trying to understand the complexity/main blocker and any approach that may be being considered

[gabrc52]

Yeah, Zoom at least for its desktop client requires really powerful hardware, but both Microsoft Teams and Google Meet have it implemented (not sure what technology all these 3 use under the hood)

[jallamsetty1]

BodyPix is very cpu intensive and gives us very low FPS even on high end MacBooks.
We haven’t been able to prioritize this because of the amount of work this needs but please feel free to give this a try -
https://ai.googleblog.com/2020/10/background-features-in-google-meet.html

[shishir1979]

Thanks @jallamsetty1 for sharing that blog

That’s very insightful !

[blaueente]

The linked "model card" on https://drive.google.com/file/d/1WvHxUONoATFJ9JpSgAF7dU_F5V58HItS/view says
"LICENSED UNDER Apache License, Version 2.0"
However, I couldn't find any source or download location.

[kirawi]

^
https://www.tensorflow.org/lite/models/segmentation/overview
Probably not the actual model, but maybe related?