Tfjs: Google Meet background segmentation model

[jameshfisher]

System information

• TensorFlow.js version (you are using): 2
• Are you willing to contribute it (Yes/No): No, it's not mine

Describe the feature and the current behavior/state.
This Google AI blog post describes the background segmentation model used in Google Meet. This model would be an excellent complement to the models in the tfjs-models collection. (The existing BodyPix model can be (ab)used for background segmentation, but has quality and performance issues for this use-case. I expect the Google Meet model improves on this.)

Will this change the current api? How?
No, it would be an addition to tfjs-models.

Who will benefit with this feature?
Apps consuming and/or displaying a user-facing camera feed. WebRTC video chat apps are the most obvious, where background blur/replacement is becoming expected. I also expect it could be a useful preprocessing step before applying e.g. PoseNet. It can also be used creatively on images as a pre-processing step -- for example, this recent app to enhance profile pictures integrates a background segmentation solution.

[rthadur]

cc @annxingyuan @tafsiri

[simon-lanf]

this would be useful for us.

[tafsiri]

I'll pass this on to our PM.

[jameshfisher]

Note: I'd also be happy if just the raw model (https://meet.google.com/_/rtcvidproc/release/336842817/segm_lite_v509.tflite) was released under a permissive license - I can figure out the model structure and JavaScript wiring :-)

[jasonmayes]

+1 to this! Would love to see this as part of the model repos for TFJS - a lot of people making Chrome Extensions to do great things in video calls etc and this would just make those experiences even more efficient when running to get higher FPS etc.

[RemarkableGuy]

+1 to this, would be a great, faster alternative to body-pix, really impressed by the performance in Google Meet :)