Twilio-video.js: [question] Possible to blur background (like in Teams)?

[aastlind]

Hello!

We're using Twilio SDKs for all platforms in our product and we want to thank you for a product that's very easy to implement!

This isn't an issue, but a question/feature enhancement request. Please remove if this is the wrong forum for this.

Since the use of video conferencing has grown a lot lately, some of our users has discovered features in other video applications. One of them is the "blur background" feature in Microsoft Teams. Is this something you have discussed to ship with your SDKs? If not, is it possible to achieve this using twilio-video.js? My guess is that you need some face tracking data to be able to know where to add blur.

[rkiller]

Hi @aastlind :

We've successfully used Chromacam https://www.chromacam.me/ for this.

Hope this helps.

[manjeshbhargav]

Hi @aastlind ,

Thanks for reaching out with your request. What you are looking for would require some image processing logic to run on every video frame to determine what part of the video is the background, which is most likely very expensive in terms of CPU especially on browsers. You can use the <canvas> element to grab video frames and apply your background detection and blurring logic, but as I said before that would be very expensive.

I'll close this issue for now. Please feel free to continue the discussion here if you need any more clarification.

Thanks,

Manjesh Malavalli
JSDK Team

[flfw]

Hi @manjeshbhargav do you ever see this as being part of the Twilio video roadmap? This is a common request from customers on our platform and suggesting Chromacam is not ideal. Over the last year most customers have come to expect video background blurring as a core feature of video platforms. Thank you!

[acro5piano]

I'm looking forward to the blur background feature!

[manjeshbhargav]

Hi @flfw , @acro5piano ,

It is under consideration, but I cannot say anything more at this point. However, the latest SDK version 2.13.1 has the Video Processor APIs, which allows you to add a processor to a LocalVideoTrack. You can use that in conjunction with libraries like bodyPix to achieve background blurring.

Thanks,

Manjesh

[acro5piano]

Thanks @manjeshbhargav

[RaptorMai]

Hi @manjeshbhargav

I noticed that there there is a Twilio video processor package posted two months ago in the link attached below.
https://www.npmjs.com/package/@twilio/video-processors

The website lists "Virtual background" and "Background blur" as features but there is no document and example.
I wonder if this is on Twilio's roadmap and will be released officially soon?

Thank you so much

[acro5piano]

Hi there,

You can use the element to grab video frames and apply your background detection and blurring logic, but as I said before that would be very expensive.

I've created a gist to implement the blur background, using canvas and TensorFlow.

https://gist.github.com/acro5piano/0972a180348643031dd54a2d29c59301

Not tested much, but it's works well on my PC.

Note that it is client-side effect, so stream data sent to Twilio is not blur. This has impact on recording. Please anyone improve it.

[acro5piano]

Related: https://github.com/twilio/twilio-video.js/issues/1413