Twilio-video.js: Guidance requested: How to have a large (10+ video & audio track) conference without burning too much CPU?

[randallb]

I think it's generally known that when you have a large multiparty group call, it's gonna get CPU intensive, especially with codecs like VP9 on the horizon that consume a lot of CPU.

Google Hangouts skirts this issue by sending a "thumbnail" view of the user consistently, and then (probably) using a selective forwarding unit to send whomever is talking. I'm assuming with VP8 Simulcast (noted in the 1.7 changelog) this might be happening for bandwidth or other reasons on the group room server side.

But what about the client side? I've done some initial experiments, and I note that pausing the video only saves a tiny bit of CPU. We've implemented our own client-side "I'm talking here!" notices to our server, which we could use on other clients to react to people who are talking or people who stop talking.

I'm wondering if anyone else has done any experiments with detaching video, but not audio, and what the lag is like in reattaching video? Would that be a significant CPU savings? Does anyone have any experience with trying to solve this problem? Group rooms seem like they can support a large number of participants from a bandwidth and processing perspective, but unfortunately clients are the bottleneck and I'm just starting to explore this.

My proposed topology is going to be to have clients join 3 different group rooms, mostly for quality reasons. I'm going to have them join a VP8 simulcast room if they're on a VP8 compatible browser. They'll also join an H.264 room for iOS users to be able to view them, but we'll probably bandwidth restrict this room heavily. We'll then also try to do a thumbnail group room where the video is extremely low bitrate, and potentially with mediastreams restricted down to low low low resolutions, just for thumbnails in our app.

I'm hoping that our "main view" can use HD quality video from the "main" room for everyone but iOS exclusive users. And our "main view" room will only have one video attached at a time. That way we can be economical for our users' CPU and still reproduce the same thumbnail-for-all main-view-for-one view we have today.

Any advice anyone has in implementing a room with up to 50-100 participants would be much appreciated.

I'll note my progress in this github issue as I'm able to attack this problem directly, likely in December.

[abhijeet1403]

According to twilio docs, i guess the maximum participants can be 50. However, On implementing I found that after 5 participants connect to the room, chrome tab memory usage spikes up and becomes unresponsive, and eventually crashes. Going through the webRTC docs, browsers should automatically handle the memory and CPU resources, but it doesn't seem to be working for me.
I'm still trying to find solutions for adding more people to the conference. For now, I think I'll use your "I'm talking here!" solution. Let us know if you successfully scale.

[diydan]

It looks like you can set bitrate on the fly:
room.localParticipant.setParameters({
maxAudioBitrate: 32000,
maxVideoBitrate: 250000
})

Can this be done for framerate? I'm thinking along a similar line, by having passive participants at a low framerate, then increase for the "active talking" participants.

[manjeshbhargav]

Hi @randallb , @abhijeet1403 , @diydan ,

You can now use the Network Bandwidth Profile API in twilio-video.[email protected] (Group Rooms only) in order to determine which VideoTracks are more important, thereby allocating more bandwidth to them.

Thanks,

Manjesh Malavalli
JSDK Team