Mumble: Video feature discussion

[davidebeatrici]

In the roadmap for almost a decade, but with no progress on it whatsoever.

I believe it's time we work on it and hopefully make it available with 1.5.

For encoding/decoding we can use libavcodec and then allow the user to choose the preferred codec.

As for video input, I came up with 3 different solutions for the implementation:

QCamera

https://doc.qt.io/qt-5/qcamera.html

Advantages

• Cross-platform, supposedly working out of the box.

Disadvantages

• Dependency on the Qt framework. Right now it's not an issue because most of our code depends on it, but we're considering replacing most of it with modern C++ and POCO libraries while keeping Qt only for the GUI.

v4l2 & DirectShow & AV Foundation

Advantages

• Low-level control of the camera devices.

Disadvantages

• Separate code for each platform.
• Probably quite long code required.
• Hard to maintain.

OBS integration

Advantages

• OBS is cross-platform, thus we would only need to write different code to interface with it on different platforms, similarly to what we do with the Link plugin.
• OBS allows a user to manipulate the video input as desired (camera settings included).
• We don't have to maintain the video input code ourselves.

Disadvantages

• Reliance on an external program.

[Krzmbrzl]

Well I think we have other priorities right now than to integrate such a completely new feature that probably implies a huge amount of work.

That being said, to me the OBS solution sounds like the best considering we plan to make the whole program working without Qt as well.

[davidebeatrici]

Indeed.

[Avatat]

In my opinion, we should focus on fixing bugs, improving the core features and UX.
We have a lot of work, to do the best VoIP communicator in the world! :)

But, I see nothing wrong in adding a video feature to the "to-do list"!

[streaps]

I don't like the idea of having a specific external application as a dependency. This might be cool for a few use cases, but the majority of users want a simple solution. Switching video on and off easily within Mumble -- not having to use a full-blown broadcast solution with Mumble as a streaming back end.

OBS uses Qt, so you would still have no video solution without Qt dependencies.

I think it would be very useful to have a very simple video implementation. low-bandwith, low-resolution video. It doesn't have to be the next Zoom, it doesn't need desktop sharing and stuff. Only some visual feedback.

"the best VoIP communicator in the world" doesn't really sell any more. Ignoring video today will push the feature another 5 years into the future.

Just my opinion, of course it's up to the developers on what you want to focus.

[felix91gr]

I have problems with Jitsi all the time because WebRTC is still on its infancy right now.

And Zoom is a no-go for me, because of the hostility and surveillancey practices they've already shown towards users. It's also closed-source.

For all of this, I would love, LOVE something with the level of quality Mumble has that was open source. I didn't imagine it could be Mumble, but hey, if it seems possible and the audio engine won't suffer because of it... then yeah, why tf no :)

PS: if you DO go for this, ping me. I don't have much money but I can make a donation for this :)

[felix91gr]

Also, regarding this disadvantage of QCamera:

will definitely result in larger static binaries.

I don't think that's much of an issue.

People are tending towards making larger static binaries because dynamic linking is really hard to get right, and its original reasons for existing are not really that important anymore. As far as I understand, those reasons were lack of good source code package managers like npm and cargo to add dependencies to code, and lack of enough disk space to put all of the big binaries in place.

I don't think that now, in 2020, large static binaries are such a big deal.

If this is the alternative you have to follow to make life easier on the maintainers, then... why not? I do think you guys deserve a break.

[davidebeatrici]

Thank you very much for the feedbacks!

@streaps We could implement video input through OBS first (less work required) and then add our own. That would allow a user to select either OBS or the direct webcam as input.

@felix91gr We only provide static client binaries for Windows and macOS (which don't provide a package manager out of the box). In future we will probably provide an AppImage file for each release.

Right now I don't know by how much our binaries' size would increase by adding Qt Multimedia, but it's definitely going to stay below 100 MB.

I removed the disadvantage from the list.

[streaps]

Requires a plugin that may not be present to be shipped in official distribution packages; for example, the Debian one doesn't seem to provide it.

I wonder which plugin that is?

[davidebeatrici]

There's at least a plugin for each platform, the one for Linux provides GStreamer integration.

I also just found out the plugin is provided by https://packages.debian.org/stable/libqt5multimedia5-plugins.

I will test https://doc.qt.io/qt-5/qtmultimedia-multimediawidgets-camera-example.html and report back.

[streaps]

What about libVLC? I have no experience with it and don't know if it can be used for this purpose.

[davidebeatrici]

Looks nice, but it's strictly focused on media player capabilities rather than encoding/decoding.

I also just found out the plugin is provided by https://packages.debian.org/stable/libqt5multimedia5-plugins.

I will test https://doc.qt.io/qt-5/qtmultimedia-multimediawidgets-camera-example.html and report back.

Turns out I already have that package installed.

It was my fault, I ran the binary from file manager forgetting about firejail.

Running the example in a non-restricted environment works fine, I updated the first message.

[ghost]

In regard to OBS integration method — virtual cam plugin exists https://obsproject.com/forum/resources/obs-virtualcam.949) for Windows and Linux.

[trudnorx]

I'd rather not have video in Mumble. Simplicity is a virtue. Programs being too bloated is a big problem.

I disagree with the idea that it could be implemented in a non-obtrusive way.

[davidebeatrici]

@Reikion I was aware of it, however a direct integration would be more efficient (Mumble <-> OBS rather than Mumble <-> V4L2/DirectShow <-> OBS).

@trudnorx What makes you think that video cannot be implemented in a non-obtrusive way?

[TerryGeng]

Hmmmm....

What about openCV?
https://stackoverflow.com/questions/2570359/cross-platform-camera-api/3909731
https://stackoverflow.com/questions/278112/webcam-library-for-c-on-linux
I know there're some people use openCV for camera capture with python.

[hhirtz]

To solve the scaling issue[1], mumble could merge incoming video streams into one, like so: https://external-content.duckduckgo.com/iu/?u=https%3A%2F%2Fi.ytimg.com%2Fvi%2FVnyitUU4DUY%2Fhqdefault.jpg&f=1&nofb=1

If the room gets bigger, it could show in big the one who's talking, and have little version of others' video streams: https://enterprise.verizon.com/content/dam/img/products/business-communications/zoom-wii-sm.png

This way tx bandwidth is reduced from O(n^2) to O(n)

[1] https://wiki.mumble.info/wiki/Planned_Features#Video

[streaps]

What about openCV?

it's huge, but very powerful

What about Gstreamer?

[streaps]

This way tx bandwidth is reduced from O(n^2) to O(n)

How would that work?

[hhirtz]

How would that work?

Instead of sending all video streams to each client, the server merges them into one and then send it to all clients. The way it is merged could be a setting, or could depend on the number of clients in the room.

For example if you have less than four people in the video conference, mumble could make a split-screen-like merge. If there is more people, it would be better to have the one talking in big and the others on the side of the video, like you can see in the images from my previous comment.

[streaps]

But how would that work?

Re-encoding everything? That wouldn't scale.

[mirh]

Look, this is your program and all.
But there is already jitsi for everyone's favorite open source meeting. There's no point in duplicating features and market positions (also somewhat forfeiting the core demographics of mumble, which is gamers AFAICT)
If you want to expand yourself, you should target the discord community. And I don't really think videos are important (or even desirable) there.

[felix91gr]

But there is already jitsi for everyone's favorite open source meeting.

Eh, I dunno about that. I've used jitsi extensively these past months and lemme tell you, there are some parts of it that really really suck.

In fact jitsi has broken our server's port 443 and so far nobody in StackOverflow seems to know what the heck is going on 😑

They also have lipsync problems that they've stopped testing for since 2-3 years ago. The other day, a friend and I managed to separate the video from the audio for about 80 seconds. Seconds!! My _audio_ was _behind_ the video by about 80 seconds!

Anywho. That's my rant about jitsi. It kinda sucks to use it atm, and it sucks doubly because it's more or less our only (for >2 people) open source alternative.

[grravity]

But there is already jitsi for everyone's favorite open source meeting.

I actually came here to suggest, maybe implementing a plugin for Jitsi to be used in the backend on Mumble.
Now I say plugin, because I think it should be optional, both client and server side whether they want to even incorporate video chat. I do see some controversy over it here though, and I know WebRTC can be a privacy concern depending on it's implementation.

Personally I am huge +1 for adding in video chat, given it's stable, does not have any insane compression like Discord, (lossless would be amazing for connections that can support it), is simple to use and integrated well and is fully encrypted.

I think the best way to do this would be, by default include it in Mumble client-side, and offer the ability to uninstall it easily without breaking anything. I am generally for adding onto programs instead of removing features when it comes to minimalism, but I think in this case it makes it easier for less technical people who generally don't read installers carefully or who may miss a checkbox, or whatever the case may be.. to have video chat "just werk" instead of having to worry about going into settings, finding the option to enable.

As for server side, I believe it should be highly optional whether it wants to be used.
When client side, the icons to video chat should not even show up if the server the user is connected to doesn't support video chat (and it should also represent this under the server's information).

[streaps]

Just use plain jitsi meet then. Create a link and post the URL in the text chat. I don't see any advantages in integrating it into Mumble.

[grravity]

Just use plain jitsi meet then. Create a link and post the URL in the text chat. I don't see any advantages in integrating it into Mumble.

It would be very nice to have a seamless integration, which is why I recommended a plugin for Jitsi for a click of a button. I feel there may be some complications to this. I do also agree with the above that Jitsi needs some work still, mainly client side stability.

I would be totally for a more minimal introduction of video chat directly through Mumble if possible though. Maybe something like FFMPEG for video capture? Not entirely sure, just throwing stuff out there.

[grravity]

I want to add onto that, even if it wasn't Jitsi being used on the backend, having our own would be great. Just something.
If there's one complaint I hear trying to switch people over to Mumble, it's the lack of screen-share capability. I can tell them to download Jitsi but it would require them installing and setting up another program, and that doesn't sound as good as a seamless all in one solution built right into Mumble.

[5trongthany]

100% this being added and would happily help in any way I can to achieve it.

[Green-Sky]

Just want to throw in my personal take on the matter:
Video steams are becoming a, kind of, hard requirement for a lot of people and to stay competitive, we need this feature sooner or later. But we should also not discard a feature I see as Mumble's strong point: Lightwight to host. Mixing, resizing and/or re-encoding on the server are kind of the opposite...
I see a problem though: as unlike opus, there is no de-facto video codec for low bandwith and hardware acceleration, which i see necessary for video. So the protocol should be designed extensible. Also decoupling video from the client and a plugin-like system would be a great fit, i think...

[davidebeatrici]

Rendering definitely has to be done client-side, especially due to VPS usually not providing hardware encoders.

By supporting multiple codecs we would also guarantee flexibility when it comes to the encoders available on the system.

For example: H.265 and H.264 are probably less efficient than AV1, but hardware encoders for them are widespread.

If the CPU is strong, using the most efficient codec is desirable.
If saving CPU is a priority, using one of the codecs supported by the hardware encoder is the way to go.

[grravity]

Ffmpeg does recording I believe, is minimal, and is very extensible. Not sure how it would work in a situation like such, but it sounds like it might work well to me.