Monogame: VideoPlayer Support Refactor

[tomspilman]

So our video support on MG has always been spotty. It works fully on a few platforms, some it works partially, some not at all. We've only gotten away with this because very few games need video playback to begin with.

It seems like the idea of using platform specific video APIs is not working out. Most platforms don't have a good solution that has all the features we need.

I think we need to find a native video decompression library then use it across all our platforms. This gives us common format support on all platforms and reliable behavior.

This library needs to be:

• Cross platform C/C++.
• Support 4K video.
• Support a well known video format.
• Also decompress audio.
• Not be GPL.

Any suggestions?

Library | Team | License | Codec
--------|-------|----------|--------
libvpx | WebM | BSD-ish | VP8,VP9

[monsieurmax]

Not sure that it's not a dumb suggestion, but what about libvlc ?
You wrote "not be GPL" and libvlc is LGPL AFAIK.

https://www.videolan.org/press/lgpl-libvlc.html

[tomspilman]

@monsieurmax

Not sure that it's not a dumb suggestion, but what about libvlc ?

Well libvlc is a wrapper around a bunch of other libs. So seems like we should maybe aim at something lower level. We likely don't need a player that supports dozens of formats. It is reasonable for our player to just support one or two specific formats.

[mrhelmut]

LGPL is fine, it's only about the base GPL license, because it is not compatible with MonoGame's Ms-PL license.

[harry-cpp]

https://github.com/cisco/openh264

[Jjagg]

There's also a newer spec, AV1.

• Reference codec: aom
• Fast decoder: dav1d
• Fast encoder: SVT-AV1 or rav1e (SVT-AV1 is generally faster I think)

There's good arguments to use it:

• There's a bunch of tech giants supporting the codec. All of Microsoft, Google, Facebook, Amazon, Intel, AMD, Nvidia, Netflix, Mozilla and Cisco are members: http://aomedia.org/about/
• It's patent and royalty free and they've even set up a legal defense fund in case things do go awry.
• dav1d supports iOS, which I think openh264 doesn't.

They're working very hard to optimize dav1d, Phoronix benchmarks it every couple months: https://www.phoronix.com/scan.php?page=news_item&px=Dav1d-0.7-Performance.
AV1 is just starting to be used in the wild for things like Youtube and Netflix.

H264 is encumbered by patents, but if you use the openh264 binaries built by cisco they pay the royalties so it's effectively free (as long as you don't use your own builds).

[tomspilman]

https://github.com/cisco/openh264

That that seemed like a pretty interesting solution, but we have to modify it to make it work on consoles/mobile devices. So i think we couldn't use it.

There's also a newer spec, AV1.

That seems really interesting.

[mrhelmut]

CPU load is maybe something to consider as well. Things like h265 can be very CPU intensive even on modern hardware (I don't know about other codecs). I also guess that most codecs won't be able to use hardware decoders on consoles, which means that in some scenario it would be fully decoded on CPU.

We're likely safe, but just thought to mention that.

AV1 looks promising. It went down a long road, I remember that it used to be unusable due to its heavy CPU requirement.

[tomspilman]

CPU load is maybe something to consider as well.

Well on desktops and desktop like systems without specialized decoder hardware we have no choice but to use a CPU decoder of some type.

But for some consoles and mobile it could be a problem.

Since generally MG titles use of video is simpler it could be the CPU overhead isn't as big a concern. Like i don't expect anyone will be decoding more than a single 4K video at most.

That said it would be great if we could decode a bunch of 4k videos at once using hardware. But making that work correctly across so many platforms is a challenge.