Exoplayer: Read data from CodecPrivate when playing AVC or HEVC

[steffenmanden]

[REQUIRED] Use case description

It has recently been decided in the mkv standard to add support for extra data in the codecprivate area when playing AVC or HEVC content.

This allows Dolby Vision data, as well as MVC data, to be properly read during playback and thus enable Dolby Vision support and more.

Proposed solution
It would be great it Exoplayer could be made possible to playback using this data allowing monitors to utilize Dolby Vision data.

The create of MakeMKV has allowed contact to him for easy explaination of how the data is stored and implemented

Alternatives considered
Workarounds are currently used through mp4 and ts files - however they lack a lot of features regarding subtitles, chapters and more! Directly play from the most common format (mkv) would make it super easy to use for the average user.

[kim-vde]

Is it standard to have Dolby Vision streams inside MKV? It is not listed in the Matroska codec IDs.

[steffenmanden]

It's not in the standard definition. Dolby Vision has not previously been in the standard.

This implementation extends the format to allow for Dolby Vision within this format. Currently solutions (mp4 and ts) however have the standard, however are limited in a lot of features.

So although its not a part of the standard of mkv, it is an extention of the standard which will improve the player capabilities in a massive way without going out of bounds in a massive way.

MKV is after all the most used format :)

[kim-vde]

Ok. I have flagged the issue as a low-priority enhancement for now. Thanks for reporting it.

[steffenmanden]

Thanks :) - referenced the MakeMKV creator to comment in here, as he can give more details as to how the format was extended! Hopefully he will comment

[MikeChenMM]

I'm author of MakeMKV. For the record, I 'm willing to provide any help to make this happen.

Now, few points:
"Matroska standard" is de-facto non-existent thing and is very outdated. :(
Dolby Vision is not a separate codec, but rather a (HEVC) stream extension - DV data (EL+RPU) are encoded in separate NAL units, according to Dolby spec.
The presence of DV data can be checked by several different methods - most obvious are:
(1) DV descriptor in codecprivate data (there is an "standard" of storing additional elements in codec private for AVC and HEVC streams, that was used for ages since 3D MVC video) or
(2)presence of DV RPU unit in first frame.

And, given that DV data is just a raw data that has to be handled to decoder and is already stored in decoder-ready format, it would take your developer literally ONE HOUR OF TIME to enable playback of such MKV files (assuming you can play back DV from other containers).

[cogira]

Really very interested in this. Hope that will soon be implemented in the exoplayer. Thanks.

[Grencola]

1000% YES!! If you need any assistance at all I will gladly help make this happen!

[pervin11111]

Honestly, I would love to see this implementation. I am so used to packing chapters, subtitles in MKV.

[ddogishere]

I'm also registering my interest as this will be epic Thx

[kcxserver]

yes it would be wonderful if the exoplayer could play dv in mkv! 😍

[doctorsirius]

It's also important to note that current support of DV on .ts files doesn't work combined with the use of PGS subtitles, this should be also tested while it is implemented. It works for HDR but not for DV.

[Wampaa]

Hi @MikeChenMM thanks for your work on MakeMKV and on DolbyVision support!

"Matroska standard" is de-facto non-existent thing and is very outdated. :(

The Matroska specifications are an ongoing effort by the cellar working group at the IETF, in its current state it is composed of:

• Matroska Media Container Format Specifications - https://www.ietf.org/id/draft-ietf-cellar-matroska-05.txt
• Matroska Media Container Codec Specifications - https://www.ietf.org/id/draft-ietf-cellar-codec-04.txt
• Matroska Media Container Tag Specifications - https://www.ietf.org/id/draft-ietf-cellar-tags-04.txt
• Extensible Binary Meta Language - https://www.ietf.org/id/draft-ietf-cellar-ebml-17.txt

The working group is on GitHub and accepts contributions https://github.com/cellar-wg/matroska-specification

If changes (or clarifications) in the specifications are needed for this particular problem it would be good to propose them!

[steffenmanden]

Hi @MikeChenMM thanks for your work on MakeMKV and on DolbyVision support!

"Matroska standard" is de-facto non-existent thing and is very outdated. :(

The Matroska specifications are an ongoing effort by the cellar working group at the IETF, in its current state it is composed of:

• Matroska Media Container Format Specifications - https://www.ietf.org/id/draft-ietf-cellar-matroska-05.txt
• Matroska Media Container Codec Specifications - https://www.ietf.org/id/draft-ietf-cellar-codec-04.txt
• Matroska Media Container Tag Specifications - https://www.ietf.org/id/draft-ietf-cellar-tags-04.txt
• Extensible Binary Meta Language - https://www.ietf.org/id/draft-ietf-cellar-ebml-17.txt

The working group is on GitHub and accepts contributions https://github.com/cellar-wg/matroska-specification

If changes (or clarifications) in the specifications are needed for this particular problem it would be good to propose them!

And they now agreed on a solution for saving data in codecprivate :-)
https://github.com/cellar-wg/matroska-specification/pull/377

@kim-vde the changes now allows for more usage than just Dolby Vision. This includes MVC data.

As you can see it is in final approval to be added to the standard.

Since this adds more functionality than just Dolby Vision, shouldnt this justify an increase in the priority ?

[kim-vde]

Feel free to make a pull request to add this feature and we will be happy to review it. If you do so, please include tests with a minimal test asset. Thanks!

[ElegyD]

https://github.com/cellar-wg/matroska-specification/pull/390 has been merged and work towards this issue can be started.

[willtrking]

I'm interested in working on this, and made changes to MatroskaExtractor locally to support Dolby Vision data in BlockAdditionMapping per the final spec here (https://github.com/cellar-wg/matroska-specification/pull/390). It seems like since the dvvc and dvcc is already supported for mp4's, all that's needed is to modify MatroskaExtractor to properly extract a DolbyVisionConfig and pass the relevant information to com.google.android.exoplayer2.Format.

Working on getting a small sample MKV to test this with before submitting a PR. On that note, @MikeChenMM you wouldn't happen to have a small test MKV in the finished spec, would you?

[kim-vde]

I will close this issue as the corresponding pull request has been merged.