Shaka-player: CEA-608 captions are garbled

[ppatlolla-turner]

Have you read the FAQ and checked for duplicate open issues?
Yes

What version of Shaka Player are you using?
v2.5.9 (uncompiled)

Can you reproduce the issue with our latest release version?
Yes

Can you reproduce the issue with the latest code from master?
Didn't try

Are you using the demo app or your own custom app?
Using the demo app.
https://shaka-player-demo.appspot.com/demo

If custom app, can you reproduce the issue using our demo app?

What browser and OS are you using?
Chrome - Version 80.0.3987.100 (Official Build) (64-bit)
OS - macOS 10.14.6 (18G2022)
For embedded devices (smart TVs, etc.), what model and firmware version are you using?
Roku, Apple TV and alternate HTML5 players do display the captions correctly.

What are the manifest and license server URIs?
Will send the manifest and license server url via email with bug number in the subject line.
What did you do?
Played the stream sent via email by populating the Custom Content button.

What did you expect to happen?
608 captions should have been displayed accurately.

What actually happened?

[joeyparrish]

@ppatlolla-turner, are those captions meant to be in English? Can you share a screenshot of the same text being rendered correctly in another player, for reference?

[ppatlolla-turner]

Yes, the captions are meant to be in English.
Shaka player via HLS manifest does identity the captions as English. Shaka player doesn’t identity the captions as English when DASH manifest is used.

Attached screen-shot displaying the captions correctly.

Thanks
Pramod

[MilosRasic]

[TheModMaker]

I confirmed the captions are displayed weirdly. We don't do the caption parsing ourselves, we use mux.js, so it is probably a bug with them. I notice nothing obviously wrong with your MP4 files.

[joeyparrish]

@TheModMaker, please try the latest mux.js release and mux.js master to see if this is already fixed upstream. I've seen a lot of PRs landing in mux lately.

[TheModMaker]

It still reproduces with both the latest release and their master.

[joeyparrish]

Because the stream is encrypted, it's possible that the CC data is garbled the encryption. Decryption happens inside the browser, but CC parsing is happening in JavaScript. JavaScript apps have no access to the keys or the decrypted media.

It could still be a bug in the parser of mux.js, but at the moment, I have no way to verify whether or not the caption data was encrypted.

So, some questions for @ppatlolla-turner:

1. Do you have another web-based player that correctly displays these captions? If so, which one?
2. What software was used to package the media?
3. Are you certain the CC data was left in the clear during encryption?
4. Can you try a clear version of the same stream and see if that makes a difference?

Thanks!

[joeyparrish]

Just got some technical details from a colleague:

CEA608 is embedded in H264 SEI NALU. For packagers that follow CENCv3 specification, SEI is not encrypted. If they follow CENCv1 specification, SEI is encrypted.

[ppatlolla-turner]

CEA608 captions do break even when clear streams are used. Sending details via email.

[joeyparrish]

Perfect, thank you. I'm sure that will help.

[joeyparrish]

With your permission, I'd like to send a single encrypted segment along with its corresponding init segment in a bug report upstream to mux.js, which we use for CEA parsing. I would be attaching those individual segments publicly in a public report, without their original URLs, their keys, or their license server URLs.

Do I have your permission to send those two segments publicly to mux.js?

[ppatlolla-turner]

Yes you do have permission to share. Captions start at 15 seconds into the stream.
Better to send first 4 segments (4 x 6.006 seconds).

[joeyparrish]

So far, the mux.js team doesn't know what is wrong with their CEA parser on this content, and neither do we.

If you have access to an open-source CEA parser in any language that works with this content, we could compare it with mux.js and see if we can identify the issue.

In the mean time, while we wait on either the mux.js team to provide a resolution or a working CEA parser to compare with, we will have to remove this issue from the v2.6 milestone.

We apologize for the inconvenience.

[joeyparrish]

After some more digging, I have determined that this is definitely caused by b-frames in the content. Mux.js does not currently support b-frames correctly when extracting captions. Re-encoding without b-frames fixes the extraction. See https://github.com/videojs/mux.js/issues/332#issuecomment-622988104 for details.

[ppatlolla-turner]

Thanks for the update.
As an alternate plan we are exploring webVTT. If we need to use 608 for captions we now know that we need to remove b-frames from the video. Removing b-frames is very unlikely.
Hope mux.js supports b-frames for 608.

[joeyparrish]

It turns out to be related to the correct parsing of trun boxes in mux.js. The v1 boxes have signed ints for composition time offset, while v0 has unsigned ints. mux.js was only correctly parsing v0 boxes. I have a PR for mux.js up for review that fixes the issue: https://github.com/videojs/mux.js/pull/338

Thanks to our colleagues on the Cast team for pointing out the root cause!

[ppatlolla-turner]

Confirming that 608 captions have been fixed.