Shaka-player: Parse Accessibility element for media type `Subtitles` in DASH

[NicolasSiver]

Have you read the FAQ and checked for duplicate open issues? - yes

Is your feature request related to a problem? Please describe.

I was on the fence should I open Bug Report or Feature Request.

Current problem is that Shaka does not recognize Accessibility element for Subtitle track. As the developer, you do not have enough signals/information to distinguish caption tracks from subtitle tracks.

Describe the solution you'd like

I think it should be an open discussion, but if somebody supports both CC and Subtitle in the media manifest (DASH), they should be equipped to analyze the data. For example, getTextLanguagesAndRoles could analyze Accessibility elements and return roles caption, subtitle. Or getTextTracks should be rich with the information about accessibility traits so the developer can build maintainable conditions to support CC + Subtitles use cases.

Describe alternatives you've considered

Trying to use methods like getTextLanguagesAndRoles or getTextTracks does not give enough metadata to operate on.

Additional context

I would like to share with you an example of how Role/Accessibility use-case looks like in DASH:

<AdaptationSet lang="en-US" mimeType="text/vtt">
    <ContentComponent contentType="text"></ContentComponent>
    <Accessibility schemeIdUri="urn:mpeg:dash:role:2011" value="caption"></Accessibility>
    <Representation id="subs-6535010" bandwidth="256">
        <BaseURL>../../texttrack/6535010.vtt?ptoken=c6dfa7a0c67abc3b03c299d9f3f0951348dd538f</BaseURL>
    </Representation>
</AdaptationSet>
<AdaptationSet lang="en-US" mimeType="text/vtt">
    <ContentComponent contentType="text"></ContentComponent>
    <Role schemeIdUri="urn:mpeg:dash:role:2011" value="subtitle"></Role>
    <Representation id="subs-6534961" bandwidth="256">
        <BaseURL>../../texttrack/6534961.vtt?ptoken=ad14492a31b98482abfb6a0c0d069eccfcc42ef1</BaseURL>
    </Representation>
</AdaptationSet>

Shaka will parse such DASH manifest without any issues, but as a developer, you will not have enough information from Shaka API to distinguish tracks.

Here are examples, how results will look for the available Shaka API:

Method getTextLanguagesAndRoles

[
  {language: "en-US", role: "subtitle"},
  {language: "en-US", role: ""}
]

Method getTextTracks

[
    {
        active: false
        audioBandwidth: null
        audioCodec: null
        audioId: null
        audioRoles: null
        bandwidth: 0
        channelsCount: null
        codecs: null
        frameRate: null
        height: null
        id: 1
        kind: "subtitle"
        label: null
        language: "en-US"
        mimeType: "text/vtt"
        originalAudioId: null
        originalTextId: "subs-6534961"
        originalVideoId: null
        primary: false
        roles: ["subtitle"]
        type: "text"
        videoBandwidth: null
        videoCodec: null
        videoId: null
        width: null
    },
    {
        active: true
        audioBandwidth: null
        audioCodec: null
        audioId: null
        audioRoles: null
        bandwidth: 0
        channelsCount: null
        codecs: null
        frameRate: null
        height: null
        id: 3
        kind: "subtitle"
        label: null
        language: "en-US"
        mimeType: "text/vtt"
        originalAudioId: null
        originalTextId: "subs-6535010"
        originalVideoId: null
        primary: false
        roles: []
        type: "text"
        videoBandwidth: null
        videoCodec: null
        videoId: null
        width: null
    }
]

[joeyparrish]

So it appears that <Accessibility> is supplying the same information that could be in <Role> (namely, captions vs subs). Is that correct?

[NicolasSiver]

When Role is used for Adaptation Set of the media type Subtitle it does not have value caption.

Possible values for Role are:

- main
- alternate
- subtitle
- supplementary
- commentary
- dub
- description
- emergency

Source: https://dashif.org/docs/DASH-IF-IOP-v4.2-clean.htm#_Toc511040753 (Scroll a bit until media type Subtitle)

[joeyparrish]

Ah, okay. I see. This should be straightforward enough to fix, then.

[joeyparrish]

We'll add parsing of the <Accessibility> element in DASH. Something of the form:

<Accessibility schemeIdUri="urn:mpeg:dash:role:2011" value="FOO">

will add "FOO" to the roles array in the track object. If the value is "caption", it will also set the kind attribute of the track.