Shaka-player: Audio Track Question

[Kae2312]

Hey,

I'm actually implementing the possibility of switching audio tracks on my player.
I've some questions about this :

• How I can be sure that selectAudioLanguage() executed well or failed? It seems to return nothing but I can see that a AdaptionEvent is triggered when I switch audio tracks, but in the data event It doesn't tell me that it was triggered by an audio track switch action.

• Also, In my DASH manifest, I can see that I have an Accessibility track, but when I call getAudioLanguagesAndRoles, it doesn't tell me if there is an accessibility track. Is there a way to recognise if my track is an accessibility track ?

• I would like to know If there also a way to get the current Audio track playing by the player? I know by default shaka player will play the main track, but sometimes I have multiple tracks who got main role and by that I dont know which track is playing. Is there an easy way to get the current track ?

Thank you :)

[vaage]

Heelo @Kae2312 , thanks for bringing your question to us; I'll do my best to answer your questions. To make it easier to follow, I'll quote part of the question, and then follow it with my answer.

How I can be sure that selectAudioLanguage() executed well or failed?

Right now, there is no simple way for you to the degree you want. It would be nice if the method return a boolean to say that it worked, we looked at that, but there was a problem. You see, when we change variants (which is what would happen when we change audio tracks), the change may not happen right away. Sometimes it is deferred if the streaming engine is busy changing between periods.

In 2.6, we are looking at aways to flatten periods. We want to player to be ignorant of periods and isolate that concept to the manifest parsers (they have the context to handle period transitions). After that, we won't have to deal with deferred changes. Which would make it possible for us to support a return value on the method.

In the meantime, I don't see any reason why adding the variant track to the adaptation event would be a problem. For this I would suggest opening a feature request asking for that. I imagine that @ismena may even be able to use that new event information with the UI. We are always open to Pull Requests, and this would be a good first contribution (and relatively simple).

Also, In my DASH manifest, I can see that I have an Accessibility track

By "Accessibility track" do you mean captions? If so, it sounds like our dash parser may not be filling in the roles correctly. If that is the case, I would suggest opening a new bug (including access to the manifest) so that we can investigate that properly. This too would be a great candidate for a Pull Request.

I would like to know If there also a way to get the current Audio track playing by the player?

Not right now. The track information is limited right now. Exposing higher level track information will be easier for us after 2.6. To ensure something like this is done, I would once again suggest opening a feature request just for this, including the specifics of the use-cases you have for it. This will make it easier for others in the community to rally behind the idea, ensuring that it gets allocated resources.

In the meantime, if the adaptation event was to include the track it changed to, this may help. Once again I would really suggest you file that feature request.

@Kae2312 was I able to answer your questions?

[TheModMaker]

How I can be sure that selectAudioLanguage() executed well or failed?

You can pass a second argument to selectVariantTrack of true to make us clear the buffer and take effect immediately.

Also, In my DASH manifest, I can see that I have an Accessibility track

How are you indicating it in the manifest? With the Role element? If so you can check the kind property on the Track returned from getVariantTracks.

I would like to know If there also a way to get the current Audio track playing by the player?

You can call getVariantTracks and look for the one where the active property is true.

[Kae2312]

Hello @vaage,

Thank you a lot for your answers! It helped me a lot.

I'll make a PR that will add the variant track to Adaption event.

By "Accessibility track" do you mean captions?

Sorry I meant an audio track, in my manifest I can see an object Accessibility like this :

<AdaptationSet
      id="1"
      group="1"
      contentType="audio"
      lang="fr"
      minBandwidth="64000"
      maxBandwidth="128000"
      segmentAlignment="true"
      audioSamplingRate="48000"
      mimeType="audio/mp4"
      codecs="mp4a.40.2"
      startWithSAP="1">
      <AudioChannelConfiguration
        schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011"
        value="2">
      </AudioChannelConfiguration>
      <Accessibility
        schemeIdUri="urn:tva:metadata:cs:AudioPurposeCS:2007"
        value="1">
      </Accessibility>
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="alternate" />
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="alternate" />
      ......

Is there a way to get this information from the shaka-player? Maybe should I use the dash parser?

I'll open a Feature request to expose the current track playing. :)

[vaage]

@Kae2312 I looked into the definition of the accessibility tag, and based on what I found it looks like it serves a very similar role as the Role tag. If my understanding is correct, updating the dash manifest parser to include that information the the variant.roles field would allow you to get access to it.

If that is something you need soon, you would be best served by implement it yourself, since I don't see that getting scheduled soon based on our current work loads.

[fadomire]

Hello, i work with @Kae2312 and wanted to add some infos to this thread

from what i understood reading
ETSI TS 103 285 v1.1.1, section 6.1.2
https://www.etsi.org/deliver/etsi_ts/103200_103299/103285/01.01.01_60/ts_103285v010101p.pdf

As shown in Table 4, the combined use of Role and Accessibility Descriptors shall identify Adaptation Sets containing audio description and clean audio streams

in our case we have
Description : Broadcast mix AD
Role element element : @schemeIdUri = "urn:mpeg:dash:role:2011" @value = "alternate"
Accessibility element : @schemeIdUri = "urn:tva:metadata:cs:AudioPurposeCS:2007" @value = "1" for the visually impaired

in Shaka if we want to make a PR, should we detect this case alternate and accessibility with this @schemeIdUri and @value ? if so, where could we add this in the AdaptationSet ? roles ? kind ? something else ?

thanks for any guidance

[vaage]

@fadomire that is a really good question. Thank you for asking. I am going to ask @TheModMaker if he has a suggestion on where this information should be expressed as he is far more experienced with the DASH spec than I am.

[TheModMaker]

We should extend the kind attribute to also apply to audio tracks based on the Role. So in your case it would be alternate. See section 5.8.5.5 of the DASH spec for these values and their interpretations. We should be careful about duplicate Role elements, since there can be main and another.

As for the Accessibility element, the DASH IOP specifies some values for this. With a @schemeIdUri of urn:mpeg:dash:role:2011: on video the value can be sign, subtitle, or caption; with audio it can be description and enhanced-audio-intelligibility; with text it can be description or caption.

Your @schemeIdUri isn't specified, but thats allowed. Since we focus on DASH and the DASH IOP, I'd be hesitant to add other specs in. How about adding the properties I described above according to the IOP guidelines?

We could then expose those as kind if Accessibility is present, falling back to Role if there isn't one. You can get the roles property to get the raw roles.

[fadomire]

after looking at DASH IOPS indeed there is a possibility of having

<Accessibility
  schemeIdUri="urn:mpeg:dash:role:2011"
  value="description">
</Accessibility>
<Role
    schemeIdUri="urn:mpeg:dash:role:2011"
    value="alternate">
</Role>

that should be taken care of

also when we look at dash js implementation we are seeing a specific case for urn:tva:metadata:cs:AudioPurposeCS:2007 but we dont know if it is standard or not

extract from dash js implementation for accessibility
src/mss/parser/MssParser.js line 124

// Map subTypes to MPEG-DASH AdaptationSet role and accessibility (see ETSI TS 103 285 v1.1.1, section 7.1.2)
        if (adaptationSet.subType) {
            if (ROLE[adaptationSet.subType]) {
                let role = {
                    schemeIdUri: 'urn:mpeg:dash:role:2011',
                    value: ROLE[adaptationSet.subType]
                };
                adaptationSet.Role = role;
                adaptationSet.Role_asArray = [role];
            }
            if (ACCESSIBILITY[adaptationSet.subType]) {
                let accessibility = {
                    schemeIdUri: 'urn:tva:metadata:cs:AudioPurposeCS:2007',
                    value: ACCESSIBILITY[adaptationSet.subType]
                };
                adaptationSet.Accessibility = accessibility;
                adaptationSet.Accessibility_asArray = [accessibility];
            }

[vaage]

@fadomire, based on the information @TheModMaker and you have assembled, do you feel comfortable filing an enhancement request summarizing the requirements and the proposed solution so that it will be easier for someone to pick-up and implement?

[fadomire]

@vaage only remaining question before is should we also include urn:tva:metadata:cs:AudioPurposeCS:2007 in addition to DASH IOP spec
i understand the hesitation.
Tho many players implement this specific spec apparently
And our packager Unified Streaming output it by default
So i dont hide that for us including it would help

[fadomire]

we had some more infos from Unified Streaming guys

<Accessibility
schemeIdUri="urn:tva:metadata:cs:AudioPurposeCS:2007" value="1">
</Accessibility>
<Role schemeIdUri="urn:mpeg:dash:role:2011" value="alternate" />

This matches latest DASH-IF spec by signalling the AD using the "DVB-DASH" syntax
https://dashif-documents.azurewebsites.net/Ingest/master/DASH-IF-Ingest.html#timed_text_and_subtitle_streams

"Additionally another example for explicitly signalling roles could be DVB DASH [DVB-DASH]. One could use schemeiduri@value and role as defined there. e.g. kind.schemeIdUri="urn:tva:metadata:cs:AudioPurposeCS:2007@1 kind.value=Alternate

[TheModMaker]

That spec seems odd to me. It isn't hosted by DASH-IF, but the repo is on their GitHub org. The repo is also only a few months old. There is no mention of it on their guidelines page (https://dashif.org/guidelines/). We focus on the DASH-IF IOP (interoperability points), which has no mention of that scheme ID.

But since dash.js implements it, we could do something. But the only use is in their SmoothStreaming parser, so it probably isn't "used" by the project, just exposed verbatim to the app. We may want to have something to expose the "Accessibility" data as raw instead of having us parse it.

[fadomire]

it seems DASH DVB would be an extension of DASH IOP ?
i found a validation tool for those different spec / format https://github.com/Dash-Industry-Forum/DASH-IF-Conformance
found in that doc https://www.hbbtv.org/wp-content/uploads/2018/11/DVB-HbbTV-DASH-Validation-Tool_explained_v1.0.pdf
did not tried it yet.

i honnestly dont know the status of DASH IOP / DVB, who promotes it, is DVB the next IOP or just a random extension

For dash.js indeed they only use it in SmoothStreaming which is kind of wired.

i'm still thinking on what would be the best regarding accessibility tag.
Implementing it in "kind" following DASH IOP ? sure.
Exposing it as raw ? why not.
Both ? maybe

[vaage]

@fadomire what I would suggest you do is summary the requirements and your proposed solution as a feature request. That request will be reviewed by those on the team who are best equipped to discuss the details. Once the details have been worked-out, the work will be allocated resources as per its priority.

[fadomire]

i'll do that thanks

[shaka-bot]

Closing due to inactivity. If this is still an issue for you or if you have further questions, you can ask us to reopen or have the bot reopen it by including @shaka-bot reopen in a comment.