Shaka-player: Codec order preference for Video and Audio tracks

[j0hnlam]

Have you read the FAQ and checked for duplicate open issues?
Yes

Is your feature request related to a problem? Please describe.

For dash manifests with multiple codecs for audio and video, Shaka player currently chooses the lowest bitrate bandwidth codec. I would like to have the ability to override that by sending an array of codecs to the shaka config.

Describe the solution you'd like
I would like the ability to pass an array of codecs that I want Shaka player to consider as my preference when choosing a codec.
videoCodecPreference: ['hevc', 'avc', etc]
audioCodecPreference: ['ac-3', 'mp4a.40.2', etc]

Describe alternatives you've considered
For audio codec, I considered the preferredAudioChannelCount shaka config but there are times where the manifest does not provide AudioChannelConfiguration channel count so that config property does not work.

For video, I have not able to find an out of the box support in the current Shaka player to let me choose what video codec to use.

Additional context
This is to be used on the Chromecast receiver where I will first check the canDisplayType method in the CAF Receiver SDK and then populate the codec preference array that will be passed to the Shaka player to choose.

It allows Chromecast users the ability to have a higher bitrate codec viewing and audio experience.

[joeyparrish]

It allows Chromecast users the ability to have a higher bitrate codec viewing and audio experience.

Why would you want to consume more bandwidth? In your example, you pass hevc ahead of avc, which would correspond to lower bitrates at the same resolution, would it not?

[j0hnlam]

In the manifest that I am provided, that is not the case. HEVC will only have 4K, 1080p, while avc will only have 1080p and lower.

<Representation id="hevc-ffa5v1-24p-primary-28800000" codecs="hvc1.1.6.H150.90" mimeType="video/mp4" width="3840" height="2160" bandwidth="28800000"></Representation>
<Representation id="hevc-ffa5v1-24p-primary-7200000" codecs="hvc1.1.6.H120.90" mimeType="video/mp4" width="1920" height="1080" bandwidth="7200000"></Representation>
<Representation id="h264-ffa5v1-24p-primary-7200000" codecs="avc1.640028" mimeType="video/mp4" width="1920" height="1080" bandwidth="7200000"></Representation>
<Representation id="h264-ffa5v1-24p-primary-3200000" codecs="avc1.64001f" mimeType="video/mp4" width="1280" height="720" bandwidth="3200000"></Representation>
<Representation id="h264-ffa5v1-24p-primary-1800000" codecs="avc1.64001f" mimeType="video/mp4" width="960" height="540" bandwidth="1800000"></Representation>
<Representation id="h264-ffa5v1-24p-primary-800000" codecs="avc1.64001e" mimeType="video/mp4" width="640" height="360" bandwidth="800000"></Representation>

[joeyparrish]

I see. And what if the user's internet connection is too poor to sustain the 1080p bitrate, but we've chosen hevc based on the codec preference before we had any bandwidth estimates?

[j0hnlam]

It's one of the risk I accept that a user can run into. However on my end, I allow the user to select the quality (ultimate, best, good) preference before establishing a cast session.

So if a user is experiencing buffering constantly, we suggest the user to lower their quality setting on the app (a message popup on receiver) and restart their Chromecast session.

[riksagar]

Hi @joeyparrish , we have a similar situation. We would like the player to prefer Dolby Digital audio over AAC , but there's no way to express a preference in the config. Shaka seems to pick one arbitrarily.

Sometimes the AAC and DDP bitrates are the same in the manifest. Often the number of channels is the same, or not reported in the manifest.

There's no good workaround. Can you provide an API we can call to force the preferred adaptation(s). We could use that once you've parsed and filtered the manifest and compiled the list of supported adaptations/variants.

[joeyparrish]

Today, you can force a particular resolution or bitrate through the tracks API (getVariantTracks(), selectVariantTrack()), but the list of tracks is fixed once we choose a codec.

We can definitely provide a way to specify a preference. Thank you both for explaining your use cases for me. This will definitely help.

This relates to our adoption of MediaCapabilities (#1391), which will open up additional ways for us to choose a codec (based on bandwidth, on smoothness, or on power efficiency). Having one more option to use an explicit preference list will be relatively simple to add at that stage.

This also relates to our adoption of codec switching (#1528), which will allow us to avoid choosing a specific codec if the platform will allow us to adapt between codecs.

[sunnyoui]

@joeyparrish Is it possible to check the supported codecs on the platform using Shaka API before passing a manifest to the player? Secondly, is Shaka using hardware or software decoding on chromecast hardware?

[TheModMaker]

We already should filter out unplayable codecs. We check the platform for support before playing the streams. So you should never see any streams that cannot be played.

We don't do anything special for software vs hardware support. We just give the data to the browser and it plays it however it wants.

[joeyparrish]

Yes, JavaScript apps have no _direct_ visibility into hardware vs software codecs. We get a hint with MediaCapabilities, in which hardware codecs should generally be labelled as "power efficient". But some software codecs could also be labeled as such, for low res or low bitrate. So there's no direct way to tell the difference from a web app.

[naktinis]

I wanted to also add that selecting priorities would be great for our use-case as well. We first generate avc, because it's faster and supported by more browsers (i.e. Safari) and then we generate vp09, because it has better efficiency. I'd like to be able to configure Shaka player to prefer vp09 whenever supported and fall back to avc when either vp09 is not supported, or it's not in the manifest.

[joeyparrish]

If your preference is for the most efficient codecs, this is what we already do today. We base this on the average bitrate in the manifest. So if your vp9 streams are lower bitrate (as they should be for consistent quality at a certain resolution), then you already get this behavior.

If you have consistent bitrates but different resolutions per codec, this is where an explicit codec preference order would help. Or maybe you have some other reason to force us to prefer a certain codec. The mechanism for that would be agnostic to your reasons.

With MediaCapabilities, we would be able to offer preferences based on other things, such as battery life or dropped frames.

[naktinis]

@joeyparrish thanks for the clarification. I think the issue is that the vp9 with our settings is not guaranteed to have a lower bitrate. The idea is roughly that we use avc mainly as a fallback, so it's ok for it to sometimes have lower quality and lower bitrate. Anyway, just wanted to add that an explicit list would also be helpful in our case.

[joeyparrish]

Understood. I'll make sure this is part of our MediaCapabilities planning, as that is when we plan to introduce a set of preferences for choosing codecs.

[mp4allday]

Would ac3 audio stream type be supported by shaka player running on tizen tv?

[joeyparrish]

Codec support is a matter of what the browser supports, not what the player supports. We don't provide any decoders in JavaScript. :-) Please see #2661 for a more detailed explanation.

[mp4allday]

Thanks for the note @joeyparrish. Most tizen tvs if not all support AC3 decoding. I am afraid that when shaka is parsing the DASH manifest it is not able to select the ac-3 adaptation set. Is it a mime type related issue?

[joeyparrish]

Check https://shaka-player-demo.appspot.com/support.html on the device to see what the browser reports support for. This is what we base codec decisions on.

[MilosRasic]

Found this issue while investigating before opening one. Our need for this feature is specific to audio codecs. We have ec-3, dtse and mp4a listed in our manifests. ec-3 and dtse both have 6 channels and same bitrate specified. The relevant tags look like this:

dtse

    <AdaptationSet
      group="1"
      contentType="audio"
      lang="en"
      segmentAlignment="true"
      subsegmentAlignment="true"
      subsegmentStartsWithSAP="1"
      audioSamplingRate="48000"
      mimeType="audio/mp4"
      codecs="dtse"
      startWithSAP="1">
      <AudioChannelConfiguration
        schemeIdUri="urn:dts:dash:audio_channel_configuration:2012"
        value="6">
...
      <Representation
        id="audio_eng_1=384000"
        bandwidth="384000">

ec-3

    <AdaptationSet
      group="1"
      contentType="audio"
      lang="en"
      segmentAlignment="true"
      subsegmentAlignment="true"
      subsegmentStartsWithSAP="1"
      audioSamplingRate="48000"
      mimeType="audio/mp4"
      codecs="ec-3"
      startWithSAP="1">
      <AudioChannelConfiguration
        schemeIdUri="urn:dolby:dash:audio_channel_configuration:2011"
        value="F801">
...
      <Representation
        id="audio_eng_2=384000"
        bandwidth="384000">

We prefer dtse for various reasons and in cases like this when codecs have matching channel count and required bandwidth, Shaka seems to choose one by inertia, which always ends up being ec-3 in our case.

We'd like to have a way to specify dtse as preferred codec to serve as a tie breaker in cases like this and can spare the bandwidth to contribute a feature, as long as we can come up with an acceptable API.

I'd suggest preferredAudioCodecs as an array of codec identifiers, with element at index 0 preferred if it's found in the supported codecs, then index 1, etc. It would have lower priority than bitrate and preferredAudioChannelCount and only be used when preferring by closeness to the specified channel count and lower bitrate yields more than one result.

Meanwhile @riksagar, any details you could share about your workaround? 🙂

[riksagar]

Hi @MilosRasic . Well it's not a pretty fix!

We basically added code to modify the manifest prior to Shaka getting it. We remove the AAC Adaptation, then Shaka is forced to choose DDP (this is assuming the platform supports DDP. We also need to perform that check too).

This is all done in the networking engine. We register a response filter callback (see player.getNeworkingEngine().registerResponseFilter()) and for the MANIFEST request modify the DOM. The manifest DOM is in the data property of the response.

[MilosRasic]

Thanks @riksagar . Agreed, not pretty at all, but good to know it works.