Webrtc-pc: Missing specification on how to assign bandwidth between encodings and/or drop simulcast layers

[murillo128]

During the IETF hackaton we have found a weird situation in chrome with latest simulcast implementation via addTransceivers in which the ordering of the encoding affects how the bandwidth is allocated to the different simulcast encoding:

https://bugs.chromium.org/p/webrtc/issues/detail?id=10469

If the reverse order of encodings is used on the addTransceiver call, all of the bandwidth will be assigned to the lower layers while the higher layer (the 720p one) will almost no bandwidth assigned.

While this is not the expected behavior there seems to be no clear specs on what should be the correct behavior.

In previous discussions we have decided that this may be an implementation detail, but after having to deal with it, I definitively think we need a consistent behavior across browsers.

[murillo128]

A possible solution would be to add the following attributes to the sendEncodings:

• minBitrate : bellow this threshold the encoding/layer will be dropped (i.e. will not be sent until there is enough bandwidth to start sending it again). Default: 0
• weight : weight of this encoding compared to the other encodings of this sender. Default: 1/scaleResolutionBy
• dropOrder : In case two (or more) encodings cannot be sent because there is not enough bandwidth which one should be dropped first. Default: array index of the encoding

So, when the total bandwidth is assigned to the sender/track (this could be up to the implementators), the total weight of the encodings should be calculated and assigned each the bandwidth according to the weight ratio. If any of the encodings does not meet the minBitrate the one with lower dropOrder will be removed and the total weight will be calculated and the bandwidth split again until either all encodings are dropped or they match the minimum bandwidth constrains.

We should also add events for layer drop/activation due to this process.

[lminiero]

I'm personally fine "privileging" lower layers (just to ensure we always have something to decode no matter the bandwidth) but I agree that once higher layers do appear, the bandwidth allocation should be more balanced as a consequence. I don't have a particular preference over the specific mechanism, but minBitrate sounds reasonable, even though in practice it may be difficult to enforce; weight is a good option as well, especially if it introduces the balance the current mechanism is lacking.

[murillo128]

@lminiero what IMHO is critical is having a consistent behavior between browser and don't have to check how each browser implements it (which may be even incompatible) in order to do our sfu logic.

[lminiero]

100% agree.

[ibc]

As Adam Roach said in Twitter, there is some spec (and I don't know which one it is) that states in section 7.2:

What simulcast streams to prioritize when allocating available bitrate among the simulcast streams in such adaptation SHOULD be taken from the simulcast stream order on the 'a=simulcast' line and ordering of alternative simulcast formats

Well, definitely sendEncodings is more than just simulcast. In the future we'll wish to declare SVC layer dependencies within multiple sendEncodings and, when that happens, there won't be any a=simulcast line in the SDP. So assuming that the order of sendEncodings determine the "bw allocation policy" for each encoding is just wrong.

Said that, we need a way to tell the encoder which simulcast streams we want to drop first if there were bw issues. Relying on the order of the given sendEncodings is just ugly (reasons given above). The order of rids in the a=simulcast line should not be determined by the order of the sendEncodings but by any other policy up to the application:

1. I may want to always send a very low resolution thumbnail video, and also send a medium and high resolution by prioritizing the high one (@murillo128 given use case in Twitter).
2. I may want to send two encodings of video with different scaleResolutionBy and maxFramerate, both with the same resulting effective bitrate, and having both the same "priority" (so instead of disabling one of them upon bw issues, just decrease the resolution and framerate of both if possible).

There is no single "algorithm" for the browser to know what the application wants, so IMHO the best way to go is having a explicit setting chosen by the app:

• In the "past" we had a priority field nobody never properly understood what it was for (BTW Chrome M75 rejects addTransceiver() if priority field is given in any encoding).
• We may reuse such a priority field or have a new one (such as the dropOrder field indicated above by Sergio).
• Whatever, but it MUST be chosen by the app. The browser CANNOT deduce it based on the order of sendEncodings nor based on maxBitrate, scaleResolutionBy, etc. values. It cannot deduce it because different legit use cases require different behaviors given the SAME encodings.

And a final proposal: Given such a dropOrder (or priority or whatever) value chosen by the app, the browser should build the a=simulcast line with an order of rids that honors those settings (rather than ordering rids based on their position within the sendEncodings array). This way we are "SDP simulcast compliant" (whatever that means) without constraining the exposed API and without conditioning the exposed API due to a wrong "simulcast" assumption (it may be SVC in the future, let's not assume what sendEncodings mean!).

[lminiero]

Just a note that this may have an impact on draft-ietf-mmusic-sdp-simulcast in the IETF, though, which is where the snippet Adam pasted comes from.

[murillo128]

Said that, we need a way to tell the encoder which simulcast streams we want to drop first if there were bw issues.

I agree, but we need to know when a layer should be dropped first, that's why I proposed the minBitrate (and it could be always set to 0). I would be happy with any other mechanism that produces reproducible and predictable results about when a layer should be paused/resumed.

[ibc]

Just a note that this may have an impact on draft-ietf-mmusic-sdp-simulcast in the IETF, though, which is where the snippet Adam pasted comes from.

Not in my comment. I said:

And a final proposal: Given such a dropOrder (or priority or whatever) value chosen by the app, the browser should build the a=simulcast line with an order of rids that honors those settings (rather than ordering rids based on their position within the sendEncodings array). This way we are "SDP simulcast compliant" (whatever that means).

[lminiero]

@ibc you're right, that proposal would be in line with the draft :+1:

I was mostly referring to any other effort that may break the current rules: in case we propose changes here, we might need the related changes made there too.

[ibc]

I agree, but we need to know when a layer should be dropped first, that's why I proposed the minBitrate (and it could be always set to 0). I would be happy with any other mechanism that produces reproducible and predictable results about when a layer should be paused/resumed.

I will delete my previous comment about the minBitrate proposal since I did not properly understand it. Anyway:

minBitrate : bellow this threshold the encoding/layer will be dropped (i.e. will not be sent until there is enough bandwidth to start sending it again). Default: 0

If there are three encodings with minBitrate': 1000,minBitrate: 2000 andminBitrate: 3000, and the TOTAL BWE is 3500, what should happen? the browser may send streams 1 and 2, or just 3. IMHO this does not fix the problem. A explicitdropOrder` looks much better.

[murillo128]

@ibc : from the pseudo algorithm I outlined above:

If any of the encodings does not meet the minBitrate the one with lower dropOrder will be removed and the total weight will be calculated and the bandwidth split again until either all encodings are dropped or they match the minimum bandwidth constrains.

[murillo128]

or another alternative, when there is a layer which minBitrate cannot be satisfied, then layers are dropped by the dropOrder until all the minBitrates of the remaining layers are satisfied.

[ibc]

I still don't know whether such a encoding needed minBitrate must be compared to the total available BWE ro just to the "fraction" that corresponds to that specific encoding based on... on things.

I mean: the assigned bandwidth of which encoding the browser should reduce if there are network issues/limitations? To clarify. You said:

If any of the encodings does not meet the minBitrate

If the current bitrate of a encoding does not satisfy its minBitrate it's because the encoder decides to decrease it. Based on what? And why not others?

[murillo128]

For example imagine that you have the following encoding settings:

• rid: a minBitrate:700 weight:4 dropOrder=1
• rid: b minBitrate:300 weight:2 dropOrder=2
• rid: c minBitrate:100 weight:1 dropOrder=3

Now, let's say that the BWE is of 2048kbps and that only this video track is present. Then the total weight of the encodings would be 7 and each rid will get the following bandwidths:

• rid a bw=(2048/7)*4=1170
• rid b bw=(2048/7)*2= 585
• rid c bw=(2048/7)*1= 293

If bw dropped to 1024kbps then each layer would get

• rid a bw=(1024/7)*4=585
• rid b bw=(1024/7)*2= 293
• rid c bw=(1024/7)*1= 146

In this case none of the layers will be able to achieve the minimum layer, so the one with lower dropOrder will be dropped and the total weight recalculated and individual layer bw:

• rid b bw=(1024/3)*2= 682
• rid c bw=(1024/3)*1= 342

[ibc]

Ok. It looks good but IMHO too complex (finally those are 3 new fields!). But indeed if we remove the minBitrate field the browser would keep sending all layers even in bad circumstances.

However, you know that current implementations assume they can stop a layer without too much rationale (they just do it). Would that be the effective behavior if minBitrate is not given?

I've also some doubts regarding the weight parameter. It looks like it may conflict with scaleResolutionBy, maxFramerate and maxBitrate. IMHO the browser should be able to estimate the "weight" of a layer/stream based on those already defined values.

This is, I assume that your suggested weight field tells the browser how much percentage of BW it must allocate for each stream. IMHO we don't need that. The encoder should know that better than us based on scaleResolutionBy, maxFramerate and maxBitrate (and I'd love to introduce as less new parameters as possible).

[murillo128]

I prefer when parameters have one and only one meaning, they introduce less complexity on the long run than parameters that are used for multiple purposes.

Calculating weight internally based on other parameters may prevent you for doing some use cases like high fps but low weight, or low fps high weight (for screensharing for example). On my initial proposal the default value was 1/scaleResolutionDownBy in case no weight was specify but we can include other values and let it be overidable.

Regarding the minBitrate we could set 0 as min value, which would mean that no layers are dropped.

[ibc]

Calculating weight internally based on other parameters may prevent you for doing some use cases like high fps but low weight, or low fps high weight (for screensharing for example).

• "high fps but low weight": Isn't better if you set high minBitrate and low dropOrder?
• "low fps and high weight": You can set low minBitrate and high dropOrder.

I think the result would be very similar and we avoid an extra parameter.

[aboba]

How is this different from Issue https://github.com/w3c/webrtc-pc/issues/2080?

[ibc]

I think this issue is much more complete than #2080 given that here we do not assume simulcast as the only valid use case for multiple encodings.

[murillo128]

@aboba #2080 is only about what layer should be dropped first, but it does not cover how the bandwidth should be split between the different layers and it's relationship with the order of the encodings.

My tentative proposal would cover both issues tough.

[ibc]

Friendly reminder that if we constraint encodings to just mean simulcast we'll have giant problems when SVC came into the spec and implementations.

[murillo128]

The current proposal for adding SVC to webrtc (https://w3c.github.io/webrtc-svc/) that has been discussed over the last meetings has diverged from the ORTC one, and does not rely on different encodings to set SVC layers but on the scalability mode within each encoding.

So my understanding is that each encoding maps 1-to-1 to a simulcast layer (even if using different codecs).

[ibc]

The current proposal for adding SVC to webrtc (https://w3c.github.io/webrtc-svc/) that has been discussed over the last meetings

It seems easy to implement:

[ibc]

BTW I just like the webrtc-svc proposal and its simple scalabilityMode parameter. It looks like encodings should probably be renamed to streams (as that is what it's about at the end) but anyway.

[murillo128]

That is one of the standard SVC modes for AV1, try to configure that scalability structure with 27 encodings in an ORTC way.. ;)

[ibc]

That is one of the standard SVC modes for AV1, try to configure that scalability structure with 27 encodings in an ORTC way.. ;)

I'm pretty sure the biggest problem wouldn't be how to fill those complex sendEncodings but making browsers correctly interpret them. But yes, you are right.

[alvestrand]

We should have some words saying that the browser needs to do "something sensible" - this will depend on the scaledownby and max-bandwidth parameters, it's complex (and likely undesirable) to specify exactly how bandwidth should be distributed under congestion, but the first example in the comment chain is clearly wrong.

[ibc]

@alvestrand may I know please which "first example in the comment chain" you mean?

[alvestrand]

@ibc https://github.com/w3c/webrtc-pc/issues/2141#issue-424592590