Webrtc-pc: Constrainable properties on remote tracks are under-specified

[jan-ivar]

What behaviors specific to remote peer connection tracks are available through the constraints APIs? There's a single paragraph on the topic, and it doesn't say.

From @guidou in https://github.com/w3c/webrtc-pc/issues/2109#issuecomment-470566522, Chrome appears to have implemented something, so we should probably specify it, like we did in https://github.com/w3c/mediacapture-screen-share/issues/31. Otherwise, the next question might be why don't we support whiteBalanceMode?

Discuss.

[henbos]

Observations or starting point-opinions:

• Constraints are usually applied to tracks, not sources. As such, "width" can make sense whether the source is a camera or a network source. However sometimes they only make sense for a particular source... (see below).
• Starting point: Unless otherwise specified or not applicable, constraints should mean the same thing independent of what the source is.
• We can either be "defensive", and say "only these constraints are supported for remote tracks" and be explicit about it, or "offensive" and allow all constraints that are still defined in a way that can be applied to remote tracks to be so. Example: "width" means the same thing independent of what source produced the frames.

Otherwise, the next question might be why don't we support whiteBalanceMode?

Definition: "White balance mode is a setting that cameras use to adjust for different color temperatures."
This is a configuration that is clearly not applicable to a remote source. A remote source gives you whatever is sent to you, whereas a camera can be configured to produce different outputs. RTCPeerConnection is not a color temperature sensing camera.

---

I think a lot of this boils down to if you view "applyConstraints()" as means to _change configurations of a source_ or _filter/transform what the source provides you with_. The former is not applicable to remote tracks, the latter is.

We may save a lot of time arguing if we can pinpoint what "applyConstraints()" should mean with regards to this!

---

Side-discussion, though perhaps we should leave this one until later, is what to do if a track gets overconstrained? Given that mute is a possible side-effect of setRemoteDescription(), I am opposed to allowing remote tracks to become muted for other reasons. Two possibilties I see:

1. Don't support "min" and "max", only support "ideal". As such we never get overconstrained; as such we never get muted for constraints-reasons.
2. Don't mute tracks on overconstrained or remove overconstrained altogether (discussion separate from the local vs remote discussions).

[henbos]

@armax00 ;)

[armax00]

My 2 cents.

With respect to the behavior of applyConstraints() as well as gUM constraints, my understanding is that they should behave as filters, not as settings. This means that for local tracks it makes sense to filter which device should be used based on some criteria but with remote tracks this seems blurry, mostly because such parameters are anyway negotiated (hence, I am not sure why, after negotiation one should further filter for the right track).

[jan-ivar]

I think a lot of this boils down to if you view "applyConstraints()" as means to change configurations of a source or filter/transform what the source provides you with.

It's clearly both. Constraints outsource (no pun intended) user requirements from a source to its sink(s); a classic "put the control surface on the result object" thing. A width might reconfigure the camera, or software downscale, doesn't matter. Tracks are inseparable from their source; multiple heads of it.

It's an API. We're in an ugly spot of asking: "What behavior should this API produce?", when we need to ask: "What remote track use-cases must be solved (and what's the best API for them)?"

In the case of https://github.com/w3c/mediacapture-screen-share/issues/31#issuecomment-367200644 we had those use-cases.

We can either be "defensive", and say "only these constraints are supported for remote tracks" and be explicit about it, or "offensive" and allow all constraints that are still defined in a way that can be applied to remote tracks to be so. Example: "width" means the same thing independent of what source produced the frames.

It doesn't. A camera track is inherently and observably different from a screen-share track here:

• width on a screen-share track preserves aspect, to not lose information around the edges.
• width on a camera track might crop the output, compromising aspectRatio, and/or frameRate.

There are other differences as well: screenshare tracks never fire overconstrained and never mute. advanced and min are effectively useless (and provoke TypeError in getDisplayMedia()). resizeMode can be used to detect retina. Many of these behaviors weren't obvious until we specified them.

If "defensive" means specifying how width should work on a peer connection track, if at all, and not leaving it to implementor whim, then count me in.

[henbos]

I've created a slide for this.

[guidou]

I agree with @jan-ivar's comment.
I propose we update mediacapture-main to say that the constrainable properties defined there apply exclusively to tracks backed by getUserMedia() calls and that other specs using MediaStreams and related objects should explicitly define the properties that apply to them while respecting the constrainable pattern in general.

[henbos]

Count me in

[henbos]

Related: I filed https://github.com/w3c/mediacapture-main/issues/578 on the gUM spec.

[dontcallmedom]

RESOLUTION: define retrieval of constraint properties for getSettings but make them not modifiable

[jan-ivar]

@henbos to provide PR?

[henbos]

Sure