Webrtc-pc: addTransceiver woes

[fippo]

given the following piece of code:

const pc = new RTCPeerConnection();
const transceiver = pc.addTransceiver('audio');
pc.createOffer()
.then((offer) => {
   console.log(offer.sdp);
   console.log(transceiver.mid);
})

1) what is the direction in the sdp? Given that the transceivers direction is sendrecv by default should it be sendrecv? But that results in an API which is sendrecv without any track information. I would expect recvonly here.

2) what is transceiver.mid inside the promise? http://w3c.github.io/webrtc-pc/#dom-rtcpeerconnection-addtransceiver says its null initially and that changes only by setting a description. However, that means the mid value in the sdp does not match the transceivers mid.

[fippo]

Similar to (1):

const pc = new RTCPeerConnection();
pc.createOffer({offerToReceiveAudio: true})
.then(offer => {
  const transceiver = pc.getTransceivers()[0];
  console.log(offer.sdp);
  console.log(transceiver.direction);
})

I would expect a recvonly sdp but there is nothing in the legacy offerToReceive text that talks about direction.

The fix (for both this and (1) from the initial question) is probably to have addTransceiver with kind set the direction to recvonly.

[alvestrand]

I would expect a transceiver created out of the blue (new RTCRTPTransceiver) to have a direction attribute of none (disabled).
the descriptions that add tracks in the various directions would then have to document what they update the direction to.
In particular, pc.addTransceiver() with kind should set the direction to recvonly.

[nils-ohlmeier]

cc @docfaraday for awareness

[nils-ohlmeier]

@fippo example pc.addTransceiver('audio') according to my reading leads in step 5.2 of the addTransceiver the track to be null. And then in step 10 a RTCRtpSender is suppose to be create with a null track. Is that going to create an "empty" RTCRtpSender?

[fippo]

@alvestrand:
addTransceiver creates a transceiver with the default for RTCTransceiverInit.direction which is sendrecv.

@nils-ohlmeier I think step 10 needs to be fixed. ORTC allows creating RTCRtpSender with trackOrKind and while webrtc-pc does not expose a constructor I assume there will be internal methods which would be called in that way.

[jan-ivar]

I think where we say "Let track be null." we need to add "set direction to 'recvonly'"

@nils-ohlmeier yes sender.track can be null, "empty" if you will.

@fippo No need for a JS constructor, I don't think. These are internal algorithms.

[adam-be]

1. what is transceiver.mid inside the promise? http://w3c.github.io/webrtc-pc/#dom-rtcpeerconnection-addtransceiver says its null initially and that changes only by setting a description. However, that means the mid value in the sdp does not match the transceivers mid.

The transceiver needs to have a "pending mid" that is used when the transceiver is offered, but that might be overridden if the remote side ends up deciding the mid. A some point I proposed that we should specify it as an internal slot, but we decided to leave it open for implementors to decide on the approach.

[stefhak]

I don't see the problem with the current way of specifying it. addTransceiver without a second argument creates a transceiver and an m line that both have direction sendrecv. If you want something else use RTCRtpTransceiverDirection when creating the transceiver.

[fippo]

@stefhak I took the SDP generated by FF nightly when using addTransceiver and fed it into Chrome:
https://jsfiddle.net/9m2xau9w/1/

Firefox itself seems to be smart to recognize that despite sendrecv "-" means no track.
Chrome is not (@henbos i can't find the bug) and fires onaddstream. Until that chrome bug is fixed this is going to result in interoperability chaos.

[jan-ivar]

Agree with @fippo we need to fix Chrome, but don't think it warrants changing the default behavior of addTransceiver.

[henbos]

This will be fixed when Chrome gets transceiver support:
https://bugs.chromium.org/p/webrtc/issues/detail?id=7600#c13

[stefhak]

I'm tempted to close this issue. I think we've established that addTransceiver is specced OK, and I think the mid stuff was clarified in https://github.com/w3c/webrtc-pc/issues/1662#issuecomment-346770769.

WDYT?

[jan-ivar]

I've been playing with transceivers a bit in Firefox Nightly, and I may be coming back to @fippo's side.

This "hold" fiddle only works correctly when I manually ensure direction is sane:

pc1.addTransceiver("video", {streams: [stream], direction: "inactive"});

but if I instead do

pc1.addTransceiver("video", {streams: [stream]});

...then nothing works, including the code to change the direction later, maybe because the change from "sendrecv" to "sendonly" doesn't cause negotiationneeded to fire? We still have bugs, so that could be a bug, but please don't close this just yet.

[stefhak]

I'll gladly keep it open. I think there must be some bug in the implementation - if you create a transceiver using kind = 'video' with direction sendrecv, surely it should start sending once you attach a live video track to its sender?
I also tend to think you should remove the tc.direction part from the onclick handler - it sort of defeats the purpose of replaceTrack since a renegotiation is triggered. (I know it is part of the 'hold' examples, but for just testing that a sendonly/sendrecv transceiver would behave correctly on replaceTract(null)/replaceTrack(livetrack) switches it should not be there IMHO).

I played a bit with your example, and it turns out that if the transceiver is created with direction = 'inactive', it's only needed to do tc.direction = "sendrecv" (I assume "sendonly" would work equally well) the first time the onclick handler is called (and skip the ="inactive" and ="sendrecv" for all subsequent calls) - and things work fine even without any more negotiation.

[stefhak]

@jan-ivar any comment to https://github.com/w3c/webrtc-pc/issues/1662#issuecomment-349640726 ?

[jan-ivar]

@docfaraday The expectation in https://github.com/w3c/webrtc-pc/issues/1662#issuecomment-349640726 seems to be that with with "sendrecv", replaceTrack from a null track to a non-null track should start sending. When I modify the fiddle, this does not appear to work in Nightly. Does that work with JSEP? If so, we should open a bug on Firefox.

[stefhak]

@henbos @taylor-b @aboba would you agree to
"The expectation in #1662 (comment) seems to be that with with "sendrecv", replaceTrack from a null track to a non-null track should start sending."?
That's my understanding, but it would be good to know if others agree before filing a bug on Firefox.

[jan-ivar]

(Ok, found the FF bug, which surprisingly was that stream arg is missing in the track event in this case)

Turns out the direction attribute matters for things other than legacy.

To recap our understanding (assuming a well-configured pc1.onnegotiationneeded):

1. pc1.addTransceiver('video'); must cause pc2.ontrack to fire, so replaceTrack will work.
2. pc1.addTransceiver('video', {direction: "recvonly"}); mustn't.

So the competing use-cases for addTransceiver here seem to be:

1. Add a null-track "sendrecv" transceiver we can use sender.replaceTrack on later @stefhak
2. Add a "recvonly" transceiver, because we don't want to offer to send anything. @fippo

If we ignore one of these use cases, then determining the default for direction is easy. With both in mind, do we feel we have the default right?

[henbos]

Is addTransceiver(kind or null) supposed to work the same as, or different from, this?

sender = pc1.addTrack(track);
sender.replaceTrack(null);
pc1.createOffer(), setLocalDescription and exchange offer/answer with pc2

Because that would result in a sendrecv transceiver that is transmitting nothing (null track).

Isn't addTransceiver() supposed to work the same such that you can replaceTrack() to immediately start sending without renegotiating?

If it isn't sendrecv, wouldn't replaceTrack(track) on a transceiver created with addTransceiver() fail because renegotiation would be needed if the direction has to change? Or should replaceTrack() be updated to say to update the direction if it's not already sendrecv or sendonly?

What happens with _"Determine if negotiation is needed to transmit withTrack in place of the sender's existing track."_ if direction is not sendrecv or sendonly?

[henbos]

About the question for what a=msid to use when we don't have a track.id to announce...

It is already the case with replaceTrack() that the sending track.id does NOT necessarily correspond to the a=msid line. Example:

sender = pc1.addTrack(track1);  // a=msid with track1.id announced in SDP
negotiate
sender.replaceTrack(track2);  // seamlessly change which track to send, same a=msid as before

In fact, if we try to use track.id we can end up with multiple senders trying to use the same ID... the ID needs to be unique to the sender, not to whatever track happens to be sending at the moment.

So what should happen?
addTransceiver() creates a sendrecv transciever with a random id announced in the SDP. The receiving end is ready to receive and replaceTrack() works without renegotiation, just like it would have if addTrack() was used.

That is, the spec is already correct for addTransceiver(). But if a=msid takes the track's ID, we might need to file a bug on track ID collisions...

[henbos]

I may have been rehashing some of the discussion already had in the two comments above wrt use case 1).

I didn't realize you could simply change the direction and re-negotiate. In that case starting in "recvonly" and then opting in for sending makes sense. In any case a new transceiver requires negotiation, so this does not add a negotiation that isn't already needed. Opt-in makes sense too.

I don't really have an opinion then. Why would you want to create a recvonly transceiver though?

[fippo]

@stefhak:

if you create a transceiver using kind = 'video' with direction sendrecv, surely it should start sending once you attach a live video track to its sender?

Ideally yes but this seems to lead to inconsistencies.
replaceTrack says:

Determine if negotiation is needed to transmit withTrack in place of the sender's existing track. Negotiation is not needed if withTrack is null or if the sender's existing track is ended

Do we need negotiation to go from "there was never a track" to "there is a track"?
Note that this is different from "we called replaceTrack with null, then want to replace again without negotiation" (which doesn't seem to be covered by the text quoted above but I think we agree this should not require negotiation).

Essentially this is:

const pc = new RTCPeerConnection();
const transceiver = pc.addTransceiver('audio');
// ... negotiate (so the transceiver becomes associated)
transceiver.sender.replaceTrack(someAudioTrack);

replaceTrack would behave more like addTrack without requiring another O/A cycle(addTrack, step 6). This is different from the use-cases I have seen it used before which are:

• changing the front/back camera
• change between camera and screen content
• stop a video track so the camera light goes off when the user presses the mute button, call GUM again to get a new video track, call replaceTrack. Note that this just stops the track and doesn't call replaceTrack(null) in the first part.

From a different angle, why do we require negotiation for this:

const pc = new RTCPeerConnection();
const transceiver = pc.addTransceiver('audio');
// ... negotiate (so the transceiver becomes associated)
pc.addTrack(someAudioTrack, someStream);

which, presumably, would reuse the transceivers sender.

[henbos]

If a transceiver that has already been negotiated is reused for addTrack(), and addTrack() essentially just does replaceTrack(), I would assume renegotiation is not needed and the spec saying it is would be a spec bug.
But if addTrack() is essentially just doing replaceTrack(), why can addTrack() be sync when replaceTrack() is said to be async?

[henbos]

There's something I don't understand about replaceTrack() and "Determine if negotiation is needed" that is keeping me from coming to a conclusion on this issue.

replaceTrack() only works if the replacing track is compatible with the already negotiated codec settings. Correct? ("Determine if negotiation is needed, if it is, reject")

So now, is a track needed for the O/A cycle to include the encoder settings, or will the O/A look the same regardless of chosen track?

• If a track isn't needed, starting with a null track and doing replaceTrack() makes sense.
• If a track is needed, starting with a null track means we do an O/A cycle without deciding what codec setting to use. And doing replaceTrack() would fail because we have yet to negotiate what encoder settings to use.

// 1. This works
sender = pc.addTrack(track1);
O/A cycle
sender.replaceTrack(track2);

// 2. What about this?
transceiver = pc.addTransceiver(); // transceiver.sender.track is null
O/A cycle
transceiver.sender.replaceTrack(track2); // Do we know if track2 is compatible?

// 3. And this?
transceiver = pc.addTransceiver();
transceiver.sender.replaceTrack(track2);
O/A cycle

I would assume 3) to work the same as 1), but from my understanding of replaceTrack() it would fail without an initial O/A cycle, so 3) would require an additional O/A cycle between addTransceiver() and replaceTrack(). But if a track is needed to negotiate codec settings, that wouldn't work either, and you would have to always use addTrack() or addTransceiver(with a track).

[henbos]

I think it would make sense if replaceTrack() could trigger a renegotiation. If not needed, great, just replace the track, otherwise treat it like an addTrack()-sender in that it doesn't start sending until the O/A cycle has occurred.

If we want it to be able to fail in case renegotation is needed because we don't want black frames until the O/A cycle has completed (in the case of already sending another track), then another operation is needed to say "I want to be able to send this track" such that when that operation completes, including the O/A, you can do replaceTrack() again.

Now if you want to say "I want to send this track" you have to addTrack(), since replaceTrack() will prematurely fail before O/A and doesn't allow you to send tracks requiring different codec settings than the default O/A set of codec settings.

[stefhak]

Going back to https://github.com/w3c/webrtc-pc/issues/1662#issuecomment-349416048, a little testing (https://jsfiddle.net/qw9dw22j/24/) indicates that
pc1.addTransceiver("video", {streams: [stream]}); creates a sendrecv transceiver and an ontrack event, and once replaceTrack() is called with a video track the video will start playing remotely in FF Nightly. All according to the spec as I read it (if the spec should be changed is another story).

[stefhak]

https://github.com/w3c/webrtc-pc/issues/1662#issuecomment-352499721 says:

To recap our understanding (assuming a well-configured pc1.onnegotiationneeded):

1. pc1.addTransceiver('video'); must cause pc2.ontrack to fire, so replaceTrack will work.

However, why should not pc1.ontrack to fire, so replaceTrack done at the remote side would work? (And we can ignore the default direction, I think the question is valid also if you do pc1.addTransceiver('video', {direction: "sendrecv"});)

[stefhak]

https://github.com/w3c/webrtc-pc/issues/1662#issuecomment-354419080 says:

Do we need negotiation to go from "there was never a track" to "there is a track"?

(assuming replaceTrack() is used) I've been arguing that it should not be needed. I also note that the "Advanced Peer-to-peer Example with Warm-up" seems to assume it is not needed.
I do however note that it's not obvious when ontrack should fire (and the spec seems vague in this respect).

[henbos]

A brief chat with @alvestrand cleared up some my confusion, that the intent is to use setParameters() to decide codec params and set direction and renegotiate as to then allow using replaceTrack() without it being rejected.

With such a conscious effort necessary to start sending, I think it makes more sense if by default it is "recvonly".

My interpretation is ontrack only fires in association with receivers being created with a track. That means at SRD() or if you addTransceiver() then no event should fire because you created it yourself.

There is no event for "now we are or are no longer receiving frames on a receiver". Receiving/not receiving frames is not associated with onmute/onunmute.

[stefhak]

I don't care that much what the default is, perhaps "inactive"? Or we let it be "sendrecv".

My interpretation is ontrack only fires in association with receivers being created with a track. That means at SRD() or if you addTransceiver() then no event should fire because you created it yourself.

When I read 4.4.1.6, "Set the RTCSessionDescription", sub bullets 2.8.2.5 and 2.8.2.6 I come to the conclusion:

• On the answerer side, ontrack will fire (as result of SRD) if the offerer side created the transceiver with sendonly or sendrecv
• On the offerer side, ontrack will fire (as result of SRD) if the answerer side created the transceiver (selected for this media description) with sendonly or sendrecv

So it seems possible to get ontrack - even if no tracks are attached - on both ends if the offerer app creates a sendonly transceiver and the answerer creates a sendrecv transceiver (that is picked for the media description).

[taylor-b]

Do we need negotiation to go from "there was never a track" to "there is a track"?

In some cases. The example of when "negotiation is needed" is "a video track differing in raw vs. pre-encoded format." So, going from "there was never a track" to "there's a pre-encoded track" would require negotiation, since it would need to negotiate the pre-encoded format. If I understand correctly.

So now, is a track needed for the O/A cycle to include the encoder settings, or will the O/A look the same regardless of chosen track?

My assumption is that, ignoring this "pre-encoded" case, the O/A will look the same regardless of the properties of the track (even whether it's webcam or screenshare). That's how Chrome currently works.

If we want it to be able to fail in case renegotation is needed because we don't want black frames until the O/A cycle has completed (in the case of already sending another track), then another operation is needed to say "I want to be able to send this track" such that when that operation completes, including the O/A, you can do replaceTrack() again.

That seems pretty complicated... I wonder if this is actually going to be an issue in practice or not. I'm still fuzzy on what things will require renegotiation.

A brief chat with @alvestrand cleared up some my confusion, that the intent is to use setParameters() to decide codec params and set direction and renegotiate as to then allow using replaceTrack() without it being rejected.

I thought setParameters was just for tweaking the knobs of how a track is sent, within the bounds negotiated by the O/A. How exactly does it interact with replaceTrack?

[jan-ivar]

So many conversations here.

From a different angle, why do we require negotiation for this:

const pc = new RTCPeerConnection();
const transceiver = pc.addTransceiver('audio');
// ... negotiate (so the transceiver becomes associated)
pc.addTrack(someAudioTrack, someStream);

which, presumably, would reuse the transceivers sender.

@fippo It would not, because it fails the following test for re-use:

"The sender has never been used to send. More precisely, the [[CurrentDirection]] slot of the RTCRtpTransceiver associated with the sender has never had a value of sendrecv or sendonly."

We decided long ago replaceTrack should never negotiate. I've heard no reason here to reopen that.

Do we need negotiation to go from "there was never a track" to "there is a track"?

In some cases.

To clarify: Not today in any implementation. We've basically absorbed API complexity here to be able to fail gracefully in case of theoretical conflicts with future "pre-encoded" cameras. But again, I'm not hearing any new information on this topic that wasn't discussed 3 years ago.

// 3. And this?
transceiver = pc.addTransceiver();
transceiver.sender.replaceTrack(track2);
O/A cycle

I would assume 3) to work the same as 1), but from my understanding of replaceTrack() it would fail without an initial O/A cycle,

@henbos replaceTrack mustn't fail in this instance. This is poor wording in the spec. The intent of the sentence "Determine if negotiation is needed to transmit withTrack in place of the sender's existing track." and "If negotiation is needed, then reject p" was to protect an already transmitting sender from entering into a failure state. The intent was never to limitreplaceTrack to working only when [[NegotiationNeeded]] == false.

@stefhak I don't have a strong opinion on the default, just pointing out the two competing use-cases.

[fippo]

@jan-ivar

@fippo It would not, because it fails the following test for re-use:

"The sender has never been used to send. More precisely, the [[CurrentDirection]] slot of the
RTCRtpTransceiver associated with the sender has never had a value of sendrecv or sendonly."

That is surprising.

Ignoring CurrentDirection, what audio data is the sender sending after addTransceiver('audio')?
Or taking CurrentDirection into account, does addTransceiver('audio') create a sendrecv offer? With what track in the msid? If there is a track, is RTCRtpSender.track still optional?

We decided long ago replaceTrack should never negotiate. I've heard no reason here to reopen that.

+1

[jan-ivar]

That is surprising.

Sure, though this is easily remedied by avoiding addTrack.

does addTransceiver('audio') create a sendrecv offer?

Yes, the webidl direction = "sendrecv"; makes it indistinguishable from

addTransceiver('audio', {direction: 'sendrecv'})

With what track in the msid?

Random uuid. Try it.

what audio data is the sender sending after addTransceiver('audio')?

Nothing is sent, but the remote end gets a receiver with a silent muted track. Support:

"If track is null then the RTCRtpSender does not send."

"Initialize track.muted to true."

"A muted or disabled MediaStreamTrack renders either silence (audio), black frames (video), or a zero-information-content equivalent."

[fippo]

The sender has never been used to send. More precisely, the [[CurrentDirection]] slot of the
RTCRtpTransceiver associated with the sender has never had a value of sendrecv or sendonly."

and

"If track is null then the RTCRtpSender does not send."

It seems we can have senders which do not send and yet have a [[CurrentDirection]] of sendrecv or sendonly?

@jan-ivar i think you need to write another blog post ;-)

[jan-ivar]

Seems we should avoid vague terms like "send" in the spec.

[stefhak]

From a different angle, why do we require negotiation for this:

const pc = new RTCPeerConnection();
const transceiver = pc.addTransceiver('audio');
// ... negotiate (so the transceiver becomes associated)
pc.addTrack(someAudioTrack, someStream);

which, presumably, would reuse the transceivers sender.

@fippo It would not, because it fails the following test for re-use:

"The sender has never been used to send. More precisely, the [[CurrentDirection]] slot of the RTCRtpTransceiver associated with the sender has never had a value of sendrecv or sendonly."

However, a const transceiver = pc.addTransceiver('audio' {direction="inactive"}); (or "recvonly") transceiver could be used by addTrack, right? Also, if the signaling state has not left stable (i.e. no setLocal since addTransceiver the [[CurrentDirection]] slot would be null.

[stefhak]

My personal summary of all this is:

• The spec is pretty OK as is
• replaceTrack to go from a non existing track to start sending is OK (given it can be done without re-negotiation, which it likely can in almost all cases)
• Maybe the term send is a bit vague, and we should update (but it seems clear to me that we can have sendrecv/sendonly/recvonly transceivers even though no RTP is being sent for the moment)
• The default direction of a transceiver coming from addTransceiver is currently "sendrecv". There may be reasons for changing it. Here we have different options, we could have another default that is always used (like "inactive"), or we could have it depend on whether the first argument to addTransceiver is a track ("sendrecv") or kind ("recvonly").
• The mid question was answered in https://github.com/w3c/webrtc-pc/issues/1662#issuecomment-346770769

What am I missing?

My personal favorite for default direction (if we decide we should change) is 'always set to "inactive"' since the app author would pretty immediately detect it does not work as intended and have to make a conscious choice.

[alvestrand]

January VI conclusion: We leave the default direction as "sendrecv".