React-native-webrtc: New remote tracks with same stream.id (in a=msid) are ignored in iOS (known problem)

[ibc]

I have a react-native-webrtc peerconnection and, initially, call pc.setRemoteDescription(offer) with an offer as follows:

v=0
o=mediasoup-client 56448044 1 IN IP4 0.0.0.0
s=-
t=0 0
a=ice-lite
a=fingerprint:sha-512 2B:19:7E:44:2A:92:E8:49:EE:FA:8E:C5:D3:EC:79:A3:E3:22:A3:E8:B3:C7:7C:9D:A5:6A:F3:F7:B3:BA:50:AF:58:1B:36:96:05:67:D8:6A:EE:F6:89:09:01:E4:98:6B:10:03:75:AD:14:7F:36:A6:6E:FD:9A:49:0A:DB:DF:BC
a=msid-semantic: WMS *
a=group:BUNDLE audio
m=audio 7 RTP/SAVPF 100
c=IN IP4 127.0.0.1
a=rtpmap:100 opus/48000/2
a=fmtp:100 useinbandfec=1
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=setup:actpass
a=mid:audio
a=sendonly
a=ice-ufrag:zzacfqcdz53izo9w
a=ice-pwd:8w3tkz1u8i0rg9ul9xwaxgavp3xc5mve
a=candidate:udpcandidate 1 udp 1082702079 192.168.1.92 40159 typ host
a=end-of-candidates
a=ice-options:renomination
a=ssrc:61428989 msid:recv-stream-61369657 consumer-audio-93108211
a=ssrc:61428989 mslabel:recv-stream-61369657
a=ssrc:61428989 label:consumer-audio-93108211
a=ssrc:61428989 cname:aJ9EdjJjV0S/hP05
a=rtcp-mux
a=rtcp-rsize

As expected, once applied the pc has a single remote stream (pc.getRemoteStreams()[0]) with id recv-stream-61369657. And such a stream has a single audio track with id consumer-audio-93108211.

After that, I run pc.createAnswer() and pc.setLocalDescription(answer).

Later, I call pc.setRemoteDescription(offer) with a new offer as follows:

v=0
o=mediasoup-client 56448044 2 IN IP4 0.0.0.0
s=-
t=0 0
a=ice-lite
a=fingerprint:sha-512 2B:19:7E:44:2A:92:E8:49:EE:FA:8E:C5:D3:EC:79:A3:E3:22:A3:E8:B3:C7:7C:9D:A5:6A:F3:F7:B3:BA:50:AF:58:1B:36:96:05:67:D8:6A:EE:F6:89:09:01:E4:98:6B:10:03:75:AD:14:7F:36:A6:6E:FD:9A:49:0A:DB:DF:BC
a=msid-semantic: WMS *
a=group:BUNDLE audio video
m=audio 7 RTP/SAVPF 100
c=IN IP4 127.0.0.1
a=rtpmap:100 opus/48000/2
a=fmtp:100 useinbandfec=1
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=setup:actpass
a=mid:audio
a=sendonly
a=ice-ufrag:zzacfqcdz53izo9w
a=ice-pwd:8w3tkz1u8i0rg9ul9xwaxgavp3xc5mve
a=candidate:udpcandidate 1 udp 1082702079 192.168.1.92 40159 typ host
a=end-of-candidates
a=ice-options:renomination
a=ssrc:61428989 msid:recv-stream-61369657 consumer-audio-93108211
a=ssrc:61428989 mslabel:recv-stream-61369657
a=ssrc:61428989 label:consumer-audio-93108211
a=ssrc:61428989 cname:aJ9EdjJjV0S/hP05
a=rtcp-mux
a=rtcp-rsize
m=video 7 RTP/SAVPF 101 102
c=IN IP4 127.0.0.1
a=rtpmap:101 VP8/90000
a=rtpmap:102 rtx/90000
a=fmtp:102 apt=101
a=rtcp-fb:101 ccm fir
a=rtcp-fb:101 nack 
a=rtcp-fb:101 nack pli
a=rtcp-fb:101 goog-remb 
a=extmap:2 urn:ietf:params:rtp-hdrext:toffset
a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:4 urn:3gpp:video-orientation
a=setup:actpass
a=mid:video
a=sendonly
a=ice-ufrag:zzacfqcdz53izo9w
a=ice-pwd:8w3tkz1u8i0rg9ul9xwaxgavp3xc5mve
a=candidate:udpcandidate 1 udp 1082702079 192.168.1.92 40159 typ host
a=end-of-candidates
a=ice-options:renomination
a=ssrc:68973725 msid:recv-stream-61369657 consumer-video-82431751
a=ssrc:68973725 mslabel:recv-stream-61369657
a=ssrc:68973725 label:consumer-video-82431751
a=ssrc:68973725 cname:aJ9EdjJjV0S/hP05
a=ssrc:67505726 msid:recv-stream-61369657 consumer-video-82431751
a=ssrc:67505726 mslabel:recv-stream-61369657
a=ssrc:67505726 label:consumer-video-82431751
a=ssrc:67505726 cname:aJ9EdjJjV0S/hP05
a=ssrc-group:FID 68973725 67505726
a=rtcp-mux
a=rtcp-rsize

Here we have a new remote track (of kind "video" and track.id: "consumer-video-82431751") within the same remote stream (`stream.id: "recv-stream-61369657").

BUT: If I iterate now the remote streams of the peerconnection:

• I get a single remote stream with id "recv-stream-61369657" (this is correct).
• The remote stream STILL has a single track (the audio track inserted above).
• So the new video track is NOT available in the remote stream.

Expected behavior

After applying the second remote offer, the single remote stream should contain TWO tracks (audio and video).

Observed behavior

After applying the second remote offer, the single remote stream JUST contains the first track (the audio one).

Platform information

• React Native version: master
• Plugin version: 1.58.3
• OS: iOS Simulator

Comments

I think this is related to how react-native-webrtc handles remote streams. AFAIS it ignores new tracks additions within an existing remote stream. Probably this is 100% related to an issue in the libwebrtc ObjC wrapper I already dealt with in the past:

• https://bugs.chromium.org/p/webrtc/issues/detail?id=4640
• https://groups.google.com/forum/#!topic/discuss-webrtc/FYJwOTJJY-U
• https://github.com/BasqueVoIPMafia/cordova-plugin-iosrtc/issues/4

In fact, I did a patch in libwebrtc for this (that fixes the issue) but was not approved by the libwebrtc team (who knows why): https://webrtc-codereview.appspot.com/50109004

Should I assume that the same issue happens here, so react-native-webrtc (when in iOS) cannot deal with track addition/removal on an already existing remote stream, right?

[zxcpoiu]

Awesome, thanks for detailed explanation. I did learn a lot. :+1:
This is a bit beyond my knowledge and will need time to investigate.
Maybe jitsi guys can help. They have some additional fixes in their branch. I will find time to do a rebase.

[ibc]

I've seen that they (Jitsi) stores remote streams/tracks on the PeerConnection but, unless they have a patched version of libwebrtc, they still suffer from the same issue since it is a limitation in the libwebrtc's ObjC wrapper.

[ibc]

To clarify, the only way to fix this limitation is by patching the ObjC wrapper of libwebrtc.

[saghul]

@ibc AFAICT you're right. There is not much we can do, however, other than raising the issue again. Keeping involved patches on top of libwebrtc is not something I'm looking forward to. Been there, done that :-P

[ibc]

I feel your pain. Managing patches or a fork of libwebrtc is not pleasant at all.

[saghul]

Update: the WebRTC objc wrapper got some new goodies and this should now be within reach. In fact there is an open PR with the fix: https://github.com/react-native-webrtc/react-native-webrtc/pull/564

Alas, not all the glitter is gold: there is no such API for Android AFAICT.

I'll consider landing the PR after the next release. After all, being correct 50% of the times is better than being correct 0% of the times. But consistency is nice. Hum.