Mixedreality-webrtc: mrwebrtc aborts when not a number mid is specified in SDP message

[tarukosu]

Describe the bug
Mrwebrtc aborts when not a number mid is specified in SDP message.

To Reproduce
Initialize PeerConnection and call HandleConnectionMessageAsync with the following offer SDP message.

await PeerConnection.HandleConnectionMessageAsync("offer", sdp);

SDP messages

Received offer message


v=0
o=test 1126184766368979219 0 IN IP4 0.0.0.0
s=-
t=0 0
a=sendrecv
a=fingerprint:sha-256 5F:02:6F:8D:D6:14:E9:03:CB:26:34:1F:F8:E2:3C:0E:E4:E1:53:DD:03:4F:1C:66:74:C0:38:04:EE:24:A0:73
a=group:BUNDLE 0 video
a=msid-semantic:WMS *
a=ice-options:trickle
m=audio 9 UDP/TLS/RTP/SAVPF 109
c=IN IP4 0.0.0.0
a=candidate:0 1 UDP 2114002687 52.246.167.126 54535 typ host
a=end-of-candidates
a=sendonly
a=ice-ufrag:aPqwjAoLRbpC
a=ice-pwd:ndjAY9W3HYwcv8UwEnqTC681gxNXMsdynxFY
a=mid:0
a=msid:8d677918-ac9b-477a-a3d5-4c23e5557780 663b768e-20f3-41e3-9f64-7a8b5e26c632
a=rtcp-rsize
a=rtcp-mux
a=rtpmap:109 opus/48000/2
a=fmtp:109 maxplaybackrate=48000;stereo=1;sprop-stereo=1;minptime=10;ptime=20;useinbandfec=1;usedtx=0
a=setup:actpass
a=ssrc:1691444826 cname:cf8336ca-0b8f-4f73-a1b8-919f29be4228
m=video 9 UDP/TLS/RTP/SAVPF 120 96
c=IN IP4 0.0.0.0
a=candidate:0 1 UDP 2114002687 52.246.167.126 54535 typ host
a=end-of-candidates
a=sendonly
a=ice-pwd:ndjAY9W3HYwcv8UwEnqTC681gxNXMsdynxFY
a=ice-ufrag:aPqwjAoLRbpC
a=mid:video
a=msid:8d677918-ac9b-477a-a3d5-4c23e5557780 fee47fcf-662a-403d-9d98-0aacd527caea
a=rtcp-rsize
a=rtcp-mux
a=rtpmap:120 VP8/90000
a=fmtp:120 max-fs=12288;max-fr=60
a=rtcp-fb:120 ccm fir
a=rtcp-fb:120 nack
a=rtcp-fb:120 nack pli
a=rtpmap:96 rtx/90000
a=fmtp:96 apt=120
a=setup:actpass
a=ssrc:3612539453 cname:cf8336ca-0b8f-4f73-a1b8-919f29be4228


Environment

• Platform: Unity Editor
• Architecture: x64
• Unity version (if applicable): 2018.4.15f

Additional context
mrwebrtc debug log

(messagequeue.cc:513): Message took 2941ms to dispatch. Posted from: Microsoft::MixedReality::WebRTC::detail::ExternalVideoTrackSourceImpl::OnMessage@d:\webrtc2\webrtcmodule\external\mixedreality-webrtc\libs\mrwebrtc\src\media\external_video_track_source.cpp:278
(p2ptransportchannel.cc:455): Set backup connection ping interval to 25000 milliseconds.
(p2ptransportchannel.cc:464): Set ICE receiving timeout to 2500 milliseconds
(p2ptransportchannel.cc:471): Set ping most likely connection to 0
(p2ptransportchannel.cc:478): Set stable_writable_connection_ping_interval to 2500
(p2ptransportchannel.cc:491): Set presume writable when fully relayed to 0
(p2ptransportchannel.cc:500): Set regather_on_failed_networks_interval to 300000
(p2ptransportchannel.cc:519): Set receiving_switching_delay to 1000
(dtlssrtptransport.cc:60): Setting RTCP Transport on 0 transport 0
(dtlssrtptransport.cc:65): Setting RTP Transport on 0 transport 1976a020
(p2ptransportchannel.cc:455): Set backup connection ping interval to 25000 milliseconds.
(p2ptransportchannel.cc:464): Set ICE receiving timeout to 2500 milliseconds
(p2ptransportchannel.cc:471): Set ping most likely connection to 0
(p2ptransportchannel.cc:478): Set stable_writable_connection_ping_interval to 2500
(p2ptransportchannel.cc:491): Set presume writable when fully relayed to 0
(p2ptransportchannel.cc:500): Set regather_on_failed_networks_interval to 300000
(p2ptransportchannel.cc:519): Set receiving_switching_delay to 1000
(dtlssrtptransport.cc:60): Setting RTCP Transport on video transport 0
(dtlssrtptransport.cc:65): Setting RTP Transport on video transport 1976a800
(p2ptransportchannel.cc:403): Received remote ICE parameters: ufrag=aPqwjAoLRbpC, renomination disabled
(p2ptransportchannel.cc:403): Received remote ICE parameters: ufrag=aPqwjAoLRbpC, renomination disabled
(peerconnection.cc:2730): Adding audio transceiver for MID=0 at i=0 in response to the remote description.
(webrtcvoiceengine.cc:1471): Setting voice channel options: AudioOptions {audio_jitter_buffer_max_packets: 50, audio_jitter_buffer_fast_accelerate: 0, }
(webrtcvoiceengine.cc:316): WebRtcVoiceEngine::ApplyOptions: AudioOptions {audio_jitter_buffer_max_packets: 50, audio_jitter_buffer_fast_accelerate: 0, }
(webrtcvoiceengine.cc:489): NetEq capacity is 50
(webrtcvoiceengine.cc:495): NetEq fast mode? 0
(webrtcvoiceengine.cc:513): Delay agnostic aec is enabled? 0
(webrtcvoiceengine.cc:523): Extended filter aec is enabled? 0
(webrtcvoiceengine.cc:533): Experimental ns is enabled? 0
(audio_processing_impl.cc:688): Highpass filter activated: 1
(fixed_gain_controller.cc:60): Gain to apply: 0 db.
(audio_processing_impl.cc:702): Gain Controller 2 activated: 0
(audio_processing_impl.cc:704): Pre-amplifier activated: 0
(webrtcvoiceengine.cc:1489): Set voice channel options. Current options: AudioOptions {audio_jitter_buffer_max_packets: 50, audio_jitter_buffer_fast_accelerate: 0, }
(channel.cc:114): Created channel for 0
(call.cc:1047): UpdateAggregateNetworkState: aggregate_state=down
(send_side_congestion_controller.cc:308): SignalNetworkState Down
(peerconnection.cc:2730): Adding video transceiver for MID=video at i=1 in response to the remote description.
(webrtcvideoengine.cc:493): CreateChannel. Options: VideoOptions {}
(channel.cc:114): Created channel for video
(call.cc:1047): UpdateAggregateNetworkState: aggregate_state=down
(send_side_congestion_controller.cc:308): SignalNetworkState Down
(paced_sender.cc:272): Elapsed time (3401 ms) longer than expected, limiting to 2000 ms
(peerconnection.cc:3512): Session: 893646457386138206 Old state: kStable New state: kHaveRemoteOffer
(channel.cc:838): Setting remote voice description
(webrtcvoiceengine.cc:1293): WebRtcVoiceMediaChannel::SetSendParameters: {codecs: [AudioCodec[109:opus:48000:0:2]], extensions: [], max_bandwidth_bps: -1, mid: 0, options: AudioOptions {}}
(webrtcvoiceengine.cc:2084): WebRtcVoiceMediaChannel::SetMaxSendBitrate.
(webrtcvoiceengine.cc:1471): Setting voice channel options: AudioOptions {}
(webrtcvoiceengine.cc:316): WebRtcVoiceEngine::ApplyOptions: AudioOptions {audio_jitter_buffer_max_packets: 50, audio_jitter_buffer_fast_accelerate: 0, }
(webrtcvoiceengine.cc:489): NetEq capacity is 50
(webrtcvoiceengine.cc:495): NetEq fast mode? 0
(webrtcvoiceengine.cc:513): Delay agnostic aec is enabled? 0
(webrtcvoiceengine.cc:523): Extended filter aec is enabled? 0
(webrtcvoiceengine.cc:533): Experimental ns is enabled? 0
(audio_processing_impl.cc:688): Highpass filter activated: 1
(fixed_gain_controller.cc:60): Gain to apply: 0 db.
(audio_processing_impl.cc:702): Gain Controller 2 activated: 0
(audio_processing_impl.cc:704): Pre-amplifier activated: 0
(webrtcvoiceengine.cc:1489): Set voice channel options. Current options: AudioOptions {audio_jitter_buffer_max_packets: 50, audio_jitter_buffer_fast_accelerate: 0, }
(webrtcvoiceengine.cc:1833): AddRecvStream: {id:663b768e-20f3-41e3-9f64-7a8b5e26c632;ssrcs:[1691444826];ssrc_groups:;cname:cf8336ca-0b8f-4f73-a1b8-919f29be4228;stream_ids:8d677918-ac9b-477a-a3d5-4c23e5557780;}
(neteq_impl.cc:114): NetEq config: sample_rate_hz=16000, enable_post_decode_vad=true, max_packets_in_buffer=50, enable_fast_accelerate=false, enable_muted_state= true
(audio_coding_module.cc:451): Created
(audio_receive_stream.cc:106): AudioReceiveStream: 1691444826
(audio_receive_stream.cc:327): AudioReceiveStream::ConfigureStream: {rtp: {remote_ssrc: 1691444826, local_ssrc: 4195875351, transport_cc: off, nack: {rtp_history_ms: 0}, extensions: []}, rtcp_send_transport: (Transport), sync_group: 8d677918-ac9b-477a-a3d5-4c23e5557780}
(call.cc:1047): UpdateAggregateNetworkState: aggregate_state=down
(send_side_congestion_controller.cc:308): SignalNetworkState Down
(channel.cc:656): Add remote ssrc: 1691444826
(channel.cc:779): Changing voice state, recv=0 send=0
(channel.cc:973): Setting remote video description
(webrtcvideoengine.cc:686): SetSendParameters: {codecs: [VideoCodec[120:VP8], VideoCodec[96:rtx]], conference_mode: no, extensions: [], max_bandwidth_bps: -1, mid: video}
(webrtcvideoengine.cc:695): Using codec: VideoCodec[120:VP8]
(webrtcvideoengine.cc:744): SetFeedbackOptions on all the receive streams because the send codec or RTCP mode has changed.
(webrtcvideoengine.cc:1137): AddRecvStream: {id:fee47fcf-662a-403d-9d98-0aacd527caea;ssrcs:[3612539453];ssrc_groups:;cname:cf8336ca-0b8f-4f73-a1b8-919f29be4228;stream_ids:8d677918-ac9b-477a-a3d5-4c23e5557780;}
(video_receive_stream.cc:119): VideoReceiveStream: {decoders: [{decoder: (VideoDecoder), payload_type: 96, name: VP8, codec_params: {}}, {decoder: (VideoDecoder), payload_type: 98, name: VP9, codec_params: {x-google-profile-id: 0}}, {decoder: (VideoDecoder), payload_type: 100, name: multiplex, codec_params: {acn: VP9x-google-profile-id: 0}}], rtp: {remote_ssrc: 3612539453, local_ssrc: 1, rtcp_mode: RtcpMode::kReducedSize, rtcp_xr: {receiver_reference_time_report: off}, remb: off, transport_cc: off, nack: {rtp_history_ms: 1000}, ulpfec_payload_type: 104, red_type: 102, rtx_ssrc: 0, rtx_payload_types: {97 (pt) -> 96 (apt), 99 (pt) -> 98 (apt), 101 (pt) -> 100 (apt), 103 (pt) -> 102 (apt), }, extensions: []}, renderer: (renderer), render_delay_ms: 10, sync_group: 8d677918-ac9b-477a-a3d5-4c23e5557780, target_delay_ms: 0}
(call.cc:1047): UpdateAggregateNetworkState: aggregate_state=down
(send_side_congestion_controller.cc:308): SignalNetworkState Down
(channel.cc:656): Add remote ssrc: 3612539453
(channel.cc:909): Changing video state, send=0
(peerconnection.cc:4763): Local and Remote descriptions must be applied to get the SSL Role of the SCTP transport.
(peerconnection.cc:2362): Processing the addition of a new track for MID=0 (added to streams=[8d677918-ac9b-477a-a3d5-4c23e5557780]).
(webrtcvoiceengine.cc:1942): SetOutputVolume() to 1 for recv stream with ssrc 1691444826
(peerconnection.cc:2362): Processing the addition of a new track for MID=video (added to streams=[8d677918-ac9b-477a-a3d5-4c23e5557780]).
(webrtcvideoengine.cc:1267): SetSink: ssrc:3612539453 (ptr)
(peer_connection.cpp:878): Added transceiver mid=#0 of type 'audio' with desired direction kRecvOnly
(peer_connection.cpp:887): Send with NULL track
(peer_connection.cpp:894): Recv with track #663b768e-20f3-41e3-9f64-7a8b5e26c632 enabled=true
(peer_connection.cpp:858): Added receiver #663b768e-20f3-41e3-9f64-7a8b5e26c632 of type 0
(peer_connection.cpp:861): + Track #663b768e-20f3-41e3-9f64-7a8b5e26c632 with stream #8d677918-ac9b-477a-a3d5-4c23e5557780
(paced_sender.cc:272): Elapsed time (7078 ms) longer than expected, limiting to 2000 ms
(messagequeue.cc:513): Message took 2203ms to dispatch. Posted from: cricket::BaseChannel::SendPacket@../../pc/channel.cc:388
(paced_sender.cc:272): Elapsed time (2133 ms) longer than expected, limiting to 2000 ms
(peer_connection.cpp:878): Added transceiver mid=#video of type 'video' with desired direction kRecvOnly
(peer_connection.cpp:887): Send with NULL track
(peer_connection.cpp:894): Recv with track #fee47fcf-662a-403d-9d98-0aacd527caea enabled=true
(peer_connection.cpp:858): Added receiver #fee47fcf-662a-403d-9d98-0aacd527caea of type 1
(peer_connection.cpp:861): + Track #fee47fcf-662a-403d-9d98-0aacd527caea with stream #8d677918-ac9b-477a-a3d5-4c23e5557780
(paced_sender.cc:272): Elapsed time (2203 ms) longer than expected, limiting to 2000 ms
スレッド 0x3230 はコード 0 (0x0) で終了しました。
(paced_sender.cc:272): Elapsed time (7492 ms) longer than expected, limiting to 2000 ms
Debug Error!

Program: ...eality.WebRTC.Unity.Plugins\Plugins\Win32\x86_64\mrwebrtc.dll

abort() has been called

[djee-ms]

Hi @tarukosu,

• How do you know that the error is related to "mid"? I don't see anything in the logs.
• Are you using Unified Plan (default) or Plan B for the SDP semantic?
• The SDP answer message looks wrong; maybe it is a copy/paste error? There is a a=sendrecv on the first line, but there is no m= line before it, which is not valid according to the SDP specification. If this is indeed the case, who generated this message? Because I am almost sure the Google implementation we use would not generate that, and would indeed fail to parse this message.

[tarukosu]

• When I replace video to 1 in the SDP message, the error didn't happen.
• I'm using Unified Plan.
• I'm using SFU Server Sora (https://sora.shiguredo.jp/). Sora generates the message.
  I'll ask the developer of Sora about a=sendrecv line.

[tarukosu]

The developer of Sora says that this is correct message and SDP generated by Firefox also contains a=sendrecv line.

https://gist.github.com/voluntas/c4504f219f233c2eece3131ab8b8830d

[djee-ms]

@tarukosu sorry I am confused, I had written a reply to you but cannot see it here, I am afraid I forgot to press Comment. In any case yes you are right the SDP message is correct, I checked the SDP specification. And the issue is that we assumed that the mline index was the same as the mid, which is true for Google's implementation but not true in general (not in the standard). I opened #328 to fix this, and I believe that should solve your issue.

[djee-ms]

Fix was merged.

[tarukosu]

@djee-ms Thank you so much!
I've tried 8ceb505b16101b59d3c0741efc588c013a20247b and found the problem is solved.