Mixedreality-webrtc: [Android] Infinite wait in SetRemoteDescriptionAsync()

[JWSon]

Hello,

I've built the android tools as instruction on master and created the libwebrtc.aar and mrwebrtc.aar.
Webrtc initialization was successful, but after receiving a SDP data from server the SetRemoteDescriptionAsync() is waiting indefinitely.

Return value of PeerConnection_SetRemoteDescriptionAsync() on PeerConnectionInterop.cs is Utils.MRS_SUCCESS, but args.completedEvent.Wait() of RemoteDescArgs still waits.
I guess it is related to synchronize transceiver in native c++ since RemoteDescArgs callback method is called on SetRemoteSessionDescObserver.

the build summary is as follows.

1. built libwebrtc.aar on Hyper-V Ubuntu 18.04
2. built mrwebrtc.aar on Windows using Android Studio.
   
   
   
   • added {mr-webrtc-native-dir}/src/utils.cpp
   
   
3. built Microsoft.MixedReality.WebRTC.dll on Visual studio 2019
4. made Unity project and configured lib and gameobject like user guide.
5. changed NodeDss to Websocket (Newtonsoft)

It would be appreciated if you would give some comments.

[djee-ms]

My guess is that the remote description fails to apply. Do you have access to the Visual Studio Output window and can see the debug logs printed by the native implementation? If so, can you please try to find the line "Remote description set. err=" and see if there's an error message? I see that the callback is not invoked if there's an error, which effectively makes SetRemoteDescriptionAsync() deadlock, we should fix that. However this only happens if the remote description does not apply correctly, so you already have some other issue here I think.

[djee-ms]

I think #320 should fix the deadlock, if you want to try it locally while the review is pending to confirm that it works for you.

[JWSon]

Thank you for your kind information.

I've applied #320 sources and added try-catch statement to SetRemoteDescriptionAsync().
now I can check a result and error message of RemoteDescriptionApplied callback. like your above comment, it seems to not apply the remote description correctly.
the result and error message are "MRS_E_INVALID_PARAMETER" and "Failed to set remote offer sdp: Failed to set remote video description send parameters".
I am suspicious about unified plan and plan b of sdp semantics.
the sdp data which is generated from my server is as follows.

v=0
o=- 8887904561871063630 0 IN IP4 0.0.0.0
s=-
t=0 0
a=ice-options:trickle
a=group:BUNDLE video0 application1
m=video 9 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=setup:actpass
a=ice-ufrag:1ifI5e7AnsFz6Ini93R//EQzPWFF1Fsq
a=ice-pwd:kcDYUHZw/GTbiTMt0vOd4dsZn9jogHyV
a=rtcp-mux
a=rtcp-rsize
a=sendrecv
a=rtpmap:96 H264/90000
a=rtcp-fb:96 nack pli
a=fmtp:96 packetization-mode=1;profile-level-id=42c01e;sprop-parameter-sets=Z0LAHtoCAEGwFqDAILSgAAADAC7msoAB4sXU,aM48gA==
a=ssrc:1836819091 msid:user500824942@host-47009fe3 webrtctransceiver0
a=ssrc:1836819091 cname:user500824942@host-47009fe3
a=mid:video0
a=fingerprint:sha-256 F9:59:6F:AA:87:A3:4C:24:F9:66:2A:E5:2D:B5:34:01:08:E5:0A:F4:C3:A4:E2:BA:1E:39:48:D2:C6:02:3C:45
m=application 0 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=setup:actpass
a=ice-ufrag:1ifI5e7AnsFz6Ini93R//EQzPWFF1Fsq
a=ice-pwd:kcDYUHZw/GTbiTMt0vOd4dsZn9jogHyV
a=bundle-only
a=mid:application1
a=sctp-port:5000
a=fingerprint:sha-256 F9:59:6F:AA:87:A3:4C:24:F9:66:2A:E5:2D:B5:34:01:08:E5:0A:F4:C3:A4:E2:BA:1E:39:48:D2:C6:02:3C:45

[djee-ms]

H.264 is not supported outside UWP, you need to use a different codec like VP8 or VP9.

[JWSon]

I thought since H.264 codec worked when I implemented webrtc lib in an Android native app, so it would work in Unity as well.
Does Unity with libwebrtc.aar need some installation elements to use H.264 hardware decoding?

[djee-ms]

The codec implementation is only compiled in the UWP build.
https://github.com/webrtc-uwp/webrtc-apis/blob/0d303d62f7e6483cf4b57c24ef295c487b0c28dd/windows/wrapper/impl_org_webRtc_WebRtcFactory.cpp#L343

[JWSon]

Thanks for the link.
Codec support seems to be related to peerConnectionFactory.
I remember setting the peerConnectionFactory using DefaultVideoDecoderFactory on Android native app.
what I understood is that since libwebrtc.aar does not have en/decoder factory like the above DefaultVideoDecoderFactory, so I need to install a separate warped Peerconnectionfatory module to support specific codec.
If this is correct, should I create a separate factory using open source such as ffmpeg?

[djee-ms]

You can indeed inject your own codec factory if you modify the code, which is the Google API's way to add actual support for pre-supported codecs, that is essentially H.264; you cannot add any random codec, the RTP implementation needs to exist for it before you can add the actual en/decoder. So you can do that but that's a non-negligible amount of work I think, and not necessarily trivial.

An non-officially-supported alternative if you feel adventurous is also to recompile Google's code (see Building the Core dependencies from sources) with built-in support for OpenH264 to get software-based en/decoding, which is disabled by default due to licensing issues. The way you do that (and I never tried, so no guarantee it works) is to add the rtc_use_h264=true flag to the Google build options in _e.g._ external\webrtc-uwp-sdk\webrtc\xplatform\webrtc\out\webrtc_win_x64_debug\args.gn. Unfortunately the WebRTC UWP overwrites args.gn each time you build with its Visual Studio build project or build script run.py, which makes changing args.gn quite the pain (easiest way is to change the template in external\webrtc-uwp-sdk\webrtc\windows\templates\gns\args.gn).

Disclaimer though, in both cases we won't provide much support for that route, this is way out of the things we want to support. We do plan to eventually add H.264 support for Windows Desktop via a different rewritten video capture module which unifies the Desktop and UWP codepaths, and there has been no talk about Android H.264 support so far (we don't even have video capture at this time yet).

[JWSon]

Thanks for the answer. You have inspired me a lot about supported codecs on webrtc.
The issue of infinite loading has been resolved, and I will test with a different codec.