Mixedreality-webrtc: H264/H265 codec is not forced in Unity

[eirikhollis]

When initializing a WebRTC connection in Unity and specifying the video codec to be "H264", the local SDP is not changed and will still priotize VP8 (96) instead of H264.
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 127 124 125

On UWP, both on desktop and HoloLens, the H264 works and the SDP is changed to
m=video 9 UDP/TLS/RTP/SAVPF 100

So the issue is only in Unity. As far as I know, this is also the case for H265.

[djee-ms]

Currently on Unity:

• The PreferredVideoCodec property from the combo box in the Editor is on the LocalVideoSource component.
  https://github.com/microsoft/MixedReality-WebRTC/blob/766d27e799e474ed784488f0f3dc937ab0ad3f7b/libs/Microsoft.MixedReality.WebRTC.Unity/Assets/Microsoft.MixedReality.WebRTC.Unity/Scripts/Media/LocalVideoSource.cs#L77
• It is assigned to the peer connection when the track is created, so when calling LocalVideoSource::AddLocalVideoTrackImpl()
  https://github.com/microsoft/MixedReality-WebRTC/blob/766d27e799e474ed784488f0f3dc937ab0ad3f7b/libs/Microsoft.MixedReality.WebRTC.Unity/Assets/Microsoft.MixedReality.WebRTC.Unity/Scripts/Media/LocalVideoSource.cs#L250-L253
• The SDP filtering is applied before the SDP message is passed to the ready event for delivery by the signaling solution.
  https://github.com/microsoft/MixedReality-WebRTC/blob/766d27e799e474ed784488f0f3dc937ab0ad3f7b/libs/Microsoft.MixedReality.WebRTC/PeerConnection.cs#L1307-L1312

So I assume you are not starting the local video before creating the offer from Unity, but somehow still adds a video track? What is the use case that doesn't work with this sequence of actions?

[eirikhollis]

Doublechecked the code now, and the process it goes through is

1. Create peer connection and assign codecs
2. Initialize local media (video/audio)
3. Create offer and subscribe to Local SDP offer event
4. Local SDP offer is raised and then i write out the values I pasted above

Same code in Unity, desktop and HL

[djee-ms]

When you mean same code on Unity, Desktop, and HoloLens, is it via Unity deployment always, or do you use also directly the C# library without Unity?

Also in step 1. where is "assign codecs" done? On the C# PeerConnection or on the Unity LocalVideoSource? Because the latter will override the former as explained in the above sequence.

[eirikhollis]

Same code as via Unity deployment.

Codecs are assigned like this

PeerConnection = new PeerConnection
{
   PreferredVideoCodec = videoCodec,
   PreferredAudioCodec = audioCodec
};

I don't use the LocalVideoSource object, but rather a custom variant of it that does not set any codecs.

[djee-ms]

So it looks like you're using directly the C# library here to create a C# PeerConnection object (not a PeerConnection Unity component). And once those values are assigned, they are used automatically:
https://github.com/microsoft/MixedReality-WebRTC/blob/766d27e799e474ed784488f0f3dc937ab0ad3f7b/libs/Microsoft.MixedReality.WebRTC/PeerConnection.cs#L1307-L1312

I'm afraid I don't see how that can be specific to Unity then, nor even how those values are not taken into account in the SDP offer. Can you try breaking into the above code, see if it does indeed filter the SDP message and what it returns before invoking the LocalSdpReadyToSend event? You should see the SDP message in the sdp variable.

[eirikhollis]

Sorry, but I'm gonna have to resume this on monday due to time constraints, but what I know is that I pass "H264" into PreferredVideoCodec on one side and nothing is modified (with regards to video) in the SDP on the other side. It is however modified if I insert "VP8" or "VP9".
I'll step through the DLL on monday.

[Daniel4144]

Same issue, I can reproduce it with the VideoChat example (connection between Unity editor and Hololens):
"PreferredVideoCodec = None": works in both directions (I think VP8 is used)
"PreferredVideoCodec = H264": the offer SDP does not change when creating the offer in the Editor. It does change, when the offer is created on the Hololens (however the Editor does not send an answer SDP).

[djee-ms]

Sorry I think I overlooked one major point in the original issue : you are trying to use H.264 on Windows Desktop, but it is not currently available. So in particular in the Unity Editor (x64 Desktop process) any filtering will not work because the original SDP offer already doesn't contain any H.264 codec, and if receiving an offer with H.264 alone then no answer is possible because the Windows Desktop peer doesn't have any support for that codec.

The reason is that the H.264 encoder/decoder code is only for UWP. This is a combination of 2 issues:

• Google on other platforms uses OpenH264, which is excluded on Windows due to licensing issue I think, or something along that line. So by default on Windows the Google implementation has no support for H.264.
• The Windows-specific H.264 encoder/decoder provided by WebRTC UWP is specific to UWP. To be honest I don't know of any good reason for this choice, but I am lacking context so may be overlooking some issue.

@eirikhollis your original post does not show H.264 on Desktop; it only shows the unfiltered list of codecs. You need to look at the a=rtpmap to find which codec corresponds to which number. This association is dynamic. I tried again just now and H.264 is nowhere in that SDP offer generated by the Unity editor peer.

Currently if you want to use H.264 with HoloLens to avail of the hardware encoder, you need a UWP app on the other side too. Otherwise you need to fall back on another codec like software VP8.

[djee-ms]

Regrettably I appreciate this is not mentioned in the README, and the v1.0.0-rc1 release notes make that even more confusing by saying that H.264 software is supported. I will fix that.

Please feel free to open a separate issue for a feature request if this is something you need, and we can discuss there the feasibility of it.

[Daniel4144]

Thanks for the info @djee-ms.
Of course H264 encoding on the Hololens would be nice (performance-wise for hardware encoding), but I can stick to the VP8-Codec.

[eirikhollis]

@djee-ms You're right, H264 does indeed not pop up in the sdp.
Thanks for the information, guess it'll be VP8 in Unity then

[djee-ms]

Closing this issue as invalid then, since it's "by design" (whether good or bad).

[iamrohit1]

• Google on other platforms uses OpenH264, which is excluded on Windows due to licensing issue I think, or something along that line. So by default on Windows the Google implementation has no support for H.264.

Hey,
I just modified h264_encoder_impl to replace OpenH264 with nvpipe. But when I compile and execute for windows x64 it is not used.. its again defaulting to vp8.. is there a way to force it to use h264?
p.s: sorry if this is not the right place to ask this..

[djee-ms]

webrtc/modules/video_coding/codecs/h264/h264_encoder_impl is not used, it's not even compiled as far as I know. On UWP we use the webrtc-uwp-sdk implementation based on Media Foundation and leveraging hardware encoding where available, and there's no implementation on Desktop. The Google implementation you are referring to uses OpenH264, which entitles you as a user to pay royalties to MPEG LA if you use it, so it is disabled by default by Google. You are free to try, but we will not offer any support for it.

[iamrohit1]

Okay, yeah I've been working on the chromium m71. I realize this is not something related to this repo and appreciate the help! Thank you!