Mixedreality-webrtc: Video Stream not rendered on HoloLens 2

[rahulk131]

Hello,

We have modified the TestAppUwp slightly to test a particular use case. In this use case the modified TestAppUwp is deployed on a Hololens 2 device and will have an audio video chat with another WebRTC client (which we've built) on desktop.

We are able to successfully have an audio and video chat when we run the TestAppUwp on a Windows laptop running Windows 10. However, when we deploy this exact same TestAppUwp to a HoloLens 2 device, Video is neither sent nor received (we don't see the video in both clients). We are using H264 as our video codec. On the HoloLens 2, the audio works fine and we're able to have a successful Audio chat.

How do we go about debugging this issue? Any pointers that we can look at to troubleshoot?

Thanks,
Rahul.

[djee-ms]

Can you have a look and/or paste the logs from the Output window of Visual Studio running in Debug config? The WebRTC implementation is printing a lot of information about what it's doing, and that should help understand why the camera is not opening. Also make sure you try without specifying any capture width/height/framerate/video profile to ensure that it tries with the default resolution and framerate, in case this is the issue. Can you see the video with the default unmodified TestAppUWP? If so, what kind of changes did you do?

[rahulk131]

Thanks for your response.

Yes, we can see video with the default unmodified TestAppUWP (running on a desktop; we haven't tested this on a Hololens 2 yet). We can also see video in the modified TestAppUWP when it is run on a laptop. Where video fails is when we run the modified TestAppUWP on the Hololens 2 device (with exactly the same defaults as on the laptop instance).

On the Hololens 2 device, in the "Tracks" tab of the app, we can see "Local Video Channel" when we hit the "Play" button. We never see the Remote Video Channel even after the call has started (audio works just fine)

However, we do see the following around the Remote video channel:
State: Opening
Receive: 0.03
Render: Infinity
Skip: 14.25

Do these help?

The modifications we've made mainly relate to:

1. signalling
2. Instead of the TestAppUWP talking to another instance of itself, in our modification the TestAppUWP talks to another webRTC client that responds with an remoteSDP and ICE candidates. These are trickled into the TestAppUWP as was done in the unmodified version.

[djee-ms]

"Skip: 14.25" means the frames are being received over the network but not pulled by the renderer, so they get discarded.
"State: Opening" seems to mean that the renderer didn't open yet.

So it might be an issue with the Media Foundation renderer on that particular TestAppUWP, and not an issue with the MixedReality-WebRTC library. One thing I can think of is, are you using H.264 for anything else than TestAppUWP? There's a limited number of work that the hardware encoder might pick at a time, though it should fallback to the software implementation in theory so would be slow but visible. Other than that, I know there's a problem with that renderer, sometimes it takes several seconds or more to start; I never observed it on the remote track, but starting and stopping the local video several times will show you some black screen on the local video for several seconds until the render pipeline finally starts. Maybe this is the same issue? How long did you wait until deciding that the video was not visible? I'd wait at least a couple of minutes to be sure.

[rahulk131]

Thanks Jerome. Will look at some of these options. and get back.

[dl4mmers]

Hello!

We have a similar issue with HoloLens 2 using H.264, while on HoloLens 1 everything works as expected. At the moment we are not using MixedReality-WebRTC, but the WebRTC-UWP-SDK and since MixedReality-WebRTC is based on WebRTC-UWP-SDK (as far as I know), I hope it is okay that I ask here. Unfortunately there is not much activity in WebRTC-UWP-SDK repo and maybe I can get help here.

We are using the latest nuget package v.1.71.0.9 of WebRTC-UWP-SDK in a DLL, used by our Unity-MixedReality-App, to establish audio and video calls with other clients in a browser.

When using HoloLens 1, we can successfully receive the HL video in the browser using VP8, VP9 and H.264 codec. When using HoloLens 2, the HL video is only showing up, when we are using VP8 or VP9 codecs. When selecting H.264, the HL video in the browser is not showing up (black screen). After the call got established with HL 2, the white LED next to HL 2 RGB-Camera is activated and also the performance is going down a bit, from constant 60 FPS to variable 45 - 60 FPS, so it seems that there is a running H.264 encoder, doing something. I also waited several minutes for the video to show up (as you mentioned in your last post), but nothing happened. I also selected different video resolutions.

We already checked possible problems with signalling, but everything looks good and local and remote SDPs on HL1 and HL2 are the same, so we think it must be a problem with WebRTC-UWP-SDK and HL2. There are also other issues, for example missing video profile selection, and since WebRTC-UWP-SDK v.1.71.0.9 is almost a year old (where HL2 wasn't released yet) and because there is no newer version, we are thinking about switching to MixedReality-WebRTC.

Is this maybe a known problem with old WebRTC-UWP-SDK, that was fixed for MixedReality-WebRTC?
Was H.264 on HL 2 tested with MixedReality-WebRTC?

[djee-ms]

Unfortunately @dl4mmers I have no idea what's inside those NuGet package, and cannot provide any support for that library.

Some of the issues you mention (video profile) have fixes in MixedReality-WebRTC via the releases/m71 branch of WebRTC UWP SDK; this is the only branch I know a bit, and the one with the most changes for HoloLens. I don't think any NuGet was ever built from it.

If you really want to stay on WebRTC UWP SDK, please open an issue on https://github.com/webrtc-uwp/webrtc-uwp-sdk/, and see if @LoadLibrary might be able to help. But I predict the answer will be that it is deprecated and you should move to either of MixedReality-WebRTC or WinRTC, which is also my recommendation.

In doubt I suggest MixedReality-WebRTC if you are targetting HoloLens 1/2, not because I have a bias, but because we offer guarantees and focus on that particular device, whereas WinRTC is more generic for all Windows platforms. In any case we are ourselves moving to WinRTC shortly and will undock from WebRTC UWP SDK, and are working closely with @LoadLibrary and the WinRTC team.

[djee-ms]

Also yes, we have partners shipping enterprise applications with H.264 on HoloLens 2, and it is very much an area of constant focus for this project to make sure things work on that device.

[dl4mmers]

Thank you for your response! We are targetting HoloLens 1/2 only, and so we decided to switch to MixedReality-WebRTC as soon as possible.

[rahulk131]

"Skip: 14.25" means the frames are being received over the network but not pulled by the renderer, so they get discarded.
"State: Opening" seems to mean that the renderer didn't open yet.

So it might be an issue with the Media Foundation renderer on that particular TestAppUWP, and not an issue with the MixedReality-WebRTC library. One thing I can think of is, are you using H.264 for anything else than TestAppUWP? There's a limited number of work that the hardware encoder might pick at a time, though it should fallback to the software implementation in theory so would be slow but visible. Other than that, I know there's a problem with that renderer, sometimes it takes several seconds or more to start; I never observed it on the remote track, but starting and stopping the local video several times will show you some black screen on the local video for several seconds until the render pipeline finally starts. Maybe this is the same issue? How long did you wait until deciding that the video was not visible? I'd wait at least a couple of minutes to be sure.

We had look at some of these. The issue still persists.

We're not using H.264 for anything else.

We waited sometimes up to 5 minutes. The Skip value goes down till about 4, but not any lesser.

Also, we're certain the WebRTC frames are being received. We keep track of the number of frames received counting the number of times PeerConnection.OnI420ARemoteVideoFrameReady() is called. It appears to steadily increase over time.

The frames returned by the WebRTC library are of type I420AVideoFrame. So is the WebRTC library decoding the H.264 frames to uncompressed I420A frames? Is it possible that the H.264 decoding is going wrong somewhere are the I420A frames received are corrupted somehow? Could that be a reason why the rendering is failing?

Could you say more about the known problem with the renderer in TestAppUWP? Are there any alternatives?

[djee-ms]

Yes, the WebRTC implementation receives H.264 frames and decodes them into I420 before calling the API callback. It is possible that H.264 decoding goes wrong, but unlikely. However it is quite easy to test, just look at the content of the I420 decoded frame and check if there's some data or it's all zeros. I would also check that the frame resolution makes sense (in particular that it's not 0x0).

The rendering issue on TestAppUWP is not well understood. For what I know the Media Foundation UWP API is poorly documented and the internal render pipeline seemingly does not restart immediately (or at all) when the source is changed, which occurs each time a track is added and/or its resolution changes. But I couldn't find a good example of hot-swapping a XAML source. I looked at this for some time, trying various permutations of the source change, and couldn't figure out what was going wrong nor any way to make it better. I also asked for help internally without success. I appreciate this is quite disappointing, even for me, to have this issue lying around. However because this is only a test app, and for lack of a clear understanding of the issue, we have been focusing our resources elsewhere so far. Do you really need that render module for your application, or only for testing? If that's a blocker for you, I can try to reach out internally once again and bump the priority, see if we're more lucky this time and someone with more experience in the UWP Media Foundation
API has any idea.

The alternative it to render the frames yourself by any other way, possibly using something like Win2D (I never tried though).

[rahulk131]

Now we're able to render the remote video stream on the HoloLens 2 device. However, this rendering happens only when we stop the local video playback.

After establishing a connection with audio and video, if we stop the local video playback, the remote video starts playing.

In the code, two things happen when we stop local media:

1. The playback stops (MediaPlayer.source = null)
2. The LocalVideoTrack is removed from the PeerConnection, and CreateOffer() is called again. This means audio and video are stopped from being sent to the remote peer. And the LocalSDP created by CreateOffer indeed has a=recvonly in the video section. (Why does audio section still have a=sendrecv)?

Any idea why these steps are helping render remote video? Both these steps appear to be necessary. Just stopping the local playback doesn't start the remote video rendering. The local tracks must be removed from the PeerConnection object and this seems to be triggering the remote media stream to be opened and rendered.

Any known fixes to this? I could re-initialize LocalVideoTrack and add it back to the PeerConnection and createOffer after remote video stream is opened. Anything else you could suggest?

(Of course, all this happens only on the HL2 device - on desktop it runs seamlessly.)

[rahulk131]

Thanks for your offer to reach out internally to see if there's someone with more experience in the UWP Media Foundation API has any idea. It'll be great if you could please do that - this rendering is a blocker for us.

[djee-ms]

I've pinged the MF team for help.

Can you please describe why/how you managed to get remote video rendering, since you were having issues at the beginning of this conversation? Just for context to understand. Also which branch/commit are you on? Are you on release/1.0 or on master (and if so, at which point, because things changed quite a lot recently there).

Aside from that, I am quite puzzled at the description. I am assuming you are using the old TestAppUWP from the 1.0 release with the two media players (left: local; right: remote), is that correct? I see no reason why they would affect each other, unless there's a mixup internally between the local and remote video feeds? The other thing I can think of is that on the first negotiation the video is not correctly negotiated, and is only properly negotiated on the second negotiation after you stopped the local track (and so in itself stopping the local track is not related, but the renegotiation is).

You are saying that you remove the LocalVideoTrack, and negotiate a new session. Why is it surprising that audio is still a=sendrecv? Or are you also removing the audio track? From your description it sounds perfectly expected, I don't understand the issue on that point. Audio and video tracks are independent, and send/receive direction is therefore negotiated independently for each track.

Can you please paste the SDP messages for the first and second negotiations, to see what they contain and what are the differences?

[rahulk131]

Thanks for pinging the MF team and your help debugging this.

We have just been able to solve the remote video rendering problem, and it's no longer a blocker for our use case. You can please close this issue now.

The problem for us appeared to be the "lazy" starting of the Remote Video Player:
https://github.com/microsoft/MixedReality-WebRTC/blob/7d47006a819c719250aa47f984052389636eab52/examples/TestAppUwp/MainPage.xaml.cs#L975

(Yes, I'm on the 1.0 Release, with the two media players)

For our use case, we know that there will be video, and also the width and height for the video frames. Therefore, we can start the Remote Video Player in the "OnLoaded" function itself. This seems to start the Remote Media Pipeline on the HoloLens device in the start itself and is able to render the remote video streams once the frames start arriving.

I think previously, when it wasn't working, there was trouble when lazily starting the Remote Video Player:
https://github.com/microsoft/MixedReality-WebRTC/blob/7d47006a819c719250aa47f984052389636eab52/examples/TestAppUwp/MainPage.xaml.cs#L1004

Somehow, that wasn't triggering the Media Pipeline.

" I see no reason why they would affect each other, unless there's a mixup internally between the local and remote video feeds? "

I was also intrigued by the seeming relationship between local and remote video feeds. But probably the simplest explanation is that it was an accident - removing the tracks and stopping a media player playback seems to have somehow triggered other media pipeline events, including the remote video stream one.

"You are saying that you remove the LocalVideoTrack, and negotiate a new session. Why is it surprising that audio is still a=sendrecv? Or are you also removing the audio track? "

My mistake. I see that in my message I specify only LocalVideoTrack. I meant to say that I was removing both LocalAudioTrack and LocalVideoTrack. So it was surprising that the audio block in the renegotiation SDP offer had a=sendrecv even after removing LocalAudioTrack from the peerConnection object. In any event, I don't need to remove the Local Tracks now.

Cheers,
Rahul