Mixedreality-webrtc: No remote video transfer in Unity

[jmc-crash]

I have followed the new Unity tutorial at https://microsoft.github.io/MixedReality-WebRTC/manual/helloworld-unity.html and it follows through (almost) perfectly except the video stream does not appear to be transferring.

I have started the node-dss server on one PC and both Unity instances are pinging it at runtime where the server logs show

GET /data/DNDO-38 404 - - 0.068 ms
GET /data/DNAO-67 404 - - 0.066 ms

respectively for each instance. I have copied the correct values into the Remote Peer Id and I am getting no errors in either Unity instance. The local video feed is working (in one instance because I only have a webcam on one of the PCs), so that isn't the problem.

I am just running the node-dss server from a powershell instance as follows:

PS C:\node-dss> npm start

[email protected] start C:\node-dss
node bin/dss.js

online @ 3000

I am not too sure where to look from here. Is there some way in Unity to establish that a connection is actually being established?

[djee-ms]

Hi @jmc-crash,

My first guess if you are trying inside the Unity editor, which is an x64 Desktop process, or a standalone PC x64 player, is to make sure that you are using either the VP8 or VP9 codecs. The H.264 codec is only supported on UWP. See the LocalVideoSource.PreferredVideoCodec which should be one of None / VP8 / VP9. The default None will fall back on VP8 too.

Also if you have a single webcam and are using the VideoChatDemo you need on one of the instances to disable (MonoBehaviour.Enabled = false) the LocalVideoSource component so that it doesn't try to capture from the same webcam that is already in use by the other instance. This might also cause issues in video initializing, which might be a reason why the remote video doesn't show up if the code throws an exception and doesn't finish the connection initializing correctly.

[jmc-crash]

Hi @djee-ms,

Good call on the desktop player however the video codecs on both instances are set to 'None' by default. To double check it I changed them explicitly to VP8 and it still isn't working.

As for the single webcam issue, I am running the instances on separate computers on the same network. One PC has a webcam built in (the one that is displaying correctly through the media player), and the other has no webcam. To ensure the PC without the webcam isn't causing issues initializing, I changed its mode to "Manual" so that it won't try to start automatically.

I'm paranoid about the node server. When I run 'npm install' I get the following output:

PS C:\node-dss> npm install
npm WARN [email protected] No repository field.

audited 532 packages in 2.684s
found 0 vulnerabilities

Is this normal? But the server does appear to be polling correct and at least receiving the data from each instance. I'm just not sure its positing it? The set debug command does not work for me so I had to go into the .js and add console.log statements so it would explicitly log the output for me.

Please let me know what other information you might need.

[djee-ms]

You're using Powershell; the set DEBUG=dss* is a cmd command. Use $env:DEBUG="dss*" instead from Powershell and you will get the debug outputs.

Once you get the node-dss debug logs, you can check the SDP offer and answer coming from the peers. They should both contain a m=video line and a a=rtmap referencing the same codec. The offer has MessageType = 1 and the answer MessageType = 2. If you don't see both this is likely a peer ID error.

[jmc-crash]

Ah! That will do it...

Well now my second instance (on the remote PC) doesn't appear to be connecting to the server according to the debug logs. I'm just getting a dss message from one PC.

PS C:node-dss> npm start

[email protected] start C:node-dss
node bin/dss.js

online @ 3000
dss GET /data/DNAO-67 404 - - 1.530 ms +0ms

I double checked and the IP is definitely set correctly on the NodeDssSignaler. No apparent error messages either.

EDIT: it does appear to be connecting, but not debug log on the node server.

[djee-ms]

Check your Windows firewall. NodeJS gets blocked sometimes.

[jmc-crash]

Okay well it is polling again correctly as it was before, however the video is still not showing on the remote instance. How do I check the offer as you mentioned above? The only debug logs I'm getting are the GET messages:

dss GET /data/DNAO-67 404 - - 0.057 ms +129ms
dss GET /data/DNDO-38 404 - - 0.052 ms +394ms

[jmc-crash]

The peerId I am using is based on what the Unity project finds on runtime. I noticed it queries SystemInfo.deviceName to auto fill local peer Id field, so this is what I am using as the remote Id on the other instance. This is the id it needs, correct?

EDIT: Extra information: I placed log statements in the 'PostToServer(Message msg)' function in Unity and it is not actually being called. Isn't this required for the video/audio to actually be sent? All the logs in node-dss are 404 messages. So it appears the 'LocalSdpReadytoSend' event is not being invoked at some point in the WebRTC PeerConnection?

[CabbageWarrior]

Hi everybody,
I have the same identical situation: I followed the complete tutorial but I get the same problem.

I have two PCs (one with a built-in webcam and the other with no cams, but I'm faking it with an OBS plugin so, in conclusion, I have a webcam for each PC), two Unity Editor instances with LocalVideoSource.PreferredVideoCodec set to none, the node-dss serve on one of the two PCs that gets the GET commands but doesn't recognise any connections between the two instances, getting only 404 responses.

I've also put a Debug.Log in the OnIceCandiateReadyToSend method (the one that calls the PostToServer one), but nothing happens in my console after the WebRTC plugin initialized successfully. log.

[djee-ms]

@jmc-crash > You can use any ID you want, provided the 2 peers have different ones. NodeDssSignaler sets the local peer ID to SystemInfo.deviceName for convenience if you didn't set anything because the ID should not be an empty string. You can use 1 and 2 or A and B or anything else. As long as the local and remote IDs are set correctly.

If PostToServer() is not called then something is wrong with the local connection. The messages are coming from Signaler.SendMessageAsync(), which is called by the Signaler_LocalSdpReadyToSend and Signaler_IceCandidateReadytoSend even handlers subscribed to the peer connection's events. Can you check those are called?

@CabbageWarrior > 404 is not an error. It means the peer asks if the node-dss server has any data for it, and the server doesn't find any (404 - not found) so says so. This is the polling mechanism at work. If you see 404 messages from 2 different peer IDs then that means they are connected correctly.

Maybe stupid question for both but did you actually press the "Create offer" button to initiate the call? Because this is not automatic. When you press on one peer "Create offer" the peer will invoke LocalSdpReadyToSend then a big chunky SDP message will be sent, visible in the node-dss logs. Shortly after the remote peer will answer with the same procedure. After that the video will start streaming (through a different network port that you cannot see in Unity).

[djee-ms]

That's my bad the part about "Create offer" is missing at the end of the tutorial 😭

[CabbageWarrior]

@djee-ms thanks for the info. I cited the 404 feedback only to say that that was the only feedback I was noticing.

I think that the problem is the lack of the "Create offer" part xD I'll copy it from the Unity project example, but it would be perfect if you could update the tutorial <3

Thanks for the help! I'll test the button!

[djee-ms]

Fixed in docs : https://microsoft.github.io/MixedReality-WebRTC/manual/helloworld-unity-connection.html#starting-the-webrtc-connection

[jmc-crash]

Good catch there @djee-ms and thanks @CabbageWarrior for the input! Mine is now working between two editors! Now the hard part of porting it to HoloLens 1! ;)

[djee-ms]

Good, glad to hear that's all fixed. Closing then.