Mixedreality-webrtc: On the browser side video is not displayed
Background of project
Now I am studying in university and making final qualifying work with my friend. Our project is a system for remote working with field-programmable gate array (FPGA), which located in the campus laboratory. Working with FPGA represents by three aspects: real-time video image of the FPGA board, loading firmware (*.sof files are generated by user on his computer in Quartus Prime) in the board and manipulation by the inputs of the board through the buttons on the web interface. Manipulation is implemented through the Arduino: Arduino is connected to the laboratory stand (computer) and receive commands from the application of laboratory stand and than generate signals on the GPIO (general purpose input/output) pins, which wired to the GPIO inputs of FPGA. My friend wrote frontend (js + react script), found TURN/STUN servers and wrote signaling server based on websockets. I should write .NET core application for the laboratory stand, which will broadcast video from webcam at least one client, load received firmware and send received commands to the Arduino through the serial port. For serialized command and firmware my friend create two WebRTC data channels on the browser side. I decided to use this Nuget packet to create WebRTC .Net core application.
Questions or I need your recommendations
I start from the creation of connection between browser and my application. I try to broadcast video from the webcam to the browser side. My application sending offer through the signaling server, ice candidates, receiving answer and ice candidates from the browser(client). On browser side and my application, it is working, no errors or warnings in browser console, pc.Connected in application fires and in browser shows message Connection established but video in browser does not show. I create LocalVideoTrack, Transciever, VideoTrackSource how it is done in User Manual of this library, my webcam turns on and this can be seen by the LED of my webcam, but video does not show in browser.
It is my primitive code:
namespace WebRTC_Remote_FPGA_stand
{
class sdp_js {
public string type { get; set; }
public string sdp { get; set; }
}
class ice_candidate_js {
public string candidate { get; set; }
public string sdpMid { get; set; }
public int sdpMLineIndex { get; set; }
}
class Program
{
// Working with sdp and ice candidate messages
public static (string, string) ConvertString(string JsonStr)
{
int index = JsonStr.IndexOf(':', JsonStr.IndexOf(':') + 1);
if (index == -1) {
return ("", JsonStr);
}
return (JsonStr.Substring(0, index + 1), JsonStr.Substring(index + 1, JsonStr.Length - index - 3));
}
private static string CreateSignalingServerUrl()
{
Console.WriteLine("Input room number:");
return $"wss://wss-signaling.herokuapp.com?room={Convert.ToInt32(Console.ReadLine())}";
}
private static string ToLiteral(string input)
{
using (var writer = new StringWriter())
{
using (var provider = CodeDomProvider.CreateProvider("CSharp"))
{
provider.GenerateCodeFromExpression(new CodePrimitiveExpression(input), writer, null);
return writer.ToString();
}
}
}
static async Task Main(string[] args)
{
Transceiver videoTransceiver = null;
VideoTrackSource videoTrackSource = null;
LocalVideoTrack localVideoTrack = null;
FileStream file = null;
// Asynchronously retrieve a list of available video capture devices (webcams).
var deviceList = await DeviceVideoTrackSource.GetCaptureDevicesAsync();
// For example, print them to the standard output
foreach (var device in deviceList)
{
Console.WriteLine($"Found webcam {device.name} (id: {device.id})");
}
// Create a new peer connection automatically disposed at the end of the program
using var pc = new PeerConnection();
// Initialize the connection with a STUN server to allow remote access
var config = new PeerConnectionConfiguration
{
IceServers = new List<IceServer> {
new IceServer{ Urls = { "stun:stun.l.google.com:19302" } },
new IceServer{ Urls = { "stun:stun1.l.google.com:19302" } },
new IceServer{ Urls = { "stun:stun2.l.google.com:19302" } },
new IceServer{ Urls = { "stun:stun3.l.google.com:19302" } },
new IceServer
{
Urls = {"turn:numb.viagenie.ca" },
TurnPassword = "9u7prU:2}R{Sut~.)d[bP7,;Pgc\'Pa",
TurnUserName = "[email protected]"
}
}
};
await pc.InitializeAsync(config);
Console.WriteLine("Peer connection initialized.");
Console.WriteLine("Opening local webcam...");
videoTrackSource = await DeviceVideoTrackSource.CreateAsync();
Console.WriteLine("Create local video track...");
var trackSettings = new LocalVideoTrackInitConfig { trackName = "webcam_track" };
localVideoTrack = LocalVideoTrack.CreateFromSource(videoTrackSource, trackSettings);
Console.WriteLine("Create video transceiver and add webcam track...");
videoTransceiver = pc.AddTransceiver(MediaKind.Video);
videoTransceiver.DesiredDirection = Transceiver.Direction.SendOnly;
videoTransceiver.LocalVideoTrack = localVideoTrack;
DataChannel chanel = await pc.AddDataChannelAsync("Data", true, true, cancellationToken: default);
videoTransceiver.Associated += (tranciever) =>
{
Console.WriteLine("Transivier: {0}, {1}", tranciever.Name, tranciever.StreamIDs);
};
WebSocketSharp.WebSocket signaling = new WebSocketSharp.WebSocket(CreateSignalingServerUrl(), "id_token", "alpine");
pc.LocalSdpReadytoSend += (SdpMessage message) =>
{
//Console.WriteLine(SdpMessage.TypeToString(message.Type));
Console.WriteLine(message.Content);
//Console.WriteLine(HttpUtility.JavaScriptStringEncode(message.Content));
Console.WriteLine("Sdp offer to send: {\"data\":{\"description\":{\"type\":\"" + SdpMessage.TypeToString(message.Type) + "\",\"sdp\":\"" + HttpUtility.JavaScriptStringEncode(message.Content) + "\"}}}");
signaling.Send("{\"data\":{\"description\":{\"type\":\"" + SdpMessage.TypeToString(message.Type) + "\",\"sdp\":\"" + HttpUtility.JavaScriptStringEncode(message.Content) + "\"}}}");
};
pc.RenegotiationNeeded += () =>
{
Console.WriteLine("Regotiation needed");
};
pc.DataChannelAdded += (DataChannel channel) =>
{
Console.WriteLine("Added data channel ID: {0}, Label: {1}", channel.ID, channel.Label);
file = new FileStream(channel.Label, FileMode.Append);
channel.MessageReceivedUnsafe += (IntPtr ptr, ulong l) =>
{
Console.WriteLine("Received unsafe data");
};
channel.MessageReceived += (byte[] arr) =>
{
Console.WriteLine("Received data in data channel");
file.Write(arr);
};
};
pc.IceCandidateReadytoSend += (IceCandidate candidate) =>
{
//Console.WriteLine("Content: {0}, SdpMid: {1}, SdpMlineIndex: {2}", candidate.Content, candidate.SdpMid, candidate.SdpMlineIndex);
ice_candidate_js candidate_send = new ice_candidate_js { candidate = candidate.Content, sdpMid = candidate.SdpMid, sdpMLineIndex = candidate.SdpMlineIndex };
try
{
Console.WriteLine("Candidate to send: Content: {0}, SdpMid: {1}, SdpMlineIndex: {2}", candidate.Content, candidate.SdpMid, candidate.SdpMlineIndex);
signaling.Send("{\"data\":{\"candidate\":" + JsonSerializer.Serialize(candidate_send) + "}}");
}
catch (Exception e)
{
Console.WriteLine("Error to send local ice candidate");
}
};
signaling.OnMessage += async (sender, message) =>
{
// Console.WriteLine(message.Data);
//Console.WriteLine("message: {0}", message.Data);
// logger.LogDebug($"Received message: {message.Data}");
(string header, string correct_message) = ConvertString(message.Data);
Console.WriteLine("Correct message: {0}", correct_message);
Console.WriteLine("Header: {0}", header);
if (header == "{\"data\":{\"getRemoteMedia\":" && correct_message == "true")
{
bool OfferCreated = pc.CreateOffer();
Console.WriteLine("OfferCreated? {0}", OfferCreated);
}
//Console.WriteLine(message.Data);
if (header.IndexOf("candidate") != -1 && correct_message != "null")
{
try
{
var candidate = JsonSerializer.Deserialize<ice_candidate_js>(correct_message);
Console.WriteLine("Content of ice: {0}, SdpMid: {1}, SdpMLineIndex: {2}", candidate.candidate, candidate.sdpMid, candidate.sdpMLineIndex);
pc.AddIceCandidate(new IceCandidate { Content = candidate.candidate, SdpMid = candidate.sdpMid, SdpMlineIndex = candidate.sdpMLineIndex });
Console.WriteLine("Deserialized by ice_candidate");
//return;
}
catch (Exception)
{
Console.WriteLine("Could not deserialize as ice candidate");
}
}
if (header.IndexOf("description") != -1)
{
try
{
var sdp_mess = JsonSerializer.Deserialize<sdp_js>(correct_message);
var mess = new SdpMessage { Content = sdp_mess.sdp, Type = SdpMessage.StringToType(sdp_mess.type) };
Console.WriteLine("Sdpmessage: {0}, Type: {1}", mess.Content, mess.Type);
await pc.SetRemoteDescriptionAsync(mess);
if (mess.Type == SdpMessageType.Offer) {
bool res = pc.CreateAnswer();
Console.WriteLine("Answer created? {0}", res);
}
Console.WriteLine("Deserialized by sdp_message");
//return;
}
catch (Exception)
{
Console.WriteLine("Could not deserialize as sdp message");
}
}
};
pc.Connected += () =>
{
Console.WriteLine("Connected");
Console.WriteLine(pc.DataChannels.Count);
};
pc.IceStateChanged += (IceConnectionState newState) =>
{
Console.WriteLine($"ICE state: {newState}");
};
signaling.Connect();
//Console.WriteLine("Press a key to terminate the application...");
Console.ReadKey(true);
Console.WriteLine("Program termined.");
file?.Close();
//(var a, var b) = ConvertString("{\"data\":{\"candidate\":null}}");
//Console.WriteLine("{0}, {1}", a, b);
}
}
}
Created offer of my application
```
v=0
o=- 6586226767117739283 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0
a=msid-semantic: WMS
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 127 124 125
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:ExvA
a=ice-pwd:6zJMWYv2G8oZuaVZQ99AGgz1
a=ice-options:trickle
a=fingerprint:sha-256 19:99:F5:49:DF:75:EC:B4:A4:71:E5:17:75:B5:52:B9:F0:9B:D0:A1:B5:9D:C6:6E:D7:DF:44:2D:1A:8C:10:EA
a=setup:actpass
a=mid:0
a=extmap:2 urn:ietf:params:rtp-hdrext:toffset
a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:4 urn:3gpp:video-orientation
a=extmap:5 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:10 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendonly
a=msid:- d5a25e7e-133e-4aff-a3c0-ccdb4f5c5a00
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96
a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=fmtp:98 x-google-profile-id=0
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:100 multiplex/90000
a=rtcp-fb:100 goog-remb
a=rtcp-fb:100 transport-cc
a=rtcp-fb:100 ccm fir
a=rtcp-fb:100 nack
a=rtcp-fb:100 nack pli
a=fmtp:100 acn=VP9;x-google-profile-id=0
a=rtpmap:101 rtx/90000
a=fmtp:101 apt=100
a=rtpmap:127 red/90000
a=rtpmap:124 rtx/90000
a=fmtp:124 apt=127
a=rtpmap:125 ulpfec/90000
a=ssrc-group:FID 251589679 3124733634
a=ssrc:251589679 cname:Nb5Shxzkeqjm/2Ae
a=ssrc:251589679 msid: d5a25e7e-133e-4aff-a3c0-ccdb4f5c5a00
a=ssrc:251589679 mslabel:
a=ssrc:251589679 label:d5a25e7e-133e-4aff-a3c0-ccdb4f5c5a00
a=ssrc:3124733634 cname:Nb5Shxzkeqjm/2Ae
a=ssrc:3124733634 msid: d5a25e7e-133e-4aff-a3c0-ccdb4f5c5a00
a=ssrc:3124733634 mslabel:
a=ssrc:3124733634 label:d5a25e7e-133e-4aff-a3c0-ccdb4f5c5a00
Received answer from browser side:
```
v=0
o=- 2357749297625213784 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0
a=msid-semantic: WMS
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 127 124 125
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:DUG4
a=ice-pwd:7wvJF4C6iMWPnbyJmQwXF8xZ
a=ice-options:trickle
a=fingerprint:sha-256 CA:AA:26:31:89:59:E2:08:E7:6E:41:47:15:86:7E:F6:1A:A9:0C:B1:D2:A2:CB:95:D8:FC:58:1B:0D:F4:31:F8
a=setup:active
a=mid:0
a=extmap:2 urn:ietf:params:rtp-hdrext:toffset
a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:4 urn:3gpp:video-orientation
a=extmap:5 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=recvonly
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96
a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=fmtp:98 profile-id=0
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:127 red/90000
a=rtpmap:124 rtx/90000
a=fmtp:124 apt=127
a=rtpmap:125 ulpfec/90000
No errors or warnings on browser side:
In the console of application:
Why video does not show? Update: i tried to send file to the application - data channel creates (fires event DataChannelAdded), but event MessageReceived doesn't fire. Where can be a problem?
Vaveform
All 9 comments
Got the same problem today, on browser event which handling streams return RTCTrackEvent object with empty stream list. Video and audio source are defined in unity and streams from browser display on HoloLens, but streams from HoloLens are empty either I use emulator or real one
Does anyone know how to fix this?
ArtemEws
on 8 Mar 2021
Not sure if this is the original issue, but it might be worth checking out the track from the RTCTrackEvent object instead of the streams. We used to use the stream a long time ago but at the time, it didn't seem to have consistent behaviour across all browsers (stream seemed to show up in Chrome at the time, but not other browsers).
According to MDN, the streams property could be an empty array, if the track does not belong to a stream.
spacecheeserocks
on 9 Mar 2021
In v1.x we assigned a single stream to the 2 tracks (A&V). This was hard-coded internally.
In v2.x we leave that to the user via the Transceiver.StreamIDs property, since this is an optional feature and streams are not needed for WebRTC to work. If you are not filling that property, then we do not associate any stream to the tracks, so it is expected that streams are empty and all working by design as far as I can tell.
djee-ms
on 9 Mar 2021
Good to know, I assume the fact we had a stream show up in Chrome and not some other browsers was possibly just some unusual chrome behaviour, or maybe some kind of Plan-B backwards compatibility or something?
Either way, the 'track' property works for us regardless :)
spacecheeserocks
on 9 Mar 2021
Thank you very much, video broadcast works! I have another one question related with DataChannels. Is it possible to receive file (250 kbytes) when browser side create DataChannel to send me this file separated to the chunks by 262144 bytes ? I tried to receive this file, DataChannelAdded event fired, but MessageReceived event of this datachannel didn't fire.
Vaveform
on 9 Mar 2021
_If I remember correctly_ the internal buffer is 16 MB to send from MixedReality-WebRTC, but you should monitor the BufferingChanged event to be sure. To receive, I don't think there's any limit as I don't think data is buffered, but I can be wrong, I didn't look closely at how the receiving side is implemented by Google. I don't know why the MessageReceived event is not raised.
djee-ms
on 9 Mar 2021
I found where problem. In JavaScript WebRTC API has opportunity to set chunk size in data channel. My friend set chunk size bigger than max chunk size of DataChannel in Mixed-Reality WebRTC. I don't know what exactly max size of chunk in MR WebRTC, but I experimentally established that max size approximately 61440-65000 bytes. So, when my friend try to send me file bigger than chunk size, data channel creates, but MessageReceived event is not raise. I simply changed chunk size in JavaScript code (on the browser side) and it is start working.
Vaveform
on 12 Mar 2021
I created simple example of making video transceiver If someone have same issue:
// pc - PeerConnection object
Transceiver videoTransceiver = null;
VideoTrackSource videoTrackSource = null;
LocalVideoTrack localVideoTrack = null;
videoTrackSource = await DeviceVideoTrackSource.CreateAsync();
Console.WriteLine("Create local video track...");
var trackSettings = new LocalVideoTrackInitConfig { trackName = "webcam_track" };
localVideoTrack = LocalVideoTrack.CreateFromSource(videoTrackSource, trackSettings);
Console.WriteLine("Create video transceiver and add webcam track...");
TransceiverInitSettings option = new TransceiverInitSettings();
option.Name = "webcam_track";
option.StreamIDs = new List<string> { "webcam_name" };
videoTransceiver = pc.AddTransceiver(MediaKind.Video, option);
videoTransceiver.DesiredDirection = Transceiver.Direction.SendOnly;
videoTransceiver.LocalVideoTrack = localVideoTrack;
I created simple example of making video transceiver If someone have same issue:
// pc - PeerConnection object Transceiver videoTransceiver = null; VideoTrackSource videoTrackSource = null; LocalVideoTrack localVideoTrack = null; videoTrackSource = await DeviceVideoTrackSource.CreateAsync(); Console.WriteLine("Create local video track..."); var trackSettings = new LocalVideoTrackInitConfig { trackName = "webcam_track" }; localVideoTrack = LocalVideoTrack.CreateFromSource(videoTrackSource, trackSettings); Console.WriteLine("Create video transceiver and add webcam track..."); TransceiverInitSettings option = new TransceiverInitSettings(); option.Name = "webcam_track"; option.StreamIDs = new List<string> { "webcam_name" }; videoTransceiver = pc.AddTransceiver(MediaKind.Video, option); videoTransceiver.DesiredDirection = Transceiver.Direction.SendOnly; videoTransceiver.LocalVideoTrack = localVideoTrack;
Now I have the same problem, how did you solve it? Set up StreamIDs?
No effect.
Thank you
Rendtime
on 22 Apr 2021
Related issues
jmc-crash
路
6Comments
amfdeluca
路
7Comments
hansenwuwu
路
4Comments
efsjouw
路
4Comments
hanseuljun
路
5Comments
Most helpful comment
In v1.x we assigned a single stream to the 2 tracks (A&V). This was hard-coded internally.
In v2.x we leave that to the user via the
Transceiver.StreamIDsproperty, since this is an optional feature and streams are not needed for WebRTC to work. If you are not filling that property, then we do not associate any stream to the tracks, so it is expected that streams are empty and all working by design as far as I can tell.