Mixedreality-webrtc: Using the C++ API with hololens

[werto165]

Hello All,

I would like to preface this by saying I am quite inexperienced in using C+. I would like some guidance on using the native C++ API, I have setup a peerconnectionhandle, peerconnectionconfiguration and put a stun server in the ice servers. However, when I call PeerConnection::create(pcc,h) passing my config and handle it crashes when deploying on hololens saying a dependent dll was not found, I have built the project from sources and statically linked mrwebrtc.lib in additional dependencies.

Includes that I have used:

include "interop_api.h"

include "peer_connection_interop.h"

include "callback.h" //source from native

include "peer_connection.h"

PeerConnectionHandle p = {}; //peer connection object PeerConnectionConfiguration pcc; pcc.encoded_ice_servers = "urls = stun.l.google.com:19302"; mrsPeerConnectionInteropHandle h = {}; //Microsoft::MixedReality::WebRTC::Result r = mrsPeerConnectionCreate(pcc, mrsPeerConnectionInteropHandle(), &p); /*I tried this didn't work*/ PeerConnection::create(pcc, h); void* userDat = {}; const char* type = "answer"; const char* sdp_data = ""; PeerConnectionLocalSdpReadytoSendCallback callb = {}; mrsPeerConnectionCreateOffer(p);

For peer connection create there is also mrsPeerConnectionCreate that is in the interop_api.h. There is the "interopp_api.h" and "peer_connection_interop.h" and "peer_connection.h", I was wondering if I'm building a C++ directx application for the hololens would the interop_api be the route to go? And sending the SDP over the singalling service(i'm using node.js ) would I have to implement the sending within the interop_api.cpp or my main file of my application(appView.cpp)?

[djee-ms]

If you're using C++ you should use the master branch and use exclusively the C methods exported in the interop files (interop_api.h and the various xxx_interop.h headers). See the tests for how it's used (for example this). There's no real C++ library anymore, only a pure C API, which works better when using DLLs.

Also you should never modify the file shipped; signaling can be implemented entirely inside your app. See for example what the C# tutorial is doing and you can I think easily adapt to C++, the method names are generally the same (with mrs prefix).

[werto165]

Thank you for the help. I thought that would be the case. I'll give that a go 👍

[werto165]

If you're using C++ you should use the master branch and use exclusively the C methods exported in the interop files (interop_api.h and the various xxx_interop.h headers). See the tests for how it's used (for example this). There's no real C++ library anymore, only a pure C API, which works better when using DLLs.

Also you should never modify the file shipped; signaling can be implemented entirely inside your app. See for example what the C# tutorial is doing and you can I think easily adapt to C++, the method names are generally the same (with mrs prefix).

I'm using the hololens 2 for development and I was wondering if these tests are just for a desktop application? Because at the bottom of the building from sources it mentions: is one of [x86, x64, ARM]. Note that ARM is only available on UWP. Correct me if I'm wrong but the win32 tests are not UWP is that correct?

I am getting this error:

HolographicAppEyeGaze.exe' (Win32): Loaded 'C:\Windows\SysArm32\ntdll.dll'.
<--- other dlls being loaded --->
'HolographicAppEyeGaze.exe' (Win32): Loaded 'C:\Windows\SysArm32\MinUser.dll'.
The program '[6980] HolographicAppEyeGaze.exe' has exited with code -1073741515 (0xc0000135) 'A dependent dll was not found'.

[djee-ms]

Yes the test executable only runs on Win32, but the API is the same for Win32 and UWP, so you can still have a look at the source code to see how the API is used.

[werto165]

I'm getting a kWrongThread error on my code when creating the peerconnection using mrsPeerConnectionCreate. I am noticing that the dll is using files in the wrong location, I have the master branch on my C:/dev/MixedReality-WebRTC. My hololens project is in the normal visual studio 2019 location. And I have the mrwebrtc.dll and mrwebrtc.lib in a folder called dependencies, these are taken from the C:/dev/MR/bin UWP ARM folder(development on hololens 2). Additionally in the dependencies directory I have the libs mrwebrtc files (source and inc files). Does the dll file assume that the source files would be in C:dev\MixedReality-WebRTC\libs\mrwebrtc\include rather than my project C:\Users(USERNAME)\source\repos\WebRTCHololens\dependencies\mrwebrtc\include.

I have limited experience using dlls, sorry if this is an obvious question. But would I need to put the implementation project of mrwebrtc-uwp in my solution directory and then have the dll linked?

[djee-ms]

I'm getting a kWrongThread error on my code when creating the peerconnection using mrsPeerConnectionCreate.

On UWP mrsPeerConnectionCreate(), or more precisely the very first call to the library which will initialize the underlying global objects and initialize the library, cannot be called from the main UI thread. This is because the implementation calls some UWP API that needs to dispatch the work to the UI thread. If the initial call is already on the UI thread, then this will deadlock (see webrtc-uwp/webrtc-uwp-sdk#143 for the full explanation, it is slightly more complex). So you need to dispatch that call to another thread.

Does the dll file assume that the source files would be in...

The DLL doesn't need/assume anything about source files. The DLL is a compiled version of the sources, and doesn't need the sources to work. Are you instead talking about the fact that your project needs to include the C++ headers to be compiled, or are you talking about the Visual Studio debugger which looks for the sources associated with the DLL so that it can offer a better debugging experience (source file based) than using the DLL alone (binary / assembly code)?

Note that if you are linking the DLL (as in, link against the import lib and use the DLL at runtime) then you only need the headers to build your project, not the sources (.cpp). In general to use a DLL into a different project a user links the import lib (.lib) with the project's executable, and compiles the project's code which includes the library headers (.h). Then at runtime the OS loads the DLL to execute its content. The original library source code is never used in this scenario.

Alternatively if you link the library statically (.lib only) and don't use the DLL, then the source files of the library are compiled with the source file of the project to create a single self-contained executable.

In parallel of all of that, the debugger generally tries to find the original sources from which the program was compiled, but this is only for debugging purpose. For DLLs the path of the source files from which the DLL was compiled is embedded into the DLL or its associated PDB, which is why I guess you can see Visual Studio loading those files and not the copy you made somewhere else in your project. But for consistency those files should be the same anyway, just at a different location.

[werto165]

Thanks for the great info, I'm trying to digest that now. Very helpful for my learning :) As for dispatching it to another thread I tried calling an async task and returning the value. Although, I don't get the kwrongthread error, I just get my program in an infinite loop somewhere, I have tried to debug it however, it creates another thread and I can't seem to step through. I have tried both of these methods:

//std::future<mrsResult> a(std::async(std::launch::async, mrsPeerConnectionCreate, &pcc, &h));
//a.get(); 
std::future<bool> b(std::async(std::launch::async, callFunc));
b.get();

which then calls for (b.get):

bool callFunc() {
    mrsPeerConnectionConfiguration pcc;
    mrsPeerConnectionHandle h;
    mrsPeerConnectionCreate(&pcc, &h); 
}

For the hololens I'm running it after the dispatcher().processevents: in appView::Run()

void AppView::Run(){
    HolographicFrame previousFrame { nullptr };

    while (!m_windowClosed)
    {
        if (m_windowVisible && (m_holographicSpace != nullptr))
        {
    CoreWindow::GetForCurrentThread().Dispatcher().ProcessEvents(CoreProcessEventsOption::ProcessAllIfPresent);
//std::future<mrsResult> a(std::async(std::launch::async, mrsPeerConnectionCreate, &pcc, &h));
            //a.get(); 
            std::future<bool> b(std::async(std::launch::async, callFunc));
            b.get();

I am sure there is a better approach to this, I'm just confused how to call from another thread without originally calling it from the UI thread. Would you instead call it from the holoens main?

[djee-ms]

All the difficulty is that you need to call it from another tread without blocking the main UI thread. And here the .get() call of std::future is blocking the current UI thread waiting for the result, which prevent the UI thread from being available to execute some UWP APIs that mrsPeerConnectionCreate() needs it to execute.

So you need some different construct than that, for example using std::promise to launch the mrsPeerConnectionCreate() call which will fulfill the promise once completed. The rest of the program should just do nothing or something unrelated to the peer connection (for example, other initializing of _e.g._ your UI, but not block!) until the promise is completed, and on completion can continue to run more code using the peer connection. The easiest (but non-optimal) way is to periodically test if the promise has been fulfilled without blocking (see Get the status of a std::future on StackOverflow for example). Unfortunately the std::future _et al._ utilities lack essential functions like continuation support (.then(), which is experimental at the moment), so using the C++ standard library for async programming is not the easiest thing.

[werto165]

Thanks a lot, I'll have a go at that, that totally makes sense that it would block the thread, now I just look a bit silly.

[djee-ms]

Multi-threaded programming is hard ^^

[werto165]

Multi-threaded programming is hard ^^

I think I have it working :D The code wasn't too hard in the end with your help of course.

if anyone else has problems setting this up:

std::future<bool> fu = std::async(std::launch::async, callFunc);
is_ready(fu);

this code before while(!m_windowClosed)

with this code:

template<typename R>
bool is_ready(std::future<R> const& f)
{
    return f.wait_for(std::chrono::seconds(0)) == std::future_status::ready;
}
bool callFunc( ) {  
    mrsPeerConnectionConfiguration pcc;
    mrsPeerConnectionHandle h;
    mrsPeerConnectionCreate(&pcc, &h); 
    return true; 
}

Hope this helps someone out.

[zhuisa]

@werto165 Hi! I am very happy to find that you use c++ API with holoLens. I am also a C++ developer, I want to use c++ API in holoLens 2 to video chat with others. But I have little knowledge that how to use? Can you give me a help!