Mixedreality-webrtc: ARM64 support progress

[eirikhollis]

I know ARM64 support for HoloLens 2 has been brought up several times in the past, but I find it difficult to follow the progress.
Is there a link to somewhere where it is being worked on?
And what is an approximate time of completion? 1 month? 6 months? 2 years? No need to be exact numbers, just nice to have an idea of where the progress is at the moment.

[djee-ms]

Hi @eirikhollis,

Thanks for opening that issue, I'll take it as an opportunity for a recap and roadmap regarding ARM/64 support.

• The current version of MixedReality-WebRTC builds upon the WebRTC UWP SDK project, which uses the M71 milestone of Google's libwebrtc implementation. This milestone is rather old and has no support for ARM64 at all.
• The WebRTC UWP SDK project uses a fork of Google's repository, which makes it very difficult to upgrade to a more recent milestone.
• Around last year or so, our best plan of action was to upgrade to M75 because the WebRTC UWP SDK project team was also working on a fork of that milestone. This was an experimental branch though, which had not been really validated in production code, and was missing all the changes we made in particular to improve support and quality of H.264 video.
• We never managed to get the proper resourcing to push that work through, and as time passed the M75 milestone was getting old itself. In parallel ownership of the WebRTC UWP SDK project changed and the new team decided instead to bootstrap the WinRTC project, which the aim of replacing the WebRTC UWP SDK project and directly building over the Google repository instead of forking it, to make future upgrades much easier.
• Today, our current plan of action is to wait for the WinRTC team to provide parity with WebRTC UWP SDK, and then make the switch, which will have the consequence of upgrading MixedReality-WebRTC internally from M71 to at least M80 if not latest (master branch). Since support for ARM64 has been added around M73 or M74 if I recall correctly, we will have ARM64 support internally, and can then work to make sure we expose it. This switch is referred to as "undocking" (the experimental/undock branch, which we use internally for this and do not support publicly, just to set expectations).
• The current expectation is to get parity (@LoadLibrary currently actively working on this) and make the move in the next few months. We will probably not wait for it for 2.0, but might get a 2.1 version after that which would leverage that undocking and all the benefits it brings, while maintaining the same feature set as 2.0.
• One consequence we noticed is that we might have to drop ARM(32) support in the process, because for ARM/64 UWP we need the Visual Studio compiler, and Google never added support for cl.exe ARM(32), only ARM64. At this point if anyone really need ARM(32) support we encourage them to get in touch, but the amount of work to continue this support is large, and involves either changes in Visual Studio itself to support a clang-cl UWP ARM toolchain (so a bit unlikely) or reverting back to the Clang toolchain, which works on ARM64 but has its own set of issues for UWP support.

I hope this makes things more clear. On our side, we'd be interested to know why you are eager to get ARM64 support. Is there a specific issue with ARM(32) running on ARM64 devices like HoloLens 2? What do you expect to get from ARM64?

[eirikhollis]

@djee-ms Thanks for the response! A great read!

What do you expect to get from ARM64?

• The main compatibility issue for us right now is Azure Remote Rendering, which does not support ARM(32). Rather than trying to implement ARR in ARM(32), I'd much rather have WebRTC move to ARM64.
• We almost never do one-to-one calls with the library. It's a lot of connected devices and/or websites in a many-to-many configuration. I hope that an ARM64 version of the library will provide a performance improvement, and if it doesn't, at least other parts of the application will run better on ARM64.
• Unity does not support ARM(32) in their new XRSDK. However, latest preview does in fact add support for ARM(32), so I'm hoping to test that out soon.

So as of right now, most, if not all components we have, run on ARM64, and it's only the WebRTC bit that is blocking us from moving over. Do you have an estimate on when the WinRTC team will achieve parity with M71? That could help us decide if ARR + WebRTC is a dream for next year or maybe possible sometime soon.

[djee-ms]

The current expectation is to get parity (@LoadLibrary currently actively working on this) and make the move in the next few months.

Current ETA is parity in the next few weeks, and then we need to integrate the change and do some extensive testing to make sure there's no regression. I don't have an estimate for that other than a few extra weeks, but I can tell you it's where our focus is with 2.0 so that will not be next year.

[kspark-scott]

HI Jerome (@djee-ms ), I just re-read your very helpful roadmap summary above and was reminded of the likelihood of ARM being dropped. I don't know much about ARM64 tooling in Visual Studio, but have observed that all C# builds appear to be dotnet native builds, regardless of how you set the dotnet native build flag in the project file (i.e. including all debug builds). I hope to learn that there is in fact a way to do an ARM64 build that is not a dotnet native build, but I'll proceed here as if ARM64 builds are dotnet native only.

As I'm sure you know, dotnet native builds can be very time consuming. If we can only use ARM64 on HoloLens 2 then every incremental change in our app will incur the dotnet native build time which can be a big productivity hit especially when working on the kinds of issues that require a lot of guess-and-check style investigation.

(And if your project dependencies aren't kept very clean, as ours admittedly are not at the moment, then builds will be done on launch even when no inputs have changed. So just re-launching the app -- e.g. to set an earlier breakpoint -- will incur the build time penalty. That at least is in our control.)

I understand that supporting both architectures is potentially very costly for you. I just wanted to be sure that you are aware of the productivity implications of ARM64-only so you can factor it into the decision making.

[djee-ms]

Thanks for the feedback @kspark-scott, I will have a look but unfortunately this is not about the support cost, this is a technical limitation of Google not supporting the MSVC toolchain on ARM. They do support MSVC but only for x86/x64/ARM64; for ARM one has to use Clang.

With the current version we use webrtc-uwp-sdk which does some magic compiling parts with clang and parts with MSVC as I remember, which is somewhat dangerous, and we had troubles in the past with it. I think we had also to disable some SIMD optimizations because of that, which increases the CPU load in a non-negligible way. Anyway we cannot continue with webrtc-uwp-sdk which is deprecated and unmaintained.

Going forward upgrading to a more recent milestone via the WinRTC patches for UWP, we are back on an (almost) clean Google repository, which means we cannot compile with MSVC on ARM. Unfortunately, Visual Studio doesn't have an ARM/64 clang-cl toolchain for UWP, and there is little chance to get one in the short term at least, which means we are stuck between Google's clang-only libwebrtc and Visual Studio's MSVC-only UWP support. As it sounds dangerous if at all possible to compile part of the code with one compiler and part with the other, we are considering dropping ARM altogether because even before the cost of maintenance, we are not even sure there is a way to make it happen in the first place (at least for UWP).

[kspark-scott]

Thanks for the clarification that it is not just about the cost of creating and supporting a customized mutli-compiler build, but also about the technical risks of doing so. Honestly I would be surprised if you would seriously consider such a thing for relatively minor issues like the one I described, but thought I should state the issue for the record if nothing else. We will plan accordingly.

[djee-ms]

I am still trying to understand why ARM has no .NET support (though I confirmed it doesn't indeed) and why Google has not added support for ARM, which apparently has something to do with Visual Studio CRT not being available unless you target Windows 10, or something in that flavor. So there's a possibility we can work around, but it's slim.

[eirikhollis]

Are there any updates on the ARM64 support progress?

[fibann]

@eirikhollis sorry, but moving away from WebRTC-UWP proved to require much more effort than we estimated and we have had to drop it from the list of features included in the 2.0 release. It is now on the backlog, but we don't have an ETA yet.

[eirikhollis]

@fibann
I'm sad to hear this, but I understand that the port process must be quite difficult.

Could I ask an unrelated question? I see that the activity in the repository, both here and on winrtc has dropped significantly the last couple of months, and I was wondering what the cause was. Has the work for WebRTC/UWP projects been dropped lately?

[fibann]

MR-WebRTC is still active - we are working on several projects at the same time, so the degree of activity on each one of them will change over time. winrtc is developed by a different team so I really don't have enough info to comment!

[Holo-Krzysztof]

I'd also be interested in ARM64 support and the winrtc project going forward.

Is there any news or maybe a rough estimate when it is ready? Insertable streams would be an interesting feature for us.

[foungyu]

Hi team, thanks for providing WebRTC tool, it works great with ARM. However some other Unity Hololens plugins only support ARM64 , any news on this ARM64 support? Thanks