Vrtk: Support Windows Mixed Reality headsets in Unity 2017.2

[thestonefox]

The Windows Mixed Reality headsets have native supported in 2017.2 so just need to get the Unity core support updated to work with that build platform:

[SDK_Description("Unity (Universal Windows Platform:Windows Mixed Reality)", null, "Windows Mixed Reality", "WSA", 6)]

(not just this is needed)

[lordvard]

very excited about this, let me know if you need help testing.
will it take much work to get the SDK manager updated with supprt for WMR?

[bddckr]

It's not about the SDK Manager. VRTK has SDKs abstracted and thus "only" new classes are needed to support the new devices.

It looks like it still needs a special patch release of Unity and is riddled with issues, so for now I guess the recommendation is to use the SteamVR SDK until support is finalized.

[SimonDarksideJ]

OK, back from hiatus with MR bootcamps and such. So I'm going to start looking in to this now.
Whats the best branch to start on this from?

[thestonefox]

@DDReaper release/3.3.0-alpha

[thestonefox]

3 things need doing for this.

Steam vr support

Unity core support (2017.2 only)

Microsoft sdk support (UnityEngine.V/XR.WSA)

[SimonDarksideJ]

Ok, started by creating the new SDK scripts but noticed a big issue in the Example scenes. None of the script references are working for the other SDK's in the examples.
Basically, getting a load of
"The referenced script on this Behaviour (Game Object 'GvrControllerMain') is missing!" everywhere. so not sure if the examples are broken or not in the 3.3.0-alpha branch atm.

On another note, I defined all the platform requirements for WindowsMR, yet the SDKSetups still indicate that the SDK isn't installed, which is wrong as there is no SDK :D, just 2017.2

[bddckr]

Script issues are fine because you didn't import all the supported SDKs. Ignore it.

For the not-installed issue:
You probably have a wrong use of the SDK attributes, one of them tells it to check via a method for existence of a specific symbol or whatever is needed to check for existence of the SDK support.

I suggest you check the simulator SDK as that one also doesn't need anything. You should be able to see a difference compared to the SteamVR SDK for example.

[lordvard]

I just got the SteamVR bridge up and running with the Acer devkit and everything in Woeful Woebots is working well with the current 3.3.0 alpha. The only problem is the SDK Transform Modify as the device picks up items at an unfortunate angle. So an update to that would be on the top of my wish list :)

[thestonefox]

@lordvard as mentioned in the list above the SteamVR plugin requires work for Windows headsets.

[SimonDarksideJ]

Got it, I've got the basic setup working now. Just testing, but expect to need to tweak things around boundary's and positional velocities.
The SDK issue was because the VRTK framework uses the Platform Defines to designate an SDK is required, setting that to null in the "SDK_Description" solved that.

[SimonDarksideJ]

P.S. @lordvard the SteamVR preview is flaky as anything at the mo, but still a step up from where it was. Issue I have is with the SteamVR client constantly losing/regaining the connection to the headset atm.

[tomwim]

Hi @DDReaper I am also currently trying to get Windows Mixed Reality run within VRTK. I think, I followed a different approach than you and was wondering if there is any branch to check out your current status to get a better understanding of MR+VRTK.

[SimonDarksideJ]

OK, next snag. After removing the Defines section from the SDK_Description setup. It's now complaining it needs the defines???
"Exception: The fallback Controller SDK is being used despite being set to 'SDK_WindowsMRController'. Its needed scripting define symbols are not added. You can click the GameObject with the VRTK_SDKManager script attached to it in Edit Mode and choose to automatically let the manager handle the scripting define symbols.
VRTK.VRTK_Logger.Log (LogLevels level, System.String message) (at Assets/VRTK/Source/Scripts/Internal/VRTK_Logger.cs:167)
VRTK.VRTK_Logger.Error (System.String message) (at Assets/VRTK/Source/Scripts/Internal/VRTK_Logger.cs:128)
VRTK.VRTK_SDKSetup.HandleSDKGetter[SDK_BaseController] (System.String prettyName, VRTK.VRTK_SDKInfo info, IEnumerable`1 installedInfos) (at Assets/VRTK/Source/Scripts/Utilities/SDK/VRTK_SDKSetup.cs:447)
VRTK.VRTK_SDKSetup.get_controllerSDK () (at Assets/VRTK/Source/Scripts/Utilities/SDK/VRTK_SDKSetup.cs:215)
VRTK.VRTK_SDKSetup.OnLoaded (VRTK.VRTK_SDKManager sender) (at Assets/VRTK/Source/Scripts/Utilities/SDK/VRTK_SDKSetup.cs:364)
VRTK.VRTK_SDKManager+c__Iterator0.MoveNext () (at Assets/VRTK/Source/Scripts/Utilities/SDK/VRTK_SDKManager.cs:756)
UnityEngine.SetupCoroutine.InvokeMoveNext (IEnumerator enumerator, IntPtr returnValueAddress) (at C:/buildslave/unity/build/Runtime/Export/Coroutines.cs:17)"

Ideas?

[SimonDarksideJ]

Hi @tomwim , happy to share, I've just not pushed up my changes yet, which only involves creating a new SDK client. As soon as I push it, I'll send you a link.

[bddckr]

@DDReaper You're doing the SDK_Description wrong.

I suggest you compare to the Simulator SDK classes. Please note how every class references the SDK_SimSystem class via an attribute on the class
https://github.com/thestonefox/VRTK/blob/288bc389771bc17421702da34d8fef129625c7df/Assets/VRTK/SDK/Simulator/SDK_SimController.cs#L10
That System class then is the only one really defining the SDK via the attribute:
https://github.com/thestonefox/VRTK/blob/288bc389771bc17421702da34d8fef129625c7df/Assets/VRTK/SDK/Simulator/SDK_SimSystem.cs#L7

[bddckr]

Documentation for the attribute:
https://github.com/thestonefox/VRTK/blob/288bc389771bc17421702da34d8fef129625c7df/Assets/VRTK/SDK/Base/SDK_DescriptionAttribute.cs#L51-L59

[tomwim]

So my current status is that I get a very rudimentary state running in which the ControllerEvents example scene works fine. I get headset and controller pose as well as all controller input properly. Boundaries are not implemented yet though.
I am not using SteamVR but only the Unity Windows MR integration cause I am waiting for a Steam VR Preview Product Key which I hopefully get within the next days. Not sure if this is the right approach...

My SDK_Description is:
[SDK_Description("WindowsMR", SDK_WindowsMRDefines.ScriptingDefineSymbol, "WindowsMR", "WSA")]

[bddckr]

@tomwim Looks good, just note that you don't need a scripting define symbol (and the whole SDK_WindowsMRDefines class probably, as you aren't relying on a third party SDK to be imported into the project, right?!

It's only needed for something like SteamVR: We can't know whether the project includes the needed SteamVR plugin, so that's why there's a Defines class that has a predicate method to check for the existance of SteamVR. 😄

[tomwim]

The idea was to check whether Virtual Reality Support with Windows MR is set in the Player Settings or not. I probably need Reflection since checking Application values is only supported on runtime.

What is your suggestion on using SteamVR? Yes and build upon SteamVR or rather depend 100% on the Unity integration without including SteamVR at all?

[bddckr]

The idea was to check whether Virtual Reality Support with Windows MR is set in the Player Settings or not.

VRTK handles this for you the SDK author, and the user already 😄
You just need to set the vrDeviceName (third parameter) of the SDK_Description to the name that shows in the PlayerSettings XR list.
VRTK will automatically manage that list based on that info (and only if the Auto Manage VR Settings checkbox is enabled on the SDK Manager). It's completely automatic 😃

You only need a scripting define symbol if your SDK implementations are relying on third party content, like in my SteamVR example above.

---

I suggest to use the SteamVR as soon as that is finalized and usable. Especially if you want to release on Steam. If you want to release on the Windows Store you have to use the MR SDK then.
Note that you can build one executable for all supported SDKs if done correctly.

May be of interest: https://www.youtube.com/watch?v=QwefwHF5E78

[bddckr]

Don't hesitate to join our discussions in the VRTK Slack at invite.vrtk.io!

[SimonDarksideJ]

As noted on the Slack. Finally got headset tracking, position and movement working.
Now on to the fun part, the controllers :P

[tomwim]

So my current status is that the headset and controllers are fully tracked and interaction works fine so far. Boundaries might be altered at some point. I am currently going through all example scenes integrating and testing Windows MR, hoping everything work.

Only Microsoft and the "amazing" Mixed Reality Portal are causing trouble since on Play Unity often (every 5-10 times) freezes...

After making sure all examples run I'll give the Controller models another try since they are loaded on runtime with GLTF which made huge problems last time I tried.

[SimonDarksideJ]

So do you have another working branch @tomwim as you seem on par with where I'm up to. if so, I can pause and comment on yours.

[tomwim]

Sorry for the super late reply @DDReaper Yes, my current working branch is on github.com/Innoactive/IA-unity-VR-toolkit-VRTK -> feature/windows-mixed-reality Maybe you can give it a look and we can decide how we continue.

[thestonefox]

Good work so far guys!

[thestonefox]

Anyone working on the steam vr and unity core support? Or is it just the unityengine.xr.wsa support at the moment?

[SimonDarksideJ]

Looking good @tomwim you've taken where I got to and moved on further, also managed to get the SDK approach working (which I failed at, but can't see why your's IS working :S)
Only issue is that I couldn't run the project as I got the following error:
"Graphics.CopyTexture called with region not fitting in destination element (dstX 0, dstY 0, srcWidth 1792, srcHeight 1767, dstMip 0)"

As your code is very similar to mine (and mine runs), I'm not sure why this is occurring.

Added some issues for code changes (and the above issue) for tracking.

[tomwim]

@DDReaper never seen that error...I'll give it a look as soon as possible but due to holidays and vacation I probably won't be able to do it before Wednesday. Thank you for the comments!

@thestonefox I am currently only working on WSA.XR support but I got a Windows MR SteamVR Beta Key 2 days ago. So I think I'll work on SteamVR and core support as soon as WSA works.

[bddckr]

For SteamVR support I'd recommend to wait until they finalized their bridge... Things like velocity reporting, model loading etc. should just be in the bridge, so we don't have to include it.

[thestonefox]

Acer Headset

Under Windows Universal Platform

UnityEngine.XR.XRDevice.model = acermixedreality
UnityEngine.XR.XRSettings.loadedDeviceName = windowsmr

Under Standalone - SteamVR Support

UnityEngine.XR.XRDevice.model = ` UnityEngine.XR.XRSettings.loadedDeviceName=openvr`

[tomwim]

Short Update:

• Currently all example scenes except for 12 (weird offset), 35 (no interaction with the cube) and 37 (no falling and height offset after climbing and teleporting) work fine.

• The controllers are not visualized yet. Needs to be done through GLTF. This is next on my list.

• No haptic feedback yet. As far as I know that's an Win API call as well.

Note: Use Unity MRTP4, which solved some problems.

[SimonDarksideJ]

I've made some notes on the issues on your repo @tomwim .
I think some of the issues you noted above are actually with the VRTK possibly, especially about falling, etc.

Haptic Feedback, not sure this is possible with the MR controllers, but would love feedback on that (no pun intended)

As for controller visualisation, the MRTK has gLTF importers which work with the editor, so probably worth grabbing that code.

[thestonefox]

Linking this in case it's of use https://github.com/if1live/VRTK-Microsoft-MotionControllers-sample

[if1live]

@thestonefox I am repository owner.
I don't have much time, but I will support VRTK repository.
My VRTK-Microsoft-MotionController is based on old VRTK, Between v3.1.0 and v3.2.0.
It requires MixedRealityToolkit
I use old VRTK, so I doubt it is useful to current VRTK.

@DDReaper I extract microsoft motion controller model.
left hand : https://github.com/if1live/VRTK-Microsoft-MotionControllers-sample/blob/master/Assets/Toko/Prefabs/GLTFScene_Left.prefab
right hand : https://github.com/if1live/VRTK-Microsoft-MotionControllers-sample/blob/master/Assets/Toko/Prefabs/GLTFScene_Right.prefab

[tomwim]

Models are great!
Controllers are now supported in Editor as well. It is in a newer MRTK version.

[SimonDarksideJ]

Have you added the controller rendering to your project @tomwim ? Couldn't see it.

Also, we should ensure we copy the necessary code from the MRTK and not make it a dependency. Each toolkit works a different way with different purposes for the moment.

[if1live]

@tomwim @DDReaper haptic is supported in windows mr. I use it my game.
https://github.com/Microsoft/MixedRealityToolkit-Unity/blob/master/Assets/HoloToolkit-Examples/Input/Scripts/HapticsTest.cs
But not supported in editor. Works in UWP build.

[tomwim]

@DDReaper no I haven't included them yet. I also think the better way is to load them via gLTF and only use the Models for default or as override. And yes, I would just copy the code and not make it dependent.

Thank you @if1live I'll include that as soon as the models are in!

[thestonefox]

I've started a PR adding SteamVR support

https://github.com/thestonefox/VRTK/pull/1622

Not tested it (because still don't have a VR dev machine). but it should work.

Plus it creates a new "TouchpadTwo" concept which we can then hijack for the core support.

The whole touchpad two rubbish needs to be replaced at some point though with the demise of the controller events script!

[wikieden]

Is there any ETA time for WMR support in 3.3? And why still don't have a vr dev machine.....Hope you have now

[thestonefox]

@wikieden no i still don't have a dev machine, someone is supposed to be loaning me a vr ready laptop but there seem to be some delays.

As with anything in VRTK, it's hard to provide an ETA for anything because development time is as and when someone can find to do it.

[tomwim]

Update from my side (without the latest alpha build changes thestonefox made - this is the next step):
I had the chance to talk to some Microsoft guys last week who were able to help me out with some issues I still faced.

• Controllers are visualized and animated now in Editor and Build, scene change still shows them in the next scene (new Newtonsoft.Json DLL is included)
• Unity does not crash anymore on scene change with WinMR
• Haptics is included (Build only) but it sometimes works and sometimes not, sometimes only one controller, sometimes long, sometimes short...
• VRTK now builds with .Net Scripting Backend, which is highly recommended by Microsoft. IL2CPP works as well of course

Sorry for the long time it takes me to do this but the biggest challenge here is actually not to include Windows MR into VRTK but to make it work with UWP...

That is my current working branch: https://github.com/Innoactive/IA-unity-VR-toolkit-VRTK/tree/feature/windows-mixed-reality

[thestonefox]

@tomwim is the plan to squash all of your commits on your branch into one commit and pull request it into vrtk?

[tomwim]

That's the plan, except if you have a better idea

[thestonefox]

That's fine, I just saw all the commits in your repo, figured a squash was in order before the pull request :)

[tomwim]

The current best working Unity Version is MRTP5:
https://forum.unity.com/threads/custom-build-2017-2-rc-mrtp-windows-mixed-reality-technical-preview.498253/

[SimonDarksideJ]

Fantastic news @tomwim
Will try and test your latest update and feedback. I believe it was close already, but certainly sounds like it's a good fit for alpha and then take it forward there.

[thestonefox]

Also, just as a side note. the SteamVR Beta released on 27th November 2017 broke the thumbstick input for the WMR controllers. So if you try the touchpad two stuff in VRTK and it's not working, blame steamvr :)

[SimonDarksideJ]

So, something to fix either during, or preferably after, the merge of the WinMR work while it's still in alpha

[thestonefox]

@DDReaper i mean the SteamVR beta software from valve is broken, it's nothing to do with VRTK

[wikieden]

MS has recommend developer to use unity 2017.3 for MR developing which is still in beta channel, I will test it with mrtp5 and beta3

[wikieden]

@tomwim I will test your package with HP MR Headset， I also have a Samsung HeadSet. And I will send you feedback

[tomwim]

@wikieden thanks for the help and testing, just open issues at the fork. That would be great. I only have a early Acer Development Headset

About the Unity version: I was at a MS event 2 weeks ago and they repeatedly emphasized to not use 2017.3 but to always use the newest MRTP since that has constant updates for Mixed Reality. But I think testing for both won't hurt :)

[thestonefox]

my personal opinion on this not using official unity for windows mr is we should only look to support it officially when it's officially supported by unity, otherwise it's a cat and mouse game

[SimonDarksideJ]

WindowsMR is supported in 2017.2. However, I agree Unity are sitting on the fence on some underlying issues which they are not exposing their proper plans to resolve, be it in a 2017.2 path or even 2017.3

[thestonefox]

I don't like the idea of supporting this off branch builds that Unity are doing, because whatever changes to suit their needs means we're just constantly playing catchup. Maybe we'll just support 2017.2 of what we know and wait for them to sort out what they're doing :)

[tomwim]

Opened a pull request for Windows Mixed Reality support (alpha) which includes

• Mixed Reality Headset and Controllers
• Mixed Reality in every example scene
• Controller visualization (with GLTF)
• MRTK scripts and plugins (where needed)
• Touchpad Two (thumbstick) in Controller Events
• Highlighting and tooltips for Mixed Reality Controller
• .NET scripting backend support

[265lutab]

What is the time frame for having Windows Mixed Reality Support added to VRTK or can I download a version now that has it?