Mixedrealitytoolkit-unity: What's the current status of using Vuforia with the MR Toolkit?

[Alexees]

I got confused by all the different version acknowledgements and recommendations (Unity 2017.2, 2017.1.2, MRTP4...). I just want to know which Unity version is capable of running MR Toolkit for Hololens along with Vuforia?

[eluchsinger]

Vuforia works with Unity even before 2017.2. The only difference I understand there is is that starting with Unity version 2017.2, it's included in the Unity installer. Before you had to add it yourself.

[eluchsinger]

Adding to that, I would recommend you using Unity 2017.2 with MRTK.

[StephenHodgson]

I'd only recommend using 2017.2 with the latest master or dev branch, but not with the release

[Alexees]

My proplem is, why I'm asking, using all that together results in the ARCamera rendering on top of the MixedRealityCamera. The former WorldPositionAnchor field is gone so you can't reference the MR Cam with the AR Cam. Trying to merge both cameras behaviours doesn't work, because the Vuforia Behaviour freezes the camera in place.
How do you set this up, did I miss setup guide?

[shllybkwrm]

@Alexees Did you read through the new Vuforia page? https://library.vuforia.com/articles/Training/getting-started-with-vuforia-in-unity-2017-2-beta.html

[Alexees]

@shllybkwrm I did, but it's Vuforia only. I'm talking about Vuforia used with the Mixed Reality Toolkit and there's no reference to it on the page

[shllybkwrm]

@Alexees There's a section on digital eyewear and on the Hololens page linked, there's a new sample project and configuration instructions. You're right that they don't specifically mention the toolkit but they do have instructions for HoloLens and other headsets.

[Alexees]

@shllybkwrm that's right, but I wouldn't have asked here if it wasn't specifically for the toolkit.

[reddo123]

Is there any updates on compatibility with Vuforia and MRTK? I'm building with Unity 2017.3.1p1 and the Hololens samples here https://developer.vuforia.com/downloads/beta.
However, it's the same issue as Alexees mentioned above on December 1:
"using all that together results in the ARCamera rendering on top of the MixedRealityCamera. The former WorldPositionAnchor field is gone so you can't reference the MR Cam with the AR Cam. Trying to merge both cameras behaviours doesn't work, because the Vuforia Behaviour freezes the camera in place."
Is there any specific setup or workaround for this?

[volpespakka]

Hello guys, same here...
I can't put together holotoolkit and AR camera. I also tried a manual initialization adding vuforia behaviour and defaulterrorinitializationhandler scripts to the mixedrealitycamera of the toolkit but in play mode it freezes . The only solution i can think of in my specific case is to have a vuforia scene with the ar camera (with the only drawback that i can't use toolkit input) and after a manual deinitialization switch to the holotoolkit scene . This is just an idea , can't make it work for now ahah

[samharper94]

I'm getting it to work on 2017.3.0p1 with a HoloLens, the AR camera replaces the HoloLens camera prefab, just make sure to change its tag to "MainCamera" and make sure you still have the InputManager prefab in the scene. Not sure about MR as I don't have access to one of those headsets.

[reddo123]

Are you getting any cursor visible or input on objects? I'm certainly not getting any toolkit input when deployed to Hololens. No cursor or anything as the ARCamera seems to override anything from the toolkit.
In the editor, cursor and input on a model semi works with AR camera/webcam in background, albeit flipped.

[samharper94]

Yes, I'm able to use my app exactly as before, except now I've got Vuforia for image targets too.

My steps for getting it to work:

1. Enable Vuforia and import the MRTK, apply scene setting etc.
2. Right click on the hierarchy > Vuforia > ARCamera.
3. Delete the HoloLens camera/MixedReality camera from the scene, remember to copy over the GazeControls child, if you want camera control in the editor be sure to copy over the Manual Gaze Control script as well.
4. Very important: Tag the AR camera as MainCamera, this means the cursor should pick it up.
5. Window > Vuforia Configuration > Digital Eyewear, select Digital Eyewear from Device Type, and HoloLens from Device Config.
6. Edit > Project Settings > Player > Vuforia Augmented Reality support, make sure this is ticked.
7. Build with 10.0.16299 as the SDK.

If these don't work, sorry I've got no idea, I've managed to make 4 or 5 applications using Vuforia since 2013.3.0p1, and I can also get it to work on 2013.3.1p1. I haven't tried using the example apps, there's a chance they may be broken. I'm willing to keep helping though, let me know how it goes.

[reddo123]

Thanks for the steps Sam. Will run some samples and get back. Cheers

EDIT: Yes those steps did the trick building a fresh project. So the Vuforia project examples have some kind of issue if building them as is. Thanks again.

[SimonB2011]

Sam, I'm trying to follow your steps but getting a bit lost. Are you able to show a screen cap of your final hierarchy?

[samharper94]

@SimonB2011

[SimonB2011]

That's fantastic and working for me now. Thanks !

[JeffreyJacobson]

Thank you, Sam! That worked for me, as well. :-)

[Alexees]

@StephenHodgson I suggest the setup guide from SimonB2011 to be put onto the main page of the MRTK

[StephenHodgson]

I'd actually like to make adjustments to the MixedRealityCamera prefab to make sure it's setup for people already, in addition to the setup steps.

@SimonB2011 any chance you could do that and open a PR with your suggested setup steps and a updated MixedRealityCamerad prefab?

[StephenHodgson]

@samharper94 How come you delete the MRCamera and replace with the ARCamera? Couldn't people just add the components needed for Vuforia?

1. Enable Vuforia and import the MRTK, apply scene setting etc.
2. Right click on the hierarchy > Vuforia > ARCamera.
3. Delete the HoloLens camera/MixedReality camera from the scene, remember to copy over the GazeControls child, if you want camera control in the editor be sure to copy over the Manual Gaze Control script as well.
4. Very important: Tag the AR camera as MainCamera, this means the cursor should pick it up.
5. Window > Vuforia Configuration > Digital Eyewear, select Digital Eyewear from Device Type, and HoloLens from Device Config.
6. Edit > Project Settings > Player > Vuforia Augmented Reality support, make sure this is ticked.
7. Build with 10.0.16299 as the SDK.

[samharper94]

Yes they can! One of them needs to be removed from the scene, I chose to delete the MRCamera and add the GazeControls instead but either way works. It's just the way I've gotten used to doing it now. It does also work the other way around.

Also, are we talking about my setup steps? Or Simon's? Forgive me I'm slightly confused.

[SimonB2011]

@samharper94 definitely yours. I can claim no credit for this! Personally I tried keeping the MRC but had no luck running that way round, just lots of crashes.

[StephenHodgson]

Okay, I'll add the setup steps to include the MRCamera.

[krreddy888]

Hi @StephenHodgson , @samharper94 . I tried using the methods mentioned in this discussion for using Vuforia with MRTK mixed reality camera. The problem that I am facing is, if I attach Vuforia components to mixed reality camera, whenever my image target is tracked, it is not showing the hologram that should be shown when tracking is found. Though the methods for tracking found, tracking lost are getting triggered.

If I have both the camera's in the scene, the ARCamera and MRTK camera, a memory leak happens. The application memory slow increases and finally a crash occurs. (This leak is happening only in Hololens and not in UnityEditor) Anyhelp on why it is happening would be of great help to me. Thanks in advance.

[samharper94]

Hi @krreddy888, you should only have one camera in the scene. Up to you whether you want to use the MR camera with Vuforia components, or the AR camera with GazeControls and the background set to black. Either works for me.

In terms of the tracking getting lost, I'm not sure. Is your camera starting at the origin (0,0,0)? Is your room well lit etc.? Does your image target have a high readability score in the Vuforia target manager?

[krreddy888]

Thanks for the reply @samharper94 . The problem that I am facing is, if I have only one camera in the scene, and when the image target is detected, there is a huge offset in the hologram, between it's actual position and desired position. It comes too close to the camera and is culled by the camera though the Culling mask is set to 0.01.

If I have two cameras the hologram comes at the exact position. I am stuck with this. I am using Vuforia's default Astronaut image target and my room is well lit. Can you suggest me what's the problem??

[samharper94]

A couple of things maybe, is the target printed at exactly the right size? If it's not, the hologram can appear way off its desired position. You can fix this by taking the actual measurements of your printed target and going to the ImageTarget GameObject > Image Target Behaviour > Advanced and type in a width. It's vital the starting position for the camera in world space is 0,0,0 as well.

[krreddy888]

Yes I resolved it. This was a silly error. So there is a script called Mixed reality camera manager attached to mixed reality camera. It also updates the clipping plane of the camera. I didn't set clipping plane there and it was clipping it. Setting clipping plane there resolves the issue. Thanks @samharper94 for your support.

[JeffreyJacobson]

@StephenHodgson , @samharper94 I followed Sam's instructions, keeping the Vuforia camera and copying over the Gaze Controls. When I run the app, the sphere appears immediately, and Vuforia works fine, popping up the cube on top of the target. But both objects shimmer and shake when I move around and being about one meter from them make them really go crazy--they look a little better when I am either closer or further away!

Here is my hierarchy:

When I use a voice command to turn off Vuforia (via a script I modified), the sphere still shakes a lot when I move. However, when delete the Default Cursor and the Input Manager, then recompile, the sphere will be rock steady. But only while Vuforia is off.

I tried this using both Unity 2017.3.1 and 2017.2.1p2 and go the same behavior. In the latter, I also tried using the MRTK camera with the Vuforia behavior script added, and again got the same behavior!

Any ideas?

The next thing I'll try is anchoring the objects before turning off Vuforia, like I'm supposed to do, anyway. But I have a feeling that won't work, because the sphere should already be well-anchored by default.

[JeffreyJacobson]

Yes! Dropping an explicit anchor is the thing to do! I added the "TapToPlace.cs" script to the Sphere (which calls the anchor-manager) and it's rock steady, now, after I tap-place it.

[krreddy888]

Hi @samharper94 , I was successfully able to set vuforia with mixed reality camera. The issue I am facing now is, in my application I need to disable and enable vuforia. For that I set the vuforiaBehaviour.instance.enabled to true or false. Whenever I set the vuforiaBehaviour.Instance.enabled to true, the application crashes, not everytime but 4/10 times. Is there any better way with which I can set vuforia active and inactive, any suggestions in this regard would be great help to me.

[samharper94]

I actually don't know, I'm sorry. But I'm also interested in hearing a solution.

[shllybkwrm]

@krreddy888 Do you actually need to turn off Vuforia, or would it be enough to not show the child object of an image target? I've done this with a toggle that changes the child object between active and not.
Otherwise I suggest posting in the Vuforia forums for help from their end.

[krreddy888]

Thanks for the reply @samharper94 and @shllybkwrm . Actually the application I am working at also requires streaming. If I don't disable vuforia at all, I cannot stream the application. Currently with disabling and enabling vuforia, I am able to stream at least the parts of the application where scanning is off, but the problem is application crashes randomly whenever the vuforia instance is enabled. I will try to get a solution from the vuforia forums. Please let me know if you find out any solutions to this particular problem.

[JeffreyJacobson]

@krreddy888 are you using a plug-in or your own code to do that streaming? Once I get my anchor code working the way I want it to, I'll be trying to mix Vuforia with the Umbra plug-in that streams data.

My guess would be that both the stream and Vuforia want to use a lot of bandwidth, and the crash is happening because of some kind of timing issue. Might be solvable with tests and error handling for boundary conditions.

[krreddy888]

@JeffreyJacobson, I am not using any plugin for streaming, I am using only the Hololens mixed reality portal to live stream. The problem is whenever vuforia scanning is switched on, live preview of Mixed reality portal wont work. That's why I am switching off vuforia whenever scanning is not needed so that I can atleast live preview some part of the application.

The problem is not related to streaming, it is related to enabling and disabling vuforia behaviour in runtime. Investigating the issue.

[krreddy888]

Hi all, I managed to fix the crash. It was an internal Vuforia issue and I had to upgrade my Unity and Vuforia to Vuforia SDK 7.0.50. This fixed the crash.

[shllybkwrm]

Hi all, has anyone tried to use Vuforia with the MixedRealityCamera prefab? I've always used the HoloLensCamera prefab, which works, but I recently updated to HLTK 2017.4.0.0, so I thought I would try the new setup. In play mode, my webcam light goes on, but I get a background like this and no webcam.

I'm attaching the Vuforia scripts to the camera like this (the same way I do with the HLCamera).

[mcboyd-bu]

@shllybkwrm I have an identical setup. It's my first time trying to add Vuforia to an existing project, so I have only ever tried it with the MixedRealityCamera. It works successfully for me, but not in Play mode or when debugging - it only works correctly when remotely deployed in "release" mode to my actual Hololens.

And for anyone else who doesn't already know, I spent a full day figuring out that Vuforia apparently requires the "microphone" as well as the "webcam" to be enabled in Capabilities. Without the microphone, Vuforia will start successfully, but the webcam won't recognize any vumarks (or presumably images or objects).

[Ariel-Feldman]

Hi, Also for me, the setup in @shllybkwrm last photo is working ).
Mind that it also successful when you preloaded the mixed reality camera prefab from the previous scene (which is usually the use case when integrating Vuforia in your app), under DoNotDestoy objects :)!.

[AbrieC]

Hi All, I was wondering if anyone has tried running Holographic Remoting with Vuforia? I can't seem to get it working.

Any advice about getting this to work would be greatly appreciated.

A

[Yoyozilla]

This would be a great question on Stack Overflow.

[MountainRains]

Hi All, I was wondering if anyone has tried running Holographic Remoting with Vuforia? I can't seem to get it working.

Any advice about getting this to work would be greatly appreciated.

A

Did you solve it?
I also encountered this problem in MRTK 2.4