Mixedrealitytoolkit-unity: Photo location data (CamToWorld and Projection matrices) on HoloLens 2 via Unity.XR SDK

[dl4mmers]

Describe the bug

Hey,

I don't know if this is the right place for this issue and I'm sorry if it's not. I don't know where to report it, because I don't know if this is caused by MRTK or Unity API itself. I will also create a Unity bug report regarding this issue.

We are currently porting our HoloLens 1 application to support using HoloLens 2. The application has a 2D-Whiteboard which displays an image taken with the HoloLens RGB Camera via Unity.XR.WSA.WebCam.PhotoCapture (https://docs.microsoft.com/en-us/windows/mixed-reality/locatable-camera-in-unity). When taking the photo we save the location data (CamToWorld and Projection matrices) by calling:

photoCaptureFrame.TryGetCameraToWorldMatrix(out c2wMatrix);
photoCaptureFrame.TryGetProjectionMatrix(out projMatrix);

On HoloLens 1 everything works fine and we are using the matrices to project annotations (freehand drawings, circles, etc.) made on the 2D-Whiteboard into the 3D world to create spatial annotations.

Now, when using the same code compiled for ARM on HoloLens 2 we are not able to get location data. Calling photoCaptureFrame.TryGetCameraToWorldMatrix(out c2wMatrix) or photoCaptureFrame.TryGetProjectionMatrix(out projMatrix) always returns false and we are getting back identity matrices. For the moment we are stuck porting this functionality to HoloLens 2.

To reproduce

Steps to reproduce the behavior:

1. Create a new MRTK project and scene
2. Add a script that starts PhotoCapture like described here: https://docs.microsoft.com/en-us/windows/mixed-reality/locatable-camera-in-unity
3. In OnCapturedPhotoToMemory add the following code

        if (result.success)
        {
            Matrix4x4 c2wMatrix;
            Matrix4x4 projMatrix;

            photoCaptureFrame.TryGetCameraToWorldMatrix(out c2wMatrix);
            photoCaptureFrame.TryGetProjectionMatrix(out projMatrix);

            Debug.Log("CamToWorld:");
            Debug.Log($"{c2wMatrix.ToString()}");
            Debug.Log("Projection:");
            Debug.Log($"{projMatrix.ToString()}");

        }
        // Clean up
        photoCaptureObject.StopPhotoModeAsync(OnStoppedPhotoMode);

1. Compile for x86 and run on HoloLens 1 (correct matrices should be logged)
2. Compile for ARM and run on HoloLens 2 (identity matrices should be logged)

Expected behavior

Location data should be available for HoloLens 1 and HoloLens 2.

Your setup (please complete the following information)

• Unity 2018.4.0f1 / Unity 2018.4.16f1 (using .NET scripting backend)
• MRTK v2.2

Target platform (please complete the following information)

• HoloLens
• HoloLens 2

[wiwei]

@keveleigh FYI, sounds kinda familiar based on some chats we have had a long time ago. I'm guessing these are not MRTK specific because they're just some underlying Unity APIs

[keveleigh]

Yep, this sounds very similar to that bug. I can make sure it's on their radar

[wiwei]

@keveleigh Alrighty - if you have a Unity bug link handy that could be helpful to share out here otherwise because this doesn't seem to be an MRTK issue I'll close this out.

[qazqaz12378]

I also encountered the corresponding problem, is it solved?

[dl4mmers]

Unity says the problem is fixed (https://issuetracker.unity3d.com/issues/photocaptureframe-fails-to-provide-camera-matrices-on-hololens-2) and indeed photoCaptureFrame.TryGetCameraToWorldMatrix or photoCaptureFrame.TryGetProjectionMatrix returning true now and matrices have values in it. Unfortunately the values are not correct and can't be used like the matrices on HoloLens 1.

Here is the new ticket: https://issuetracker.unity3d.com/issues/photocaptureframe-provides-wrong-camera-matrices-on-hololens-2

PhotoCaptureFrame.TryGetCameraToWorldMatrix and PhotoCaptureFrame.TryGetProjectionMatrix returning true now, but matrices are not correct. When calculating rays from 2D PhotoCaptureFrame into the 3D world by using these matrices, on HoloLens 1 the rays are correct, while on HoloLens 2 the rays are totally wrong.

When taking two PhotoCaptureFrames with different viewing angles, it can be seen that the HL1 CameraToWorldMatrix has different values while the HL2 CameraToWorldMatrix is nearly the same. That can't be correct.

Example:

PhotoCaptureFrame's CameraToWorld matrices from both HoloLenses viewing north:

HL1:

0.96491 -0.06117 -0.25534 -0.08812
0.01769 0.98542 -0.16920 0.15655
-0.26197 -0.15875 -0.95193 -0.25989
0.00000 0.00000 0.00000 1.00000

HL2:

0.01313 -0.99939 -0.03227 0.00257
-0.99966 -0.01239 -0.02304 -0.05001
0.02263 0.03257 -0.99921 -0.01091
0.00000 0.00000 0.00000 1.00000

PhotoCaptureFrame's CameraToWorld matrices from both HoloLenses viewing south:

HL1:

-0.96757 0.05874 0.24569 -0.24733
0.03725 0.99513 -0.09121 0.11127
0.24985 0.07910 0.96505 -0.14452
0.00000 0.00000 0.00000 1.00000

HL2:

0.01313 -0.99939 -0.03227 0.00257
-0.99966 -0.01239 -0.02304 -0.05001
0.02263 0.03257 -0.99921 -0.01091
0.00000 0.00000 0.00000 1.00000

[qazqaz12378]

I also saw this article，But this explanation can’t be used in actual projects at all

[keveleigh]

The first phase of a fix went out, where (as you've seen) data is flowing on HL2.
Unfortunately, (as you've also seen) that data isn't completely correct just yet. It's also been observed that every photo after the first continues to return the same matrix. Both cases are under investigation by Unity and I can provide updates here as things are resolved.

[skyzdev]

still experiencing incorrect repeating matrices with the latest Unity LTS versions (2019.4.14f1 and 2018.4.29f1)

[keveleigh]

I'm planning to schedule some time this week or early next week to work with Unity on this issue. Will keep this thread updated!

[ketankapila2]

Hi @keveleigh, can you please provide an update on this issue?

[keveleigh]

@ketankapila2 We've identified the source of the issue and are working through the fix now.

[ketankapila2]

@keveleigh, thank you for your quick response. Are you able to share the expected timeline for release?

[ketankapila2]

@keveleigh, following up on this

[mr0ng]

@wiwei looks like this issue might still be open. Let me know if there's another open issue that needs to be linked to this one or this one reopened

[keveleigh]

@mr0ng I think we've been keeping this one closed since it's not an MRTK issue (we don't have any scripts using this Unity API). It's always been a fine balance between using these issues to track Windows/Unity issues vs only tracking MRTK issues 😕

@ketankapila2 I don't yet have a timeline for release. I'm getting the fix reviewed and into the release pipeline, at which point I should have a better idea!

[zorderjohn]

I'm trying to figure out a workaround for this issue. In theory we should be able to get the matrices using the winrt API.
Is there any code we can try while this is solved on the Unity side?