Gvr-unity-sdk: Front-facing camera access

[IanPhilips]

I've seen #962 but I'd like to put my own spin on it. We're currently writing VR software for people with movement disabilities who can't easily take off and on the headset themselves. Access to the front-facing cameras would be extremely useful for them to check in on their surroundings if something changes, i.e. a person enters the room.

Imagine someone entering the room while you have the headset on and they don't say anything - it would be creepy if you can't take off the headset to see!

I'd like to file a feature request if that's possible, thanks!

[IanPhilips]

Here's Microsoft's feature: https://uploadvr.com/microsoft-mixed-reality-flashlight/

[jadware]

here it is on the Mirage Solo: https://developers.googleblog.com/2018/09/new-experimental-features-for-daydream.html

[IanPhilips]

It says those features are coming but I haven't seen anything in the API about them, when is their ETA?

[jadware]

don't know when but it definitely looks like it's coming

[rusmaxham]

There will be an API to enable limited compositing-layer only access to the front facing cameras. Initially the app will not have any access to the camera buffers, only turn on compositing of an underlying camera layer to enable SeeThrough mode.

[dymdev]

There will be an API to enable limited compositing-layer only access to the front facing cameras. Initially the app will not have any access to the camera buffers, only turn on compositing of an underlying camera layer to enable SeeThrough mode.

Hi rusmaxham,
Could you explain how to do it on Unity?
Thank you.

[rusmaxham]

The API is not yet available.

[ehsuun]

Any updates regarding the see-through mode API?
2D apps on daydream is now available and they were announced on the same blog post.

[rusmaxham]

This will be in the next release of the SDK due by the end of March.

[jadware]

Cannot wait for this feature :) We have such an excellent use case to share once we can build it into a release...

[andybak]

Initially the app will not have any access to the camera buffers,

Sounds like we need this to do anything interesting. If we're getting the very limited version in March, any ETA on buffer access?

[rusmaxham]

Enabling see-through mode is now available in the latest release. Camera image buffer access is a more complicated issue, but we will look into it.

[floe]

Thanks @rusmaxham ! For the time being, would it be possible to do texture readback on the background layer?

[rusmaxham]

No, the see-through image is composited out of process by VR Services. The background layer is in the VR Services process, not the app's process, thus inaccessible to the app.

[floe]

OK, thanks for the clarification. Are you planning to expose something like a shared-memory area with the camera data in the future?

[rusmaxham]

We are looking into it, but there are some security and privacy policies around see-through that are enforced deep in the OS that we will have to reevaluate before we're able to implement this feature.

[andybak]

Related question - how about AR Core support on the Lenovo Mirage. I can think of some incredible use-cases around this and it would put Daydream into a very unique category. Considering the existing close relationship between ARCore and Worldsense (by my understanding they are both descendants of Project Tango) - I'm assuming this is on your radar?

With the hype around AR and the lack of any plausible head mounted display at a reasonable cost - being able to do pass-through AR is extremely exciting.

[jadware]

My simplest request that shouldn't cross any privacy policies would be to track a marker in the real world. The app provides an image to the system which then places a unity object where it "sees" the marker in the real world. Would allow some blending of both for AR experiences without the app getting access to the video feed.

[floe]

@rusmaxham seems like Daydream is getting axed, no? Any chance of parts being open-sourced for the research crowd?