Webxr: Get video background in Webgl AR session

[devilinthedata]

Hi

Is it possible to get hold of the video passthrough layer as a texture map within WebGL, when running an WebXR AR session? This would be useful for lighting / rendering effects, and possibly enhance the lighting effects to make the object 'sit' in the scene more realistically. I can't find anything in the API to do this, but it seems like it ought to be doable. We render into the baselayer, and I wondered if there were other layers we could get hold of somehow?

Thanks for any help, and best regards

Matt

[cabanier]

The API intentionally does not allow access to the real world cameras for privacy reasons.

[Manishearth]

Yes, this is out of scope, however I believe @blairmacintyre was somewhat interested in this for CV, and @mounirlamouri was interested in this too (?). I could swear there were proposals for this but I can't find them right now.

For lighting estimation there's a separate proposal: https://github.com/immersive-web/lighting-estimation/

[blairmacintyre]

In https://github.com/immersive-web/computer-vision and in the experimental API we created for the WebXR Viewer https://github.com/MozillaReality/webxr-ios-js we proposed an API that returns video frames (obviously, with user-permission required).

Getting access to the video layer itself is the wrong approach here: it isn't compatible with see-through displays, which would have video cameras running at a different framerate than the rendering system (e.g., hololens, ML, etc). So, a reasonable API would return frames with a timestamp and camera extrinsics (pose) using the same monotonically increasing timestamp the frames use.

It would be relatively easy to say that in the case where the camera and the rendering frames are synchronized, the camera frame will be return immediately before the frame callback (for example).

[cabanier]

I agree. If an author wants the scene to be more part of the real world, we need APIs that return information about the real world (like geometry and lighting).
Returning a raw camera feed and then relying on JS computer vision will not be as good as tying into the system's native world understanding.

[devilinthedata]

Thanks a lot for all the helpful comments. I will keep an eye on the lighting estimation proposal stuff.

[cabanier]

Thanks a lot for all the helpful comments. I will keep an eye on the lighting estimation proposal stuff.

Was that the feature you were looking for?
If not (or you have other asks), it's really helpful to hear about your use case.

[devilinthedata]

Of course, I'm glad to share. We're doing 3d product visualisation in webgl, and the realism of our 3D rendering is a big selling point for us. We're keen to integrate an AR mode into our presentations, which webXR hopefully allows us to do. However having the lighting of the product be harmonious with the environment it appears in, adds greatly to the realism - grounding the model in the scene, so to speak.

We mostly use environment based lighting - but any info we could get regarding the lighting env. of the scene would be very helpful for us - simple average colour , single light direction, spherical gaussians.

I've a lot of experience in such realtime lighting schemes, and am definitely up for discussing possibilities, if that would be useful.

Cheers

Matt