Three.js: WebXR implementation is capped at two views

[Manishearth]

Description of the problem

https://github.com/mrdoob/three.js/blob/f76e149e681cdf33a16623248376c04a1038dae7/src/renderers/webxr/WebXRManager.js#L31-L43##### Three.js version

This code caps the webxr implementation at two views.

However, the webxr spec allows for any number of views!

CAVE displays will have a high number of views. Within head-mounted displays, "first person observer" views for AR capture exist (e.g. on the Hololens 2), as well as "bionic displays" on the Varjo VR-2.

The camera object should perhaps preallocate two cameras but append additonal cameras as more views are discovered.

(I came across this issue after implementing the "first person observer" view for Hololens 2 in Servo and had to patch threejs to be able to run demos)

• [x] Dev
• [ ] r116
• [ ] ...

Browser

• [x] All of them
• [ ] Chrome
• [ ] Firefox
• [ ] Internet Explorer

OS

• [x] All of them
• [ ] Windows
• [ ] macOS
• [ ] Linux
• [ ] Android
• [ ] iOS

Hardware Requirements (graphics card, VR Device, ...)

[mrdoob]

(I came across this issue after implementing the "first person observer" view for Hololens 2 in Servo and had to patch threejs to be able to run demos)

Where can I read more about that?

[Manishearth]

https://microsoft.github.io/OpenXR-MixedReality/openxr_preview/specs/openxr.html#XR_MSFT_first_person_observer_preview

[Manishearth]

I've also filed https://github.com/immersive-web/webxr/issues/1045 to have more information about such views in the spec itself

[mrdoob]

It would be nice if the WebXR spec could help with frustum culling somehow.

Right now we create a single frustum so we don't do the frustum culling pass twice. This works when both cameras are pointing the the same direction but would break in the CAVE case.

This 3rd person view seems to also complicate things.

[Manishearth]

The third person view in most cases is a secondary view that cannot see things that left/right cannot, so culling should not be an issue.

[asajeffrey]

A case when we'd need this is recording 360 video from webxr content. This is implemented in Servo, by rendering the webxr using six views which are the faces of a cubemap, then post-processing via gstreamer. An example is at https://www.youtube.com/watch?v=74mX-FB4FdA

In order to do this, I modified three.js to hard-wire six cameras rather than two, but ideally that shouldn't be necessary! One wrinkle is that in webxr you don't know how many views there are until you begin a session, so the three.js cameras can't be allocated until the session is created.

See also https://twitter.com/mrdoob/status/1284340064521199618 :smile:

[Manishearth]

Oh, an update:

The spec now talks about this explicitly. There is now a "secondary-views" feature that can be used to opt in to extra views showing up. Threejs should be tweaked to handle views > 2 and set the "secondary-views" optional feature unconditionally.

As @asajeffrey mentions you'll need to handle this anyway for rendering to non-binocular systems like CAVE systems and cubemap rendering. These are not considered secondary views and may be surfaced even if the feature isn't requested.

[mrdoob]

So seems like in order to cull only once we'll have to parse the frustums for each view provided by the WebXR API every frame. Is that correct?

[asajeffrey]

@mrdoob yeah, at the moment webxr doesn't provide any data about the views saying whether or not they can share culling. Moreover the relative transforms between the views are allowed to change every frame, even though for most devices they don't change much (or indeed at all).

[Manishearth]

@mrdoob You can cache it though.

But this is already the case, you can change the frustum of a view by e.g. changing IPD on the oculus using the slider.

[mrdoob]

@mrdoob You can cache it though.

Hmm, how? Does the view have an id?

[Manishearth]

Oh, no it doesn't.