Three.js: Feature Request: allow ArrayCameras to opt in to multiview rendering

[lojjic]

Description of the problem

When the OVR_multiview2 extension support landed (#16316), it was limited to running only in VR contexts (#17596). This was done, as I understand it, to prevent unexpected console warnings for all ArrayCameras, and was probably prudent as a risk mitigator anyway.

This is a request to open that up slightly, by giving ArrayCamera a way to explicitly opt in to multiview rendering outside of the vr.enabled scenario. This could be as simple as a multiviewEnabled or similarly named boolean property on ArrayCamera, which could then be checked alongside vr.enabled in the appropriate spots in WebGLRenderer:

if ( ( vr.enabled || camera.multiviewEnabled ) && multiview.isAvailable() ) {...}

This approach should have no impact on code that doesn't opt in.

Three.js version

• [x] Dev
• [x] r110

Browser

• [x] All of them
• [ ] Chrome
• [ ] Firefox
• [ ] Internet Explorer

OS

• [x] All of them
• [ ] Windows
• [ ] macOS
• [ ] Linux
• [ ] Android
• [ ] iOS

Hardware Requirements (graphics card, VR Device, ...)

N/A

[mrdoob]

This is a request to open that up slightly, by giving ArrayCamera a way to explicitly opt in to multiview rendering outside of the vr.enabled scenario.

Out of curiosity, what's your use case for wanting to using multiview without vr.enabled?

[lojjic]

Out of curiosity, what's your use case for wanting to using multiview without vr.enabled?

Legit question. I'm actually still wanting it for the XR use case, but in the context of our own framework. For various reasons (different scene graph abstractions, needing control over the frame loop and object lifecycles, etc.) we bypass WebXRManager and replicate the relevant bits of its behavior, including an ArrayCamera-like camera of our own. That's what I'm hoping to allow multiview on, even though the lack of multisampling limits its utility.

Otherwise, to be totally honest, I'm just speculating that anywhere someone might use an ArrayCamera they might also benefit from a multiview option. I don't have any specific cases in mind, but it seems probable they exist?

[marcatec]

Maybe it could be used for a CAD like Quadview, showing the scene from front, side, top and perspective at the same time.

[mrdoob]

@lojjic

Thanks for sharing your use case. How do you think the API should look like?

[lojjic]

@mrdoob Sorry, I'm not sure if you're asking about API for exposing multiview capability, or API for WebXR management in general based on my use case?

If the former, I imagine a boolean property on ArrayCamera as described above should suffice, unless there are other details I'm not aware of.

If the latter, I'd be happy to try putting together some thoughts on that based on my experience, but I wouldn't say it's a failure of Three's WebXRManager API in any way, just a difference in assumptions about the way scene objects are managed within our Troika framework. For example all objects are updated in a strictly controlled way that lets it optimize matrix updates, and it controls the frame loop itself so it can call its own lifecycle hooks, etc... not necessarily things Three's API can or should care about. But I'll give some thought to whether WebXRManager could better support these things, and if I come up with anything I'll open a separate issue.

[mrdoob]

The thing is... I have yet to see an example where multiview solves a performance issue that InstancedMesh couldn't 🤔

[fernandojsg]

fwiw the current implementation won't work with a CAD use case unless you disable frustum culling (afaik it's not possible right now) as the frustum culling test is done against just one camera. So we should create a generic frustum culling implementation that will do the test against all the cameras, or use a specific combined test as the VR camera already does.

[lojjic]

I have yet to see an example where multiview solves a performance issue that InstancedMesh couldn't

InstancedMesh is a great tool, for sure, but it's got some pretty narrow parameters to when you can use it (same geometry, same material). Troika has an instancing solution of its own (which also lets you instance other things like colors of builtin materials), and we make extensive use of it wherever possible, but there are many things you just can't do without extra draw calls.

In XR, the frame budget is very tight as you know, so there's often a balancing act between performance and scene complexity. Multiview is no panacea either of course, but halving the draw calls changes the math in a way that favors more featureful scenes.

It sounds like you're maybe arguing for removing multiview support entirely from Three.js, not against this specific request per se. Do you think its performance benefit is not worth the complexity it introduced?

---

@fernandojsg That's a great point about the frustum culling, and may be a legitimate argument against my feature request, at least in the short term. It severely narrows the cases where an ArrayCamera could use multiview without unexpected frustum culling issues. It would work for my WebXR-without-WebXRManager use case, but not much else from the sound of it. 🤔

[fernandojsg]

Here you can see a proposal of the generic frustum culling implementation in case you find it useful https://github.com/mrdoob/three.js/pull/11523

[mrdoob]

It sounds like you're maybe arguing for removing multiview support entirely from Three.js, not against this specific request per se. Do you think its performance benefit is not worth the complexity it introduced?

Well, I don't know yet, because it's limited to antialias off and I don't know who wants to go into VR and see aliasing. Right now it doesn't seem very useful imho and it feels like the addition was premature.

[fernandojsg]

The thing is... I have yet to see an example where multiview solves a performance issue that InstancedMesh couldn't 🤔

I don't get that, even if you use instancedMesh in your app, you could still use multiview and you will be dropping the number of drawcalls the same way. For sure for an specific example as just using 1 drawcall to paint plenty of cubes you will probably won't get a lot of improvement (still you will get something), but that's not a common use case I believe.

Regarding adding multiview or not, honestly I thought that we would have antialias on MV way earlier, although Oculus browser has already an implementation on that and I know another browser vendor is working on that too so I'm hopeful that it will become a thing in all the browsers soon.
Once we get antialias support in MV the current code and API will remain practically the same with just some few changes on the implementation.

And currently you could still give it a try with antialias off to determine your frame budget for whenever we will get the antialias MV so I'd vote for keeping while we wait for that to happen.

[mrdoob]

@lojjic

If we want to have progress in this issue I need to be convinced to invest time on redesigning all this. Any chance you can create an example that convinces me? Ideally using Oculus Quest as the target.

[lojjic]

@mrdoob That's fair. This felt like a very simple enhancement to the current solution to me (I definitely wasn't asking for a redesign of anything!) But if doing it correctly will indeed have cascading effects, like requiring a new frustum culling solution as @fernandojsg pointed out, then you have to decide if it's worth that effort. I'm not sure I can prove that, especially given the current antialiasing situation.

FWIW, I do have a hacktacular workaround for my XR use case: overwrite all of WebXRManager's methods with empty functions and set vr.enabled like normal so it triggers the multiview code paths but doesn't do anything else. Gross but it seems to work.

[Mugen87]

Since OVR_multiview2 support has been removed (see #18750), I guess this issue can be closed for now?

[lojjic]

Yes indeed. Closing.