three.js VR renderer does frustum culling based on only the left eye, causing premature clipping on the right eye.

[ngokevin]

Description of the problem

The VR renderer does frustum culling based on the frustum of the left eye camera. It applies the same culling to the right eye. As a result, when an object leaves the left eye's view, the user can see the object flash out on the corner of the right eye periphery.

Saw https://github.com/w3c/webvr/issues/203 was linked.

https://github.com/mrdoob/three.js/pull/11523

Three.js version

• [x] Dev
• [ ] r89
• [ ] ...

Browser

• [ ] All of them
• [ ] Chrome
• [x] Firefox
• [ ] Internet Explorer

OS

• [x] All of them
• [ ] Windows
• [ ] macOS
• [ ] Linux
• [ ] Android
• [ ] iOS

Hardware Requirements (graphics card, VR Device, ...)

VR Device

[ngokevin]

@mrdoob From talking yesterday, yeah, I can notice in VR from the Rift in my right eye as objects leave the left eye's frustum.

[mrdoob]

Is that with https://threejs.org/examples/webvr_cubes.html?

[ngokevin]

Yeah, I can notice it there and on our own projects with Rift. Should be more apparent if you have longer objects.

[mrdoob]

Longer vertically I guess?

[amakaseev]

you can see this problem in a code

_projScreenMatrix.multiplyMatrices( camera.projectionMatrix, camera.matrixWorldInverse );
_frustum.setFromMatrix( _projScreenMatrix );

but

// HACK @mrdoob
// https://github.com/w3c/webvr/issues/203

cameraVR.projectionMatrix.copy( cameraL.projectionMatrix );

[amakaseev]

I think increase fov in cameraVR.projectionMatrix, will solve this problem

[mkeblx]

Just changing FOV not what wanted. The linked PR at top looked good conceptually to work in the general case. Would that not work?

[amakaseev]

Yes there almost everything is truly described except the picture in the first post
Correct frustum has to look so:

[ngokevin]

The culling issue has seemed to worsen on landing of r90?

Try https://threejs.org/examples/webvr_vive_paint.html in a Vive, all objects disappear including the controlller unless they are the far left side of field of view.

[mrdoob]

The culling issue has seemed to worsen on landing of r90?

Did it worsen for all headsets or just for Vive?

Try https://threejs.org/examples/webvr_vive_paint.html in a Vive, all objects disappear including the controlller unless they are the far left side of field of view.

I don't have a Vive. @dmarcos do you mind testing this? I have the feeling #13158 has something to do with this.

[dmarcos]

I can see the regression and narrowed down the changes that caused it: https://github.com/dmarcos/three.js/commit/7bf42e19134f26e99bf555d19a934b9f2ff973c8

Looking closer to understand what happened. @brianpeiris ?

I opened a PR in case you want to get it fixed for now (https://github.com/mrdoob/three.js/pull/13350). I tested on Vive / Oculus.

[brianpeiris]

Not sure what's going on here. I'll have to get my hands on a Vive to understand it. I'll add comments to #13350 as well.

[brianpeiris]

I haven't been able to reproduce any culling issues in my Vive with r90 (d55897b). I see a culling issue in my Rift with r90 and r89 as well, so I don't think this is the standing matrix change after all.

[brianpeiris]

Correction: dmarcos helped me repro it. It's very intermittent for me. But when it does occur it shows an obvious culling issue on the left of the frustum, where it culls too early.

dmarcos came up with these steps to repro:

1. Exit SteamVR and Firefox
2. Start Firefox
3. Visit https://threejs.org/examples/webvr_vive_paint.html
4. Enter VR
5. Exit Firefox
6. Visit https://threejs.org/examples/webvr_vive_paint.html

[brianpeiris]

Alright, I think I've solved this correctly in #13414.
I was able to reproduce it much more consistently if I setup my Vive playspace so that it was rotated 90 degrees (or any angle other than near-zero) from my screen.

[LAGENCECREE]

I confirm https://threejs.org/examples/webvr_cubes.html have a camera culling issue on window 10/vive/last firefox , far distance is extremely wrong on both eyes + left eye controlling the culling ( gfx card, Radeon rx 480 )

[dmarcos]

https://github.com/mrdoob/three.js/pull/13414 solves part of the problem. Frustum culling is still performed with just one eye resulting in disappearing objects. @mrdoob should we reopen or you prefer to have a discussion somewhere else?

[brianpeiris]

Thanks for the clarification @dmarcos. I didn't realize there was still a culling issue after my fix because it's not very obvious in the "cubes" demo. It's visible, but not when you're in the HMD -- you can see it more clearly in the browser's stereo view though.

@omgitsraven provided a clear demo in aframevr/aframe#3488 which I've converted to three.js code here:
https://edge-disappear.glitch.me/
https://glitch.com/edit/#!/edge-disappear

[ngokevin]

This still happens (e.g., on Oculus Rift). Objects disappear out of the corner of the right eye while turning head. Is there an issue this is being tracked?

[mrdoob]

Oh, I guess this issue got closed automatically...