Three.js: setProjectionFromUnion makes incorrect assumptions

[cabanier]

The comment from setProjectionFromUnion states:

Assumes 2 cameras that are parallel and share an X-axis

For Magic Leap devices, that is not the case and because of this, this function calculates incorrect values which results in the scene not rendering correctly.
It's observable by virtually pinning content to a wall or whiteboard and then walking away. You will then see that the content will rotate or move from its location.

WebXR specifies that projection matrices are not supposed to be decomposed:

It is strongly recommended that applications use this matrix without modification or decomposition. Failure to use the provided projection matrices when rendering may cause the presented frame to be distorted or badly aligned, resulting in varying degrees of user discomfort.

[cabanier]

There is a way to calculate the half-angles from the projection matrix when the cameras are not parallel, but maybe it would be better if the browser provides these values since it's quite complicated.

@mrdoob, why is three.js calculating another projection matrix?

[cabanier]

We looked the code over and we understand what it's doing now.
We're planning on updating the algorithm to work with any set of projection matrices.

[mrdoob]

/cc @jsantell

[jsantell]

Any assumptions were for optimization reasons; if that doesn't hold true on magic leap, then we probably need to decompose both projection matrices -- decomposing the projection seems OK as this is only used for frustum culling, yeah?

What assumptions break on magic leap, are the cameras not parallel?

It's observable by virtually pinning content to a wall or whiteboard and then walking away. You will then see that the content will rotate or move from its location.

Hm this should only affect culling..

[cabanier]

Any assumptions were for optimization reasons; if that doesn't hold true on magic leap, then we probably need to decompose both projection matrices -- decomposing the projection seems OK as this is only used for frustum culling, yeah?

Yes. The link to the stackoverflow article proposes a more generic solution. We're using that to create another algorithm.

What assumptions break on magic leap, are the cameras not parallel?

Correct. Because of eye tracking, the projection matrices will contain a unique ipd, fov values and an additional rotation component.

It's observable by virtually pinning content to a wall or whiteboard and then walking away. You will then see that the content will rotate or move from its location.

Hm this should only affect culling..

You're correct. This is caused by something else. We're investigating if this is a bug on our side or three.js. Since other sites look fine, we suspect it's three.js but we won't file an issue until we're sure.