Webxr: Should we just remove `viewMatrix` and rename `transform` to `viewTransform`?

[thetuvix]

In looking through #440, I wonder if any non-trivial engine with an actual scene graph will ever make use of viewMatrix, or if transform is _actually_ all they will evaluate each frame for all purposes, including positioning the camera in the scene relative to the app's chosen reference space origin.

As a case study, in the Windows MR native APIs, we exposed a similar member to viewMatrix: HolographicCameraPose.TryGetViewTransform(), returning you a Left and Right view matrix, each as a Matrix4x4. However, when engines like Unity get a hold of this each frame, they immediately decompose it into a position and orientation for the view, which they then apply as the world transform for their Camera's GameObject in their scene graph. At the end of the frame, the engine then just calculates its own view matrix from the Camera's final position and orientation, identically to how it would for a non-XR game where all camera movement is app-driven.

My own mental model here is that WebXR is not so much the first 10% of an opinionated XR render engine (e.g. dealing directly with view matrices, etc.), as it is a source of low-policy poses for tracked objects, such as viewers, input sources, planes, meshes and more. UAs can consistently deliver those poses (including viewer poses) as positions and orientations within app-created reference spaces, with engines then picking up from there to position their scene cameras appropriately, with the engine then calculating view matrices relative to their scene origin precisely as they've done for many years.

In that model, do we see a clear use case for engines to use viewMatrix without just decomposing it into transform and manually recomposing it before rendering anyway? If not, perhaps we should just remove the viewMatrix attribute and rename transform to viewTransform, defining it simply as you have here.

_Originally posted by @thetuvix in https://github.com/immersive-web/webxr/pull/440_

[thetuvix]

Here is a version of drawScene from the explainer, updated for the proposal above:

function drawScene(view) {
  if (view) {
    scene.camera.setTransform(view.viewTransform.position, view.viewTransform.orientation);
    scene.camera.projectionMatrix = view.projectionMatrix;
  } else {
    scene.camera.setTransform(defaultCameraPosition, defaultCameraOrientation);
    scene.camera.projectionMatrix = defaultProjectionMatrix;
  }

  // Set uniforms as appropriate for shaders being used

  // Draw Scene
}

[toji]

Even if we do drop the viewMatrix I don't know that I feel renaming transform to viewTransform is necessary, since it should be contextually clear what it's transforming. Looking at your code above, I don't find this any more confusing to read, and the repetition of the word "view" is eliminated:

scene.camera.setTransform(view.transform.position, view.transform.orientation);
scene.camera.projectionMatrix = view.projectionMatrix;

It would be more important IMO if we had another transform on the view, but I can't really see that happening?

It does feel a little weird to me that we have a matrix for projection and a decomposed transform for the view, but that's a relatively minor quibble. Maybe we should add an 'inverseMatrixtoXRRigidTransform` (lazily evaluated, of course) to accommodate any cases where the app actually does want a matrix instead, while still representing the view's transform as a single object?

[thetuvix]

Just keeping the name transform does make sense to me after seeing the code.

The inverseMatrix sibling attribute to matrix also makes much more sense to me than having viewMatrix one level up. Engine developers can then get a pose from an XRRigidTransform in whatever form is most directly useful for them: as a position/orientation, or as a transform matrix in the direction that fits their use case. While something did feel nice about having sibling viewMatrix and projectionMatrix attributes (which is why many of our native APIs had chosen that design), my take is that it's proven to be false symmetry - engines generally do just decompose the view matrix provided and create a model transform for the camera, either as a position/orientation or as a forward model matrix, so we should probably just accept that reality.

One additional benefit to having all of those attributes and them being lazily evaluated is that internal UA implementations can avoid double transformations for apps choosing any option, regardless of whether the underlying native XR API uses view matrices, model matrices or position/orientation to report view poses.

[toji]

Slight tangent: In your experience have you seen the same tendency to attempt and decompose projection matrices as well?

[thetuvix]

I've found it more common than with view matrices for engines to have their scene camera just support custom projection matrices directly, allowing apps to implement their own exotic effects. (e.g. Unity's Camera.projectionmatrix property)

However, I know of no VR/AR platforms that currently require an app to render a single eye buffer with a projection matrix more complex than can be expressed with 6 frustum planes.

For example:

• Wide FOV headsets like StarVR ask apps to render more than one view per eye, but each such view gives out a projection expressible as 6 frustum planes.
• HoloLens originally expressed a shear factor in its native projection matrices, but we quickly moved away from that, as it made app code more complicated with little benefit. We now give out a common projection matrix that can be expressed as 6 frustum planes.

A WebXR app that believes it needs to harden against arbitrary projection matrices may unnecessarily skip optimizations or complicate shaders for no effective benefit. In a world where app developers may already come to the web with skepticism about XR performance through JS, we should help web engines make all the optimizations they can.

WebXR could structurally express such a constraint on its view projections by returning each view's projection without the excess generality implied by a Matrix4x4 - for example, as the angle of the left/right/top/bottom frustum planes. This could require some extra code for engines as they ingest the projection data, but hopefully those engines can then more confidently optimize in a manner supported across all UAs and XR platforms.

[toji]

I see, thanks!

HoloLens originally expressed a shear factor in its native projection matrices, but we quickly moved away from that, as it made app code more complicated with little benefit. We now give out a common projection matrix that can be expressed as 6 frustum planes.

I think that this :arrow_up: comment may possibly have broken my co-editor. :wink:

[NellWaliczek]

Indeed it did. I'm ... good now.... Aaaaaanyway.

1. I'm open to removing the viewMatrix, though I'm strongly inclined not to add an inverseMatrix property because anyone using WebXR/WebGL will already be using matrix math library.
2. Sounds like we should file a separate issue to track removing the projection matrix in favor of FOV.

[blairmacintyre]

Sounds like we should file a separate issue to track removing the projection matrix in favor of FOV.

Not really in favor of that. It means we can never support things that have off-axis projection, like true magic window displays (e.g., zSpace), correctly rendered dioramas, or projection-based AR. Only simple HMDs and simple-camera-based handhelds.

[NellWaliczek]

I've filed #461 to discuss the projection matrix issue.

[darktears]

I'm fine with removing viewMatrix because the developer can just access the same information using the transform.matrix from the XRView. Repeated information is not a good pattern.

Personally I'm fine keeping the transform name because it's easier to grasp when someone get started with WebXR, for example one would read it as "that's how the scene is transformed".

[toji]

I'm fine with removing viewMatrix because the developer can just access the same information using the transform.matrix from the XRView. Repeated information is not a good pattern.

@darktears, that's actually not quite correct. The transform.matrix would represent the inverse of the viewMatrix. (It's essentially the transform that would would need to apply to a camera object.) That's part of why I would be nervous about dropping viewMatrix entirely is that I think you'd get a lot of devs who make the same assumption. (Though to be fair, you're not gonna get very far in your application development if you make that mistake...)

[darktears]

I had a chat with @toji to clarify why I misinterpreted in the first place and thought the two were equal values. Provided that I myself uses viewMatrix often I think it's a good convenience. And while @NellWaliczek pointed out most of the developers will have some ways to quickly invert a matrix if needed, it doesn't add much complexity in the browser implementation to keep that convenience for folks operating with the viewMatrix directly. Also I expect that most devs will just head to the viewMatrix property as their first choice....

[thetuvix]

@darktears:

Provided that I myself uses viewMatrix often I think it's a good convenience.

Interesting, thanks for the insight! Is this because you are building out a direct WebXR-first rendering engine, rather than integrating into an engine that already treats cameras as an arbitrarily-positionable scene object?

Also, if we do have WebXR directly provide the inverted matrix, which of the following schemes would you find most sensible for providing the three formats of view transform to apps?

Scheme A:

1. view.transform.position/orientation
2. view.transform.matrix
3. view.viewMatrix

Scheme B:

1. view.transform.position/orientation
2. view.transform.matrix
3. view.transform.inverseMatrix

@NellWaliczek:

I'm open to removing the viewMatrix, though I'm strongly inclined not to add an inverseMatrix property because anyone using WebXR/WebGL will already be using matrix math library.

While I agree that anyone implementing a render engine will have easy access to matrix inversion, providing all three representations on XRRigidTransform could help avoid UAs needless matrix conversions if it turned out the app wanted the same format as the underlying XR system providing the pose data.

For example, if the underlying XR platform expresses view transforms to apps as view matrices (i.e. inverse matrices), and then @darktears' render engine will be using them as view matrices, it would be a shame for a UA to convert them to either position/orientation or matrix, just to have the app immediately convert them back. Letting the app directly request the format it intends to use could allow a UA to lazily convert system-provided transforms only as necessary, though perhaps that optimization is in the noise.