Webxr: Encourage use of the depth buffer for the XR WebGL context

[rcabanier]

For good headpose prediction, it helps if the author provides a depth buffer.
The spec is currently silent on this requirement.

Can we add some language to the current spec to encourage authors to create a valid depth buffer for the scene? Maybe there could also be a boolean on the XRSessionCreationOptions if the depth buffer should be used by the graphics backend.

[toji]

I would be supportive of a boolean somewhere that indicates that the developer intends to leave the depth buffer in a valid state, sure. We've talked about it a few times, but it's never been a primary concern. It could be used by other systems for improved reprojection and system UI rendering as well.

I'm interested in your assertion that it helps pose prediction, could you clarify that? Are you talking about a form of reprojection? Also, does Magic Leap's API require validity of the depth buffer to be declared early on (at the equivalent of session creation time?) or can it be toggled dynamically. Such a flag would feel more at home to me on either the XRWebGLLayer or the XRRenderState.

[rcabanier]

Are you talking about a form of reprojection?

Yes, it's for reprojection which results in a more stable headpose.

Also, does Magic Leap's API require validity of the depth buffer to be declared early on (at the equivalent of session creation time?) or can it be toggled dynamically

We always require that there is a depth buffer but headpose still works if none is provided. It just won't be optimal.
If the author provides garbage data in the depth buffer, it would give bad results so we need to know if we should ignore it. This can be done dynamically after the session creation.

Such a flag would feel more at home to me on either the XRWebGLLayer or the XRRenderState.

I agree

[avaer]

+1 for the language change, because there's no other way to ensure valid depth buffer other than to politely ask.

This might belong to the language issue, but it might be nice for the UA to signal that without the depth buffer you will have a bad time, maybe being able to reject the session or layer on the basis of such a flag. I can imagine compositing architectures that won't work if the client isn't providing depth info.

[rcabanier]

Rejecting sessions that provide no depth buffer might result in lots of broken experiences if developers only test on platforms that don't require one.
If we made it a requirement, I suspect it should probably only be for AR sessions

[thetuvix]

+💯

Agreed on the critical value of having a useful depth buffer from the app! This is key for allowing per-pixel reprojection on devices that support it, which provides a large increase in visual quality, especially when apps are dropping frames.

Even on today's HoloLens, which supports only full-image reprojection using an app-driven stabilization plane, we've found that the platform calculating that plane from the app's depth buffer works far better than relying on app developers to submit such planes themselves. There are just too many corner cases to cover (e.g. the user looking away from the main subject of their scene, the user standing very close to a subject that only covers 5% of their FOV) where an app developer calculating the plane in the most obvious way (e.g. calculate offset from main subject to the camera) actually makes the effective stability worse. We've basically stopped telling developers to specify planes manually.

Given that, if I had to choose between #525 and this issue for WebXR 1.0, I'd definitely prefer that we cover the 99% case by just letting developers "submit" the depth buffer they filled in. We could then add in an additional affordance for #525 for expert developers that really need to go further and specify a custom stabilization plane, though I question whether web developers thinking across a wide range of devices will really do better than they would just submitting their depth buffer 99% of the time.

[cabanier]

That's great to hear.
Magic Leap currently doesn't require a depth buffer but we might tighten that requirement in the future because it's very hard to render a stable scene without it.

Given this, should we make it a requirement or are we ok with a suboptimal path that is easier to render?

[klausw]

I'm all in favor of encouraging apps to provide a useful depth buffer if they have it, but making this a platform-specific requirement would mean that some apps just wouldn't work on those platforms. Making it a hard requirement for all WebXR apps seems like an extra burden for developers that may make some types of apps either not work or be less efficient.

Some rendering methods may not have a useful depth buffer, or would need to do extra steps to make it available. A deferred shader may be doing rendering in a separate FBO where the final copy into the output buffer doesn't involve depth testing. Other rendering methods such as stereo video or volumetric rendering may also not use a traditional depth buffer.

If an application were to provide a depth buffer with partially filled in data, would that be worse than not providing one at all?

[cabanier]

@klausw I was proposing to make it a requirement for all immersive AR sessions.

If an application were to provide a depth buffer with partially filled in data, would that be worse than not providing one at all?

I'm unsure. I will ask internally

[toji]

Given the discussion in this thread and my understanding of the various platform's usage of depth buffers, I do feel like it's generally something we want to encourage applications to provide. It also seems like the average web application, given that deferred rendering is not as popular on the platform, is probably naturally producing the necessary depth values anyway.

As a result, I would suggest that if we're going to add a method to allow the depth buffer to be used, it would be beneficial if we made it the default state, requiring applications to opt out if they know that their depth buffers do not contain valid values. If that's going to be the case, it's necessary that we do so early, before any serious applications are built atop the API. (The UA, of course, would be under no obligation to make use of the depth buffer.)

My initial impression would be to add it to the XRWebGLLayerInit like so:

dictionary XRWebGLLayerInit {
  boolean invalidDepthValues = false;
};

Such that in order to opt out of the UA utilizing those values they'd need to do something like this:

session.updateRenderState({
  baseLayer: new XRWebGLLayer(session, gl, { invalidDepthValues: true })
});

[NellWaliczek]

Bikeshed alert!! What about ignoreDepthValues instead?

[toji]

Actually, I prefer that because the term "invalid" seems overly broad given the intent. You can have "valid" depth values that nevertheless wouldn't be appropriate for something like reprojection.

[ddorwin]

Speaking of bikeshedding, would it be better to use a positive attribute that defaults to true instead of a negative attribute? For example, useDepthValues.

[toji]

Web best practices are to have Boolean dictionary values default to false whenever possible because it meshes better with developer understanding that undefined evaluates as false. The WebGL init values are an exception to that rule because they're trying to match a native interface more directly.