Webxr: Encouraging apps to avoid mid-frame flushes.

[toji]

This is an issue of GPU specific behavior that I don't have a great answer for, so for the moment I'll focus on describing the problem and we can bounce potential methods of addressing it off of each other from there.

On some tiled architectures various heuristics are used to determine that the depth/stencil buffer doesn't need to be persistent and therefore can live entirely in the tile cache. This provides a good memory and performance benefit on mobile GPUs, but comes with a problem: If the GPU context switches away from the current set of rendering commands, switches framebuffers, or flushes mid-frame then the depth buffer information may be lost entirely. This manifests as a variety of visual artifacts.

In general the solution for this is to do all of your rendering to a framebuffer at once before changing bindings or flushing. This should be pretty manageable, but definitely requires an understanding of what's going on and adherence to best practices. For example, the following pattern is asking for trouble (for more reasons than just this artifact):

bindFramebuffer(shadowbuffer);
renderShadows(light1);
bindFramebuffer(xrLayer.framebuffer);
renderSceneWithShadows(light1);
bindFramebuffer(shadowbuffer);
renderShadows(light2);
bindFramebuffer(xrLayer.framebuffer);
renderSceneWithShadows(light2);

That's gonna hurt you just because of pipeline stalls, but on certain mobile chipsets you'd could end up with serious artifacts as well.

Technically this is a problem for any WebGL content, but we've seen that it's more problematic for WebXR content for a variety of reasons. One specific example is that we've seen that enabling foveated rendering extensions on Qualcomm chipsets tends to make applications more sensitive to this. (I'm going to defer to @klausw for questions regarding that. I'm relaying this information second hand.)

So the question is: Should we do anything in the spec, normatively or otherwise, to try and mitigate this? Especially since it would be pretty easy to build content that looks fine on desktop but has artifacts on mobile VR. At minimum we could add some non-normative spec text that describes the problem and recommends that developers do all rendering to the layer framebuffer in a single shot. If we wanted to be more aggressive about it we could actually enforce that the XRWebGLLayer's framebuffer can only be bound once per frame, but that seems like a frustration and over-reaction. Or we could just say "WebGL be crazy, yo!" and do nothing.

Thoughts?

[klausw]

To clarify, mid-frame flushes generally don't cause corruption when using standard framebuffers, but it causes "unresolves" which are extremely inefficient. It can cause undefined results and corruption when some GLES extensions are active.

When there's a mid-frame flush, the GPU needs to save tile memory to main memory for all pixel samples and depth values, then read them back later. For efficient tile-based rendering, you'd want to do all sample and depth computations inside the fast tile memory, and just save the final resolved color samples to the destination framebuffer.

For example, if you're using 2xMSAA, you'd have 2 samples per pixels with 4 bytes RGBA and 4 bytes depth+stencil each, for a total of 16 bytes written, 16 bytes read back, and then 4 bytes resolved to the output framebuffer. That overhead usually wrecks performance. Interestingly, the effect of this may not be as dramatic for synthetic test scenes with flat colors since the GPU may be using compression to avoid saving the full data, but more natural scenes are heavily affected.

Some newer GLES extensions try to avoid the cost of unresolves by prohibiting them or declaring the result as undefined behavior. Since these extensions are opt-in, this doesn't break existing applications, but it makes it impossible to transparently use such extensions as part of automated optimizations since they may not be safe.

For example, the QCOM_framebuffer_foveated extension says:

    6. How does unresolving from a foveated framebuffer work?

    Loading from a foveated framebuffer is undefined, so the app must be
    sure not to trigger mid frame flushes.

(The newer QCOM_texture_foveated extension which replaces it has slightly different semantics, but mid-frame flushes would still cause bad side effects.)

The EXT_multisampled_render_to_texture2 extension says:

    Remove the following wording describing FramebufferTexture2DMultisampleEXT:

        "and have the same restrictions.  attachment must be COLOR_ATTACHMENT0."

    In the description of FramebufferTexture2DMultisampleEXT, after the sentence
    "After such a resolve, the contents of the multisample buffer become undefined.",
    add the following sentence:

        "If texture is a depth or stencil texture, the contents of the multisample
        buffer is discarded rather than resolved - equivalent to the application
        calling InvalidateFramebuffer for this attachment."

[...]

    1. How should downsampling depth/stencil surfaces be handled?

    Proposed: Ideally, when using this extension, depth/stencil attachments should
    always be discarded/invalidated by the application *before* the rendering is
    submitted.  If the application fails to do so, the implementation is not required
    to preserve the contents of those attachments.  A depth/stencil resolve is
    equivalent to InvalidateFramebuffer for those attachments.

Additional references:

• Qualcomm® Adreno™ OpenGL ES (PDF), Qualcomm Developer Network, section 6.4.11 "Avoid intra-frame resolves

[toji]

Thanks, Klaus, for correcting my fuzzy explanation! 👍

[klausw]

Another example sequence would be using an offscreen FBO for reflection. This triggers mid-frame flushes when drawing each eye independently:

  bind reflection FBO for left eye
  draw reflected scene for left eye
  bind output FBO
  draw left eye viewport
  bind reflection FBO for right eye <== mid-frame flush
  draw reflected scene for right eye
  bind output FBO <== expensive unresolve of left-eye multisamples + depth data
  draw right eye viewport

To avoid the costly (and potentially optimization-preventing) mid-frame flush, applications would need to change the sequence to complete all offscreen FBOs first before binding the output FBO.

Does that sound like an acceptable restriction if it has the potential to significantly enhance performance on tiling GPUs, and could enable optimizations such as foveated rendering?

@Artyom17 - Oculus Go supports foveated rendering for native apps, have you considered using that for WebXR also? If yes, this scenario would likely affect you also.

[RafaelCintron]

I am fine with the spec offering developers suggestions for optimizing content, but only if those optimizations do not have adverse affects on some platforms. If suggestions help some platforms and hurt others, then there needs to be a way for developers to detect and branch accordingly.

I do not think the WebXR API should restrict the number of times WebGL framebuffers should be bound.

[klausw]

I think this is an interesting case. Enforcing this restriction would allow efficiency gains on some platforms, and as far as I know there aren't any platforms where incrementally updating framebuffers would be more efficient, so it's more like "help some platforms, no difference on others".

Of course there is a cost to developers if the rendering sequence needs to be adjusted, but I also suspect that many were not aware of the high cost of incremental rendering on some platforms.

The opaque drawing buffer used for WebXR is already not quite a regular FBO due to having lifecycle restrictions, so I was hoping there may be more flexibility in making the API facilitate optimizations.

One problem is that it's not enough for the application to simply refrain from multiply binding the buffer if the goal is to activate optimizations, since some require setup at framebuffer creation time, and at that time the system doesn't know yet how the buffer will be used. Using heuristics to observe application behavior and then adjusting the render pipeline on the fly would in theory be possible, but this seems hackish and is also not entirely safe, i.e. the app could start using reflection buffers only once a reflective surface appears in view.

Is there any good precedent for an API with opt-in restrictions, where the app could declare ahead of time that it's going to cooperate? For example, a session creation argument single-bind-framebuffer: true or similar? Implementations that benefit from this could use it to activate optimizations depending on this, but if it's ignored on other implementations it would be a potential source of bugs if this hint affects semantics inconsistently.

[Artyom17]

I would love to have such restriction in place since I've ate enough of this issue with WebVR on mobile. On good drivers mid-frame flushes do not cause artifacts, just wasting a lot of GPU time on unresolves (2+ ms per frame - is normal wastage of such kind). But on 'bad' drivers, drivers with bugs, mid-frame flushes may cause extreme artifacts, the good example is Samsung S7 with SD820: mid-frame flushes cause black patches (tiles) flashing during the rendering. And even though QCOM has fixed this issue 2 years ago, Samsung & vendors haven't updated them OTA.
My only question how that gonna be enforced or even detected. WebXR and WebGL are completely separate APIs and WebGL definitely allows to shoot yourself in a foot by various ways.

[Artyom17]

@klausw: shoudn't that flag -single-bind-framebuffer - be specified what you create a WebGL context rather than WebXR session?

[klausw]

@Artyom17, I did mean WebXR session specifically for the opaque WebXR drawing framebuffer. That would only help solve the specific issue of repeatedly binding that one, so people could still shoot themselves in the foot with other framebuffer usage, but I think it would catch very common cases, including drawing eyes separately where each eye's drawing involves a separate source framebuffer. And since the WebXR drawing buffer is already special, i.e. it's not possible to rebind its attachments, changes at that level may be less controversial.

A more general WebGL-level API would be useful also, but I think that changes to semantics of standard WebGL FBOs would need to be done as part of a WebGL spec change or extension.

[Artyom17]

@klausw I see. I was under impression that the whole idea of opaque framebuffer is under the scrutiny right now, at least for multiview / WebGL 2 cases. If we are going to get rid of the opaque framebuffers or even if we have a way around it then this restriction won't do anything....

[RafaelCintron]

@artyom17 brings up a good point about the WebGL 2 alternative. If the WebGL 2 alternative becomes part of the spec, then the only "special" object is the output texture, not a framebuffer. Developers can attach the output texture to many framebuffers they create themselves. They can (and probably should) reuse depth buffers across multiple renders during a frame. Implementations that want to manage framebuffer based extensions under the covers will need to “upgrade” all framebuffers that attach the output texture and “downgrade” ones that detach the output texture.

I do not think the API should enforce single bind semantics. For all we know, the textures and renderbuffers may be part of a texture cache used to make generated content during scene loading. As the browser, I do not think we should second guess the developer’s intent.

I would be in favor of developer tool warnings when the browser sees problematic behavior.

[casseveritt]

The multisampled_render_to_texture extensions were unfortunate because they exposed a reality of the tiler abstraction that had otherwise been hidden in an API (OpenGL) designed to express immediate mode operation. It would have been possible at the cost of extra (mostly unused) system memory to ensure that mid-frame flushes were only catastrophic to performance and had no impact on image quality, but that is not how the extension was written.
In general mobile GLES apps don't have the luxury of not enabling multisampling in tile memory only, as its value proposition is compelling despite the flaws in its specification. I don't think there is an obvious answer for WebXR except that it will somehow have to expose this functionality for apps that rely on it. I can't say whether it needs to be part of the core API, but it needs to at least be available.
In the longer term, perhaps other low level APIs will provide a better way of exposing this without the vagaries of mid-frame flushes.

[toji]

As we're nearing the "VR complete" milestone, I think we should clarify how we approach the types of optimizations discussed in this issue.

First off, I don't see a path to making a change such that the _default_ WebGL layer behavior is one that forbids things like mid-frame flushes so that optimizations can be applied silently. It's a difficult thing for developers to understand if they're not immersed in graphics development minutia, and I imagine that we'd get a lot of people picking up a functioning bit of Three.js code and throwing it in XR and then wondering why it's broken.

I'm fully in support of allowing developers to opt-in to optimizations that carry with them various restrictions on how you are allowed to interact with them. I do think we should aggressively experiment, likely through either a layer module or a module more explicitly dedicated to rendering optimizations, with enabling things like fixed foveated rendering, multiview rendering, and other similar approaches. At that point, if developers turn on fixed foveation and then proceed to complain that it's not compatible with their multi-pass soft shadowing technique, well... 🤷‍♀

We should also document best practices where necessary to explain how to avoid performance traps in the context of WebXR, but I don't see that there's much we as the Immersive Web group can or should be doing to address one of the most fundamental issues of GPU development.

In short: _"WebGL be crazy, yo!"_