Druid: Roadmap for 3D

[raphlinus]

This issue captures my thinking on 3D, but I'm open to discussion, especially if people bring a lot of energy and motivation.

There's a lot of interest in allowing access to 3D graphics from druid, but at the same time it's not in any way blocking Runebender, so it's hard to prioritize.

That said, one reason I'm very interested in 3D API's is to support better 2D rendering. There are at least two paths to that right now - Pathfinder and piet-gpu. These are fairly different in their approaches, as Pathfinder is designed for compatibility with installed GPU hardware and drivers, while piet-gpu explores cutting-edge compute capabilities. As of recently Pathfinder exposes enough of the 2D imaging model that we could consider it, and there is already a Pathfinder piet backend in progress.

Thus, we need to consider the approach to 3D in layers. One is: what should druid-shell expose? That layer can be consumed by druid to provide the best piet experience possible, even without exposing 3D. The other is: what should druid expose?

There is also the question of complexity. There are many, many 3D API's out there, with a complex matrix of compatibility and capabilities. Any approach to 3D must involve runtime detection, with some sort of fallback. Adding to the complexity, using 3D codepaths creates integration problems unique to desktop apps, not shared by the more typical game use cases: incremental present capability, low latency present modes (based on wait objects), smooth resize, etc.

One very appealing approach is to adopt wgpu as the primary integration point. The runtime detection would be wgpu or no wgpu, plus of course finer grained feature detection as provided by wgpu. Not all platforms can support wgpu, but compatibility work is envisioned (from the wgpu web page, OpenGL is currently unsupported but in progress).

There is another question of how to composite 3D content with the GUI. Again, two main approaches. One is to leverage the compositor capabilities of the platform, having loose coupling between the 2D and 3D pathways. Another is to use a GPU-resident Texture as the integration point. This would involve synchronization primitives to signal a frame request to the 3D subsystem (and similarly to negotiate resizes, which can get quite tricky with asynchrony), and a semaphore or fence of some kind to signal back to the 2D world that the texture is ready. Then the 2D world can consume that texture as it likes, applying clipping and translation (needed for scrolling), drawing other UI on top of it, etc. My preference is fairly strongly for the latter, though as always there are tradeoffs.

Since wgpu is not really mature yet (among other things, Pathfinder does not yet have a wgpu back-end, though it likely will soon), if we want to make faster progress we would need to add dynamic negotiation for a broader range of GPU interfaces. It's possible, but I'm certainly not enthusiastic enough about that to put time into it myself.

Discussion is welcome, we can use this issue.

[nicoburns]

Again, two main approaches. One is to leverage the compositor capabilities of the platform, having loose coupling between the 2D and 3D pathways. Another is to use a GPU-resident Texture as the integration point

I believe Firefox recently went through a relatively involved process of integrating with the system compositor as it was the only way that they could get decent power usage on macOS (Firefox was all-but-unusable on macs high-DPI displays scaled to a non-native resolution until this patch landed. Chrome also uses this approach on macOS.)

[raphlinus]

I'm aware of this (we've discussed it on Zulip). These are very difficult and complex tradeoffs. At the risk of oversimplifying, Apple pulls you into their way of building apps (slow rendering, fix everything in the compositor) and punishes you if you don't fit into that, while the direction other platforms are moving rewards rendering the final appearance into a swapchain buffer and presenting that with minimal friction.

It's very difficult to plumb a platform compositor abstraction up to apps; I find the simplicity of the non-compositor approach really appealing. It will increase power usage on mac for certain use cases, but it depends on the workload. The biggest win for the compositor is scrolling of otherwise static content. I think we can get back some of the power by having very efficient rendering.

[msiglreith]

An hard (and interesting) issue in this regard is also priority handling of the different workloads on the gpu. Newer hardware supports more fine grained priority mechanism and graphics API also expose queue priorities - nonetheless ensuring a smooth UI experience is not easy to achieve depending on the gpu load imposed by the 3D scene ):

[SimonSapin]

The biggest win for the compositor is scrolling of otherwise static content.

I think another scenario where this Firefox work had significant impact is video playback

[sztomi]

Just wanted to chime in here mentioning my own use-case: I'd like to create an mpv gui. The best way to embed libmpv is to give it access to an opengl context.