Druid: Prevent background redraw for transparent widget (embedding wgpu inside druid)

[djeedai]

Hi,

I am trying to write a small proof of concept of embedding a wgpu surface inside a druid app, as a custom widget. I've started druid yesterday and I am really liking this project, but would like to make sure it is possible to render a 3D view inside a desktop app. The typical use case is a 3D viewport with some 2D UI around it (think something like a game editor / Unity3D / UnrealEngine, or any CAD application, or some 3D modeller like Maya or Max).

I've started from the multiwin example and _lightly_ modified druid for WindowHandle to implement the HasRawWindowHandle trait, which allow creating a wgpu instance from it (is that of interest for #891 by the way? the patch is 10 lines (Windows only)). Then I more or less copy-pasted the "cube" example of wgpu-rs into the multiwin example of druid.

Now it almost works, but it seems that some background redraw is still active, so I cannot get a stable image. When the mouse moves out and in of the window borders, or other repaint is triggered, I briefly see the 3D cube, but then immediately get back the normal 2D druid UI on top of it.

Does someone has any idea how I could fix this, or could point me in the direction of where widgets are cleared? I don't do that background repaint myself. I've tried with PaintCtx::paint_with_z_index(9999) but that doesn't work either.

Many thanks for any pointer! :)

[djeedai]

I've just seen #1249 and changed theme::WINDOW_BACKGROUND_COLOR to (0,0,0,0) but this doesn't do anything in my case, presumably because the 3D clear from wgpu is clearing the window anyway. But it seems druid is still painting it.

[jneem]

The background is drawn at druid/src/window.rs:354. Does commenting it out help at all?

[djeedai]

Thanks @jneem; no not really, I see a change (it flickers to black instead of dark gray) but there's still some bad interaction between wgpu and druid. So it's not a problem with the background. I need to dig into the wgpu rendering and the wgpu::Queue to understand what is being submitted when.

[djeedai]

Sadly I think this is a composition issue. The way I patched up things, wgpu reuses the same HWND created by Druid:

• wgpu creates its own composition pipeline (I don't know the details) from the HWND
• Druid (Windows; piet-direct2d) creates some DXGI surface from the HWND it created. Actually I'm not quite sure exactly of the details because I looked at the code for 1 hour and I am now more confused than before; I've seen some DXGI swapchain, some HWND, some Direct2D, some Direct3D, ... a bit of everything.

Anyway I think the two composition pipelines are targeting the same HWND and rendering in parallel, fighting to present to the window.

One workaround I am thinking of trying when I have time is to create a child window (HWND) inside the Druid one, and use that one for wgpu. I think that could possibly work around the issue. Otherwise unfortunately I don't see how to do for now, except waiting for piet to have a wgpu backend that is exposed and can be accessed for manual rendering.

[raphlinus]

I'm very interested in wgpu, but think that having embedded 3d widgets is a pretty hard problem. Either you use wgpu for everything, which I think is fine (but involves building that), or you start having to expose composition up through the widget hierarchy, so for example a wgpu inside a scroll view can get translated and clipped appropriately as it scrolls. Doing that in a cross-platform way is hard. I'm not sure there's a simple answer to this, or what might be the best path forward.

See #891 for more discussion.

[djeedai]

Yes I've read #891 actually before starting my experiment. This is why I was trying with wgpu, and why I've tried a simpler workaround than using wgpu for everything, which I agree is probably the best way but also the most involved in terms of workload. Instead I was trying to come up with a (temporary) workaround in the meantime, even if that means limited features (I am not interested in scrolling personally, and I think it would be fine anyway if an embedded 3D view doesn't handle it, as opposed to not having any solution).

I've looked at how Qt does it nowadays and it is interesting to see they went the child window approach back in 2013 (I couldn't find anything more recent on the subject; I think this is still the recommended approach), which is my next plan for when I have time. This delegates the clipping to the OS compositor by leveraging the parent-child relationship of the native windows (HWND on Windows). Then this native child window (with no border etc.) can be embedded via a custom Druid widget which controls its position and size.

In fact the Qt documentation has a list of restrictions in this case, and they explicitly mention the scrolling one (highlights mine):

The window container is attached as a native child window to the toplevel window it is a child of. When a window container is used as a child of a QAbstractScrollArea or QMdiArea, it will create a native window for every widget in its parent chain to allow for proper stacking and clipping in this use case. [...] Applications with many native child windows may suffer from performance issues.

I know this approach is also used sometimes when _e.g._ the inner window is rendered by a separate process (for example, a game engine) to avoid crashes to the renderer affecting the stability of the surrounding application. This is doable on Windows for sure, and I believe on other OSes too.

So I think this may be a more feasible approach short term, even if it means it has limitations compared to other widgets. I'll see if I can try it when I have some spare time. Would you be open for Druid to take this kind of change? I appreciate this somehow goes against the philosophy of "druid is safe high-level abstraction, druid-shell contains the low-level platform-dependent implementation", since it would leak the native platform handle to end users for them to embed their content (unless the change is restricted to embedding wgpu only, in which case the widget can act as an abstraction barrier).

[raphlinus]

Thanks for looking into Qt, I think that's good background.

Overall I am open to landing this in Druid, as I think it's good to get experience; it's clearly a use case people care about. I think if we're up front about the limitations and don't let it constrain our development too much, it can be ok. We're already partly down the path of doing subwindows for things like menus.

I also think it's a fair amount of work, especially as it has to be rethought on different platforms, and even within Windows, as subwindow is compatible with Windows 7, but "swapchain for composition" might be a richer and more performant experience on 10. Similarly for X11 and Wayland on Linux. But if you're willing to take it on, I certainly won't stop you :)

[djeedai]

I am willing to try it on Windows 10 for now, see if that works at all :)

[dhardy]

If you'd like to try things the other way around, both Iced and KAS are WGPU-native. Iced supports embedded usage (GUIs inside a WGPU app) and KAS supports custom WGPU render pipes (embedded WGPU). (Yes, this is a shameless plug — don't let me stop you working on Druid.)