Wlroots: Multi GPU is broken

[emersion]

Workaround: set WLR_DRM_DEVICES=/dev/dri/card0

[DawidLoubser]

Man, the moment this gets fixed, I'm switching to sway :-) If there is any way of helping you to test with my setup (Intel + Nvidia), please let me know.

[ascent12]

I was asked to do a write up about this issue, and actually looking at the reported message and thinking about it, I've noticed something wrong:

https://github.com/swaywm/wlroots/blob/master/backend/multi/backend.c#L146-L150
Says we can't have multiple renderers, and this is partially true. Each of the secondary GPUs will have their own renderer internally, because they need to do a copy with the dma-buf they receive. This is just an internal detail, and shouldn't be exposed outside of the backend.

Basically, https://github.com/swaywm/wlroots/blob/master/backend/drm/backend.c#L51-L55 is wrong, and should be returning the renderer of the primary GPU, instead of its own renderer, since that's the thing which actually does all of the "real" rendering.

If we're lucky, the fix will just be to change DRM's backend_get_renderer. I'm not in the position to test this out as I'm writing this. I'll try it later.

Before the check in wlr_multi_backend_add was added, the place the error happened was https://github.com/swaywm/wlroots/blob/master/render/gles2/renderer.c#L28. Maybe what I said above would explain why this used to fail too.

Otherwise there is some weird misuse of renderers and EGL contexts somewhere, which needs to be tracked down. I tried looking before, but it certainly wasn't obvious.

---

Another important thing is that this needs to be tested on a wide range of setups. I originally got it working on an AMD HD 7950 and an Nvidia GTX 5XX card, but when it used to work someone did report that it did not work for them. So I'm not really confident that the actual implementation is correct.

I have a couple of more GPUs to work with and test now (AMD R9 2XX and an RX 5XX), so hopefully I can be more confident with the results I get. I'm really not looking forwards to having to keep turning my computer off and replacing GPUs though.

However, I don't have a laptop with a discrete GPU, or even an Intel iGPU desktop to work with (although I may be able to get my hands on one easily). Once this is actually thought to be fixed, any extra help testing would be great. The more hardware setups tested, the better.

---

That said, I'm not particularly happy with the way multi-GPU is currently implemented. Most of it is fine. but it makes the DRM code is fair bit more complicated to follow (it was already complicated as hell). The way it was originally implemented was to basically bolt it on top of the single-GPU code, and reuses variables is weird ways.
I think there needs to be a refactor of DRM to make that whole bit a lot easier to follow.

[braatvedt]

I'm experiencing the same issues on my side, can't wait for this to be fixed.

[DawidLoubser]

@ascent12 Thanks for the write-up! Happy to help testing this side.
My use-case is the following: Need all rendering to happen on the one card (intel embedded), and then need to output those display buffers via the other card (nvidia, via nouveau) in order to output to an external 4k monitor connected to a display port which is wired to the nvidia chip.

In that sense, I'm not sure if "multi gpu" is even the correct terminology, depending on interpretation. But I just wanted to clarify that I'm not trying to get multiple GPUs to actually do rendering, it's just about copying display buffers.

Thanks!

[Ongy]

I can test AMD+Nvidia dedicated cards (6950+580) and can easily get my ands on an intel-iGPU+nvidia.
But I can't use either of those setups for a long time.

@DawidLoubser iirc your usecase is what should be supported by wlroots currently.
The other step, to send single applications to the dedicated card didn't work at any point iirc (s.t some vulkan magic)

[ascent12]

I tried changing backend_get_renderer and wlr_multi_backend_add, but that alone didn't fix the issue (but is still needed), and lead to the assertions inside of the renderer.

After a bit more debugging by running rootston and the example compositors, the functions in particular that are triggering the assertions are rendering/egl functions which aren't directly related to rendering (e.g. creating a texture, binding the wl_display).

So as I currently understand it, after the secondary backend is made, it will leave its own internal renderer as the current EGL context. Then the user tries to do some other EGL operation without explicitly making the primary backend current, causing the assertion failure.

The solutions that come to mind are:

• Require the user to explicitly make the context current for every rendering related function. We'd need to be careful to not accidentally change the context out from under the user's feet.
• Make the secondary DRM backend set the context to the primary DRM backend before exiting any API function, so the details of the internal renderer are never leaked.
• Automatically set the correct context inside of rendering related functions.

I'm not really sure which solution I like better, but we need to be a bit more organised about how we manage the EGL context.

[emersion]

How is multiple renderers in different GPUs different from multiple renderers in different backends?

Why are these multiple renderers even exposed to the user via backend_get_renderer? I think something's going wrong here, backend_get_renderer should not return internal renderers.

When I implemented multiple renderers, I chose "Automatically set the correct context inside of rendering related functions" and it worked well. Note that it changed the context only for functions that _don't_ render like uploading the texture or wlr_renderer_begin, not for functions like drawing a texture or a rectangle (since those are between wlr_renderer_begin and wlr_renderer_end anyway).

[ascent12]

I already answered these on IRC, but I'll re-answer them here just so it's somewhere more permanent, and everybody can see it.

How is multiple renderers in different GPUs different from multiple renderers in different backends?

The various DRM renderers are being used in a very specific way that is actually guaranteed to work (assuming dmabufs are actually supported), and only is being used for a simple dmabuf export/import and a quick copy to the scanout buffer. The more general multiple renderer design before was a lot more complicated and required excessive amounts of resource duplication.

I don't personally have a problem with multiple EGLContexts, but trying to mix together EGLDisplays in a general way is what was causing the issues for me.

Why are these multiple renderers even exposed to the user via backend_get_renderer? I think something's going wrong here, backend_get_renderer should not return internal renderers.

Yes, that is a mistake.

I chose "Automatically set the correct context inside of rendering related functions" and it worked well. Note that it changed the context only for functions that don't render like uploading the texture or wlr_renderer_begin, not for functions like drawing a texture or a rectangle

From a user API perspective, I suppose that's less hassle.

[DawidLoubser]

Any update on this issue? Really looking forward to having hybrid graphics working in wlroots-based window managers... Any way which I can help?

[progandy]

I'll be able to test Intel Core i7-8705G with Radeon RX Vega M GL.
Edit: Might not be of any use, the outputs are all connected to the Intel iGPU. wlroots is also unable to open the vega gpu, but it can be used as a render-node for clients with DRI_PRIME=1.

[n3rdopolis]

I will be able to test on qemu with

qemu-system-x86_64 -vga none -device VGA,id=video0  -device secondary-vga,id=video1

as well

[manio]

And I will also be able to test it on my dual nvidia cards (4 monitors in total) which I am currently using but on x11/xinearama. The thing here is that @ascent12 surely need a time for debugging this...

btw: @ascent12 I was discussing this problem on IRC and we agreed that it is rather a complex problem to debug/solve for a developer who is not familiar with all this DRM-related stuff (me), but say a word and I'll try to do my best to help you...

[ddevault]

"I can test" isn't half as a good as "I can work on a patch"

@ascent12 you should stop discouraging people on the basis of difficulty, it's not unsolvable by someone unfamiliar with DRM imo.

[manio]

@SirCmpwn,
It's not that @ascent12 was discouraging people (or me) because this issue difficulty, I just got this feeling after an IRC discussion (not even with ascend12) about it... Sorry for misleading. If it's not that hard, then great :)

@ascent12,
Do you maybe have a branch or patch with your starting work? I mean the one where you've got assertions inside of the renderer?

[martinetd]

We have the assertion on current master already, they fail as he said because when creating multiple sub-backends (one per card in attempt_drm_backend) the latest backend created leaves the current renderer to one that isn't the masters.

If we're going for the solution where the backends automatically set the correct renderer, a simple solution would be to replace the asserts by checking if it's not the correct renderer then set the correct one and continue.
If you can reproduce the current assert failures that actually doesn't look like it would be too difficult, assuming they're happy with that direction :P

[manio]

Hi!
I just played with it a little. My plan was at least to make sway starting/working somehow:
My changes in the code was:

• Removing the check for multiple renderers:
  https://github.com/swaywm/wlroots/blob/master/backend/multi/backend.c#L149
• Removing assert in gles2_get_renderer_in_context():
  https://github.com/swaywm/wlroots/blob/master/render/gles2/renderer.c#L28
• Removing assert in get_gles2_texture_in_context():
  https://github.com/swaywm/wlroots/blob/master/render/gles2/texture.c#L28
• Removing assert in wlr_gles2_texture_from_pixels():
  https://github.com/swaywm/wlroots/blob/master/render/gles2/texture.c#L146
• Removing assert in wlr_gles2_texture_from_wl_drm():
  https://github.com/swaywm/wlroots/blob/master/render/gles2/texture.c#L183

The results are:

• sway is starting! :)
• sway is painting the wallpapers on all my four monitors :)
• sway is correctly assigning the workspaces to those monitors (1,2,3,4)
• rendering clock on swaybar is working on different GPUs in the same time (seconds are ticking)!
• I was able to run weston-terminal and then move it between all monitors/workspaces! :)

Problems:

• two monitors (from the one drm/gpu) are wrongly rotated 180 degree left, the others two are ok (when starting sway independently on every single GPU - no such problem).
• one monitor is not rendering swaybar at all
• runtime reloading sway configuration is causing troubles - flickering and other problems...
• urxvt (xwayland) is not rendering correctly, while weston-terminal looks definitely better
• general other rendering problems - eg I have to move the window several times to get it's content rendered properly, etc

If i understand you correctly in some calls (not all?) to those gles2 functions the wlr_egl is wrong and I have to change it in those cases (when assert is occuring) to the correct one?

[emersion]

Removing the check for multiple renderers

This one should be left untouched.

Removing assert in gles2_get_renderer_in_context()
Removing assert in get_gles2_texture_in_context()

These should be left untouched.

Removing assert in wlr_gles2_texture_from_pixels()
Removing assert in wlr_gles2_texture_from_wl_drm()

These should instead check if the context is current, and make the context current if not.

[ddevault]

Regarding rendering issues, can you try to reproduce them running sway with -Ddamage=rerender?

[manio]

@emersion
I see, thanks for clarification, I'll try to do something with those contexts... :)

@SirCmpwn
I tried it with the -Ddamage=rerender, but it doesn't help much - maybe the only difference I see is that the swaybar was rendered even on the monitor on which it was not before... but the problem was that i have to move cursor to that screen to update the seconds ticking of the swaybar's clock - on the rest workspaces it was ticking properly...

Update: Two monitors are not only rotated 180 degree counterclockwise but also mirrored (horizontal flip)

[manio]

@emersion
I was unable to start sway when I got rid of the asserts you are telling that should be left untouched, specifically the check in wlr_multi_backend_add() imo has to be removed when multigpu will be supported, isn't it?

The minimal changes in code which leads to proper compilation and not asserting/crashing at startup are:
https://github.com/manio/wlroots/commit/7ba9b88f9f3050a6e4a98b3789d8276fc0ad43e0
The code at this commits works as described above - it is driving all displays but with rendering problems.

I was also trying to make really really stupid things like changing the EGLs when the context is not current (please don't comment the code - I know that it is totally wrong - I just wanted to test if it would help):
https://github.com/manio/wlroots/commit/bfab2d5bd0de80ae8a6620316e5eda00893f10d8
This also doesn't changed much.

My last approach was to make it as you told - so trying to make the context current if it is not:
https://github.com/manio/wlroots/commit/59170e717eda8079e593ce86c31d38defa2a3b60
And I had to do it also in gles2_get_renderer_in_context() and get_gles2_texture_in_context() - without it, the assertion was raising there.
But this approach leads to:

2018-07-24 06:12:07 - [EGL] eglSwapBuffers: eglSwapBuffers
2018-07-24 06:12:07 - [render/egl.c:363] eglSwapBuffers failed
2018-07-24 06:12:07 - [EGL] eglSwapBuffers: no front buffer
swaybg: ../swaybg/main.c:215: main: Assertion `state.compositor && state.layer_shell && state.output && state.shm' failed.

Maybe I am wrongly calling the wlr_egl_make_current()? Should I pass the surface instead of EGL_NO_SURFACE?

ps. full log is here: https://skyboo.net/temp/sway/sway.log

[emersion]

I was unable to start sway when I got rid of the asserts you are telling that should be left untouched, specifically the check in wlr_multi_backend_add() imo has to be removed when multigpu will be supported, isn't it?

No: while there will be multiple internal renderers inside the DRM backend, these shouldn't be exposed to users.

Maybe I am wrongly calling the wlr_egl_make_current()? Should I pass the surface instead of EGL_NO_SURFACE?

There are two families of functions in the renderer:

• Rendering functions, that must be called between wlr_renderer_begin and wlr_renderer_end. These should not change the context, and should assert that the current context is the right one.
• Texture functions, that create and manipulate textures. These should switch the current context if necessary. I think it's fine to use EGL_NO_SURFACE.

The problem you're running into is probably because you're always switching the context - with EGL_NO_SURFACE - in all renderer functions, regardless whether the context is already current or not. Not switching context in rendering functions should prevent these functions to unset the current surface (since you set the surface to EGL_NO_SURFACE). Only switching the context if necessary (ie. don't switch the context if it's already current) in texture functions would allow to use texture functions functions between wlr_renderer_begin and wlr_renderer_end.

[manio]

@emersion
As far as I can tell I was changing the context only when it is was not current. Anyway - I reworked the patch one more time to make this sure. The summary changes are:
https://skyboo.net/temp/sway/changes.diff

Results: I can see that changing the context is really working, because the two asserts in texture.c are not raising anymore, however the following problems are occurring:

2018-07-31 12:48:06 - [GLES2] GL_INVALID_FRAMEBUFFER_OPERATION in glDrawArrays(incomplete framebuffer)
2018-07-31 12:48:06 - [EGL] eglSwapBuffers: eglSwapBuffers
2018-07-31 12:48:06 - [render/egl.c:347] eglSwapBuffers failed
2018-07-31 12:48:06 - [EGL] eglSwapBuffers: no front buffer

Full log:
https://skyboo.net/temp/sway/sway.log

I also removed the assert in gles2_get_renderer() - otherwise it was raising...

[manio]

Hi guys,
I have it finally working! :)

Pull request: https://github.com/swaywm/wlroots/pull/1168

[ddevault]

Nice! That patch is a lot smaller than I expected. Can we get the people in this thread who have been asking for this feature to test that patch, please?

[J0nnyMak0]

@SirCmpwn what is the best way to test the patch? Will it appear on https://aur.archlinux.org/packages/sway-wlroots-git/ immediately after merge?

[ddevault]

Just compile it yourself. There are instructions in the readme.

[J0nnyMak0]

I took a quick look:

1. All 4 screens come to life. yay!
2. Is there a way to specify the order of the screens. Right now, VGA (wired to intel) comes up as primary (with mouse cursor), followed by laptop's internal dispaly to its right, followed by DP-2 and DP-1 all to the right. The way I have the screens arranged, this is totally disorienting, but presumably there is a way to specify the order.
3. Screens connected to the discrete (Nvidia) card have a weird black square cursor.
4. Opening apps and trying to do stuff crashes Sway,
   even though it may not be related to this patch (logfile attached)
   sway.log
   cheers!

[manio]

@JonnyMako

All 4 screens come to life. yay!

Great!

Is there a way to specify the order of the screens. Right now, VGA (wired to intel) comes up as primary (with mouse cursor), followed by laptop's internal dispaly to its right, followed by DP-2 and DP-1 all to the right. The way I have the screens arranged, this is totally disorienting, but presumably there is a way to specify the order.

I also have this problem - thanks for confirming.
There is probably a way to specify the order:
https://github.com/swaywm/sway/wiki#multihead
But I don't know if it is related with pointer entry/out points from screen :)
btw. the order is correct for me - I mean workspace numeration on each screen is ok.
Do you also have it correctly assigned?

Screens connected to the discrete (Nvidia) card have a weird black square cursor.

This is a known problem, not yet fixed...

Opening apps and trying to do stuff crashes Sway,
even though it may not be related to this patch (logfile attached)
sway.log

Thanks for the log - I was suspecting the problem might be there but was waiting for confirmation :)
You just confirmed it :)
PR updated, please rebuild and check if it is not crashing anymore...

[ddevault]

To specify which output is focused on startup just stick a workspace n command at the end of your sway config where n is some workspace on that output (you can assign them with workspace n output x)

[J0nnyMak0]

@manio I'm travelling and don't currently have access to multi monitor setup. I did try the newer patch (albeit with no monitors attached) and crashes seem to have gone away! Will test more when I get back.

@SirCmpwn thanks. I'll try that.

Also, is specifying which card to use on a per app basis in the scope of this pull request #1168 ? If so how do I specify that? DRI_PRIME=1?

[DawidLoubser]

Haven't plugged into an external monitor yet (can't wait to) but I can confirm that after this update, for the first time ever sway runs just fine on my Nvidia Optimus laptop in "Hybrid graphics mode" (both Intel and Nvidia cards active) so I'm very positive about the multi-monitor situation.

You guys are doing amazing work, thanks so much!

I'm trying to run sway for daily work, and it crashes pretty much every day a couple of hours in.
I'm going to start a concerted effort to grab stacktraces and report these properly - because I'm totally sold.

[J0nnyMak0]

Ok, I'm back. I can confirm all four monitors work. I've arranged them with output commands in config and everything works great! Hot pulgging works great! Just set my notebook down in the dock and the monitors all sprung to life!!!

I'll be using Sway full time as my primary environment from now on and will report any issues I run across. Thanks!!

[rhysperry111]

Hi, I'm not sure if this is the right place to put this, but I can't seem to get a second monitor to work on Sway. I am not using the NVidia card for rendering as it is absolutely blown away by the performance of the Intel integrated graphics. I am on an optimus laptop (Intel graphics + Nvidia 1050Ti (nouveau)) which has the HDMI port wired to the Nvidia card.

When I plug my laptop into the external display Sway detects the screen but it's output is completely black. I am running Sway 1.4, wlroots 0.10.1 and linux-zen 5.6.11

[emersion]

Your bug report is missing debug logs. Please add a link to the full debug log file.

[rhysperry111]

Here is the output of sudo journalctl | grep sway as I am running sway as a unit:
sway.txt
EDIT: Sway run with -d
sway-debug.txt

[manio]

Do you have the proper outputs configuration in sway/config?
How about running:
swaymsg -t get_outputs

[rhysperry111]

Here is the output of swaymsg -t get_outputs:

Output eDP-1 'Unknown 0x07A1 0x00000000' (focused)
  Current mode: 1920x1080 @ 60.004002 Hz
  Position: 3840,0
  Scale factor: 1.000000
  Scale filter: nearest
  Subpixel hinting: rgb
  Transform: normal
  Workspace: 1
  Max render time: off
  Available modes:
    1920x1080 @ 40.000999 Hz
    1920x1080 @ 60.004002 Hz

Output HDMI-A-2 'Samsung Electric Company SAMSUNG 0x00000100'
  Current mode: 3840x2160 @ 30.000000 Hz
  Position: 0,0
  Scale factor: 1.000000
  Scale filter: nearest
  Subpixel hinting: unknown
  Transform: normal
  Workspace: 2
  Max render time: off
  Available modes:
    720x400 @ 70.082001 Hz
    640x480 @ 59.939999 Hz
    640x480 @ 60.000000 Hz
    640x480 @ 66.667000 Hz
    640x480 @ 72.808998 Hz
    640x480 @ 75.000000 Hz
    720x480 @ 59.939999 Hz
    720x480 @ 60.000000 Hz
    720x576 @ 50.000000 Hz
    800x600 @ 60.317001 Hz
    800x600 @ 72.188004 Hz
    800x600 @ 75.000000 Hz
    832x624 @ 74.551003 Hz
    1024x768 @ 60.004002 Hz
    1024x768 @ 70.069000 Hz
    1024x768 @ 75.028999 Hz
    1280x720 @ 50.000000 Hz
    1280x720 @ 59.939999 Hz
    1280x720 @ 60.000000 Hz
    1152x864 @ 75.000000 Hz
    1280x800 @ 59.910000 Hz
    1366x768 @ 59.790001 Hz
    1440x900 @ 59.901001 Hz
    1280x1024 @ 60.020000 Hz
    1280x1024 @ 75.025002 Hz
    1600x900 @ 60.000000 Hz
    1680x1050 @ 59.882999 Hz
    1920x1080 @ 23.976000 Hz
    1920x1080 @ 24.000000 Hz
    1920x1080 @ 25.000000 Hz
    1920x1080 @ 29.969999 Hz
    1920x1080 @ 30.000000 Hz
    1920x1080 @ 50.000000 Hz
    1920x1080 @ 59.939999 Hz
    1920x1080 @ 60.000000 Hz
    3840x2160 @ 23.976000 Hz
    3840x2160 @ 24.000000 Hz
    3840x2160 @ 25.000000 Hz
    3840x2160 @ 29.969999 Hz
    4096x2160 @ 23.976000 Hz
    4096x2160 @ 24.000000 Hz
    4096x2160 @ 25.000000 Hz
    4096x2160 @ 29.969999 Hz
    4096x2160 @ 30.000000 Hz
    3840x2160 @ 30.000000 Hz

I have not configured the display (have never needed to do this on my desktop computer). I will see if choosing a different output mode fixes it

[rhysperry111]

Sorry for creating a fuss in this thread, turns out lowering the resolution fixed the issue