Wlroots: Low refresh rate on external monitors with multi-gpu setup

[J0nnyMak0]

Starting with https://github.com/swaywm/wlroots/commit/91cb0fc4432592a8c0806d3e63dc77ce16c73cb6 I'm experiencing sluggish performance. And the cpu is running a good 20-25 degrees hotter for the most part. I'm attaching good and bad logs.

sway-good.log
sway-bad.log

[emersion]

Hm. Can you try to figure out what consumes CPU time with perf(1)?

perf record sway …
perf report

[J0nnyMak0]

Nothing stands out in perf report as consuming excessive cpu. I have a feeling it has something to to with cursor. Cursor movement is very choppy starting with the said commit. It feels like the more I move the cursor, the more cpu is being taxed, but it isn't showing up any of the processes. Upon exiting sway, I get:

KHR, error: EGL_NOT_INITIALIZED (0x3001), message: "_eglDestroyImageCommon" is too slow

Sorry for the vagueness, but hopefully there is something here of help.

Edit: And the bad log has a lot more of these.

00:00:06.625 [ERROR] [wlr] [libinput] event13 - Logitech MX Master: client bug: event processing lagging behind by 46ms, your system is too slow

[emersion]

OK, I'll try to reproduce on my setup.

KHR, error: EGL_NOT_INITIALIZED (0x3001), message: "_eglDestroyImageCommon" is too slow

This is unrelated and should be fixed with https://github.com/swaywm/wlroots/commit/2585f322cba40baa1dc3741f9871571bd2db8b78.

[emersion]

Some findings:

• This only happens on a multi-GPU setup. Running with only the Nouveau card is fine.
• WLR_NO_HARDWARE_CURSOR=1 helps with CPU usage, but output is still choppy.

[emersion]

I managed to track it down to:

https://github.com/swaywm/wlroots/blob/5d054258af1bbdad43758d0161ef4ca10f41aaec/backend/drm/renderer.c#L156

Nouveau doesn't like it when we create/destroy DMA-BUF textures like this. Caching the textures fixes it.

[emersion]

Can you try this very hacky patch? If it fixes the issue for you, it'll be worth it to clean it up.

diff --git a/backend/drm/renderer.c b/backend/drm/renderer.c
index f2d7677c01ca..22d65dd22666 100644
--- a/backend/drm/renderer.c
+++ b/backend/drm/renderer.c
@@ -153,13 +153,17 @@ static struct wlr_buffer *drm_surface_blit(struct wlr_drm_surface *surf,
        return NULL;
    }

-   struct wlr_texture *tex = wlr_texture_from_dmabuf(renderer, &attribs);
+   struct wlr_texture *tex = buffer->mgpu_tex;
    if (tex == NULL) {
-       return NULL;
+       tex = wlr_texture_from_dmabuf(renderer, &attribs);
+       if (tex == NULL) {
+           return NULL;
+       }
+       buffer->mgpu_tex = tex;
    }

    if (!drm_surface_make_current(surf, NULL)) {
-       wlr_texture_destroy(tex);
+       //wlr_texture_destroy(tex);
        return NULL;
    }

@@ -176,7 +180,7 @@ static struct wlr_buffer *drm_surface_blit(struct wlr_drm_surface *surf,

    drm_surface_unset_current(surf);

-   wlr_texture_destroy(tex);
+   //wlr_texture_destroy(tex);

    return out;
 }
@@ -445,6 +449,7 @@ bool drm_fb_import(struct wlr_drm_fb **fb_ptr, struct wlr_drm_backend *drm,
    struct wlr_buffer *local_buf;
    if (drm->parent && mgpu) {
        // Perform a copy across GPUs
+       wlr_log(WLR_DEBUG, "BLIT %p", buf);
        local_buf = drm_surface_blit(mgpu, buf);
        if (!local_buf) {
            wlr_log(WLR_ERROR, "Failed to blit buffer across GPUs");
@@ -456,6 +461,7 @@ bool drm_fb_import(struct wlr_drm_fb **fb_ptr, struct wlr_drm_backend *drm,

    struct wlr_drm_fb *fb = drm_fb_get(drm, local_buf);
    if (!fb) {
+       wlr_log(WLR_DEBUG, "DRM FB CREATE");
        fb = drm_fb_create(drm, local_buf, formats);
        if (!fb) {
            wlr_buffer_unlock(local_buf);
diff --git a/include/wlr/types/wlr_buffer.h b/include/wlr/types/wlr_buffer.h
index 8455589999ff..70ef8e8d95c6 100644
--- a/include/wlr/types/wlr_buffer.h
+++ b/include/wlr/types/wlr_buffer.h
@@ -41,6 +41,8 @@ struct wlr_buffer {
        struct wl_signal destroy;
        struct wl_signal release;
    } events;
+
+   void *mgpu_tex;
 };

 /**

[J0nnyMak0]

Can you try this very hacky patch?

Unfortunately, this patch didn't make any difference for me. Sorry.

Edit: If perf.data will be of any use, I can upload it.

[J0nnyMak0]

Here is a sample. Fan constantly on and battery draining real fast:

  32.33%  sway     [unknown]                   [k] 0xffffffff9a000163
  14.43%  sway     libc-2.32.so                [.] __memmove_avx_unaligned_erms
   6.58%  sway     i965_dri.so                 [.] 0x0000000000642413
   3.94%  sway     i965_dri.so                 [.] 0x000000000064241c
   3.58%  sway     i965_dri.so                 [.] 0x0000000000642433
   2.62%  sway     libpixman-1.so.0.40.0       [.] pixman_region32_init
   1.91%  sway     i965_dri.so                 [.] 0x000000000055bd18

Samples: 110  of event 'cycles:u', Event count (approx.): 26624113, DSO: libc-2.32.so
Overhead  Comm  Symbol
  14.43%  sway  [.] __memmove_avx_unaligned_erms                                                                                       
   1.50%  sway  [.] malloc_consolidate                                                                                                 
   1.24%  sway  [.] __libc_recvmsg                                                                                                     
   0.67%  sway  [.] epoll_wait  

[emersion]

I've tried to reproduce again with amdgpu+nouveau, but can't with my patch. I've also tried radeon+amdgpu multi-GPU and that worked fine without the patch. Unfortunately I don't have the hardware to test the i915+nouveau combination.

  32.33%  sway     [unknown]                   [k] 0xffffffff9a000163

It seems like we're missing some debug symbols here. Compiling Mesa could help getting more useful information:

git clone https://gitlab.freedesktop.org/mesa/mesa.git
cd mesa
meson build/ -Dprefix=$PWD/build/prefix # also select the right drivers for your hardware
ninja -C build/
export LD_LIBRARY_PATH="$PREFIX/lib"
export LIBGL_DRIVERS_PATH="$PREFIX/lib/dri"
# then run sway

I find it pretty strange that 91cb0fc causes the issue. The only thing it should change is how long we keep a ref on the Intel buffer...

[emersion]

This patch tries to restore 91cb0fc's behaviour, can you give it a go?

diff --git a/backend/drm/renderer.c b/backend/drm/renderer.c
index f2d7677c01ca..82f537bdf4f7 100644
--- a/backend/drm/renderer.c
+++ b/backend/drm/renderer.c
@@ -297,6 +297,7 @@ void drm_fb_clear(struct wlr_drm_fb **fb_ptr) {
    }

    struct wlr_drm_fb *fb = *fb_ptr;
+   wlr_buffer_unlock(fb->original_wlr_buf);
    wlr_buffer_unlock(fb->wlr_buf); // may destroy the buffer

    *fb_ptr = NULL;
@@ -463,6 +464,9 @@ bool drm_fb_import(struct wlr_drm_fb **fb_ptr, struct wlr_drm_backend *drm,
        }
    }

+   wlr_buffer_unlock(fb->original_wlr_buf);
+   fb->original_wlr_buf = wlr_buffer_lock(buf);
+
    drm_fb_move(fb_ptr, &fb);
    return true;
 }
diff --git a/include/backend/drm/renderer.h b/include/backend/drm/renderer.h
index fdc8a75d68de..415654e07da5 100644
--- a/include/backend/drm/renderer.h
+++ b/include/backend/drm/renderer.h
@@ -31,6 +31,7 @@ struct wlr_drm_surface {

 struct wlr_drm_fb {
    struct wlr_buffer *wlr_buf;
+   struct wlr_buffer *original_wlr_buf;
    struct wl_list link; // wlr_drm_backend.fbs

    struct gbm_bo *bo;

[J0nnyMak0]

That patch completely freezes up the system on sway launch. I'm attaching the log. I'm sorry I've been putting off building mesa as I'm unable to schedule a couple of hours off on the machine for that to happen. I'll try and do it tonight and hopefully get more useful info for this issue as well as the other segfault issue. Cheers.

[J0nnyMak0]

That was the wrong log file. Here is the correct one.
sway.log

sway: types/wlr_buffer.c:49: wlr_buffer_unlock: Assertion `buffer->n_locks > 0' failed.`

[emersion]

Oops. Here's another try (sorry, I don't have access to my multi-GPU setup right now):

diff --git a/backend/drm/renderer.c b/backend/drm/renderer.c
index f2d7677c01ca..073360238ff8 100644
--- a/backend/drm/renderer.c
+++ b/backend/drm/renderer.c
@@ -297,6 +297,8 @@ void drm_fb_clear(struct wlr_drm_fb **fb_ptr) {
    }

    struct wlr_drm_fb *fb = *fb_ptr;
+   wlr_buffer_unlock(fb->original_wlr_buf);
+   fb->original_wlr_buf = NULL;
    wlr_buffer_unlock(fb->wlr_buf); // may destroy the buffer

    *fb_ptr = NULL;
@@ -424,6 +426,8 @@ void drm_fb_destroy(struct wlr_drm_fb *fb) {
        wlr_log(WLR_ERROR, "drmModeRmFB failed");
    }

+   wlr_buffer_unlock(fb->original_wlr_buf);
+
    gbm_bo_destroy(fb->bo);
    free(fb);
 }
@@ -463,6 +467,9 @@ bool drm_fb_import(struct wlr_drm_fb **fb_ptr, struct wlr_drm_backend *drm,
        }
    }

+   wlr_buffer_unlock(fb->original_wlr_buf);
+   fb->original_wlr_buf = wlr_buffer_lock(buf);
+
    drm_fb_move(fb_ptr, &fb);
    return true;
 }
diff --git a/include/backend/drm/renderer.h b/include/backend/drm/renderer.h
index fdc8a75d68de..415654e07da5 100644
--- a/include/backend/drm/renderer.h
+++ b/include/backend/drm/renderer.h
@@ -31,6 +31,7 @@ struct wlr_drm_surface {

 struct wlr_drm_fb {
    struct wlr_buffer *wlr_buf;
+   struct wlr_buffer *original_wlr_buf;
    struct wl_list link; // wlr_drm_backend.fbs

    struct gbm_bo *bo;

[J0nnyMak0]

Thank you for trying to fix this even while not having access to i915+nouveau. I appreciate it. The new patch also did not fix the issue with lag and overheating of cpu. I"ll get back with perf data and other logs after I've built mesa locally with debug symbols and hopefully that will shed some light on what might be going on. Thanks!

[J0nnyMak0]

I double checked to make sure my bisection was correct. I can reaffirm https://github.com/swaywm/wlroots/commit/91cb0fc4432592a8c0806d3e63dc77ce16c73cb6 is the problematic commit.

Here are the perf data samples (with mesa debug) for the good (smooth and cool) and bad (choppy and hot) sway runs. This is with load. Idle temperatures also have the same delta. Around 55 for the smooth case and around 75 for the choppy cursor case.

Perf report comparison:
Good (CPU ~75 C)

Overhead  Command  Shared Object               Symbol
   7.05%  sway     [i915]                      [k] i915_gem_throttle_ioctl
   7.05%  sway     [nouveau]                   [k] nvkm_object_ctor
   6.48%  sway     [drm]                       [k] drm_prime_sg_to_page_addr_arrays
   5.57%  sway     [kernel.vmlinux]            [k] __kmalloc
   4.70%  sway     i965_dri.so                 [.] brw_prepare_drawing
   4.21%  sway     [i915]                      [k] __i915_sw_fence_complete
   4.09%  sway     i965_dri.so                 [.] _mesa_get_incomplete_framebuffer
   3.74%  sway     i965_dri.so                 [.] _mesa_reference_framebuffer_
   3.65%  sway     i965_dri.so                 [.] linear_to_ytiled_faster.lto_priv.0
   3.34%  sway     [kernel.vmlinux]            [k] __mod_memcg_state.part.0
   3.34%  sway     libwayland-server.so.0.1.0  [.] 0x000000000000be80
   3.34%  sway     libwlroots.so.7             [.] wlr_buffer_lock

Bad (CPU ~95 C)
Overhead Command Shared Object Symbol 5.78% sway [kernel.vmlinux] [k] ioread32 4.62% sway [nouveau] [k] nv50_instobj_wr32 4.19% sway [nouveau] [k] gf100_vmm_pgt_sgl 3.36% sway [drm] [k] drm_prime_sg_to_page_addr_arrays 3.26% sway [kernel.vmlinux] [k] dma_sync_single_for_device 2.84% sway [kernel.vmlinux] [k] memset_erms 2.67% sway [kernel.vmlinux] [k] iowrite32 2.45% sway [kernel.vmlinux] [k] __x86_retpoline_rax 1.93% sway libc-2.32.so [.] _int_malloc 1.79% sway i965_dri.so [.] linear_to_ytiled_faster.lto_priv.0 1.59% sway [kernel.vmlinux] [k] __sg_page_iter_next 1.55% sway [nouveau] [k] nouveau_bo_sync_for_device

[J0nnyMak0]

It may be a red herring, but to my untrained eye it looks like i915 is being used a lot less in the bad case. Even with load sway usage of i915 barely gets above 1% in the bad case.

[J0nnyMak0]

perf.tar.gz

[J0nnyMak0]

Just pinging to see if there is anything useful in perf data? Is there anything else I can try to shed some light on what might be causing this issue? I'm stuck at https://github.com/swaywm/wlroots/commit/5bd86b94f9a6f3b41d1bc4bbf2985632fdd6d0a7 and would like to get up to speed with recent changes if possible. I think the main issue here is the choppy cursor. I'm updating the title to reflect that.

Edit: The lagginess happens only on external monitors.

[imirkin]

No real comment on the underlying issues, but high ioread32() cost means that nouveau is "doing more work". By the looks of it, it's doing more mapping/unmapping of pages than in the "good" case. Changing the VM requires various stalls, so it should be avoided as much as possible. For linear buffers, the mesa driver employs sub-allocation to avoid this type of overhead, for example.

I'm not sure what the original change did exactly, but you're much better off keeping a cache of a handful of buffers than deleting/creating each time.

[emersion]

We already re-use buffers. We don't re-use textures yet, so the current code repeatedly imports EGLImages coming from the primary GPU into nouveau. https://github.com/swaywm/wlroots/issues/2636#issuecomment-758635062 is a hacky patch to cache the textures and fixed the issue on my setup, but didn't seem to help @J0nnyMak0.

[J0nnyMak0]

hacky patch to cache the textures and fixed the issue on my setup, but didn't seem to help @J0nnyMak0.

I'll double check to see the patch was in fact applied correctly and retry.

[J0nnyMak0]

I double checked and neither of the patches provided helped. W.r.t @imirkin 's comment about nouveau doing more work, here is the sensors reading. It almost look like the load that was on i915 is being picked up by the CPU:

Adapter: PCI adapter
GPU core:    868.00 mV (min =  +0.60 V, max =  +1.20 V)
temp1:        +60.0°C  (high = +95.0°C, hyst =  +3.0°C)
                       (crit = +105.0°C, hyst =  +5.0°C)
                       (emerg = +135.0°C, hyst =  +5.0°C)
power1:       17.02 W  

BAT0-acpi-0
Adapter: ACPI interface
in0:          16.84 V  
curr1:         0.00 A  

coretemp-isa-0000
Adapter: ISA adapter
Package id 0:  +89.0°C  (high = +84.0°C, crit = +100.0°C)
Core 0:        +85.0°C  (high = +84.0°C, crit = +100.0°C)
Core 1:        +82.0°C  (high = +84.0°C, crit = +100.0°C)
Core 2:        +83.0°C  (high = +84.0°C, crit = +100.0°C)
Core 3:        +82.0°C  (high = +84.0°C, crit = +100.0°C)

[fwSmit]

I don't know if it helps, but I can't reproduce this with my very similar setup. I have a laptop with intel+nouveau. I could reproduce https://github.com/swaywm/wlroots/issues/2526 with this laptop, but I don't have any problems with a laggy cursor. At least not on the integrated display (connected to iGPU). I would have to check for and external display (connected to dGPU).

[J0nnyMak0]

I would have to check for and external display (connected to dGPU).

Thanks for trying to replicate this.

To be clear, the lagginess only happens on my external displays. Both my external displays are HiDPI if that matters. But it should be easy to spot by just moving the cursor around on your external displays. With https://github.com/swaywm/wlroots/commit/5bd86b94f9a6f3b41d1bc4bbf2985632fdd6d0a7 the cursor movement is smooth as butter even when moving it around fast. Switching to https://github.com/swaywm/wlroots/commit/91cb0fc4432592a8c0806d3e63dc77ce16c73cb6 you should be able to see the cursor getting choppy. This affects all elements on external monitors, so browser scrolling, video playback etc. are all choppy.

It'd be interesting to see if you notice any change in cpu temps.

[fwSmit]

I don't notice a choppy cursor, but cpu temps definitely goes up. The cpu goes from around 40°C to >50°C, still rising. I can feel my laptop is warm to the touch now. My dGPU temp goes from N/A to 43°C, so that's something. It's normal that my laptop gets warmer when connected to an external display, because the dGPU has to spin up, but I noticed in top that there were two kernel processes taking a significant amount of cpu power: events_unbound and events_power_efficient.
I can confirm your bisect as well. 5bd86b9 works fine, 91cb0fc not.

Also, a bit unrelated, screenshotting doesn't work anymore. It only screenshots the background (swaybg), screenshotting done with grim. I guess that's because of the WLR_DRM_NO_MODIFIERS=1.

[J0nnyMak0]

I don't notice a choppy cursor, but cpu temps definitely goes up.

Interesting. Maybe my CPU is too old to pick up the slack. But I think @emersion is able to reproduce the choppy cursor behavior.

My screenshots are working fine even with WLR_DRM_NO_MODIFIERS=1 when using for example grim -g "$(slurp)" screenshot.png so maybe you are running into some other issue there.

[fwSmit]

I don't notice a choppy cursor, but cpu temps definitely goes up.

Interesting. Maybe my CPU is too old to pick up the slack. But I think @emersion is able to reproduce the choppy cursor behavior.

Yeah I think that might be the difference. My cpu is quite new and high end.

My screenshots are working fine even with WLR_DRM_NO_MODIFIERS=1 when using for example grim -g "$(slurp)" screenshot.png so maybe you are running into some other issue there.

I can't reproduce it anymore. I think it may have something to do with plugging in the external display (and maybe the 91cb0fc commit). I'm not gonna test now, but I'll be sure to keep an eye out if I see again.

[J0nnyMak0]

If I may gently prod the powers that be on this issue. I know this is probably not a priority for the upcoming release and personally, I can't wait to move away from nvidia as soon as I can. In the meantime though, I am stuck with this mult-gpu situation and if there is anything I can do to provide more info on what might be causing this, I'd be happy to do it. I am trying to keep up with the new updates (to wlroots, nouveau, mesa etc.), but since my working environment is stuck at https://github.com/swaywm/wlroots/commit/5bd86b94f9a6f3b41d1bc4bbf2985632fdd6d0a7 I'm having to do it less often than I'd like. Also with the upcoming release, I'm having major FOMO :)

Edit: correct link to commit.

[emersion]

Sorry, I still have no clue why this bug is happening...

[J0nnyMak0]

Sorry, I still have no clue why this bug is happening...

Oh well. Thanks for the response.

[emersion]

NOUVEAU_GEM_PUSHBUF is taking ~9s to complete sometimes.

[emersion]

Opened a Nouveau issue, because I don't think we're doing anything wrong: https://gitlab.freedesktop.org/drm/nouveau/-/issues/77

[emersion]

It seems like this isn't Nouveau-specific, even if it's more noticeable with Nouveau. Ref https://github.com/swaywm/sway/issues/6196

[J0nnyMak0]

I've changed the issue since this is a more accurate description of what is happening. On my setup with Nouvea+i915 I seem to be getting approximately 15fps refresh on external monitors.

[nwg-piotr]

BTW: I can't confirm any significant refresh rate decrease in swaywm/sway#6196, which has been closed as a duplicate.

[J0nnyMak0]

BTW: I can't confirm any significant refresh rate decrease in swaywm/sway#6196, which has been closed as a duplicate.

I think this is consistent with what others have reported elsewhere in this (admittedly long) issue comments. My guess at the time was that older hardware in my computer may be exacerbating this problem including the low refresh rate / choppy cursor etc. Others with newer hardware are only able to reproduce the CPU load/temp increases without experiencing performance loss.

[nwg-piotr]

Alright, just wanted to make sure we're having the same problem. :)

[emersion]

For me, https://github.com/swaywm/wlroots/pull/2851 helps a little. It still seems like cursor movement has low FPS, but I no longer get long freezes when the cursor image changes. CPU usage is also normal.

[J0nnyMak0]

I hate being the person to keep pouring cold water over this issue, but on my setup, I don't see a marked improvement with #2851. What I usually do to test this issue is, to play a youtube (avc1, hd) video on my external QHD monitor in full screen mode at 2x speed (using firefox native wayland). This seems to tax cache and re-use logic. With https://github.com/swaywm/wlroots/commit/5bd86b94f9a6f3b41d1bc4bbf2985632fdd6d0a7 and older everything is fine and my CPU is at ~75 deg C. With newer commits, you can clearly see the video struggling to keep up. Video and audio quickly get out of sync and CPU is at ~90 deg C. I'm just putting this out there because this is a somewhat objective method for testing the issue and any possible fix.

[emersion]

OK, sounds like I have an additional, unrelated issue causing long freezes then.

CPU usage is quite complicated to make sense of on my machine because it's quite beefy, but I can definitely notice slow FPS when moving the cursor.

[nwg-piotr]

CPU usage is quite complicated to make sense of on my machine because it's quite beefy,

On my Ryzen 7 3750H increased CPU usage / temperature are the only visible symptoms. The sway process would take 6-8% never before.

[J0nnyMak0]

CPU usage is quite complicated to make sense of on my machine because it's quite beefy, but I can definitely notice slow FPS when moving the cursor.

Ok, so the latest commits to #2851 since my last comment have in fact improved CPU temps on my machine!! With the same full-screen video test, the temps now hover around 80-85.

The FPS however haven't improved. I'm getting roughly half the FPS. Here are the screenshots of "Stats for nerds" of a random youtube video:

bad:

good:

[emersion]

Note to self: checked again that our primary surface → secondary surface → scanout pipeline is correct. Checked that the assert introduced in #2900 passes. It seems like we keep re-using the same swapchain buffer, but rotating between them doesn't make a difference.

Debug patch applied on top of #2851: https://paste.sr.ht/~emersion/9229637497211a1b1a4b26a6b1a625152dae54d1

[geekuillaume]

Is there any news regarding this issue? This is blocking me from updating to the latest version of Sway :/

[mpsq]

Some news on this, I hope this is valuable feedback: I use intel+nouveau on a powerful laptop with an external 2560 x 1440 (QHD) Dell monitor (P2418D) without issues. For some reasons I recently tried another Dell monitor (P3421W) with a 3440 x 1440 (WQHD) resolution and I am experiencing this issue with that particular monitor. I used the same port (HDMI) for both, same version of Sway / wlroots, the setup is entirely the same apart from that monitor. Not sure what is causing this but I confirm the external display is unusable while the laptop screen is fine. For what it is worth, the laptop itself is displaying on a 3840 x 2160 screen + note that it is one of those where only the Nvidia card is wired to the ports.

If it helps @emersion, from a user perspective, the issue feels very much like #2991.

[mpsq]

Some news on this, using latest master (as of today, ad7651a3), this issue is not occurring anymore for me (maybe this was fixed somewhere else? nouveau?).

EDIT: Some extra info

sway: f67ed677
wlroots: ad7651a3 (WLR_DRM_NO_MODIFIERS=1)
kernel: 5.13.9
mesa: 21.1.6
xf86-video-nouveau: 1.0.17

CPU: Intel i9-10980HK (16) @ 5.300GHz
GPU: NVIDIA GeForce RTX 2080 SUPER Mobile / Max-Q

[nwg-piotr]

Seems to be true on my all-AMD hybrid laptop, too. My two external displays are connected to Radeon Pro 5500M. On sway version 1.6-4cdc4ac6 (Aug 14 2021, branch 'master') CPU load and temperature look like on the good old sway 1.5.