Sway: Unable to start sway on Arch Linux running in VMWare Fusion

[gpanders]

Please fill out the following:

• Sway Version: 1.5.1

• Debug Log: https://paste.sr.ht/~gpanders/64de4204fcd197701c1750ffcd02cf2397b37e37

• Configuration File: Default sway config file

• Description: This is a fresh install of Arch Linux on VMWare Fusion. I'm running sway from the command line and then the screen goes black and unresponsive. The only way for me to kill it is to switch to another tty using Ctrl+Alt+F3 and running killall sway.

These are the errors I see in the log file:

00:00:00.178 [ERROR] [backend/drm/atomic.c:36] Virtual-1: Atomic commit failed (pageflip): Invalid argument
00:00:00.190 [ERROR] [backend/drm/renderer.c:381] Tried to acquire an FB with a NULL BO
00:00:00.190 [ERROR] [backend/drm/atomic.c:164] Failed to set plane 35 properties
00:00:00.190 [ERROR] [backend/drm/renderer.c:381] Tried to acquire an FB with a NULL BO
00:00:00.190 [ERROR] [backend/drm/atomic.c:164] Failed to set plane 35 properties
00:00:00.192 [ERROR] [backend/drm/renderer.c:381] Tried to acquire an FB with a NULL BO
00:00:00.192 [ERROR] [backend/drm/atomic.c:164] Failed to set plane 35 properties
00:00:00.207 [ERROR] [backend/drm/renderer.c:381] Tried to acquire an FB with a NULL BO
00:00:00.207 [ERROR] [backend/drm/atomic.c:164] Failed to set plane 35 properties
00:00:01.196 [ERROR] [backend/drm/renderer.c:381] Tried to acquire an FB with a NULL BO
00:00:01.196 [ERROR] [backend/drm/atomic.c:164] Failed to set plane 35 properties
00:00:02.200 [ERROR] [backend/drm/renderer.c:381] Tried to acquire an FB with a NULL BO
00:00:02.200 [ERROR] [backend/drm/atomic.c:164] Failed to set plane 35 properties
00:00:12.225 [ERROR] [backend/drm/atomic.c:36] Virtual-1: Atomic commit failed (modeset): Permission denied

I've tried with and without VMWare's "Accelerate 3D Graphics" option. It doesn't seem to make a difference.

[emersion]

Can you obtain DRM debug logs? See https://github.com/swaywm/wlroots/wiki/DRM-Debugging

[gpanders]

dmesg.log

[emersion]

[ 2965.440924] [drm:vmw_du_cursor_plane_atomic_check [vmwgfx]] *ERROR* surface not suitable for cursor
[ 2965.440934] [drm:drm_atomic_helper_check_planes [drm_kms_helper]] [PLANE:35:plane-1] atomic driver check failed
[ 2965.440952] [drm:drm_atomic_check_only [drm]] atomic driver check for 000000002c4d38b2 failed: -22

[gpanders]

$ lsmod | grep drm_kms_helper
drm_kms_helper        266240  1 vmwgfx
cec                    73728  1 drm_kms_helper
syscopyarea            16384  1 drm_kms_helper
sysfillrect            16384  1 drm_kms_helper
sysimgblt              16384  1 drm_kms_helper
fb_sys_fops            16384  1 drm_kms_helper
drm                   585728  5 vmwgfx,drm_kms_helper,ttm

I've scoured the package manager for every permutation of mesa, drm, dri, and vmware I can think of and it appears I have all appropriate packages installed.

Also, I don't know if this is relevant/helpful, but I can start sway from an already running Gnome session (i.e. running sway from Gnome Terminal). It opens sway in a separate window but everything seems to be working as expected. It's only when started from the tty that it fails to start. That makes me think it's not a driver problem (though I could be wrong).

[hedgepigdaniel]

WLR_NO_HARDWARE_CURSORS=1 sway is a workaround for me.

[emersion]

It sounds like this check fails in the driver:

    if (surface && !surface->snooper.image) {
        DRM_ERROR("surface not suitable for cursor\n");
        ret = -EINVAL;
    }

The snooper image is created in these conditions:

    if (metadata->scanout &&
        metadata->num_sizes == 1 &&
        metadata->sizes[0].width == 64 &&
        metadata->sizes[0].height == 64 &&
        metadata->format == SVGA3D_A8R8G8B8) {

        srf->snooper.image = kzalloc(64 * 64 * 4, GFP_KERNEL);
        if (!srf->snooper.image) {
            DRM_ERROR("Failed to allocate cursor_image\n");
            ret = -ENOMEM;
            goto out_no_copy;
        }
    } else {
        srf->snooper.image = NULL;
    }

We need to figure out which one of these conditions fail.

Can you provide a drm_info log?

[hedgepigdaniel]

drm_info, in windowed mode: https://gist.github.com/hedgepigdaniel/63c37724779f70a5415efe8237f61bca

[hedgepigdaniel]

The "preferred" mode changes in drm_info when resizing the host window, or when switching to and from full screen modes, or enabling/disabling the use of multiple screens in full screen mode.

However, the modes seen by sway according to swaymsg -t get_outputs don't seem to update consistently. When new displays are added or removed they appear (I think usually with the correct resolution), but resizing an existing host window (guest screen) does not seem to cause sway to update the available modes.

Also when guest use of multiple screens is enabled, the cursor is sometimes not able to travel across the entire area of all the screens.

LMK if any of these deserve separate issues - just posting here in case it's related.

[hedgepigdaniel]

The symptom is different when VMware is set to full screen mode - sway starts and works, but the cursor is not visible as in https://github.com/swaywm/sway/issues/3814

drm_info: https://gist.github.com/hedgepigdaniel/bf4a121cace8d9eb0e0e83393dd71a82

[hedgepigdaniel]

I see many messages like this in my kernel log:

[drm:vmw_du_cursor_plane_atomic_check [vmwgfx]] *ERROR* surface not suitable for cursor

[gpanders]

Thanks @hedgepigdaniel, setting WLR_NO_HARDWARE_CURSORS=1 works for me as well. I had seen that recommended in other issues (such as #3814) but I didn't think to try it since the problems in those issues were different than mine.

[hedgepigdaniel]

The snooper image is created in these conditions:

  if (metadata->scanout &&
      metadata->num_sizes == 1 &&
      metadata->sizes[0].width == 64 &&
      metadata->sizes[0].height == 64 &&
      metadata->format == SVGA3D_A8R8G8B8) {

      srf->snooper.image = kzalloc(64 * 64 * 4, GFP_KERNEL);
      if (!srf->snooper.image) {
          DRM_ERROR("Failed to allocate cursor_image\n");
          ret = -ENOMEM;
          goto out_no_copy;
      }
  } else {
      srf->snooper.image = NULL;
  }

We need to figure out which one of these conditions fail.

Preliminary findings to be confirmed: None of this code runs at all. I put a DRM_ERROR in the else block and didn't see any log messages.

Next step is to put more of them at all the early returns earlier in that function and see of one of them is the culprit.

[hedgepigdaniel]

To save me a bunch of confusion - what code in wlroots would you expect to be calling vmw_surface_define_ioctl?

[emersion]

what code in wlroots would you expect to be calling vmw_surface_define_ioctl?

This function is hooked up to the ioctl VMW_CREATE_SURFACE, which I expect would be called when creating a GBM BO (so as a result of a gbm_bo_create call, which should call the driver-specific function to allocate a buffer in Mesa).

[emersion]

Mesa call, if that helps: https://gitlab.freedesktop.org/mesa/mesa/-/blob/dc3505f87ed69ac843cd4ce7d269b2ab01a32ff7/src/gallium/winsys/svga/drm/vmw_screen_ioctl.c#L187

[hedgepigdaniel]

Well, it turns out that mesa doesn't call vmw_surface_define_ioctl - instead it calls vmw_gb_surface_define_ext_ioctl, and that does not set snooper.image.

The choice between the two ioctls is made here on the basis that have_gb_objects is true.

[emersion]

OK. Can you report a Mesa bug about this?

[hedgepigdaniel]

Sure, https://gitlab.freedesktop.org/mesa/mesa/-/issues/4003

[IvoMajic]

WLR_NO_HARDWARE_CURSORS=1 doesn't help in VMWare Fusion 12.1.0 (17195230) (macOS Big Sur host, fresh Arch Linux guest). I get the following error when starting from tty1:

00:00:00.004 [sway/main.c:349] Unable to retrieve socket path

Same issue when starting from GDM. If started from gnome-terminal sway successfully runs in a separate window.

[emersion]

@IvoMajic Please redirect the output of Sway to a file to get full logs: sway -d >sway.log 2>&1. The console is messed up when Sway switches it to graphics mode.

[IvoMajic]

@emersion full log:

https://gist.github.com/IvoMajic/e09a49d12510e1efa61d1c67dfa3c90b

The result is just a black screen, which can only be exited by switching to other tty or shutdown.

[emersion]

You're also getting this:

00:00:00.221 [ERROR] [backend/drm/atomic.c:36] Virtual-1: Atomic commit failed (pageflip): Invalid argument