Sokol: Mouse capture? (maybe separate feature than mouse lock)

[meshula]

I'd like to request that sokol do a mouse capture (see below). I have a bunch of cases with UI widgets, like camera navigators, where it's important that the mouse is not lost if the mouse goes out of the window. Would you consider a simple patch like this?

            case WM_LBUTTONDOWN:
                _sapp_win32_mouse_event(SAPP_EVENTTYPE_MOUSE_DOWN, SAPP_MOUSEBUTTON_LEFT);
                if (!mouse_buttons && ::GetCapture() == NULL) ::SetCapture(hWnd);
                mouse_buttons |= 1;
                break;
            case WM_RBUTTONDOWN:
                _sapp_win32_mouse_event(SAPP_EVENTTYPE_MOUSE_DOWN, SAPP_MOUSEBUTTON_RIGHT);
                if (!mouse_buttons && ::GetCapture() == NULL) ::SetCapture(hWnd);
                mouse_buttons |= 2;
                break;
            case WM_MBUTTONDOWN:
                _sapp_win32_mouse_event(SAPP_EVENTTYPE_MOUSE_DOWN, SAPP_MOUSEBUTTON_MIDDLE);
                if (!mouse_buttons && ::GetCapture() == NULL) ::SetCapture(hWnd);
                mouse_buttons |= 4;
                break;
            case WM_LBUTTONUP:
                _sapp_win32_mouse_event(SAPP_EVENTTYPE_MOUSE_UP, SAPP_MOUSEBUTTON_LEFT);
                if (!mouse_buttons && ::GetCapture() == hWnd) ::ReleaseCapture();
                mouse_buttons &= ~1;
                break;
            case WM_RBUTTONUP:
                _sapp_win32_mouse_event(SAPP_EVENTTYPE_MOUSE_UP, SAPP_MOUSEBUTTON_RIGHT);
                if (!mouse_buttons && ::GetCapture() == hWnd) ::ReleaseCapture();
                mouse_buttons &= ~2;
                break;
            case WM_MBUTTONUP:
                _sapp_win32_mouse_event(SAPP_EVENTTYPE_MOUSE_UP, SAPP_MOUSEBUTTON_MIDDLE);
                if (!mouse_buttons && ::GetCapture() == hWnd) ::ReleaseCapture();
                mouse_buttons &= ~4;
                break;

[floooh]

Yes, but the whole topic is wider than just calling SetCapture and ReleaseCapture on Windows though (but if it works for your use case now, I'd recommend doing just that).

To be cross-platform compatible, I'd like to implement mouse-capture as "pointer-lock", similar to how I've done it in Oryol, which in turn was inspired by GLFW plus some additional tweaks for the web platform.

Pointer-lock basically means forcing the absolute mouse position to the center of the screen and only emitting movement information (this is fine for camera-dolly controllers, first-person-shooter-controls etc...), but unfortunately isn't useful for UI slider wigdets, because for some platforms it's impossible to get absolute mouse coordinates outside the own window area (thus the "force mouse position to the center" trick).

This also means that pointer-lock is mostly only useful with an invisible mouse pointer.

On the web, pointer-lock is a special input mode with a lot of restrictions (see here: https://developer.mozilla.org/en-US/docs/Web/API/Pointer_Lock_API)

• mouse pointer is made invisible automatically
• must be requested, and can only be activated from an input event handler
• only movement is reported, not absolute position
• a browser popup might occur

(you can check the behaviour in browsers here: http://floooh.github.io/oryol-samples/wasm/BulletPhysicsBasic.html, click, hold and move the left mouse button to take over the camera).

...so since the web is the most restrictive platform for this, it would make sense to make the other platforms behave similar, and to model the sokol-app API after those restrictions. I'm not really happy with the API I had in Oryol (it was callback based), so I'd need to experiment around a bit what feels right for the sokol-app API.

So in short: if the SetCapture() and ReleaseCapture() method is currently working for you, I'd recommend that you use that in your own copy for sokol_app.h for now, and I'll start thinking about the "proper" pointer-lock feature :)

[meshula]

Sounds good to me. I was actually looking to see if you'd done a pointer-lock that I could build mouselook over. In the end, I implemented it app-side on top of the Set/Release code above. I didn't want to spend time investigating cross platform until I'd heard your thoughts. A solutionspecific to Windows will be fine for my use case for a little while. If I circle back and do something more elaborate, I'll let you know.

[floooh]

Ok, I have a first working implementation of pointer-lock for macOS in the pointer-lock-2 branch (I called it "mouse-lock" to be more in line with the other function names).

It's just two new functions:

void sapp_lock_mouse(bool lock);
bool sapp_mouse_locked(void);

I have converted the cgltf-sapp sample to have a "orbit camera" (similar to the Maya camera, but activated by keeping the left mouse button pressed down), here's a code example how the mouse-lock would be used within an sokol-app event handler function (although it will not be required to call those functions from within the event handler, at least if my plan for WASM/emscripten works as intended):

// input event handler for camera manipulation
static void input(const sapp_event* ev) {
    if (__dbgui_event(ev)) {
        return;
    }
    switch (ev->type) {
        case SAPP_EVENTTYPE_MOUSE_DOWN:
            if (ev->mouse_button == SAPP_MOUSEBUTTON_LEFT) {
                sapp_lock_mouse(true);
            }
            break;

        case SAPP_EVENTTYPE_MOUSE_UP:
            if (ev->mouse_button == SAPP_MOUSEBUTTON_LEFT) {
                sapp_lock_mouse(false);
            }
            break;

        case SAPP_EVENTTYPE_MOUSE_SCROLL:
            cam_zoom(&state.camera, ev->scroll_y * 0.5f);
            break;

        case SAPP_EVENTTYPE_MOUSE_MOVE:
            if (sapp_mouse_locked()) {
                cam_orbit(&state.camera, ev->mouse_dx * 0.25f, ev->mouse_dy * 0.25f);
            }
            break;

        default:
            break;
    }
}

(code is here: https://github.com/floooh/sokol-samples/blob/b2b24ffa80ab7dce93eac8361d8ce7d68f939fbe/sapp/cgltf-sapp.c#L395-L426).

Basically, while mouse-lock is active, MOUSE_MOVE events only report movement in the new mouse_dx and mouse_dy members. The mouse position appears to be frozen (and the mouse pointer will be hidden, not sure if I can make the hiding an optional feature, that depends mostly on the pointer-lock web-API).

The mouse_dx/dy members are also valid while the mouse is unlocked, but in that case they are computed by subtracting the reported mouse positions (so they will "stop" either when hitting the screen edge (on macOS), or when leaving the window (on the web and maybe other platforms)). With mouse lock active, dx/dy is always reported no matter how far the mouse is moved, BUT the units will/may be different from the dx/dy values while mouse lock is inactive (because the mouse-lock mouse movement isn't necessarily in screen-pixels.

My current plan is to at least implement the Windows and WASM mouse-lock before merging the branch into master (and I also need to write documentation before the any merge can happen).

[floooh]

...just updating this issue to incorporate your feedback on the mouse-lock PR here:

https://github.com/floooh/sokol/pull/348#issuecomment-667638215

https://github.com/floooh/sokol/pull/348#issuecomment-667672684

...I think it makes sense to implement a separate mouse-capture feature which allows to capture the mouse for UI interaction which works like the Windows SetCapture() functions (I'll need to check what the equivalent on the other platforms is).

[floooh]

@meshula ping: I have added "regular" SetCapture/ReleaseCapture calls now for Win32 here in this commit https://github.com/floooh/sokol/commit/78b9b6c4787ae4569e636168ac8ce33adec18941.

I noticed that the new UWP backend behaves that way automatically, I still need to check whether macOS and Linux need a similar fix.

PS: reading through the related issue comments: I noticed while SetCapture/ReleaseCapture provides mouse input outside the window area, it still "stops" in one dimension when hitting the screen border. Hopefully the fix is still "good enough" (it immediately feels better in ImGui applications because one can now grab a slider and scrollbar and yank it around also when the mouse is outside the window).

[floooh]

...and a small fix on macOS: while the mouse is "dragged" (which is the same as "capture" on Win32), no enter/leave events are generated when the mouse crosses the window border. This behaves the same as on Windows. Without this, sokol_imgui.h would lose the mouse button status because of this code:

https://github.com/floooh/sokol/blob/fcacc89864793f5cc571f23e909a6a246a8718a3/util/sokol_imgui.h#L2019-L2026

...the intent of this code is to prevent the mouse buttons from being stuck when the mouse exits the window and any mouse-button-up events might not arrive. I'm actually not sure if this behaviour is needed anymore (I think it was a problem on Windows before adding the SetCapture/ReleaseCapture calls).

Last platform I need to check is Linux...

[floooh]

...ok, and Linux also behaves the same now. No enter/leave events are sent while mouse buttons are down (other then that behaviour was already correct, while mouse buttons are pressed down, mouse events are sent also when the mouse is outside the window, otherwise not).