Monogame: [Mac OS X] Tap on trackpad not works correctly

[mgkcorty]

Hello, guys!

I trying to migrate my iOS game to MacOS, and my code:
'mouse.LeftButton == ButtonState.Pressed' only is executed when I click the trackpad and not when I tap it.

My MacOS is configured to "Tap to click - Tap with one finger".

Sometimes on trackpad-Tap state of left mouse button changed on 'Pressed' state, but it's very unstable (1 of 10 taps) in other cases in update method we get always 'Released' mouse state like we don't Tap ever.

Looks like MonoGame not catch very fast clicks (Tap-s). (i.e. 'Pressed' state changed very fast from 'Pressed' to 'Released', and Update method don't catch it).

Tested on last stable MonoGame Release 3.6 and with develop branch. Same problem.

Thank in advance!

[harry-cpp]

Hmm... this might be due to how XNA input API works, it might be missing the state when the touchpad is pressed.

What platform MG assembly are you using, DesktopGL?

[mgkcorty]

@cra0zy thank you for answer.

Yes, DesktopGL.

it might be missing the state when the touchpad is pressed.

No, input works fine.

I make some research and can say that ->
Update method calls slower than Mac trackpad change state (chain) ['Released' -> 'Pressed' -> 'Released']. i.e. between Update calls mouse state changed from 'Released' to 'Released' and we can't catch that use make tap.

I check this with snippet, in my LoadContent method i call it:

    private void CheckActualMouseState()
    {
        Task.Run(() =>
        {
            while (true)
            {
                var mouseState = Mouse.GetState();
                if(mouseState.LeftButton == ButtonState.Pressed)
                    Debug.WriteLine(mouseState.LeftButton);
            }
        });
    }

forever while catch my trackpad taps correctly, but it's ugly solution, because new thread, and while block.

[harry-cpp]

I make some research and can say that ->
Update method calls slower than Mac trackpad change state (chain) ['Released' -> 'Pressed' -> 'Released']. i.e. between Update calls mouse state changed from 'Released' to 'Released' and we can't catch that use make tap.

That is exactly what I said... its missing the state when its pressed...

@tomspilman @KonajuGames @dellis1972 So... anybody for an event driven input API?, in all seriousness this does sound like a big problem and I'm just tagging you along for an idea on how to fix it.

[mgkcorty]

@cra0zy Thank you again, yes all right.

Also. It's very interesting check this case on Windows, with same behavior of mouse click. But firstly tune mouse button like 'make click on left mouse down'.

I think we again catch this.

[harry-cpp]

Good idea :+1:

[lotziko]

@mgkcortyw I also encountered this issue when making double tap in my GUI. Did you find a solution?

[Halofreak1990]

@mgkcortyw what Mac system are you using to test this? If I recall correctly Mac trackpads don't support tapping. Only clicking. It's one of the many things that annoy me about Macs.

[AidanMcK]

This is a huge problem - has nobody released anything for Macs ever using MonoGame?

Having this code in the update loop:

MouseState curMouseState = Mouse.GetState();
if (curMouseState.LeftButton != ButtonState.Released) Console.WriteLine("Left Button Pressed");

results in a trackpad tap being recognised some of the time only - most of the time it is ignored.

Anyone know any way to work around this problem? I can't release any product for Macs with this problem present.

[HaikuJock]

I'm looking into fixing this. I'm currently trying capturing mouse events with SDL_SetEventFilter, adding button-press events to a concurrent-queue and pumping the queue each game tick.

[harry-cpp]

There is no bug here, the API does what its meant to do.

[HaikuJock]

The API was designed before mouse events could happen faster than 60Hz.

I have a working solution on my fork but it has a flaw where simultaneous mouse events will be delivered on successive game ticks. This is not a problem for my game so I won't bother fixing it. If you change your mind I'll be happy to fix the flaw and make a PR.

[harry-cpp]

Again, not a flaw, that's how the API is supposed to act, you get the current state of the mouse when you poll for it, getting anything other than that would be considered wrong.

As for what the proper solution would be, I would say its to add few more events to the GameWindow like the TextInput event.

[HaikuJock]

Not a flaw, agreed. :-)

I started on a solution similar to the TextInput event and then noticed the TouchPanel. Is there a good reason to not use TouchPanel? It would be nice to get gesture support for free.

[harry-cpp]

TouchPanel is the touch screen API, I didn't really have any touchscreen desktop devices to implement and test it out. Not sure if XNA also treated touchpad as a TouchPanel.

[HaikuJock]

I also don't have a touchscreen desktop device. The SDL documentation and tutorials I've been looking at suggest both the mac touchpad and touchscreen devices generate the finger events. I'll see how I get on with the mac touchpad updating TouchPanel.

[nkast]

@HaikuJock,
monogame can emulate a touch events from mouse, TouchPanel.EnableMouseTouchPoint.
You can ignore the mouse and use the touch input. Each touchlocation has a complete lifecycle Pressed-Moved*-Release.

[HaikuJock]

EnableMouseTouchPoint doesn't currently work on the DesktopGL version. Adding it I find that TouchPanelState doesn't distinguish between left and right mouse-buttons.

Adding a new event-handler for finger-touches would put more work on client applications to then convert finger input into taps. SDL is already converting two-finger taps into right mouse-button presses so there's no point in doing that work again.

I'd like to add event handlers for left and right mouse button touch events.

[HaikuJock]

Mac touchpad mouse-button events are not marked as being touches, so the event handlers would be for all left and right mouse button events. I'm suggesting this would only be for the DesktopGL version of MonoGame.

[HaikuJock]

I've made a PR but I'm open to discussing alternative solutions.

[AidanMcK]

I changed SDL2.cs and SDLGamePlatform.cs (from the 3.7.1 release) locally to allow for Input events for my Mac releases and all works fine. They current situation is not acceptable on Macs at the moment. I can post the code if needed, but you'd probably want to find some other solution that does not involve using Events.

[HaikuJock]

Hi @AidanMcK
I'm curious to see how you solved this problem and what is your objection to using events?

[AidanMcK]

@HaikuJock I've no objection to using events and use them, but I would imagine the MonoGame guys would prefer a solution that is consistent across all platforms.

Here's the code:
Game.cs

public delegate void MouseButtonEventDelegate(byte button, bool down, float posX, float posY);
public delegate void MouseMoveEventDelegate(float posX, float posY);
public delegate void MouseWheelEventDelegate(float scrollX, float scrollY);
public delegate void KeyEventDelegate(Input.Keys key, bool pressed, bool repeat);

public MouseButtonEventDelegate  MouseButtonEventCB;
public MouseMoveEventDelegate    MouseMoveEventCB;
public MouseWheelEventDelegate   MouseWheelEventCB;
public KeyEventDelegate          KeyEventCB;

SDL2.cs

[FieldOffset(0)]
public Mouse.ButtonEvent Button;

[StructLayout(LayoutKind.Sequential)]
public struct ButtonEvent
{
    public EventType Type;
    public uint Timestamp;
    public uint WindowID;
    public uint Which;
    public byte Button;
    public byte State;
    private byte _padding1;
    private byte _padding2;
    public int X;
    public int Y;
}

SDLGamePlatform.cs

private void SdlRunLoop()
{
...
    else if (ev.Type == Sdl.EventType.MouseWheel)
    {
        const int wheelDelta = 120;
        Mouse.ScrollY += ev.Wheel.Y * wheelDelta;
        Mouse.ScrollX += ev.Wheel.X * wheelDelta;
        Game.MouseWheelEventCB?.Invoke(ev.Wheel.X * wheelDelta, ev.Wheel.Y * wheelDelta);
    }
    else if (ev.Type == Sdl.EventType.MouseMotion)
    {
        Window.MouseState.X = ev.Motion.X;
        Window.MouseState.Y = ev.Motion.Y;
        Game.MouseMoveEventCB?.Invoke(ev.Motion.X, ev.Motion.Y);
    }
    else if (ev.Type == Sdl.EventType.MouseButtonDown)
    {
        switch (ev.Button.Button)
        {
            case 1: Game.MouseButtonEventCB?.Invoke(0, true, ev.Button.X, ev.Button.Y); break;
            case 3: Game.MouseButtonEventCB?.Invoke(1, true, ev.Button.X, ev.Button.Y); break;
            case 2: Game.MouseButtonEventCB?.Invoke(2, true, ev.Button.X, ev.Button.Y); break;
        }
    }
    else if (ev.Type == Sdl.EventType.MouseButtonup)
    {
        switch (ev.Button.Button)
        {
            case 1: Game.MouseButtonEventCB?.Invoke(0, false, ev.Button.X, ev.Button.Y); break;
            case 3: Game.MouseButtonEventCB?.Invoke(1, false, ev.Button.X, ev.Button.Y); break;
            case 2: Game.MouseButtonEventCB?.Invoke(2, false, ev.Button.X, ev.Button.Y); break;
        }
    }
    else if (ev.Type == Sdl.EventType.KeyDown)
    {
        var key = KeyboardUtil.ToXna(ev.Key.Keysym.Sym);
        if (!_keys.Contains(key))
            _keys.Add(key);
        Game.KeyEventCB?.Invoke(key, true, ev.Key.Repeat > 0);
        ...
    }
    else if (ev.Type == Sdl.EventType.KeyUp)
    {
        var key = KeyboardUtil.ToXna(ev.Key.Keysym.Sym);
        _keys.Remove(key);
        Game.KeyEventCB?.Invoke(key, false, ev.Key.Repeat > 0);
        ...
    }
    ...
}

[AidanMcK]

And can someone change the title to say "does not work correctly" please.

[HaikuJock]

@tomspilman @harry-cpp API change proposal for mouse-button events.

Problem statement
The current Mouse.GetState() is inadequate for mouse devices that allow for sub-frame presses and releases such as Mac TouchPads.

Proposed change
Add two event handlers to the game window for button presses and releases. The parameters to these events to include:

• Button (left, right, middle, x1 and x2)
• Game window coordinates

As the problem has only been reported for Mac TouchPads, I'm suggesting that the change is only added to the DesktopGL version of MonoGame. There's a trade-off between maintaining API consistency and adding complexity to the Windows API where it's not needed. I'm happy to add handlers for the Windows API if that's the consensus.

[HaikuJock]

@AidanMcK Did you also find Mouse.GetState() inadequate for mouse motion and mouse wheel? Or did you change all your input handling to be event-driven?

[jnoyola]

Do we have this same polling problem for all inputs? Should we have a broader solution that can handle keyboard events as well?

[AidanMcK]

@HaikuJock I use events entirely for the Mac versions only.

[harry-cpp]

@jnoyola The problem comes from the fact that you can do stuff like touch to click, and it just sends both a mouse down and mouse up event, so you can't really catch it.

All XNA games will miss it, and probably a lot of other stuff miss it as they are not designed to catch it in any way.

[nkast]

A button has inertial that put a limit on how fast it can change state. There are other factors that put extra delay like polling by mouse microprocessor, usb polling, etc.
Even when a button in directly connected to an input/interrupt line, you get a capacitor in-parallel to prevent multiple signals from sparks. That also put a delay on how fast a button can change to Released state.

The touchpad driver probably sends a Pressed msg followed by Released msg without any delay.
That fools most programs but it's not a good simulation of a button.

You can simulate that inertia in MG by keeping a timestamp of the last Press event.
in GetState you return Pressed either if the last event was pressed or if the last timestamp is not older than ,say, 20ms.

[harry-cpp]

You can simulate that inertia in MG by keeping a timestamp of the last Press event.
in GetState you return Pressed either if the last event was pressed or if the last timestamp is not older than ,say, 20ms.

That would make an input delay, but that is a wonderful idea, it just needs a small tweek!

All we have to do is check SDLs timestamps when we receive MouseButtonEvent during our SdlRunLoop, and if the interval is small enough, we mark the button as Pressed until next SdlRunLoop.

We don't care about previous or next invocations of SdlRunLoops as the issue can only occur during single pass of SdlRunLoop where it receives both a Pressed and Released signals in a row.

Unless you are using touch to click it will not create an extra Release event.

I'll make a PR for this tomorrow (I do use a mac btw.).

[nkast]

About the event based API, will the events fire from another thread or get queued and fire in succession before each frame/update()?
Either way it's not a clean-cut solution for the end user. All it does is to expose the events and let the user solve the problem.

A simple implementation like the one below will fail on a mac trackpad for the same reasons.
Window.LeftButtonInput += (s,e) => { spaceShip.Engine.Enable = (e == ButtonState.Pressed); }

The user will have to set a boolean and clean it on each frame to solve the problem of trackpad.
Basically adding a one-frame-lag between press/release.
Window.LeftButtonInput += (s,e) => { fireEngines = (e == ButtonState.Pressed); }
bool fireEngines = false;
Update()
{
spaceShip.Engine.Enable = fireEngines;
fireEngines = false;
}

[HaikuJock]

@nkast Yes, the event-based API is not ideal and requires more work on the user's part.
I think what yourself and @harry-cpp have come up with is a better solution.
If that doesn't work another option would be to capture the finger touch events, send the finger events to TouchPanel and add two new gesture-types to TouchPanel: TwoFingerTap and ThreeFingerTap. (EDIT: Or add a button identifier to GestureSample)