Sway: Simulate input events with sway commands

[ddevault]

seat "default" cursor move +10,0
seat "default" cursor set 100,100
seat "default" cursor press|release left|right|1|2|3...
seat "default" keyboard press|release H
seat "default" keyboard type "hello world"

[emersion]

Note that the keyboard part of this issue could be achieved with Wayland protocols: virtual-keyboard and input-method.

[emersion]

Additional note: if the currently focused client supports text-input, we can use that to type text.

[athas]

I have interest in making this time, as well as possibly the time. However, I have no experience with sway's codebase, or Wayland programming in general. Can anyone suggest a good point to start looking?

[NilsIrl]

Would it also be possible to have virtual seats?

Seats which don't have any input device attached to them and can only be controlled through sway commands.

[mvdan]

I could almost get this to work with commands like:

seat - cursor set 50 50
seat - cursor press button1
seat - cursor release button1

If I understand the docs correctly, this should move the cursor at 50x50 and do a full mouse left-click.

However, it doesn't quite work. The mouse does move, but the click happens at the last position the mouse was in. I tried adding sleeps, in between the calls to swaymsg, but still the same.

[mvdan]

Thinking about it some more, I think what I tried above would work if I was able to use a virtual seat, much like what @NilsIrl suggested.

[mvdan]

I was wondering if https://github.com/swaywm/sway/pull/4674 could help here, but I don't see a way to create a virtual pointer.

[paulolieuthier]

@emersion forgive my ignorance but why isn't something like wlr_seat_keyboard_send_key enough, instead of relying on a text input protocol?

[emersion]

This depends on the keymap. If you want to send a Hangul character while having your keyboard configured as en-us, it just won't work (same the other way around).

[paulolieuthier]

That makes sense, but I don't think the text input protocol can be used to enter keys like arrows (the input method protocol has a "keysym" request that looks promising).

Before trying to use those protocols, I'm trying to implement the new commands using wlr_seat_keyboard_send_key, but can't get it to enter the arrow keys (using key codes from xev). Can you help you me?

[emersion]

enter keys like arrows

Yeah, for these we'll still need raw keycodes.

but can't get it to enter the arrow keys

There's an offset of 8 between raw Linux key codes and libxkbcommon key codes. Maybe that's the issue? Can you try getting a WAYLAND_DEBUG=1 log of wev?

[paulolieuthier]

The offset hint nailed it, subtracting 8 from the numbers got from wev produces the expected keys. Now I can finally bind capslock+hjkl to arrow keys (without messing with the system keymap)! :smile:

I'll try to experiment with the protocols and send PR that also handles complex texts besides raw key codes.

[emersion]

Note, you don't need to edit system keymaps to do this. See https://github.com/swaywm/sway/issues/4250

[paulolieuthier]

Yeah, thanks. I had seen that. But I also want to have it layout-independent and all inside sway's config (which I can achieve using xkb_options lv3:caps_switch and bindsym Mod5+).

[Tezkerek]

Can I send modifier keys with wlr_seat_keyboard_send_key?
I tried press ctrl, press c, release c, release ctrl, and wev receives those events in order, but in Alacritty for example I just get the letter c, rather than ctrl+c.

[MatthiasGrandl]

I could almost get this to work with commands like:

seat - cursor set 50 50
seat - cursor press button1
seat - cursor release button1

If I understand the docs correctly, this should move the cursor at 50x50 and do a full mouse left-click.

However, it doesn't quite work. The mouse does move, but the click happens at the last position the mouse was in. I tried adding sleeps, in between the calls to swaymsg, but still the same.

I am having exactly the same issue. Mouse moves, but clicks occur at the physical mouses last location. Also I can't get scroll events (button 4 and 5) to happen at all.

[sloonz]

seat "default" cursor move +10,0
seat "default" cursor set 100,100

Those commands move the cursor but without emiting any cursor move event (as seen in wev/xev). Is that intended ?