Terminal: [Conpty] Add support for mouse input

[wez]

On: Microsoft Windows [Version 10.0.17763.134]

I have implemented conpty in my terminal emulator:
https://github.com/wez/wezterm

I can successfully run target\debug\wezterm.exe to spawn console applications such as cmd.exe powershell.exe and bash.

The issue I'm seeing is that when I launch bash, either indirectly via cmd.exe or directly via the bash launcher, conpty seems to swallow the mouse reporting escape sequences; I don't see them being received by my terminal parser, and thus vim has no effective mouse support despite being configured with set mouse=a.

Running the same WSL install via wsl-terminal does have working mouse support, and wezterm has been my daily driver on Linux for about a year with working mouse support, so we can rule out an obvious misconfiguration with vim and the parser in wezterm.

I've also tried echo -e "\e[?1000h" to manually enable mouse reporting from the shell; normally (on linux and via wsl-terminal) this causes clicks in the terminal to send data to the shell (which appears as garbage input), but when running my terminal with conpty, these are also swallowed up somewhere.

Is there something special needed for the apps that I spawn into my pty to be able to work with the mouse?

In case you want to double check the key portion of the code, the relevant file is:
https://github.com/wez/wezterm/blob/master/src/winpty.rs
The flow is to CreatePipe a pair of pipes, CreatePseudoConsole, and then pass that to a spawned child via the threadproc attributes, as the samples in the MSDN docs and this repo also do.

[DHowett-MSFT]

Hey @wez,
Unfortunately, ConPTY won't transit mouse reports (or, from a hosted application, _requests for mouse reporting_). We've got a backlog item tracking this that we're hoping to get to soon.

Unfortunately, we can't just pass encoded mouse events through: since ConPTY can host standard windows console applications, the likes of which would be expecting MOUSE_EVENTs to come in through ReadConsoleInput, we'll need to do some translation.

In all other regards, though, it sounds like you're setting up the pseudoconsole correctly.

---

Tracking: MSFT:20469462

[wez]

@DHowett-MSFT thanks for the response!
It's a bit of a bummer that the mouse reporting isn't there yet, but still good that the rest of the pty stuff is possible now!

[orlp]

I have nothing to add other than really wanting mouse support with ConPTY. With alacritty now having ConPTY support using alacritty + ssh + tmux is an incredible linux terminal on Windows, only mouse support is lacking now.

[nurbles]

I am a heavy user of midnight commander in my Ubuntu bash shell where it works GREAT. Unfortunately, the Ubuntu shell tab in Windows Terminal 0.3.2171.0 does not appear to send ANY mouse events through to the mc application, making it extremely difficult for me to use. I was going to post a bug, but I this it would duplicate this.

[Dijir]

For my use, sending mouse events is of utmost importance for a good experience with vim and tmux.

[knuckolls]

Just dropping my support for this here, plus a little bit of context. Over the past month I have stopped dual booting and started using the Windows 10 insider builds "fast" ring as my primary personal machine. On that I have wsl2 working perfectly, x410 working perfectly, microsoft terminal working great, visual studio code wsl2 interop working perfectly, and the explorer.exe interop working perfectly.

This mouse input ticket is pretty much the only thing stopping me from calling wsl2 / microsoft terminal a reasonable replacement for having a separate partition / development box. Since some spots here are open source, are there any pointers as to how to see the mouse device inputs from somewhere in /dev or should I just hang tight?

Thanks for all of this though! I really like it otherwise :)

[damnskippy]

One more vote to say/request that mouse support would be so sweet, and is sorely missed by anyone using tmux, though can sort of get by without it.

[r-rojo]

Please please please enable mouse support 😄

[damnskippy]

@DHowett-MSFT, sorry about the nag, is there any plan to fix this any time soon? Seeing labels/priorities assigned on other issues, but this is not one of them, so just checking. Thank you.

[DHowett-MSFT]

@damnskippy Priority is being assigned to things we need to fix in the in-box windows console host (due to internal bug deadlines). This is a _huge_ feature (and requires an appropriately-scaled specification) and while we know we need it for v1.0 we're not actively working on it today.

If we could just "fix" it like a bug, I'd love that -- but it needs a lot more than a fix.

[tjhowse]

Thanks for the update. Your work is appreciated.

[davericher]

I love what you are doing here, I do, but until we get mouse support, we have to compliment this terminal with another terminal, somewhat defeating the purpose of this terminal.

[rix1337]

Midnight Commander without this is a nightmare.

[mdtauk]

Mouse Input in general could be useful, say if someone in the community, were to create a DosBox add-in and profile which could run in Windows Terminal.

[thisguychris]

I'm a heavy vim user; even I miss mouse support 😢

I just read @cinnamon-msft blog update, seems the team is aiming for 1.0 by the end of this year? Does this mean we get mouse support by the end of the year? If so, is it actively being worked on right now?

[zadjii-msft]

Does this mean we get mouse support by the end of the year?

Maybe, but no hard commits. Estimating how long it takes software to get done is just about as hard as proving P==NP. I'd go so far as to say that the terminal wouldn't be 1.0-ready without this.

If so, is it actively being worked on right now?

Not currently. No one is assigned to the task, and usually when someone on the team bites off a task they'll assign themselves.

[SamuelEnglard]

Out of curiosity, is everything needed to do this here? Could an ambitious (read: crazy) developer take it on and make a PR?

[zadjii-msft]

Sure, someone ambitious absolutely could try this on their own. Where to begin?

First, lets outline some scope. There's a lot of work to do with mouse input, so it'd be best to start small, and work on pieces to get to a complete solution. I think the first thing we should get working is just simple SGR-encoded mouse up/down sequences. We can work on mouse wheel events, then maybe hover, after that, but clicking I think will solve _most_ use cases.

Start by taking a look at InputStateMachineEngine.cpp. The InputStateMachineEngine is responsible for parsing input sent over conpty, and translating it into INPUT_RECORDs. An enterprising young developer would want to modify that class to be able to also parse those mouse sequences, and translate them into INPUT_RECORDs. Once you have the INPUT_RECORDs, call InteractDispatch::WriteInput. This will add those INPUT_RECORDs to the input buffer. Once they are in the input buffer, they'll be delivered to the attached console client application normally.

Things I'd watch out for:

• conhost might not insert MouseEvents into the buffer at all unless we're in mouse input mode. If that's the case, then we should make sure to just ignore these sequences as well.
• Apps that want mouse input from VT will get not a stream of INPUT_RECORDs, but a stream of characters. At some point in conhost, we attempt to translate those mouse INPUT_RECORDs into a stream of characters, if the attached application is in VT mouse mode. If we do that translation _before_ the mouse events are in the buffer, then doing the above might not work for VT applications (read: wsl). If that's the case, then we'll need to make sure that the translation from mouse INPUT_RECORD to VT-encoded mouse events is done manually for the mouse events generated by the InputStateMachineEngine.
  
  
  
  • Looking into it more, that does look like the case. We only translate mouse input for events that were initiated by the window unfortunately. See this code:
    
    https://github.com/microsoft/terminal/blob/2c8b3243dca0c48dd05ecd7b420a7a03b3e19c93/src/interactivity/win32/windowio.cpp#L113-L129
    
    terminalMouseInput.HandleMouse will synthesize VT sequences for the client application, but it's only called from the window proc unfortunately. So, we'll need to somehow expose a way for the InputStateMachineEngine to call that (via a new method on InteractDispatch), and if that method fails, then generate the appropriate INPUT_RECORDs.
  
  
  • Technically, someone could move the terminalMouseInput.HandleMouse call instead to the reading of the InputBuffer, and have it try to translate INPUT_RECORDs right as they're read, but that might be more complicated.
  
  

[tungwaiyip]

What the deal with this issue? Mouse works perfectly in in the ancient Windows Console. Does it mean I have to stuck with console until this is fixed?

[DHowett-MSFT]

If you need VT mouse support, yes.

[DHowett-MSFT]

Just an FYI to any community members who may have been looking into this (/cc @SamuelEnglard!) we’ve officially booked work on the dev team to do this. I hope we’re not stepping on your toes!

[SamuelEnglard]

I thought about it, but haven't been able to book my own time so perfect lol!

[rsinuhe]

I was just playing with VSCode with the remote development extensions and found out that the integrated terminal in it actually supports mouse mode in tmux! Panel selection, window selection, panel resizing and scroll wheel support all work. I'm new to these projects though so I don't know if the terminal is part of the open source VS Codium and can be used as a starting point... sorry if this is not really useful info

[thinkjrs]

Just an FYI to any community members who may have been looking into this (/cc @SamuelEnglard!) we’ve officially booked work on the dev team to do this. I hope we’re not stepping on your toes!

@DHowett-MSFT @zadjii-msft @bitcrazed Your communication on this issue herein and elsewhere has been fantastic; this is an exemplar to successfully involve the community in building your software and it shows. Your team(s) (console/WSL/msft-linux) is personally responsible for my business having Windows (non-nix) installs, at all. Keep up the outstanding work 🥇

[bitcrazed]

@thinkjrs Many thanks for your kind words.

And our sincerest thanks to you and everyone in our community who runs & tests / files bugs / submits asks, ideas, and pull-requests for Terminal, Cascadia Code, WSL, etc. Your feedback directly influences us as we prioritize work, and plan and design features.

We weren't kidding when we say that we build these features for and with our community 😜

[dmxt]

What is the process? Are there any functional mouses in WSL? I.e. tmux panel switching, clicking to change channel/server in weechat&irssi, (n)vim clicks, aptitude clicking, htop clicking, etc.

[offero]

What is the process? Are there any functional mouses in WSL? I.e. tmux panel switching, clicking to change channel/server in weechat&irssi, (n)vim clicks, aptitude clicking, htop clicking, etc.

@dmxt Currently I use wsltty which has been the most compatible of all the terminal options for me. I look forward to switching to Terminal once some of these features come around.

[dmxt]

What is the process? Are there any functional mouses in WSL? I.e. tmux panel switching, clicking to change channel/server in weechat&irssi, (n)vim clicks, aptitude clicking, htop clicking, etc.

@dmxt Currently I use wsltty which has been the most compatible of all the terminal options for me. I look forward to switching to Terminal once some of these features come around.

I agree with your comment, after trying out all terminal emulators known to the public for Windows in the past several years, for now at the time of writing, wsltty is the best out there. Their official repo is great as well, they have a wonderful guide for me to get started out in a speedy manner. You couldn't ask for better with witty, it got full mouse support with no hiccups throughout various different workflows and tools.

I noticed the slight latency in I/O and I think it's the bottleneck of the WSL1 system. I'm on bare-metal Linux and there's 0ms latency with mouse input.

[iivmok]

@dmxt @offero I have had success with XShell (real command line from experimental features, or ssh into WSL) - has mouse support etc, just FYI. Also apps that needs mouse support badly are midnight commander and micro editor

[SiarheiKuchukBelitsoft]

Please take a look how it is solved in ConEmu

[kvnxiao]

Any roadmap or timeline for when exactly this will hopefully be implemented? Seems like quite an important feature to release. It's kind of surprising to me that v1.0 is being released without this being resolved. I guess versioning doesn't mean anything nowadays.

[fat0troll]

@kvnxiao in latest Microsoft Store version of Windows Terminal mouse input is supported (at least in Vim), as far as I can tell

[kvnxiao]

@fat0troll As far as I can tell, this is definitely not the case. Even in vim with set mouse=a, mouse input works on old conhost but not Windows Terminal 1.0.1401.0.

[fat0troll]

set nocompatible
syntax on
set number
set mouse=a
set backspace=indent,eol,start

With that vim config I can click inside vim's window and cursor will move to place where I've clicked. 1.0.1401.0, Windows build 18368.836 (if it have any effect on this).

[DHowett]

@kvnxiao I'm going to guess you're using OpenSSH_For_Windows_7.7. There's a bug in it (resolved in 8.x) that prevents it from working for mouse mode.

We explicitly implemented this for all VT applications that want to receive mouse input.

I guess versioning doesn't mean anything nowadays.

There's no need to be unkind.

[kvnxiao]

Regarding vim, I've attempted it using neovim built for windows as a windows executable. If others are saying that mouse support does work for vim (e.g. through ssh / wsl, etc.), then I am not doubting you, but this shows that "full" support doesn't exist as of yet, which poses a more specific question:

What would be left in the roadmap for "full" mouse support in comparison with what conhost is currently capable of?

I am speaking in terms of wanting to launch generic terminal-based applications that support mouse input (and do work in conhost) through WT. For example, running several text-based/terminal user-interface applications built directly as a windows executable. These do work fine when double-clicked on to run, which resort to using conhost, but when ran through Windows Terminal they end up with just the mouse selecting the text displayed.

@niklaskorz What is the point of upvoting and downvoting the aforementioned comments when people like me have relevant questions pertaining to this topic, that may or may not still be unresolved?

[DHowett]

You’re totally right. This workitem, which you have correctly identified as being for Win32 console applications receiving mouse events from any terminal, is slated for “terminal 1.x” (milestone), which indicates that we want to tackle it between now and 2.0. I don’t have a finer-grained estimate than that.

[kvnxiao]

Thanks for the clarification! A bit sad that this workitem wasn't able to be competed for 1.0 but I'm eagerly awaiting for when it does, and hopefully it won't be too long of a wait 😀.

[tig]

FIWW, in case you weren't aware of it, the new Powershell Out-ConsoleGridView (https://github.com/PowerShell/GraphicalTools) is a killer test case for this. See Mouse related tracking bug there: https://github.com/PowerShell/GraphicalTools/issues/95

It is built on top of Terminal.Gui (https://github.com/tig/gui.cs).

In addition, we just built a new sample app for Terminal.Gui that y'all should be able to use to test mouse support as it comes to life in WT.

Really looking forward to it!

[tig]

Is there any way I can help get this issue fixed? It is a real bummer for GUI console apps built with Terminal.Gui (https://github.com/tig/gui.cs).

[ipcjs]

@kvnxiao I'm going to guess you're using OpenSSH_For_Windows_7.7. There's a bug in it (resolved in 8.x) that prevents it from working for mouse mode.

We explicitly implemented this for all VT applications that want to receive mouse input.

I guess versioning doesn't mean anything nowadays.

There's no need to be unkind.

How can i update build-in openssh to latest version?

[thochra]

@kvnxiao I'm going to guess you're using OpenSSH_For_Windows_7.7. There's a bug in it (resolved in 8.x) that prevents it from working for mouse mode.
We explicitly implemented this for all VT applications that want to receive mouse input.

I guess versioning doesn't mean anything nowadays.

There's no need to be unkind.

How can i update build-in openssh to latest version?

I guess you're looking for something described in this blog post (Openssh install from chocolatey) : https://blog.frankfu.com.au/2019/03/21/moving-from-windows-1809s-openssh-to-openssh-portable/

[shaneschulte]

I'm going to guess you're using OpenSSH_For_Windows_7.7. There's a bug in it (resolved in 8.x) that prevents it from working for mouse mode.

We explicitly implemented this for all VT applications that want to receive mouse input.

@DHowett from this context, it sounds like using OpenSSH_for_Windows_8.0p1, LibreSSL 2.6.5 would be expected to work?

I'm using the Terminal preview build to SSH to an Ubuntu machine with tmux mouse mode enabled, but my mouse inputs still seem to only be controlling the terminal itself.

I've also tried upgrading the server to OpenSSH 8.0, but that didn't help either.

Is this issue still blocking that sort of thing from working?

[DHowett]

Ah, the x in 8.x might be 1. I was not aware that they released an 8.0 build.

[shaneschulte]

Ah, the x in 8.x might be 1. I was not aware that they released an 8.0 build.

I'll give that a shot. Thanks for the incredibly fast response!

It works! Amazing, thank you!

[Hamayama]

I began to understand this issue.

Sadly, MS doesn't implement mouse input function of Win32 API on Windows Terminal yet.
(Only VT escape sequence is supported.)

I tried to replace ReadConsoleInputW and PeekConsoleInputW in my application
to work with mouse on Windows Terminal.

First, I run the following code.

SetConsoleMode(hin,  ENABLE_VIRTUAL_TERMINAL_INPUT);
SetConsoleMode(hout, ENABLE_PROCESSED_OUTPUT | ENABLE_VIRTUAL_TERMINAL_PROCESSING);
char *vt_mouse_input_enable_cmd  = "\x1b[?1000h\x1b[?1003h\x1b[?1006h";
DWORD written;
WriteConsoleA(hout, vt_mouse_input_enable_cmd, strlen(vt_mouse_input_enable_cmd), &written, NULL);

Then, mouse input can be received as VT escape sequence (sgr-1006) (e.g. \x1b[<0;10;20M ).

But, another problem occurred.

Some key input (such as arrow keys) are also received as VT escape sequence (e.g. \x1b[A ).

I tried to convert these unwanted VT escape sequece to key event of win32 API,
but this is incomplete.
(virtual key code and virtual scan code casually become zero, etc.)

My code is here.
https://gist.github.com/Hamayama/6add968870269f2426716fad79724b31
( PDC_read_console_input_w and PDC_peek_console_input_w are alternative functions. )

I want a method to disable VT escape sequece except for mouse input.

for example

char *vt_key_input_disable_cmd = "\x1b[?9XXXl";
DWORD written;
WriteConsoleA(hout, vt_key_input_disable_cmd, strlen(vt_key_input_disable_cmd), &written, NULL);

or

SetConsoleMode(hin, ENABLE_VIRTUAL_TERMINAL_MOUSE_INPUT_ONLY);

But, this might be a wrong idea for the future ...

[o-sdn-o]

I found that there is no tracking of mouse movement in src/terminal/parser/InputStateMachineEngine.cpp: 391 (translation of SGR VT-sequences into INPUT_RECORDs).

In other words, the Terminal sends VT-sequences for ConPTY, but ConPTY only monitors the state of the mouse buttons. Changes in mouse coordinates are ignored:

src/terminal/parser/InputStateMachineEngine.cpp: 391:

success = _UpdateSGRMouseButtonState(id, firstParameter, buttonState, eventFlags);
success = success && _WriteMouseEvent(parameters.at(1), parameters.at(2), buttonState, modifierState, eventFlags);

With the current state of support for mouse input, the following option is possible.

You can add coordinate tracking and mouse movement will start working in classic console applications.

src/terminal/parser/InputStateMachineEngine.hpp: 172:

+ size_t _mouseColumn = 0;
+ size_t _mouseLine = 0;

src/terminal/parser/InputStateMachineEngine.cpp: 391:

- success = success && _WriteMouseEvent(parameters.at(1), parameters.at(2), buttonState, modifierState, eventFlags);
+ auto mouseColumn = parameters.at(1).value_or(0);
+ auto mouseLine = parameters.at(2).value_or(0);
+ auto isMoved = mouseColumn! = _mouseColumn || mouseLine! = _mouseLine;
+ if (isMoved)
+ {
+     _mouseColumn = mouseColumn;
+     _mouseLine = mouseLine;
+ }
+ success = (success || isMoved) && _WriteMouseEvent(mouseColumn, mouseLine, buttonState, modifierState, eventFlags);

Note: Before starting a classic console application, you need to request mouse tracking in SGR format:

Windows PowerShell

PS C:\Users> [char]0x1b + "[?1003;1004;1006h"

Command Prompt

C:\Users> echoCtrl+[[?1003;1004;1006h

Parameters

• 1003 - ANY_EVENT_MOUSE_MODE
• 1004 - any unsupported mode (e.g. 1001 or 9999) to forward the sequence through ConPTY to the Terminal itself
• 1006 - SGR_EXTENDED_MODE

as a result, the Terminal will start sending mouse events to ConpTY, which will start generating INPUT_RECORDs for the classic console application.

[DHowett]

Good catch. We'll have to make sure we fix that when we actually move to support this :smile: