Terminal: Alt+Numpad input is broken (since Windows 7)

[alabuzhev]

Prior to Windows 7 it was possible to enter any "Unicode" (UCS-2 0-65535 range) character into a console app using Alt + Numpad keys.
This feature has been broken since Windows 7 (probably due to moving some console code into conhost.exe).

Environment

Any OS version since Windows 7.

Any other software?
No.

Steps to reproduce

Run the attached app:
readkey.zip

• Press and hold the Alt key
• Type any large enough decimal number on the numeric keyboard, say, 8888 or 65535.
• Release the Alt key
• Press and hold the Alt key
• Type any small enough decimal number on the numeric keyboard, say, 1 or 7.
• Release the Alt key

Expected behavior

Windows 2000 / XP / 2003 / Vista / 2008

The release of the Alt key generates a KEY_EVENT_RECORD with wVirtualKeyCode = VK_MENU and uChar.UnicodeChar = .

The expected output of the app for Alt+8888:
KEY_EVENT_RECORD: Up, Count=1, Vk=VK_MENU [18/0x12], Scan=56, uChar=[U='⊸'(0x22b8) A='¸'(0xb8)], Control=0x04000020 (casac - ecNs)

The expected output of the app for Alt+1:
KEY_EVENT_RECORD: Up, Count=1, Vk=VK_MENU [18/0x12], Scan=56, uChar=[U='☺'(0x0001) A='☺'(0x01)], Control=0x04000020 (casac - ecNs)

Actual behavior

Windows 7 / 8 / 10 classic / 10 new

The release of the Alt key generates a KEY_EVENT_RECORD with wVirtualKeyCode = VK_MENU and some rubbish in uChar.UnicodeChar.

The actual output of the app for Alt+8888:
KEY_EVENT_RECORD: Up, Count=1, Vk=VK_MENU [18/0x12], Scan=56, uChar=[U='©'(0x00a9) A='©'(0xa9)], Control=0x00000020 (casac - ecNs)

The actual output of the app for Alt+1:
KEY_EVENT_RECORD: Up, Count=1, Vk=VK_MENU [18/0x12], Scan=56, uChar=[U='☺'(0x263a) A=':'(0x3a)], Control=0x00000020 (casac - ecNs)

It looks like the host performs some dodgy internal conversions on the entered Unicode character.

And it's especially mental in the case of Alt+1 (and other "control" characters below 0x20):
instead of setting both UnicodeChar and AsciiChar to 0x1, it somehow takes a Unicode "replacement" for \1 - '☺'(0x263a), as if the wollowing code has been executed somewhere:

wchar_t W;
MultiByteToWideChar(CP_OEMCP, MB_USEGLYPHCHARS, "\1", 1, &W, 1);
assert(W == 0x263a);

Is it possible to stop doing those weird conversions and return to the pre-Windows 7 behaviour?

Thanks.

[zadjii-msft]

Let me start off - great breakdown here.

For the record, there wasn't a console team in the Windows Vista-7 timeframe. We only took over modification of the console at the start of Windows 10 development, so the fact that this is something that regressed in that timeframe is surprising to say the least.

I'd agree the new behavior doesn't seem to make any sense.

This might also have some tie-in to conversations happening in #3101/#3117, though those are more Terminal-specific and not conhost specific.

I'd love to hear the rest of the team's input on this, we'll probably discuss in triage early next week. If someone wants to try digging in deeper, I believe the following is where we handle input from the window, and convert it to INPUT_RECORDs. This is where I'd start investigating what's happening.
https://github.com/microsoft/terminal/blob/634687bae3f88f883360d322265c792ef143ac50/src/interactivity/win32/windowio.cpp#L131-L449

[alabuzhev]

the fact that this is something that regressed in that timeframe is surprising to say the least

As I mentioned, somewhere between Vista and 7 at least one major thing happened - some parts were extracted from csrss into conhost, which was, supposedly, not a trivial copy&paste, so probably not that surprising after all.

I noticed this in 2009, but, since you didn't have this handy bugtracker back then, implemented a workaround and moved on. Today it resurfaced in a slightly different context, and while implementing another workaround I realised that now I can (finally) report it :)

[alabuzhev]

I think I found it.
https://github.com/microsoft/terminal/blob/634687bae3f88f883360d322265c792ef143ac50/src/host/stream.cpp#L101-L121

CharToWchar is implemented in terms of ConvertOutputToUnicode, which is implemented in terms of MultiByteToWideChar with the MB_USEGLYPHCHARS flag:
https://github.com/microsoft/terminal/blob/634687bae3f88f883360d322265c792ef143ac50/src/host/misc.cpp#L316

This explains 0x1 -> 0x263a.

And static_cast<char>(LOBYTE(L'\x22b8')) is 0xb8, MultiByteToWideChar(0xb8) is 0x00a9.

I think HIBYTE(keyEvent->GetCharData()) is 0 even for 0x22b8 here, otherwise CharToWchar would have picked the 0x22 byte and returned L'\' - probably some other code zeroes the high byte elsewhere.

P.S. It's probably not a good design choice to use ConvertOutputToUnicode for the input stream (e.g. due to unexpected effects of MB_USEGLYPHCHARS, as in this case), but that's a different question.

[alabuzhev]

some other code zeroes the high byte elsewhere

Probably here:
https://github.com/microsoft/terminal/blob/634687bae3f88f883360d322265c792ef143ac50/src/host/directio.cpp#L92-L98

But there are also quite a few other occurrences of static_cast<char>(...).

[DHowett-MSFT]

This is definitely something we should look at, so I'm taking the Triage tag off. Thanks for the great investigation.

[Shorotshishir]

hello, sorry to make this old post alive, I faced an interesting behavior today,

Widows 10 Pro 1909
Windows Terminal 1.0.1401.0 (Installed via store)

In default PowerShell,

• if you go Alt+22, it pastes whatever you copied last.
• Alt+26 removes whatever you wrote or pasted,
• Alt+25 and Alt+29 gives a knocking sound
• Alt+18 gives you a searching option. bck-i-search. Which I figured allows you to search through your terminal input history.

In Commandline:

• Alt+22 shows ^V which we used to see in cmd when we typed Ctrl+V

Not sure why this is happening, but seemed interesting, :D