Psreadline: Ctrl-C doesn't work if PowerShell was started with Russian keyboard layout

[ForNeVeR]

Environment

PS version: 7.0.0
PSReadline version: 2.0.0
os: 10.0.18362.1 (WinBuild.160101.0800)
PS file version: 7.0.0.0
HostName: ConsoleHost
BufferWidth: 170
BufferHeight: 32766

Steps to reproduce

1. Enable Russian keyboard layout on Windows (I use Alt-Shift to do that)
2. Start PowerShell Core 7.0 (which includes PSReadLine 2.0)
3. Press Ctrl-C in the shell

Expected behavior

Ctrl-C should terminate current command/input/whatever, it should work as Ctrl-C usually do.

Actual behavior

Russian letter "с" getting inserted into the command line, and no actual Ctrl-C behavior happens.

Ctrl-C is still broken even if I switch to the English keyboard layout (then English letter "c" gets inserted inctead of actual Ctrl-C behavior heppening).

[iSazonov]

@ForNeVeR Do you see the issue without PSReadline? If no please move the issue to PSReadline repo.

[ForNeVeR]

@iSazonov, no, I cannot reproduce the issue without PSReadLine.

Isn't this repository already a PSReadLine repo? I can see that some repository movement has been involved, but this is definitely a repo I have been reporting PSReadLine issues earlier.

Please direct me to a right repo if this isn't right. I've tried searching for other PSReadLine repo but wasn't successful, sorry.

[iSazonov]

Isn't this repository already a PSReadLine repo?

Oh, sorry.

[daxian-dbw]

Ctrl-C is still broken even if I switch to the English keyboard layout (then English letter "c" gets inserted inctead of actual Ctrl-C behavior heppening).

@ForNeVeR Thanks for the reporting. Just want some clarification on this behavior, was it broken for you when using the English keyboard from the very beginning?

[ForNeVeR]

Ctrl-C works fine if I start PowerShell session when English keyboard layout is active. It is only broken if I start the session with Russian layout.

[daxian-dbw]

A duplicate of this issue is reported in #1442 with a different repro -- ctrl+r results in the Russian letter к.

[ForNeVeR]

Yes, actually all of the Ctrl-hotkeys are broken.

[ForNeVeR]

Exact steps to reproduce:

1. Start PSReadline (e.g. by starting PSReadline-net461 from sources).
2. Enable Russian keyboard layout, and then press Ctrl-C or Ctrl-R one time (it won't work).
3. After that, you may change the current keyboard layout as many times as you want, Ctrl-hotkeys won't work for you in this session. The session is absolutely broken.

Also, I've been able to reproduce the issue both in PowerShell Core and in PowerShell 5.1.

I was able to determine that the culprit is likely Microsoft.PowerShell.PSKeyInfo.KeyInfoAsString: this method returns a string "Ctrl+к" even when I press Ctrl-r with English keyboard layout enabled (and the input argument properly indicates Ctrl+r as being pressed). Most likely, there're two different issues:

1. Something in PSKeyInfo is static and wrongly so, so it remembers my (Russian) keyboard layout when I enter this method for the first time, and then reuses this layout even if I change it afterwards.
2. It just doesn't work as it should: it relies on the current keyboard layout for some reason, when the expected behavior would be to ignore it and always use the default one (at least, every single program I know behaves that way with the Russian+English keyboard layouts; there may be some exceptions from this rule, but I'm not aware of them yet).

I'll try to investigate further.

[ForNeVeR]

This call of ToUnicode causes problems: https://github.com/PowerShell/PSReadLine/blob/70249682ed8d87ddc3869e5d0ba852a0d6da99da/PSReadLine/Keys.cs#L150

On my system, it looks like it remembers the first keyboard layout with which it is called, and then it just uses it for all subsequent calls. So, while it obviously should hold "r" for input values of ConsoleKey.R, 19, (an array of 256 zeros), (an array of 2 zeros), 2, 4, actually it returns a Russian letter "к".

In the documentation, it isn't specified which layout it will use, but I could try to guess it uses current thread layout or something like that, and, since this code is called on a background thread (which may not process keyboard layout change messages properly), it may actually auto-initialize the keyboard layout for the first usage, and then leave it as-is. And this behavior could be dependent on OS version, installed language pack, or whatever else factor comes into play here.

I believe we could try to load an English keyboard layout and pass it to ToUnicodeEx (which takes a layout as input instead of relying on some vague global value), since our hotkeys seem to be specified in English.

@lzybkr, am I right? Would that be a good fix for the issue? I'm not quite sure why do you call all this ToUnicode business instead of e.g. creating a table like ConsoleKey.R => "r", but it surely was a good reason, I presume?

[lzybkr]

Console applications are "sticky" with regards to the keyboard layout. I reported this as a bug to the appropriate internal team but there were concerns of appcompat issues.

I don't know if specifying the English keyboard layout works correctly or not. Hopefully it would, but I vaguely recall experimenting specifying an explicit keyboard layout and it not working.

That said, I don't know under what conditions you would specify the English keyboard layout. How would PSReadLine know, for any given keyboard layout, that you really wanted a shortcut given that some normal keys use modifiers to type. This is why PSReadLine uses ToUnicode in the first place.

[iSazonov]

Perhaps the problem is more fundamental: in https://github.com/dotnet/runtime/issues/802 keyboard layout is discussed too.

[ForNeVeR]

@lzybkr, well, if there's a bug with sticky keyboard layout, then I can't experience it in usual console application. Only in PSReadLine-enabled PowerShell instance.

I agree that my plan about English keyboard doesn't look very good, so I'd like to take it back. What other options do we have?

Look, Windows applications usually set their hotkeys not based on the national keyboard, but based on the virtual key codes (see the RegisterHotKey function as an example). This mean that my initial use case described in the first post _makes sense_, and it should work (without any need to define additional keys for Russian keyboard layout or any others; it should just work out of the box).

And, as I can see in the current code in Keys.cs, PSReadLine for some reason decided to move away from this solution, and tries to interpret keys that are essentially the hotkeys in their national keyboard layout context (which may be frozen on background thread, but this is another issue).

Was this done by purpose? If not, then I believe we could fix the issue (by _not_ using ToUnicode WinAPI function for cases when there's any known modifiers (Ctrl/Alt) pressed with the key, or by performing an additional search with raw, "ununicoded" hotkey sequence).

What do you think of this plan?

@iSazonov, I'm only experiencing an issue on Windows (and everything seems to be okay on my Mac, where I use pwsh+PSReadLine, too), so I don't think that bug is relevant here.

[lzybkr]

If the console application uses cooked mode, then conhost (a WIndows GUI application, not a console application) implements reading input, so it doesn't have any issues with a sticky keyboard layout.

In v1.2, PSReadLine used ConsoleKeyInfo as the key for dispatch. The Key property is the virtual key code (wrapped in a .Net enum). This mostly worked, but did have usability issuses. For example, ; is Oem1 with the en-us keyboard layout. PSReadLine even lets you bind keys this way:

Set-PSReadLineKeyHandler -Key 'Ctrl+Oem1' -Function Complete

And it converts Oem1 to ; using the initial keyboard layout, so when you run Get-PSReadLineKeyHandler you'll see:

Ctrl+;        Complete            Complete the input if there is a single completion, otherwise complete the input with common prefix for all completions.  Show
                                  possible completions if pressed a second time.

In v2, I moved to using a canonical string representation of the shortcut for better portability because ConsoleKeyInfo values differed between platforms. Unfortunately I don't recall specifics now.

I considered doing something special when Alt is in use, but the .Net api Console.ReadKey reports the use of Alt when using AltGr which does not really indicate a modified key, so I considered this approach a non-starter.

@ForNeVeR - Can you this setting in a new instance of PowerShell (needs to be set before PSReadLine is loaded):

$env:PSREADLINE_VTINPUT = 1

This will use a code path that is more similar to what runs on Linux/Mac. Some key bindings won't work because Windows key bindings are richer than what can be expressed by VT escape sequences, but it might be a reasonable workaround for some if it works.

[ForNeVeR]

Alright, thanks for discussion. Now, when I understand the details a little bit better, I could try to improve the keyboard handling without breaking everything else.

I've tried to set $env:PSREADLINE_VTINPUT = 1 (and made sure I set this environment variable _before_ starting the PowerShell process, and it properly inherits it), but it didn't have any effect on this issue: it still reproduces.

[ForNeVeR]

Interestingly enough, only Ctrl-containing shortcuts are affected. Alt-containing characters (e.g. Alt-d: KillWord) seem to work. And we even enter the erroneous block after we know that the user has pressed a control character! https://github.com/PowerShell/PSReadLine/blob/70249682ed8d87ddc3869e5d0ba852a0d6da99da/PSReadLine/Keys.cs#L231-L253

Another interesting fact is that 44 tests for en_US_Windows fail on my system (after I enable the English keyboard layout).

[ForNeVeR]

And it starts to work if I set control: true instead of control: false.

Alright, it's time to summarize. I believe that _probably_ Windows itself has the code that uses the correct (English?) keyboard layout if it sees that the user has pressed the Ctrl key. For some reason, we destroy this knowledge by overriding control: false.

Why is this code required currently is beyond me, since it has something to do with a "dead key", a concept foreign to me, since I, unfortunately, have no dead keys in any of the keyboard layouts I use (though I understand its purpose and basics of how it works). So, I have no intuition about how the key bindings should work with a dead key.

@lzybkr, it looks like you're the original author of this snippet and control: false. Could you please explain why is it written like this? Are there any examples of input/output for this part of the code? (Even if I'll have to install some additional keyboard layout to try them out, it's okay.)

If the purpose of this code was to only detect a dead key, could it be written in a way to not override the c character?

I think it should never override a control character with a non-control one, would this be sufficient?

var keySansControl = new ConsoleKeyInfo(key.KeyChar, consoleKey, isShift, isAlt, control: false);
char c1 = c;
TryGetCharFromConsoleKey(keySansControl, ref c1, ref isDeadKey);
if (char.IsControl(c1)) c = c1; // only override a control character with another control character

Or we could try to drop this ref c completely. I have verified that this fix works well with both of my keyboard layouts (tests are even more broken after the patch: 47 vs 44, should we care though?).

[lzybkr]

It's been too long since I wrote that code so I won't pretend to remember details or to defend the code in any way.

Maybe the safest change is to avoid that code for the unambiguous control characters.

I think you could safely add c > 26 && to this test: https://github.com/PowerShell/PSReadLine/blob/70249682ed8d87ddc3869e5d0ba852a0d6da99da/PSReadLine/Keys.cs#L237