Psreadline: Set-PSReadlineKeyHandler fails with certain key chords on Unix

[joeyaiello]

_From @mklement0 on April 17, 2017 19:44_

Steps to reproduce

Define Control and Alt-key-based chords for doubling characters ', ", and ( when typed with Control or Alt held down:

Set-PSReadlineKeyHandler -Chord "Ctrl+'" { [Microsoft.PowerShell.PSConsoleReadLine]::Insert("''") }
Set-PSReadlineKeyHandler -Chord 'Ctrl+"' { [Microsoft.PowerShell.PSConsoleReadLine]::Insert('""') }
Set-PSReadlineKeyHandler -Chord 'Ctrl+(' { [Microsoft.PowerShell.PSConsoleReadLine]::Insert('()') }
Set-PSReadlineKeyHandler -Chord "Alt+'" { [Microsoft.PowerShell.PSConsoleReadLine]::Insert("''") }
Set-PSReadlineKeyHandler -Chord 'Alt+"' { [Microsoft.PowerShell.PSConsoleReadLine]::Insert('""') }
Set-PSReadlineKeyHandler -Chord 'Alt+(' { [Microsoft.PowerShell.PSConsoleReadLine]::Insert('()') }

Expected behavior

The definitions should be accepted on all platforms and function as defined.
E.g. Ctrl+' should result in ''.

Actual behavior

On Windows, the actual behavior matches the expected behavior.

On Unix platforms (verified on macOS 10.12.4 and Ubuntu 16.04 with the respective default terminal applications), the definitions fail with (repetitions of) the following error messages (note that omitting modifier keys ctrl and alt from the definitions would work fine):

Set-PSReadlineKeyHandler : Unrecognized key 'ctrl'. Please use a character literal or a well-known key name from the System.ConsoleKey enumeration.
Set-PSReadlineKeyHandler : Unrecognized key 'alt'. Please use a character literal or a well-known key name from the System.ConsoleKey enumeration.

The error messages suggest that ctrl / alt are not recognized as _modifier keys_ in this case.

Note that something like Ctrl+A _does_ work, however; the problem appears to be the specific characters (', ", () that the modifiers are paired with.

Environment data

PowerShell Core v6.0.0-alpha (v6.0.0-alpha.18) on Microsoft Windows 10 Pro (64-bit; v10.0.14393)
PowerShell Core v6.0.0-alpha (v6.0.0-alpha.18) on Darwin Kernel Version 16.5.0: Fri Mar  3 16:52:33 PST 2017; root:xnu-3789.51.2~3/RELEASE_X86_64
PowerShell Core v6.0.0-alpha (v6.0.0-alpha.18) on Darwin Kernel Version 16.5.0: Fri Mar  3 16:52:33 PST 2017; root:xnu-3789.51.2~3/RELEASE_X86_64

_Copied from original issue: PowerShell/PowerShell#3584_

[lzybkr]

I believe this is fixed in #561, but some bindings with Alt will not work correctly because the CLR does not set they KeyChar property properly - tracked by https://github.com/dotnet/corefx/issues/12336.

I'll publish a pre-release version to the PowerShell Gallery as soon as they support pre-releases.

If you'd like to try a fix before that, you can build the lastest yourself from the master branch, or
grab a build from AppVeyor like this one.

Note the version number is now 2.0, so be sure to copy the contents to \PSReadLine\2.0.

[lzybkr]

I'll have an official prerelease build out next week if you want to verify this is fixed, otherwise you can grab a build out of appveyor here.

[mklement0]

Thank, @lzybkr, but it still doesn't seem to work with the 2.0.0-beta1 version from the gallery (published on December 06 2017).

[lzybkr]

@mklement0 - this might have been fixed later - try the latest build from AppVeyor (linked from the build status badge from the repo readme.md.)

[mklement0]

Thanks, @lzybkr - just tried with the latest AppVeyor build on both macOS 10.13.4 and Ubuntu 16.04, but it still doesn't work, I'm afraid.

The current symptoms are:

• The definitions are accepted and do show up in Get-PSReadLineKeyHandler's output.

• On pressing something like Alt+', nothing happens.

[lzybkr]

It looks like Alt+' is affected by the CLR issue I mentioned above, and I hint at where a fix is probably necessary in this comment.

[mklement0]

I see. Unfortunately, it doesn't work with Ctrl-based chords such as Ctrl+' either - or is this the same issue?

[lzybkr]

Not quite - on Linux, Ctrl-' is indistinguishable from '. The command Linux showkeys -a is useful to help me figure out what is possible, what might be a CLR bug, or my bug, in this case, it shows those keys generate the same char sequence which is what the CLR sees after the key press. The CLR then translates the char sequence into a ConsoleKeyInfo.

I thought there was an issue for rejecting key bindings that can't work, but I can't find it at the moment, and to be honest, I'm not exactly sure how I'd implement that on non-Windows platforms.

[mklement0]

Thanks, @lzybkr - showkey -a was illuminating.
sudo showkey -k - note the need for sudo - does show that the actual individual keypresses are all available at a lower level, however.

I'm not sure what's actionable about my Unix-related findings below, but hopefully it provides a few pointers:

Alt-based key chords (mapped from Esc-based key _sequences_):

showkey -a tells us that something like Alt-" is translated into a key _sequence_:

• ESC (code point 0x1b)
• " (code point 0x22)

And that is indeed how readline functions are often defined; e.g., to represent the above _sequence_ in a readline definition in bash and map it onto inserting "<cursor-pos>", you'd run (I'm using the builtin bind here so that the command can be experimented with interactively, but such a definition is usually placed in ~/.inputrc on Unix platforms):

bind '"\e\"": "\"\"\e[D"'

To invoke this binding, you'd press Esc _first_ and _then_ ".

However, many _terminal programs_ translate Alt-based key _chords_ into these Esc-based key _sequences_, so that pressing Alt-" - simultaneously - has the same effect.

As for where this translation happens:

• Windows:
  
  
  
  • mapped by default in Bash on Ubuntu for Windows (WSL)
  
  
• Linux:
  
  
  
  • Mapped by default in the default terminals of many major distros:
    
    
    
    
    
    • Ubuntu
    
    
    
    • Debian
    
    
    
    • Fedora (presumably therefore also RHEL)
    
    
    
    • CentOS
    
    
    
  
  
  • TBD:
    
    
    
    
    
    • OpenSUSE
    
    
    
    • RHEL (presumably mapped, assuming it uses the same terminal as Fedora)
    
    
    
    • Arch Linux
    
    
    
    • Kali Linux
    
    
    
    • Others?
    
    
    
  
  
  • Alternative terminal applications:
    
    
    
    
    
    • Note: There may be terminals where the mapping may not even be available on an opt-in basis, through configuration.
    
    
    
    • NOT mapped:
      
      
      
      
      
      
      
      • XTerm and UXTerm - no obvious way to configure the behavior
      
      
      
      
    
    
    
  
  
• macOS:
  
  
  
  • Requires configuration, unfortunately:
    
    
    
    
    
    • Terminal.app (the default terminal): only if option Use Option as Meta key is turned on via Preferences > Profiles > {active-profile} > Keyboard
    
    
    
    • iTerm.app (iTerm 2): only if Esc+ is chosen for ⌥-key mapping in Preferences > Profiles > {active-profile} > Keys
    
    
    
  
  

So the question is: can PSReadLine conversely recognize Esc-based key _sequences_ as Alt-based key _chords_?

---

Control-based key chords:

Control-based chords - expressed as \C-<char> - are tricky in the Unix world:

• _Fundamentally_, the repertoire of available Control-based chords is limited to caret notation, which means the combinations are limited to the set of ASCII C0 control characters (plus (0x7F))

  • In a nutshell (but see caveat below):
  • Control-A through Control-Z (whether Shift is involved doesn't matter).
  • plus a _few_ punctuation-based chords:
    
    
    
    • \C-@ (0x0) _also_ maps to: ⌃`
    
    
    • \C-[ (0x1b); i.e, ESC - DO NOT REDEFINE
    
    
    • \C-\ (0x1c) _also_ maps to: ⌃4
    
    
    • \C-] (0x1d) _also_ maps to: ⌃5
    
    
    • \C-^ (0x1e) _also_ maps to: ⌃~
    
    
    • \C-_ (0x1f) _also_ maps to: ⌃7 and ⌃/
    
    
    • \C-? (0x7f)(⇧-based!!) also maps to: ⌃8
    
    
  • Notably, something like Control-' is _not_ included.

• Even this restricted repertoire above _cannot be fully utilized_, lest vital terminal functions be interfered with:

  • Since the available Control-based chords generate _control characters_, they implement functions that are vital for an interactive terminal experience:
    
    
    
    • At the very least, the following chords should NOT be redefined:
      
      
      
      
      
      • Control-M ... this maps to CR and therefore ENTER - required to submit commands on the command line.
      
      
      
      • Control-I ... this maps to Tab - required for tab completion
      
      
      
      • Control-[ ... this maps to Esc
      
      
      
      • Control-H ... this maps to Backspace
      
      
      
    
    
    • In general, any Control-based chord you redefine takes away the ability to use the _keyboard_ to generate the ASCII control character the chord was originally mapped to.
    
    

That said, the above is currently hypothetical, because none of the fundamentally available Control-based chords can currently be redefined with PSReadLine on _Unix_-like platforms:

Example: Redefinition of Control-J to insert the string jump:

# As of PSReadLine v2.0.0-beta.1: 
#   * OK on Windows
#   * BROKEN on Unix-like platforms
Set-PSReadLineKeyHandler -Chord 'Ctrl+j' -ScriptBlock { [Microsoft.PowerShell.PSConsoleReadLine]::Insert('jump') }

By contrast, the equivalent bash definition works fine:

bind '"\C-j": "jump"'

[mklement0]

@lzybkr:

I thought there was an issue for rejecting key bindings that can't work, but I can't find it at the moment, and to be honest, I'm not exactly sure how I'd implement that on non-Windows platforms.

In light of the above (assuming that no attempt is made to intercept lower-level keyboard events , which may require even more support in CoreFX, if I understand correctly), at least _fundamentally_ unavailable chords could be ruled out, namely:

• Control-based chords that do not map onto C0 control characters / 0x7f via caret notation (as described above).

Note that among the Alt-based chords - assuming we get them to work - there may still be _situationally_ unavailable chords, depending on the specific platform (e.g., there are Alt-based chords with system-wide meaning on Ubuntu), but trying to anticipate all of those is impractical.

[lzybkr]

@mklement0 - A console application can't really depend on the raw keyboard input - a pty usually sits between the application and the hardware. That's why I mentioned showkey -a and not showkey -k.

The character sequences generated by a key press depend on the terminal type. A proper console application use the terminfo database (see man terminfo or man infocmp) to map the sequences to what a user thinks of as a key press. For example, F1 might generate very different sequences for some old obscure terminal or a more typical VT100 terminal.

In the .Net world, we rely on the Console api to do this for us, though it is not yet mature (hence the bug I mentioned previously, also your Ctrl+j example which produces a ConsoleKeyInfo that is identical to Enter.)

When investigating these issues, I start by comparing the info returned by showkey -a with what I get from Console.ReadKey() (which returns a ConsoleKeyInfo).

PSReadLine makes some attempt to normalize the ConsoleKeyInfo to deal with platform specific differences and differing keyboard layouts, but there are still some issues.

[mklement0]

@lzybkr:

Good to know, thanks.

The character sequences generated by a key press depend on the terminal type.

And many terminals translate _chords_ such as Alt-' into _sequences_ such as Esc, ', so perhaps ConsoleKeyInfo only ever sees the latter?

If so, one conceivable approach is for PSReadLine to listen for the latter instead (while still allowing them to be _declared_ as Alt chords) - is that doable?

---

Small correction: Ctrl+J maps onto \n (0xA, LF), not ENTER (Ctrl+M,0xD, CR), that's why it's OK to use.

[lzybkr]

Right - an application reading from stdin only sees the sequences. Keyboards haven't always had an Alt key, so the Escape key was an obvious way to add useful key bindings, and when keyboards evolved, it made sense to generate the same sequence.

PSReadLine could be implemented as you suggest, but it would take on more complexity than I'm willing to take on and there are issues, some unsolvable, if PSReadLine were to rely on sequences everywhere:

• Sequences are somewhat limiting, e.g. some popular key bindings on Windows can't work on Linux, see #638 and #639.
• Sequences would not work on Win7, Win8, or earlier Win10 systems because the console does not provide a way to generate sequences.
• Without complex terminfo support, sequences would only support the most common terminals.

PSReadLine actually has some support for sequences so that it can run under tmux on WSL (using the Windows version of PowerShell). The code is here, but it is doesn't (can't) use terminfo because Windows doesn't have terminfo support.

I'm not perfectly happy with how things are today, but I think I've made reasonable choices considering the alternatives.

[mklement0]

@lzybkr:

Thanks for the background info, as usual.

On gaining a better understanding of the issue you've opened in the CoreFX repo, I do agree that CoreFX is the right place to address these issue, so I've posted a follow-up comment there.