Wezterm: Ctrl-Space inserts quotes using the German keyboard layout

[mardukbp]

Describe the bug

In PowerShell 7.0.3 with the module PSReadline 2.1 and the following configuration

%HOMEDIR%\Documents\PowerShell\Microsoft.PowerShell_profile.ps1

Import-Module PSReadLine
set-psreadlineoption -predictionsource history

pressing Ctrl+Space inserts " when the German keyboard layout is selected.
With the Spanish layout this key chord is passed to PowerShell, which then
suggests a command based on the history (like fish shell).

Environment (please complete the following information):

• OS: Windows 10
• Version: wezterm 20201101-103216-403d002d

To Reproduce

Assuming wezterm is configured to execute PowerShell.

1. Install the above versions of PowerShell and PSReadline
2. Create the configuration script mentioned above
3. Select the German keyboard layout
4. Launch wezterm
5. Press Ctrl+Space

Configuration

local wezterm = require 'wezterm';

return {
   font = wezterm.font_with_fallback({
    "Consolas",
    "DengXian"
   }),
   font_size = 16.0,
   hide_tab_bar_if_only_one_tab = true,
   default_prog = {"pwsh.exe"},  --PowerShell 7
   keys = {
      {key="F", mods="SHIFT|CTRL", action=wezterm.action{Search={CaseInSensitiveString=""}}},
      {key="l", mods="ALT", action="ShowLauncher"},
    },
}

Expected behavior

The key chord Ctrl+Space should be passed to the shell or any other process running in wezterm.

Screenshots

None.

Session Recording

None.

Additional context

PowerShell 7 is cross-platform. However, the keybinding Ctrl+Space is only defined on Windows.

[wez]

I can confirm that this only appears to happen for me when using wezterm + pwsh. I couldn't reproduce this with cmd.exe or the windows terminal preview.

I don't really know anything about powershell, but this little program:

while (1) { echo $Host.UI.RawUI.ReadKey() }

appears to show keypresses and it looks like it is not interpreting the keypresses the same way under wezterm as under the others.

If you run wsl and run xxd and press CTRL-space, it shows that wezterm is correct emitting ASCII NUL for CTRL-space ^@.

Turning up debugging in wezterm itself I can see that it is definitely recognizing this keypress as CTRL-space and definitely sending ASCII NUL.

My best guess is that this is something funky with codepages; the new windows PTY layer has a lot of issues around this, and it isn't clear how much of those are terminal "host" side vs. "client" side. I was thinking of this one https://github.com/microsoft/terminal/issues/1802 and that is in turned linked to https://github.com/microsoft/terminal/issues/7777

@miniksa / @DHowett-MSFT: do you have guidance on how a terminal emulator should be sending input data to the PTY? Today wezterm writes a UTF-8 byte sequence. Is there something special that should be done by wezterm as part of setting up the PTY to ensure that this is interpreted correctly?

[DHowett]

ConPTY is always UTF-8 in and out. This may be related to changes we made in conhost that aren't yet available in a Windows build? I recall us doing something to disambiguate Ctrl+Shift+2 from Ctrl+Space, but I cannot find it on account of I'm on my phone 😄

[zadjii-msft]

I'm pretty sure what you're seeing here is actually part of what was solved with microsoft/terminal#4999. Ctrl+space is one of those tricky things, because in Win32, NUL != Ctrl+space, but in VT, Ctrl+space == NUL. See also https://github.com/microsoft/terminal/issues/2865

[wez]

FWIW, wezterm bundles and uses a build of OpenConsole.exe and conpty.dll from this rev: https://github.com/microsoft/terminal/commit/90452664ff4541a417ccc1f50d38faca39ec89bb

[wez]

Not on my windows machine right now, so I'm speculating: it might be worth trying to run that bundled openconsole.exe directly and see if CTRL-space behaves the same way there? I'm wondering if this might be something that needs consideration in pwsh.exe's input handling?

[wez]

https://github.com/microsoft/terminal/issues/6454 sounds like exactly this scenario: DEU, Ctrl-space -> " in powershell

[mardukbp]

@wez I just downloaded OpenConsole.exe from the wezterm repo. Ctrl-Space works as expected using the German keyboard layout in pwsh.exe.

[wez]

After digging through the linked issues, I found that although they are marked as resolved the resolution requires that a terminal implement a proprietary extension; this really wasn't made clear in the issues.
https://github.com/microsoft/terminal/blob/main/doc/specs/%234999%20-%20Improved%20keyboard%20handling%20in%20Conpty.md

That spec suggests that conpty unconditionally sends ^[[?9001h to the terminal to request a win32 specific key encoding mode, however, I've not seen it sent into wezterm so something feels incomplete in the implementation.

Additionally, the extension is problematic because the encoded data is not something that eg: a linux based terminal can produce for a remote win32 host because the encoding requires knowing virtual key codes and scan codes from win32 keyboard and language selection. This is referred to as a hypothetical future but this is in fact the situation today for any terminal remoting in to the windows host. This also applies for win32 native wezterm: we use a portable high fidelity representation of the key press but there isn't a way to portably resolve the win32 specific encoding required to represent that key.

FWIW, I think it's a shame that the CSI-u encoding (referred to as libtickit in the linked spec) wasn't selected as that has support in neovim and iTerm2, resolves the ambiguous key encoding problem and doesn't require the remote terminal to have inner knowledge of opaque details like win32 keyboard layouts.

As a workaround:
I found that starting pwsh.exe with ENG selected and then switching to DEU allows ctrl-space to be interpreted correctly(!)

So the state of things for now is that this isn't fixable within wezterm given the current implementation in conpty.

[mardukbp]

I found that starting pwsh.exe with ENG selected and then switching to DEU allows ctrl-space to be interpreted correctly(!)

Yeah, I know. Thanks anyway.

So the state of things for now is that this isn't fixable within wezterm given the current implementation in conpty.

I don't understand how openconsole.exe is used by wezterm. If openconsole.exe does not have this issue, then why does
wezterm do?

[wez]

wezterm uses conpty.dll to create a pty. Due to the way that those things work in windows, that causes openconsole.exe to be spawned in the background to act as a server for processing pty activity. When run in that mode it doesn't spawn a UI; it just processes data going across the pty pipes.

when openconsole.exe has a UI, it performs key input processing and encodes the results as appropriate input records, and powershell can use the win32 console APIs to directly access those. When run without a UI it doesn't (and cannot!) do those things and has to convert to/from the underlying win32 structures.

The issue here is that in the DEU keymap the standard terminal encoding for CTRL-Space is ambiguous, and that the way that the pty layer wants to resolve the ambiguity is by requiring the terminal emulator to use a proprietary, non-portable encoding for keyboard input to disambiguate it.

[mardukbp]

I see. Thank you so much for taking the time to explain!

As a workaround I defined a different shortcut for the completion menu in my $profile file:

Set-PSReadLineKeyHandler -Key Ctrl+r -Function MenuComplete