Terminal: Escape sequences behave strangely over conpty when VirtualTerminalLevel is set

[j4james]

Environment

Windows build number: Version 10.0.18362.175
Windows Terminal version (if applicable): 0.2.1831.0

Steps to reproduce

1. Set the HKCU\Console\VirtualTerminalLevel registry entry to 1, to enable support for VT escape sequences by default.
2. Makes sure the initialCols option in the Windows Terminal profile is set to 120.
3. Open a cmd shell in the Windows Terminal (this can't be a WSL bash shell - it must be something that would usually not have VT support enabled without the registry setting).

4. Run the following Python script:

   import sys
   sys.stdout.write('X'*(120-10))
   sys.stdout.write('\033[1;34;47mHello\033[m\n')

Expected behavior

I'd expect this to display 110 X's followed by the word _Hello_, in blue on white, on the same line. This is what it looks like in a regular console cmd shell.

Actual behavior

What actually happens in the Windows Terminal is the _Hello_ is displayed on the following line.

I don't have a good understanding of how the conpty stuff works, but my theory is that the escape sequences aren't actually being processed initially, and are just written out to the screen as literal characters. As a result the _Hello_ is forced to wrap onto the next line, because there isn't enough space remaining. This screen structure is then passed through conpty somehow, at which point the escape sequences are processed and the text colors changed. But by that stage the wrapping has already taken effect, so the text is in the wrong place.

It's easier to see what's going on if you apply PR #1964 first. In that case, the ESC character gets translated to a ← when it isn't initially interpreted as an escape sequence. The other side of the conpty pipe then doesn't get a second chance to process those values as an escape sequence, so you just see the raw characters being written out.

This may be a good reason to reject PR #1964 for now, since it's clearly making the situation worse, but I think the real issue is in the conpty code, and if that were fixed then the PR might well be safe to merge.

[DHowett-MSFT]

No, this is actually _great_! We've had a number of reports come through that some vt things "act weird", when it's just because they're printing escape sequences with ENABLE_VIRTUAL_TERMINAL_PROCESSING turned off.

Bit more info in this thread:

https://github.com/microsoft/terminal/issues/1616#issuecomment-507443706
https://github.com/microsoft/terminal/issues/1616#issuecomment-507453315

I had no idea that VirtualTerminalLevel was a thing. It almost certainly explains all of this behavior.

I've always thought that if we got a raw ESC for the buffer that we should _store it as an inert rendering character_. This is exactly what that does :grin:

[DHowett-MSFT]

FWIW, just to note, it is _incorrect_ that Windows Terminal (or any conpty consumer) have a "second chance" to process the vt sequence. Since conhost is printing out what it thinks is in the screen buffer, double-processing some escape sequences can result in the cursor misaligning, vt sequences wrapping and being split into multiple lines, etc.

[DHowett-MSFT]

We took a fix in 19H1 to make conpty _not load user settings_, because that could impact how pseudoconsole hosts work in weird and difficult to debug ways. It looks like in doing so, we dropped VirtualTerminalLevel, the dropping of which impacts how pseudoconsole hosts work in weird and difficult to debug ways.

Thank you.

[j4james]

I guess this is technically a duplicate then. I did wonder if it might have been related to some of the other weird behaviour bugs, but nobody else seemed to be mentioning VirtualTerminalLevel so I wasn't sure. I'm glad if it has helped clarify the issue, though.

[DHowett-MSFT]

We may want to just consider enabling VT processing by default for all conpty hosts.

[zadjii-msft]

Okay, so looks like we have options here:

1. Even though we're not loading the rest of the users's setting in conpty mode, still check for VirtualTerminalLevel

   • Pro: Terminal and Conhost will behave consistently when VirtualTerminalLevel is enabled
   • Con: unsure of how hard this might be
   • Con: Without setting VirtualTerminalLevel, ESC is still going to be rendered via VT/conpty, causing weird and difficult to debug differences between the terminal and conhost

2. Automatically enable ENABLE_VIRTUAL_TERMINAL_PROCESSING for conpty

   • pro: ESC will always be processed^caveat by conhost instead of just placed in the buffer to be rendered by conpty.
   • con: (caveat) What happens when an application wants to _disable_ VT mode in conpty? We'll be right back in this scenario again, where the terminal has the state different from the conpty.
   • pro: We think that everyone should be doing VT by default, and this makes everyone do VT by default
   • con: It was not enabled by default for conhost for legacy reasons. Some legacy applications will behave strangely in conpty now.
   • (I feel like there are more pros here @DHowett-MSFT help me out)
     

3. Convert the ESC into some sort of inert character when rendering via VT

   • Pro: Terminal and Conhost look consistent
   • pro: console buffer remains unmodified, as we're only converting on write
   • Does nothing about the existing VirtualTerminalLevel problem - if it's set, then it won't be set in conpty, and the terminal and console will behave differently.

I'm sure I'm missing scenarios here, but I'm throwing this out for feedback. I don't think any of these changes are particularly complicated code changes, but there's not a clear solution to me.
/cc @miniksa

[DHowett-MSFT]

2+3 is the most correct, honestly. There's an entire range of "printable" control pictures, which provides a bunch of sweet iconography like ␛ and ␃.

We could do 2 today and punt 3 for windows 2xHy, honestly.

[DHowett-MSFT]

I mean, technically 1+3 is the _most_ correct, but 2+3 is the most forward-looking...

[j4james]

If we did PR #1964, would that not address option 3 as well as the third con in option 1, at least for the DOS code pages?

As for enabling vt processing by default, I'd be in favour of doing that for conhost itself, which I assume would then make it the default for conpty as well if we went with option 1. If anyone doesn't want vt processing for some reason, they still can choose to disable it via the registry, and have that work consistently for both conhost and conpty.

That said, I don't know what the downsides are. I would have thought there were more legacy applications that needed some form of ANSI support (thus the popularity of alternatives like ANSICON). So I'm curious to know what kind of legacy applications would be broken by having the vt processing enabled.

[zadjii-msft]

It's been admittedly like 4 years since the issue came up originally, but when we first introduced VT support, we got some sort of major internal complaint that it was on by default, and it didn't break a tool, but it made is way slower. (since then, we did some optimization to the parser to make that not so much of a problem.) I honestly don't remember all the details, but the decision to change it to off by default was made, much to our dismay. I don't really want to try and fight that battle again.

However, I think that conpty gives us a fair opportunity to say that this is the new world, and as much as we'd love to provide 100% backwards compatibility with it, this is also the new way of doing things. There are other minor differences that running in a conpty session introduces, so I don't think this is terribly bad.

In conclusion, let's do 2+3. #1964 does something like what I was thinking for 3. I was picturing that we do it on the way out of conpty, in the vt renderer, but this seems reasonable. I'd want to compare how that behaves (esp. in regards to API buffer queries) compared to the v1 console, but that seems reasonable.

I'll use this issue to do "Conpty activates VT mode by default". Apps can still request VT _off_ when they're in conpty mode, and if they write an ESC then, it'll still be rendered to the terminal correctly. There will be behavioral differences between conhost and conpty, but if something doesn't work in the terminal, it'll be unchanged for the legacy console, so use that. We sidestep the VirtualTerminalLevel problem, because it's just on by default in conpty.

@miniksa can @ me if this seems unreasonable or has any other arguments.

[msftbot[bot]]

:tada:This issue was addressed in #2824, which has now been successfully released as Windows Terminal Preview v0.5.2661.0.:tada:

Handy links:

• Release Notes
• Store Download