Terminal: ConPTY inserts unexpected new line character

[maxhora]

• Microsoft Windows [Version 10.0.17763.316] ( also reproducible on 10.0.18841.1000 )

• What you're doing and what's happening:
  ConPTY MiniTerm sample receives extra new line character to display tmux bottom status bar.
  

• What's wrong / what should be happening instead:
  ConPTY should process input VT sequences without inserting new line character.
  The screenshot of the same input, but inside MSYS2 terminal emulator:
  

• Steps to reproduce:

  • Build console CPP app to read input data from file and send to standard output
    main.txt
  • File with input data input.txt to be placed in the same directory with console app executable from previous step
  • Build adjusted version of MiniTerm https://github.com/MaxRis/console/commit/82f36fd39b4fc7e12752d226dd5d70c0b3159824 , run it, launch executable from first step.

• Additional info:

  • Similar issue is reproducible with ConEmu. Extra new line is added after tmux status line:
    
  • Following conclusion made on top of following observations might be important:
    Changing SGR command length has straightforward impact on position where the extra new line character is generated by ConPTY
    
    
    
    ConPTY might has an issue to process SGR command and count symbols related to the SGR command as output terminal emulator buffer symbols. Since the length of line is 130 symbols in above case, ConPTY inserts new line character every time when total count of SGR command symbols and visible output buffer symbols exceeds 130 symbols.
    Here it's simplified input data used to generate above output with SGR command of different length
    input.txt

[DHowett-MSFT]

Hey, about this "ConsolePrinter": is it turning on ENABLE_VIRTUAL_TERMINAL_PROCESSING?

If it _isn't_, something fun is happening with your VT sequences. When VT processing is disabled, those VT sequences are being treated as raw characters to display in the console. Since those characters _are_ being displayed, they're counted towards the width of the output.

When you're connected over ConPTY, two strange (but by-design) things happen:

1. Those raw escape sequences are line-wrapped, because you didn't tell conhost that it should process them
2. The ConPTY consumer receives those sequences _and then parses them._ Since they may have been broken by the line wrapping ro any other console screen buffer manipulation, they _will_ be rendered incorrectly.

In short, if you're not setting ENABLE_VIRTUAL_TERMINAL_OUTPUT but you are in fact using virtual terminal output, this is by design.

[maxhora]

Hey @DHowett-MSFT thank you for looking into this!

ENABLE_VIRTUAL_TERMINAL_PROCESSING is set for MiniTerm.exe https://github.com/Microsoft/console/blob/master/samples/ConPTY/MiniTerm/MiniTerm/Terminal.cs#L36 and I have believed that the console instance and custom set attributes are shared with launched CMD.exe and further with ConsolePrinter.exe process. Does it make any sense here?

ConsolePrinter doesn't modify console mode and just writes to standard output:

#include <iostream>
#include <fstream>
#include <string>

void main() {
    std::ifstream dataFile("input.txt");
    if (dataFile.is_open()) {
        std::string dataLine;
        while (std::getline(dataFile, dataLine)) {
            std::cout << dataLine;
        }
    }
}

[DHowett-MSFT]

Ah, so the code in MiniTerm is enabling VT processing _for the terminal itself_. Since ConsolePrinter is on the other side:

+-------+                      +-------+
| Mini  | <----------(pipe)--- | pty   |
|  Term | -----------(pipe)--> |  host | - vt processing OFF.
+-------+                      +-------+
  |   ^                           |  ^  
  v   |                           v  |
+-------+                      +---------+
| con   |                      | Console |
|  host | - VT processing on.  | Printer |
+-------+                      +---------+
  |   ^ 
  v   | 
/ - - - +
: user  :
:       :
+ - - - /

it needs to enable virtual terminal processing _for its console host_.

[maxhora]

@DHowett-MSFT thanks! will give it a try!

[maxhora]

@DHowett-MSFT thank you for finding out the source of issue!

If I'm extending ConsolePrinter with something like following:

        HANDLE hOutput = GetStdHandle(STD_OUTPUT_HANDLE);
    DWORD dwMode;
    GetConsoleMode(hOutput, &dwMode);
    dwMode |= ENABLE_PROCESSED_OUTPUT | ENABLE_VIRTUAL_TERMINAL_PROCESSING;
    if (!SetConsoleMode(hOutput, dwMode)) {
        exit(1);
    }

Everything is rendered correctly, even under ConEmu! Thanks!

[oising]

Hey, about this "ConsolePrinter": is it turning on ENABLE_VIRTUAL_TERMINAL_PROCESSING?

If it _isn't_, something fun is happening with your VT sequences. When VT processing is disabled, those VT sequences are being treated as raw characters to display in the console. Since those characters _are_ being displayed, they're counted towards the width of the output.

When you're connected over ConPTY, two strange (but by-design) things happen:

1. Those raw escape sequences are line-wrapped, because you didn't tell conhost that it should process them
2. The ConPTY consumer receives those sequences _and then parses them._ Since they may have been broken by the line wrapping ro any other console screen buffer manipulation, they _will_ be rendered incorrectly.

In short, if you're not setting ENABLE_VIRTUAL_TERMINAL_OUTPUT but you are in fact using virtual terminal output, this is by design.

This is the kind of information that belongs on Docs, along with all of the PseudoConsole stuff.

[miniksa]

Please file a docs bug at https://github.com/MicrosoftDocs/Console-Docs. Feel free to make a branch and add suggested text to a pull request if you have a good idea of what you think it should say.

[Biswa96]

Inspired from @DHowett-MSFT's ascii art. Does ConPTY work with Duplex Named Pipe? As following:

+-------+                      +-------+
| Mini  |                      | pty   |
|  Term | <-------(pipe)-----> |  host | - vt processing OFF.
+-------+                      +-------+
    ^                              ^  
    |                              |
    v                              v  
+-------+                      +---------+
| con   |                      | Console |
|  host | - VT processing on.  | Printer |
+-------+                      +---------+
    ^ 
    | 
    v
/ - - - +
: user  :
:       :
+ - - - /

[DHowett-MSFT]

Give it a shot!