Terminal: ConPTY does not support black background

[Tyriar]

bash repro: echo -e '\x1b[40mabc'
powershell repro: Write-Host -BackgroundColor Black "Hello"

Expected (conhost/powershell.exe):

Actual (vscode/conpty):

[Tyriar]

/cc @sbatten

[DHowett-MSFT]

Copying @zadjii-msft's response from #362:

If I had to guess, I'd think that maybe you're emitting something like 37m or 40m, and that's silently getting translated to 39m/49m (or maybe even 0m). In that case, your client app would say "I would like to write text with a foreground of dark white" (37m), but the conpty's "default foreground" color is also set to "dark white", so conpty then emits "I want default foreground text" (39m).

Since RS4 and RS5 conhost do not support "default" colors that lie outside of the stock 0-16 palette (as explained here), ConPTY starts up with "white on black" (37 on 40) set as the default.

This does mean that attempts to set the background to black or the foreground to white will result in ConPTY emitting "set background to default" (49, or 0 if no foreground was set) and "set foreground to default" (39, or 0 if no background was set.)

In 19H1, conhost _does_ support default background/foreground being outside of the palette, but we did _not_ enable it by default for ConPTY consumers as it is an experimental feature with a potential impact on application compatibility.

[DHowett-MSFT]

I'm tagging this Issue-Question because we need to have a discussion around the best way to solve it. We're taking a bigger dependency on ConPTY ourselves in Windows Terminal, and we absolutely want to support the full range of colors.

The problem is, as always, backwards compatibility. Let's put together a design?

[TBBle]

Would it make sense for the theme-loader or applier to use a different foreground/background index when the given colour is in the pallette? That would restore compatibility for situations which were previously handled by updating the shortcut properties or registry key to choose a different colour, e.g., https://github.com/neilpa/cmd-colors-solarized/blob/master/Update-Link.ps1

[antoineco]

For people landing here after wondering why highlighted backgrounds are invisible with Solarized palettes, here a summary of what does not work currently.

reference Solarized colors

Windows Terminal (preview) rendering

(the script)

• the column labeled 40m has an invisible background
  Probably the most striking difference when running typical apps in the terminal because all highlights are gone (vim, tmux, status bars, ...). In the Solarized palette this color is base02, which corresponds to "black" (color 0). As already explained by @DHowett-MSFT, "black background" is always translated to the default background color, which for us is base03 a.k.a. "bright black" (color 8).

• the row labeled 1m has white text
  This text should have the same base0 color for "foreground" (same as "bright blue" in the Solarized palette) as the row above, but making it bold automatically translates it to base3 for "bright white" (color 15), making it identical to the 1;37m row.

• the row labeled 37m has gray text
  This text should have the base2 color for "white" (color 7), but it was translated to the base0 color for "foreground" instead, making it identical to the m row.

[hanubeki]

I figured out ESC ] 11 ; rgb:12/34/56 ESC \ can be used to separate background color and ESC ] 10 ; rgb:12/34/56 ESC \ for foreground color, but I'm not sure this is a related to this issue.

[hanubeki]

After using ESC ] 11 ; rgb:00/2B/36 ESC \ and ESC ] 10 ; rgb:83/94/96 ESC \, My terminal just looks like this.

[DHowett-MSFT]

That just changes your foreground and background colors, it doesn’t fix the underlying issue.

[TBBle]

It's close to solving the problem at least for Solarized, since a request for 37m is being translated into 39m, and 40m is being translated into 49m, and those OSC codes (10 and 11) are setting the VT100 text foreground color" and "VT100 text background color", (presumably applied by 30m and 40m) to be the colours expected by 37m and 49m? It's interesting, that these OSC codes are not listed on MS's Console Virtual Terminal Sequences listing.

However, I assume this now means most applications will get the wrong default colours, i.e. showing 40m when the screen is cleared or otherwise, instead of the desired (Solarized Dark) background, 100m.

I'm not sure if Terminal is holding the correct pipe to mangle the stream from application to ConPTY, but if so it could work around ConPTY's behaviour here by turning any call to ESC[37m (or compound code containing 37) into ESC]10;rgb:<themeWhite>ESC\ESC[37mESC]10;rgb:<themeFG>ESC\ (and same for ESC[40m) which seems like a pretty awful hack, but ought to work...

Since I understand the issue lies in ConPTY, any fix therein, e.g., removing this remapping behaviour, or allowing the host to specify (and update?) which colour indexes to remap into 'foreground' and 'background'.

[erossetto]

Y'all got any more of them... updates?

[zadjii-msft]

@erossetto Not at this time, no. This bug is a pretty high priority for us, but actually requires a decent amount of work and unfortunately didn't make the cut for 1.0. Hopefully this should land soon after 1.0. When we do have updates, we'll make sure to update this thread.

[blami]

Wouldn't be possible to make this togglable/negotiable between client and conpty? I assume most of us are interested in this in WSL space and not in legacy cmd.exe.

[DHowett]

@j4james you mentioned that your set of color-strengthening PRs fixed most cases of this, and the ones that they didn't fix that they may not be able to. What are the remaining cases, if you know?
I'm looking to close this out -- if the remaining known issues are palatable I'll just send this one to join the choir invisible.

[j4james]

The remaining issues (that I'm aware of) are the ones where PowerShell is using the console APIs to set its colors, so it can't differentiate between black and default. I don't think there's anything we can do about that.

I think there might also have been some linked Solarized issues that were just Solarized doing its thing, but anything that was a legitimate bug should be fixed now.

Bottom line is I think it's OK to close this.

[TBBle]

Is there a schedule for the next Preview release? I'd love to see the improvements for myself, and see what's left in the "Solarized doing its thing" bucket.

[TBBle]

Apart from PowerShell (will be fixed by #6807 pending an upstream fix) it all seems to be working quite nicely now. That said, I have a correctly-setup Solarized environment, including overriding the as-of-2-days-ago-default Solarized Dark theme with one that matches the spec (i.e. I've overridden the change in #6985).

Per Terminal Preview 1.2.2022.0:

and to contrast with the state 12 months ago:

Screenshots taken with either MSYS2 or WSL2 Ubuntu. It might have been one of each... they both give the same results anyway.

So yeah, looks good, let's mark this one done-and-dusted. Huge thank you to @j4james for driving the colour system work (AFAIK), and @DHowett for (amongst other things) telling me in another issue where I was going wrong with my initial testing.

[DHowett]

And on that note: thanks, everyone, for playing! This has been fixed and should be out in 1. the Terminal previews and 2. the next full-scale Windows update, so VSCode can take advantage of it.