Powershell: Discussion: PowerShell needs to support 24bit color. Where? How?

[Jaykul]

Since the Windows console now has 24bit color support, it seems obvious to me that PowerShell needs to build support for that in the same release timeframe.

As a reminder, 24-bit color means 256 values each for red, green, blue -- without an alpha channel.

For the sake of argument, let's assume that at a minimum, Write-Host needs to support Foreground and Background colors as 24bit values ...

What should the syntax be for 24bit color?

The most obvious suggestion is the HTML #rrggbb syntax -- but that would be a comment. We could pass it as a quoted string, but we could also choose to accept simple unadorned rrggbb ...

A richer choice would be to add a [Color] type, which would allow for syntax like: [color]"336699" or even RGB values like [color](51, 102, 153) or even [color]@{h=210; s=50; l=40} (all of which represent the same color)...

Any other thoughts?

Where should colors work?

The most obvious place where this is _definitely_ needed is in the Write-Host command.

However, I would also like support in the colors that are specified for $Host.PrivateData ... where I think it would be particularly helpful to be able to pick colors that are not one of the 16 colors, and of course, it would be wonderful if the core 16 colors could be available there as well, so that we could theme our console by just setting those values ;-)

Additionally, we need to be able to support colors in format files, perhaps we need foreground/background settings on the table/row/column/cell elements, or perhaps we just need a function like Get-AnsiCode, or a property on a type like the aforementioned [Color] so we could put the VT escape sequences into a string, like: Write-Host "$(([color](51, 102, 153)).Foreground)This is blue$([Color]::reset) and this isn't" ...

Finally, many modules (like PSReadLine and PowerLine) need to support colors, and I would really like them to all support the same syntax for setting colors (and/or the same Color type).

Have I missed anything?

Is there a better syntax? Other commands or classes that need colors? Do you wish I would just go off and do this in a module instead of asking for it in the core shell? Please speak up here!

[bobfrankly]

I feel like a [color] type supporting RGB and HSL would be the most functional option, and agree that Write-Host is the obvious target for this. I would already be using it if it was available.

[joeyaiello]

This all looks pretty sound to me. Support casting multiple string and collection formats into a [Color] type.

If you're ready to implement this, @Jaykul, feel free to submit it as an RFC.

[rkeithhill]

If folks want to use 24-bit color with Write-Host then it seems that PowerShell and/or .NET Core would have to adapt. Right now System.ConsoleColor is an enum defined as:

    [Serializable]
    public enum ConsoleColor
    {
        Black = 0,
        DarkBlue = 1,
        DarkGreen = 2,
        DarkCyan = 3,
        DarkRed = 4,
        DarkMagenta = 5,
        DarkYellow = 6,
        Gray = 7,
        DarkGray = 8,
        Blue = 9,
        Green = 10,
        Cyan = 11,
        Red = 12,
        Magenta = 13,
        Yellow = 14,
        White = 15
    }

Where the int values identify a index in the current 16-color palette for the console.

In the blog post announcing 24-bit color support for the Windows Console, they mentioned they have not updated the Console property page to support 24-bit color yet. It would be good to know what they have planned especially when it comes to surfacing this functionality through Win32 API and/or .NET Core.

That said, I could imagine new parameter sets on Write-Host with new parameters like -Background and -Foreground (or maybe -Back/ForegroundTrueColor) that are of type System.Drawing.Color. PowerShell already knows how to convert int values to that type:

97> [System.Drawing.Color]0xff0000


R             : 255
G             : 0
B             : 0
A             : 0
IsKnownColor  : False
IsEmpty       : False
IsNamedColor  : False
IsSystemColor : False
Name          : ff0000

And this type supports known color names:

98> [System.Drawing.Color]::AliceBlue


R             : 240
G             : 248
B             : 255
A             : 255
IsKnownColor  : True
IsEmpty       : False
IsNamedColor  : True
IsSystemColor : False
Name          : AliceBlue

It has methods to return Hue, Saturation and Brightness. Unfortunately, I don't see a ctor or static method to construct an object from those values.

Now for coloring strings _not_ using Write-Host, then a class to help with the ANSI esc sequence would be very nice to have. Not sure what exactly that would look like but we should take a peek to see what the folks in the node community (chalk, ansi-256-colors, ansi-escapes) have done. I could see the community building modules to provide this functionality. No need for integration into PowerShell Core AFAICT.

[rkeithhill]

Doh! Looks like System.Drawing.Color is available only in .NET Core App >= 1.1. I assume PowerShell Core is currently .NET Core App 1.0?

[rkeithhill]

Also, FWIW I would want to specify unnamed colors like so 0xe90c1b but I think it would be easy to make most approaches accept this as it _is_ the actual color value.

[Jaykul]

I suppose that using the existing type (that most of us are familiar with) is worth putting up with the useless alpha value, especially since it can already cast 0xrrggbb and names.

I assume that PowerShell will update it's .Net Core eventually, but we _could_ always just copy the code from the CoreFx repo. We could also add the type adapters or constructors or From___ methods etc. -- there are MIT licensed conversion routines on Colourful (and also ColorMine).

We also need an XtermConsoleColor table. We could use that _in place of_ the ConsoleColor since the first 16 of the Xterm color table are the same as the basic 16 ...

However, the crucial part is a we also need to be able to map RGB values to the XtermConsoleColors, and even to the basic 16, because, you know ... not every terminal supports full color. I have to think about how to do that right. Colourful has a few difference implementations, but this would need to be fast...

And yes, @rkeithhill, there is, in fact, no API or anything that allows using the new colors in the Console currently, _except_ VT ANSI escape sequences ... and that's really the only way I can see to implement it in format files anyway -- unless we extend them to support a color attribute all over the place.

I'm ok with that. In fact I have already used them in file table formats the hard way, and that's how I implemented colors for PowerLine obviously.

[joeyaiello]

@rkeithhill thanks for the heads up on Core's type. That seems like the way to go.

we _could_ always just copy the code from the CoreFx repo.

I'd much rather we upgrade PowerShell Core to use .NET Core 1.1. Unless we're absolutely blocked and something is critical on a short time-span, we should avoid duplicating code as a general rule of thumb.

[lzybkr]

Regarding the Win32 API - we shouldn't care, it's not portable. That said, I believe the plan is no 24bit color support in the api, escapes sequences are the new api.

There is mapping support in conhost to support the screen scraping apis which only have 4 bit color support. Maybe it's possible to use that code somehow, but maybe the right api is to report errors if the terminal doesn't support the colors you ask for.

[Jaykul]

Well @lzybkr, I think the right thing to do is probably to (copy what chalk does in this situation and) convert down. They use a javascript color-convert library for that, which has a nice simple down-sampling algorithm.

Or just borrow from ObscureWare/Console.Core

[be5invis]

@lzybkr

Just a mention, Example code used to produce colors:

bool EnableVTMode()
{
    // Set output mode to handle virtual terminal sequences
    HANDLE hOut = GetStdHandle(STD_OUTPUT_HANDLE);
    if (hOut == INVALID_HANDLE_VALUE)
    {
        return false;
    }

    DWORD dwMode = 0;
    if (!GetConsoleMode(hOut, &dwMode))
    {
        return false;
    }

    dwMode |= ENABLE_VIRTUAL_TERMINAL_PROCESSING;
    if (!SetConsoleMode(hOut, dwMode))
    {
        return false;
    }
    return true;
}
int __cdecl wmain(int argc, WCHAR* argv[])
{   
    argc; // unused
    argv; // unused
    //First, enable VT mode
    bool fSuccess = EnableVTMode();
    if (!fSuccess)
    {
        printf("Unable to enter VT processing mode. Quitting.\n");
        return -1;
    }
    int red = 0; // set these to whatever
    int green = 0;
    int blue = 0;
    printf("\x1b[38;2;%d;%d;%dm", red, green, blue); // produces RGB foreground
    printf("\x1b[48;2;%d;%d;%dm", red, green, blue); // produces RGB background

    int index = 0;
    printf("\x1b[38;5;%dm", index); // produces xterm color table index foreground
    printf("\x1b[48;5;%dm", index); // produces xterm color table index background
}

cc. @zadjii-msft

[zadjii-msft]

Just to add my comments, I'm backing up @lzybkr on what he said. We're not going to extend the Console API to add support for this. That just creates another API that .NET and others would have to try and translate manually when they port to other platforms. Emitting VT sequences is the new standard for console features (and it has been for decades, Windows is finally on the train).

It's also notoriously tricky to determine what kind of actual color support a particular terminal provides. Most terminals set themselves as TERM=xterm, but then also support all the way to 24-bit color, while some applications like tmux only support 256color. I haven't found a good way to pick which of the sequences to go with, or determine what's possible, so I usually stick to just emitting xterm table sequences.

That's my 2 cents.

[Jaykul]

Hey, @lzybkr and @zadjii-msft -- I'm working on research for this, and was looking at related console APIs and the IShellLink API.

Do either of you know if it's possible to determine (from within a PowerShell instance) whether a process was launched from a link or not, and if so, which link? That is, specifically, the "link" one was launched from, so as to change the properties (for future instances) the way the property dialog does?

For instance, this MSDN page documents new shell features which have registry settings... _unless_ the shell was launched from a shortcut. How could I tell this was the case?

[zadjii-msft]

@Jaykul I don't actually know if that's possible from within Powershell, though PS isn't really my area of expertise. @adiviness may know better if it's possible.

From a cursory glance at the API, there's nothing that you can query if the properties at launch came from a link or the registry. We only know at launch if we were launched from a link or not, but that value isn't exposed externally at all. Unless their's something else that powershell might expose, I'd guess it's not possible.

[Jaykul]

Thanks @zadjii-msft that's what I figured. It just means that knowing when to customize the PowerShell shortcut in the start menu is a pain 😖

[rkeithhill]

The way console settings are persisted (registry, lnk) is a mess. I wish the conhost.exe folks could come up with something more manageable.

[zadjii-msft]

@rkeithhill We're very well aware of the mess. The problem is that there's not a clear solution - either we end up breaking a lot of people's workflows (bad) or we introduce another system for managing it (relevant xkcd).

We'd love to fix it. Absolutely. But it's a big problem and will likely not get prioritized for a while :(

[Jaykul]

Could PowerShell 6 fix this problem by putting settings in the registry instead of in the shortcut, so there's only one source of truth (unless users modify the shortcut deliberately)? Or would using the property pages _still_ change the shortcut instead?

[zadjii-msft]

@Jaykul Nope. The settings are handled by conhost itself, and they're persisted based on how you launched it.

For example, if you launch powershell by using Win+X, I (or Win+X, A), then no matter what, the settings are going to be saved to the shortcut. This is the same shortcut that (by default) is used for launching powershell from the start menu. Because these types of launches use the shortcut to launch, conhost will only ever put any changes back in that shortcut. It's not something that's configurable per-app.

And no matter what, using Win+R "powershell" will persist to the registry, never knowing about the existence of that link.

[rkeithhill]

Well that `splains why my registry-based conhost theming only seems to work via Win+R.

[Jaykul]

Can I submit my Pansies module (gallery) as a RFC? ;-)

It's purely about the user interface and how you can specify colors (and use them in format files), and I'm still thinking hard about how to make this work in down-level Windows and Linux when you only have 16 (or even 8?) colors.

Anyway, my first few thoughts are in code at this point:

• An RgbColor class for parameter types
• A Text class that supports coloring so you can do it in format files
• A rough version of Write-Host that uses the RgbColor and respects a static property that controls downsampling to xterm256 or the 16 console colors...

Note that currently I'm using ColorMine, but that's purely for the fun of supporting color spaces -- in PowerShell we'd presumably only have RgbColor (and I would rebuild Pansies to use and extend that so you can do color space shifting and palette generating, etc).

[powercode]

How do we perform feature detection to determine the underlying platforms capabilities?

[lzybkr]

@Jaykul - you can propose any RFC as you see fit, but I think in this area, the preference is not a new module, but new apis and parameters to existing cmdlets.

@powercode - feature detection often isn't easy, sometimes impossible. Here's a good discussion for *nix platforms: https://gist.github.com/XVilka/8346728

[Jaykul]

@lzybkr yeah -- I absolutely intend to put the RgbColor class into the core and update the relevant cmdlets. I just wanted a way to try out some ideas -- and in any case, I want a module for PS5 where I expect I'll still be spending most of my day, for a while :wink:

@powercode Currently I'm not even trying to do feature detection. I just put a static property in that you can set. My intention is to change the _default_ based on testing the OS, but allow you to change either the static property or an environment variable.

Windows would default to 16 except on Windows 10: build 1607 (AU) 256color, build 1703 (CU) 24bit.
Linux and OSX would probably default to 256 colors. Not totally sure whether it's worth checking for an environment variable like $Env:TERM or just using a preference variable like $PSColorDepth.

[iSazonov]

Related https://github.com/PowerShell/PowerShell/issues/3611

[zadjii-msft]

@Jaykul actually, just to be clear, I don't think 1607 supported 256 color. We had a dumb translator from 256/rgb to the 16 color table, but real support for both 256color and 24 bit came with the Creator's Update, 1703.

[9Rune5]

I recently had to reacquaint myself with bash, and to my delight discovered a repository of colours for various file extensions: https://github.com/trapd00r/LS_COLORS

I'd love to see support for that in PS, even if it is limited to "only" 256 colours. OTOH, I'm sure a converter would be easy to implement, no matter what scheme you guys land on.

[rkeithhill]

@9Rune5 This exists. Install the DirColors module for the PSGallery. It even supports a LS_COLORS style configuration.

[Jaykul]

UPDATE: As of Windows 10 build 18298, when you open the properties page of any Console window, you’ll notice an additional “Terminal” tab -- which among other things, allows one to set the _default_ ForegroundColor and BackgroundColor to RGB values which are _separate_ from the 16 color "ConsoleColor" palette.

If these are set, then the PowerShell $Host.UI.RawUI values BackgroundColor and ForegroundColor are just plain wrong, and basically impossible to get right, since they only allow setting to ConsoleColor values (i.e. 0-16)

By adding this setting, the Windows team has really thrown down the gauntlet on support for VT and RGB values instead of ConsoleColors -- there's no possible value for the current [ConsoleColor] properties that won't be wrong.

[ExE-Boss]

I use the Pansies module, which allows for using objects to store ANSI escape sequence data.

[Jaykul]

@joeyaiello regarding your comments on the community call -- color is NOT something you can leave to the terminal. The terminal is built on top of the console. It supports and uses RGB color values and the XTerm 216-color (web-safe palette) but there's no API for them except VT escape sequences.

PowerShell currently only supports the base 16 "ConsoleColors" for any of it's color parameters and properties, and switching to RGB requires using the VT escape sequences, which is not backwards-compatible with older versions of Windows.

For instance, see what happens when the default foreground color is changed, in my stream:
https://www.twitch.tv/videos/424738436?t=00h05m36s

[Delta456]

Currently with the latest release of Powershell 7. I can do 24 bit color via echo but it doesn't via os.system function of Python or print().

I am working on Powershell 7 24 bit support of my module and I am stuck on this limitation :c Will love to see this fully supported.