Powershell: ANSI escape handling and PowerShell

[SteveL-MSFT]

User scenario

Since we've added more ANSI escape sequences for coloring in PS7, if you pipe or redirect that text output, the embedded ANSI escape sequences are sent down the pipeline. This can make log files harder to read. Also, for those wanting to leverage ANSI escape sequences, they are difficult to create/read for scripters and also harder to use formatting presets rather than hardcoding specific colors.

Fundamental philosophy

On Linux, if you use ls --color then different file types use ANSI escape sequences as color indicators. If you pipe this output to less, then you get paging while retaining the color information. If you redirect this output to a file, that file contains the ANSI escape sequences. If you then use the cat command on the file, you see the coloring as the ANSI escape sequences are rendered by the terminal.

On macOS, the similar command is ls -G, however, piping this output to less or redirecting to a file you lose the ANSI escape sequences so it is just plain text. I believe ls on macOS is detecting if output is redirected and turning off coloring automatically.

So for PowerShell, we should have a consistent experience regardless if Windows, Linux, or macOS and interop with native commands that may output ANSI escape sequences in addition to cmdlets/formatting.

Since ANSI escape sequences are de facto standardized, we should not introduce a new intermediate format and should just embed ANSI escape sequences within strings. This will ensure compatibility with other tools that emit or handle ANSI escape sequences.

Proposed technical implementation details

A new automatic variable $PSStyle will be added:

   TypeName: System.Management.Automation.PSStyle

Name            MemberType Definition
----            ---------- ----------
AttributesOff   Property   string AttributesOff {get;set;}
Background      Property   System.Management.Automation.PSStyle+BackgroundColor Background {get;set;}
Blink           Property   string Blink {get;set;}
BlinkOff        Property   string BlinkOff {get;set;}
Bold            Property   string Bold {get;set;}
BoldOff         Property   string BoldOff {get;set;}
Foreground      Property   System.Management.Automation.PSStyle+ForegroundColor Foreground {get;set;}
Formatting      Property   System.Management.Automation.PSStyle+FormattingData Formatting {get;set;}
Hidden          Property   string Hidden {get;set;}
HiddenOff       Property   string HiddenOff {get;set;}
OutputRendering Property   System.Management.Automation.OutputRendering OutputRendering {get;set;}
Reverse         Property   string Reverse {get;set;}
ReverseOff      Property   string ReverseOff {get;set;}
Standout        Property   string Standout {get;set;}
StandoutOff     Property   string StandoutOff {get;set;}
Underlined      Property   string Underlined {get;set;}
UnderlinedOff   Property   string UnderlinedOff {get;set;}

$PSStyle

The base members return ANSI escape sequences mapped to their names. These are also settable so the user can change bold to underlined, for example. This makes it easier for scripters to author decorated strings with tab completion:

"$($PSStyle.Background.LightCyan)Power$($PSStyle.Underlined)$($PSStyle.Bold)Shell$($PSStyle.AttributesOff)"

$PSStyle.OutputRendering

This is of type System.Management.Automation.OutputRendering which is an enum with the values:

• Automatic This is the default. If the host supports VirtualTerminal, then ANSI is always passed as-is, otherwise plaintext
• ANSI ANSI is always passed through as-is
• PlainText ANSI escape sequences are always stripped so that it is only plain text
• HostOnly This would be the macOS behavior where redirected or piped output the ANSI escape sequences are removed

$PSStyle.Formatting

This effectively replaces $Host.PrivateData as the way to read or configure colors for formatting rendering. $Host.PrivateData will continue to exist for backwards compatibility, but is not connected to $PSStyle.Formatting.

   TypeName: System.Management.Automation.PSStyle+FormattingData

Name         MemberType Definition
----         ---------- ----------
Debug        Property   string Debug {get;set;}
Error        Property   string Error {get;set;}
ErrorAccent  Property   string ErrorAccent {get;set;}
FormatAccent Property   string FormatAccent {get;set;}
Progress     Property   string Progress {get;set;}
Verbose      Property   string Verbose {get;set;}
Warning      Property   string Warning {get;set;}

One difference here is that instead of being of type ConsoleColor, these are all strings and doesn't separate foreground and background colors. This means that a single member can have foreground and background colors defined as well as other attributes like bold, underlined, etc...

Scripters can easily leverage this:

"$($PSStyle.Formatting.ErrorAccent)Power$($PSStyle.Formatting.Verbose)Shell$($PSStyle.AttributesOff)"

$PSStyle.Foreground and $PSStyle.Background

These members contain the standard 16 console colors as well as a RGB() methods to specify 24-bit color. For the colors, the values are settable and because they are strings can be any string content and any number of ANSI escape sequences.

   TypeName: System.Management.Automation.PSStyle+ForegroundColor

Name         MemberType Definition
----         ---------- ----------
Rgb          Method     string Rgb(byte red, byte green, byte blue), string Rgb(int rgb)
Black        Property   string Black {get;set;}
Blue         Property   string Blue {get;set;}
Cyan         Property   string Cyan {get;set;}
DarkGray     Property   string DarkGray {get;set;}
Green        Property   string Green {get;set;}
LightBlue    Property   string LightBlue {get;set;}
LightCyan    Property   string LightCyan {get;set;}
LightGray    Property   string LightGray {get;set;}
LightGreen   Property   string LightGreen {get;set;}
LightMagenta Property   string LightMagenta {get;set;}
LightRed     Property   string LightRed {get;set;}
LightYellow  Property   string LightYellow {get;set;}
Magenta      Property   string Magenta {get;set;}
Red          Property   string Red {get;set;}
White        Property   string White {get;set;}
Yellow       Property   string Yellow {get;set;}

PowerShell engine changes

The formatting system, pipelining, and redirection will be updated to respect the value of $PSStyle.OutputRendering.

StringDecorated type

This is an internal type to handle ANSI escaped strings.

Constructor

Single constructor taking a string as input.

bool IsDecorated property

Returns if the string contains ANSI escape sequences based on if the string contains ESC or C1 CSI.

int Length property

Returns the length of just the text content of the string sans ANSI escape sequences.

StringDecorated Substring(int contentLength) method

Returns a substring starting at index 0 up to the contentLength for content that is not part of an ANSI escape sequence. This is needed for table formatting to truncate strings preserving ANSI escape sequences that don't take up printable character space.

string ToString() method

Returns the plaintext version of the string.

string ToString(bool Ansi) method

Returns the raw ANSI embedded string if Ansi parameter is true, otherwise returns plain text with ANSI escape sequences removed.

Out-String update

To enable scripts to easily allow plain text redirection, Out-String will have a -RemoveAnsi switch.

Select-String update

By default, it would make sense for Select-String to ignore ANSI escape sequences, perhaps a -IncludeAnsi switch should be added.

$Host.PrivateData

This will be available for legacy scripts/modules that read from it. However, the engine changes (and console host changes) would mean these settings will no longer be observed and instead use the settings from $PSStyle.

Alternate considerations

Original prototype was built on top of System.CommandLine.Rendering, however, that is still a work in progress and is not targeted to be 1.0 in time for 7.1. It is also geared towards C# developers and the user experience is not great for scripts.

Related Issues

https://github.com/PowerShell/PowerShell/issues/10811
https://github.com/PowerShell/PowerShell/issues/7744
https://github.com/powershell/powershell/issues/3611

[adamskt]

Possibly stupid question here: Will I be able to use Export-Clixml on the $PSSyle automatic variable to "round-trip" my thoroughly tricked-out and gorgeous settings to a file that I could use to move them to another environment? Will the ANSI escape codes in the string members get serialized in such a way as to not trip up version control systems? In any case, this seems like a fabulous idea!

[SteveL-MSFT]

@adamskt good question. I don't see any reason why you couldn't serialize $PSStyle to Clixml and deserialize it to overwrite $PSStyle as a way to have the same settings across systems.

[jhoneill]

So in summary.
(1) A standard, fairly readable way for scripts to do formatting
(2) A simple way to turn off ANSI formatting whether the author used (1) or did it their own way
That solves what I see as the two main problems with ansi codes (wanting to turn it off - sometimes - and ugly scripts) .

I liked the early draft of this idea, and I like this even more. A simple thumbs up wasn't enough :-)

Since common parameters set the preference variables for the local scope, what would your view be on adding a common parameter which sets output-rendering for a single command ? Among other things that would mean if one really hates the authors choice of colors a simple entry in $PSDefaultParamterValues gets rid of it.

I also wonder if more choices should be available so that a user can set their own warning, error, emphasis styles and the author then says "format this as a warning" not "use orange text"

[SeeminglyScience]

If PowerShell is going to tackle theming I'd really really like to see some extra settings that a TUI might be able to tap into. Taking bootstrap as an example, I'd like to see the styles be under names like:

• Primary
• Secondary
• Success
• Danger
• Warning
• Info
• Light
• Dark
• Muted

Every time I've started making a TUI of any sort, trying to make something that looks nice and also keeps to the spirit of the user's color choices is very challenging.

[mklement0]

Re terminology: I wonder if we should use "Virtual Terminal (VT) [Escape] Sequences" rather than "ANSI escape codes" in order to avoid confusion with the "ANSI" _character encoding_ - though, arguably, that confusion is more likely for Windows users and, conversely, Unix users may be less familiar with the term "Virtual Terminal".

FWIW, looking at a few man pages for the various Unix utilities capable of producing colored output suggests that they rarely mention the coloring _mechanism_ and often just talk about "color", and less frequently "escape sequences".

Avoiding "ANSI" would also mean: string ToString(bool Ansi) -> string ToString(bool decorated)

---

Re output modes (controlling when to use the sequences, $PSStyle.OutputRendering):

Note that GNU ls (and implicitly also macOS/BSD ls) has _3_ modes (as do both GNU and macOS/BSD grep):

• auto (the default): if stdout is connected to a (VT-enabled) _terminal_, use color; otherwise (redirected output (pipe, file)) don't.
• never: never use color
• always: always use color, irrespective of where the output goes.

GNU ls _without a color option_ defaults to auto; using --color is the same as --color=always (the other options obviously being --color=never and --color=auto); note that this default is an unfortunate choice; GNU and macOS/BSD grep more sensibly interpret --color as --color=auto.

macOS/BSD ls, via _arguments_, offers auto behavior (only) via -G (default is never); environment variable CLICOLOR_FORCE can be set to achieve always behavior.

In short: I think these modes are sufficient and I think they're worth adopting - including the terminology.

---

Use of environment variables, new CLI parameter, new common parameter to control coloring:

We should consider exposing setting $PSStyle.OutputRendering and the potential theming proposed by @SeeminglyScience via environment variables as well, so that there's an easy way for automation environments that call the PowerShell CLI to preset unconditional (always) coloring (e.g., from a POSIX-like shell; name negotiable, but it should be reasonably short: PSSTYLE_MODE=always pwsh -NoProfile -File ...)

_Update_: https://github.com/PowerShell/PowerShell/issues/10811#issuecomment-546126662 mentions @lzybkr proposing a new _CLI_ parameter:

@lzybkr had a good suggestion to have a param on pwsh like -color with always, auto, never values.

@jhoneill above suggested a new _common parameter_ for _cmdlet-individual_ control (caveat is that unless this parameter is opt-in, conflicts with existing user-defined parameters are possible).

[mklement0]

@SteveL-MSFT, this issue mistakenly links to itself under "Related Issues"; I think you meant to link to #10811.

[vexx32]

This already seems kind of similar to what @jaykul has done with PANSIES, only less flexible and harder to work with. In that module, a PSProvider is added that is capable of generating escape codes for common (named) colors as well as arbitrary hex color values. That can then be used mid-string to insert values in a flexible way.

If users are still going to have to look up the escape sequences for these settings / download a third party module to handle that anyway, I'm not sure this really adds much. Having a base theming framework is a good start, but without some more user-friendly features (VT escapes are _not_ user friendly, period) I don't see it getting a ton of use.

[SteveL-MSFT]

@mklement0 you're right, had the wrong URL in my clipboard at the time. Fixed.

[SteveL-MSFT]

@vexx32 the $PSStyle variable is explicitly to handle not knowing ESC sequences and allowing for tab-completion in strings. It also has the RGB method for arbitrary 24-bit color.

[SteveL-MSFT]

@SeeminglyScience I think those are interesting style names and can see how they may be more future proof. Were you thinking those would replace the ones I have under the Formatting member or in addition? How would you map yours to the existing ones (some are obvious, others not as much like ErrorAccent).

[mklement0]

@vexx32, as implied by @SteveL-MSFT's comment, unless you want to redefine the default colors (which probably isn't a good idea) or redefine the $PSStyle.Formatting properties, you won't have to deal with escape sequences.

However, in order to better support these - advanced - use cases, perhaps we could extend Write-Host to support all the styles supported by $PSStyle and add an -AsDecorated switch that outputs the result as a string with escape sequences.

[SeeminglyScience]

@SeeminglyScience I think those are interesting style names and can see how they may be more future proof. Were you thinking those would replace the ones I have under the Formatting member or in addition? How would you map yours to the existing ones (some are obvious, others not as much like ErrorAccent).

The way I have it in my head is something like this:

class Theme
{
    public Style Default { get; set; }
    public Style Primary { get; set; }
    public Style Secondary { get; set; }
    public Style Success { get; set; }
    public Style Danger { get; set; }
    public Style Warning { get; set; }
    public Style Info { get; set; }
    public Style Light { get; set; }
    public Style Dark { get; set; }
    public Style Muted { get; set; }
}

class Style
{
    public Color Foreground { get; set; }
    public Color Background { get; set; }
    public Decoration Decoration { get; set; }
}

class Color
{
    // Maybe just the RGB value here.

    public bool TryWriteAnsiPrefix(Span<char> buffer, out int amountWritten, bool isBackground = false);

    public bool TryWriteAnsiReset(Span<char> buffer, out int amountWritten, bool isBackground = false);
}

class Decoration
{
    public bool IsUnderlined { get; set; }

    // etc

    public bool TryWriteAnsiPrefix(Span<char> buffer, out int amountWritten);

    public bool TryWriteAnsiReset(Span<char> buffer, out int amountWritten);
}

Where $PSStyle points to Theme.Default maybe with an ETS property called Theme that points to the parent?

_{I realize this is a drastically different design in a few places that aren't relevant, it's just for illustration purposes.}

[mklement0]

The formatting system, pipelining, and redirection will be updated to respect the value of $PSStyle.OutputRendering

I assume this means, as also implied by @jhoneill's comment, that the engine will have to check every output object for whether it is a string and, if so, whether it contains VT escape sequences - do we need to worry about performance, especially with sophisticated detection (see below)?

Returns if the string contains ANSI escape sequences based on if the string contains ESC.

I suggest a more sophisticated test, along the lines of using regex (?:\e\[|\u009b)[^m]*m, so as to not accidentally strip unrelated escape sequences used for different purposes (rare as that may be); this regex should cover all SGR (Select Graphic Rendition) sequences, which, however may still not be enough, given that other sequences do exist: see https://en.wikipedia.org/wiki/ANSI_escape_code#Escape_sequences; remember the discussion in #7744, initiated by @felixfbecker, in the context of an OSC (Operating System Command) sequence where not only the actual start and end sequences must be stripped, but also the text _in between_ (it is a hyperlink that shouldn't print when rendering as plain text).

At any rate, we need to clearly document what escape sequences we do detect as $PSStyle.OutputRendering-relevant, and what the limitations are.

ANSI escape code - Wikipedia

[mikeTWC1984]

$PSStyle.OutputRendering would also solve https://github.com/PowerShell/PowerShell/issues/12703
Other thoughts :
Some of these can be implemented as string extenstions (e.g. "Powershell".Green.Bold.Bg("Yellow") )
Rgb method could accept named colors from System.Drawing.Color (256 colors)
Some out of the box color testing cmdlet would be nice to have (maybe lolcat ? :) )
Converto-Html should be able to convert ansi to colors

[iSazonov]

The formatting system, pipelining, and redirection will be updated to respect the value of $PSStyle.OutputRendering

We forget about Jason's PSMore idea. Formatting system based on the idea will be great. I do not think that it will require more efforts than this PSStyle proposal.

Also I'd think about generalizing of the rendering - OutputRendering could be IRendering/IPSRendering so that developers can easily address web and other scenarios like Markdown, HTML (5? :-) ) and other outputs.

[felixfbecker]

The API looks good to me, but the proposal seems to be focused a lot on creating "decorated strings" for the pipeline and doesn't talk about the distinction between formatting and pipeline data. Colors are imo a formatting concern and should generally not be in strings in the pipeline. The API can of course be used inside formatting files, so there isn't a problem with the API, I just wished the proposal here targeted that use case explicitly and less so putting strings with ANSI sequences into the pipeline. Some people will do that of course, and that is ok, because the distinction doesn't matter for every quick shell script hacked together. But I expect PowerShell itself and many modules to use this feature the most, and they'll want to do it the proper way.

Since this issue is very broadly named "ANSI escape handling and PowerShell", I'd also like to point out that there are some big issues with the way the formatting system handles color escapes that would need to be solved too (as others have also pointed out): https://github.com/PowerShell/PowerShell/issues/7744#issuecomment-489771728

Some comment on the API:
Does AttributesOff only turn text attributes like bold and underline off? Or is it the RESET sequence that resets everything, including color? If the latter, I think Reset would be a less confusing name.

It also seems like it would make sense to add -OutputRendering also as a parameter to Out-String, Out-Host, etc cmdlets

Overall would love to see such an API though!

[jhoneill]

The API looks good to me, but the proposal seems to be focused a lot on creating "decorated strings" for the pipeline and doesn't talk about the distinction between formatting and pipeline data. Colors are imo a formatting concern and should generally not be in strings in the pipeline.

YES! I have marked myself out the position of "the curmudgeon who wants monochrome" but data has no colour - we add it to give humans something better to look at - therefore it should be applied at the very last moment via the formatting xml. Nothing in this prevents that - I think it makes things easier.
However, whether it is write-host with -foregroundcolor or embedding ANSI escape codes in strings writers, DO put colour in their output, and this gives a less bad way of doing it... It should still come with a very clear warning that when you turn data into formatted text getting back to the data may not be simple.

[SteveL-MSFT]

@felixfbecker the SubString() method is explicitly to handle proper truncated for decorated strings used with formatting. AttributesOff is the RESET ESC sequence. The name was borrowed from the .NET API, but can be changed to Reset if that's more obvious. -OutputRendering to Out-String makes sense as a way to control it within the pipeline. Will add. I think Out-Host rendering should be implicit, is there a scenario where a script would control it vs the user?

[jazzdelightsme]

I second the comment about "Reset" being a better name than "AttributesOff". (I also prefer "VT sequences" to "ANSI escape sequences", but not quite as strongly.)

Re: the IsDecorated method that looks for ESC characters: don't forget about C1 codes (the C1 CSI is a single 0x9b--no ESC (0x1b) character).

Oh wait... did you say this class would be internal only? That seems like it would be a shame... the PowerShell engine is not going to be the only thing that wants to be able to deal with this sort of stuff.

(Feel free to borrow any code from my related CaStringUtil class.)

The "fg" and "bg" PSProviders in jaykul's Pansies has a really nice terse syntax--${fg:red} versus $PSStyle.Foreground.Red--though maybe the latter is more descriptive. Maybe it's a "why not both?" situation.

Side note (only very tangentially related to this): another thing we need is editors that understand VT SGR sequences. Once or twice I've taken to embedding SGR sequences in my comment-based help comments, which looks great when displayed, but gets real ugly real quick when reading the comments in the source and trying to edit it.

[SteveL-MSFT]

@jazzdelightsme updated to also include C1 CSI. The intent of keeping the class internal is that .NET team is also working on some APIs to solve this problem, so would prefer not to have 2 public APIs once theirs is available. Perhaps we can put it in the internal namespace rather than making it internal. Foreground and Background could be just FG and BG, but wanted to follow PowerShell convention of verbosity.

You should open that editor complaint in the VSCode repo!

[SeeminglyScience]

The API looks good to me, but the proposal seems to be focused a lot on creating "decorated strings" for the pipeline and doesn't talk about the distinction between formatting and pipeline data. Colors are imo a formatting concern and should generally not be in strings in the pipeline. The API can of course be used inside formatting files, so there isn't a problem with the API, I just wished the proposal here targeted that use case explicitly and less so putting strings with ANSI sequences into the pipeline.

I think this is a very important point. Ideally the API would be designed in a way that doesn't make it appear that the best practice is to just output decorated strings. With the current design I think we'd see an uptick in this sort of pattern.

---

@SteveL-MSFT

I think Out-Host rendering should be implicit, is there a scenario where a script would control it vs the user?

I can definitely think of scenarios, but none that wouldn't be solved easily enough with Out-String @formatting -Stream | Out-Host

The intent of keeping the class internal is that .NET team is also working on some APIs to solve this problem, so would prefer not to have 2 public APIs once theirs is available.

Should PowerShell wait then? Once the dotnet version comes out we'll have one public API that doesn't work with PS and one internal type we still can't use right?

Realistically PowerShell is probably going to need it's own version anyway since the dotnet one will likely be pretty byref-like heavy. So my opinion would be make the PowerShell-centric version public, and just switch the implementation to use the dotnet version once it becomes available. Maybe make them implicitly convertable to each other for extra points.

Perhaps we can put it in the internal namespace rather than making it internal.

I mean we can use reflection if we don't care about using unsupported API's. I feel like pubternal API's don't really work in PowerShell, most users won't understand what that means. Especially with the already mixed messaging around them via AutomationNull.

[ThomasNieto]

Forgive my ignorance but why are not solving this in the formatting layer like #11890? PowerShell has kept data and formatting separate and it seems like this is trying to combine formatting (color) into the data (strings).

[mklement0]

To add to the comments by @felixfbecker, @SeeminglyScience, and @ThomasNieto re the problematic use of VT strings _in the pipeline_:

• I think that the engine itself should _not_ be in the business of (conditionally) cleaning VT sequences from strings _in the pipeline_. It is expensive, can lead to false positives, and we generally shouldn't _guess_ whether or not a command actually meant to output _data_ that happens to contain VT sequences.

  • Besides, it is impractical, as the engine would have to walk an object's _properties_ if the VT string is not the output object itself, but in a property (something like [pscustomobject] @{ one = "`e[31mred`e[m" }).

• Instead, the use of VT sequences for _formatting_ should be a _deliberate act_ by _each command_, in one of two ways:

  • Preferably, via the formatting layer (format.ps1xml files), where the use of $PSStyle will by definition affect formatted output only.

  • Simpler alternatives for scripters:

    • In the simplest form, allow signaling that a string's VT sequences are solely for formatting, e.g., by adding a new -HostOnlyVT switch (or some such thing) to Write-Output, which would tell it to strip the VT sequences (by default) except when writing to the host:

      "It ain't easy being $($PSStyle.Foreground.Green}green$($PSStyle.AttributesOff}." | Write-Output -HostOnlyVT

  • For more control, let's revisit @KirkMunro's proposal to make output formatting easier by extending the [OutputType()] attribute, which could fit in here as well: #10463 (inspired by his earlier FormatPx project). It could be integrated part of #11890 to provide property-specific formatting via calculated properties containing newly introduced formatting instructions.

• The formatting engine then needs to be VT-aware in order to strip sequences on demand properly handle truncation of values to avoid the problems described in #7744, which is what the proposed StringDecorated class would do.

  • All Out-* cmdlets should then respect $PSStyle.OutputRendering, which should default to auto (VT sequences only when printing to the host).

  • All Out-* cmdlets could get a new -PreserveVT <mode> (name negotiable) parameter to allow controlling the output mode on a per-call basis, with <mode> being one of the values supported by $PSStyle.OutputRendering (which I propose be auto, always, and never).

    • As proposed, a new CLI parameter could help when explicitly wanting colored output even when capturing output. For instance, from Bash: capturedWithColor="$(pwsh -preserveVT always -c '"foo" | sls "o"')" (remember that what an external caller sees on stdout when calling the PowerShell CLI is _formatted_ output, unless -OutputFormat XML is used).

---

@SteveL-MSFT:

• Out-String -PreserveVT never would then supersede -RemoveAnsi

• I don't think Select-String should ignore VT sequences _by default_:

  • We should stick with the assumption that what comes through the pipeline / from a file is _data_.
  • Asking to ignore VT sequences should therefore be _opt-in_.

---

@jazzdelightsme:

${fg:red} is definitely appealing for its concision and not having to use the ugly $($...) syntax.

The problem is that, by virtue of using namespace notation via a provider-based implementation, fg and bg are of necessity PS drives by that name; if we expose additional formatting capabilities, as proposed, we'll get a proliferation of PS drives with the potential for naming conflicts. (The nice thing about having drives is discoverability with Get-ChildItem).

@Jaykul himself has proposed a single, hierarchical text drive as an alternative at https://github.com/PowerShell/PowerShell/issues/11890#issuecomment-592667804:

make the _provider_ a "text effect" and the drive "text" and make "background" (and "bg") and "foreground" (and "fg") as folders, so you could Write-Host "${text:fg\red}${text:bg\white}${text:underline}Hello World" or use _that_ in the format files and cmdlets ...

Re SGR (VT escape sequences) in comment-based help:

Given that we want to move away from raw VT escape sequences, I think a more promising thing to consider is the use of (perhaps a limited subset of) _Markdown_ in comment-based help.

[Jaykul]

It's a mistake to talk about "moving away from raw VT escape sequences" -- that is impossible, this is just UX.

Also, remember the color palette is always owned by the terminal. Practically all terminals support a base 16 color palette which can be configured. Most support changing the full 256 color xterm palette, nowadays. A growing number support using specific RGB values directly without a palette. But in no case should the shell try to redefine any of those palette colors.

As far as the user experience goes, I think there are two separate concerns in the original pos that should _perhaps_ be separated.

The first part is the OutputRendering enum, and support for it (and overriding it) on all the output commands. I think that variable should be it's _own_ enum preference variable, and possibly a switch on the pwsh executable, as someone already suggested. Any discussion about adding a fancy class with named styles and so-on is also tangential, and not necessary for this. Any discussion about alternate ways of applying formats is tangential and not necessary for this. Remember, the only thing that truly exists for formatting are the VT escape sequences! For the purposes of this enum, we're just deciding when and whether or not to strip them off.

The second part is theming, and I think the design of a struct/class with specific named properties that are _primarily for use in formatting files_, and configured by the user according to their own preferences is a good one. There's no need for people to worry about when and where these things will be put into strings. We all know where things are converted into strings in PowerShell, and this does not change that. As a corollary, any discussion about alternate ways of applying formats is still tangential here!

We've always had an enum for colors for the output streams, but it doesn't support xterm or full color, and I like the idea of not only extending the color to "any sequence supported by your terminal" but also extending the list of colors from Default, Warning, Error, Verbose, Debug, Progress to include a few extra colors like Success, and even some "accent" colors like PrimaryAccent, PrimaryMuted, SecondaryAccent, SecondaryMuted, or whatever.

If we add these, then everything PowerShell outputs by default should only use the values defined here -- so that users can create specific combinations that they can read, and expect them to be respected throughout the shell.

For the sake of usability and accessibility, I'm not sure it's a good idea to have the foreground and background defined separately for these -- there's very little chance that combining the foreground from one with a different background is a good idea, so why bother requiring people to write both out every time they want to use one of these styles? Wouldn't it be better to just have a "Warning" style which can include a foreground _and_ background, depending on the preference of the user?

I am not excited about the extra named formats (i.e. PSStyle) -- I think most of these are still too rarely functional to be of any use (for instance, bold is normally just "bright", which is actually rendered as a _totally different color_, and only supported on the base 8 colors -- see this comment). Frankly, we're probably better off using more generic names like "PrimaryAccent" and letting people define those to include blinking if they really want to ...

In case it's not obvious from what I've already written:

PowerShell should _not_ define it's own _base 16 color palette_!

That is, please do not implement anything like the proposed PSStyle+ForegroundColor. It would be a mistake. The color palette belongs to the _terminal_, not the shell. Implementing a palette where the names of the base 16 colors are defined as a _different VT escape sequence_ than the default 16 color escape sequences would just confuse people and would wreck interaction with native apps. If you want to expose color names for simplicity, do what I did in Pansies and expose a shortcut way to use color names alongside rrggbb values to get the VT escape sequence for a color name, so that people don't need to learn the sequences -- but _don't make the sequence customizable_.

[mklement0]

Good points, @Jaykul, but just to be clear re:

It's a mistake to talk about "moving away from raw VT escape sequences"

What the quoted statement meant is (a) that we don't want to _require_ that users deal directly with the underlying VT escape sequences (they should only have to deal with abstractions) and (b) for those advanced users that do, we should make it easy to generate such sequences with something like Write-Host -AsDecoratedString.

[SteveL-MSFT]

@Jaykul I agree that the terminal owns the definitions of the palette and this proposal doesn't change that. The only difference is that $PSStyle.Foreground.Red means a specific string which probably contains ANSI escape sequences. The ANSI escape sequence for "red" as it shows up is still owned by the terminal. In this case, a user may decide that they want Write-Error to show with red background, black foreground, italics, and blinking. They can do that with $PSStyle.Formatting. (Red and black would, of course, be whatever colors the terminal has defined for those entries in the palette).

@PowerShell/powershell-committee had a discussion about this proposal today and some points made:

• would like to see a complete example of how a user would use $PSStyle with custom formatting
• consider a different setting for redirection vs host instead of all-in-one
• does it make sense to have ANSI controlled at individual cmdlets rather than Out-String?
• make clear that ANSI should only be used with formatting or text output and not as part of string members of objects which are just treated as pass-thru

Given that there doesn't seem to be consensus on this, rather than rushing this in, we're moving this to 7.2.

[Jaykul]

@SteveL-MSFT as long as those Foreground.Red colors are _not settable_, and always result in the simple 16 color codes `e[31m etc., then we're on the same page 😁 I just don't want to deal with git outputting one version of red, and PoshGit outputting a different shade.

I totally agree that "ANSI should only be used with formatting or text output and not as part of string members of objects" but given that statement, I don't understand what you mean about having "ANSI controlled at individual cmdlets"? If we "control" ANSI outside of the formatting or output commands, wouldn't that break the last point?

I'm really not sure I'm totally on board with all this work going in to basically just create sets of colors. I can do that in a module and just tack it on to the Host.PrivateData for people to find....

Personally, I think it would be best to add ANSI parameters to the formatting commands and color properties to the formatting objects they output. Enhancing things with color there would be far less backward compatible (you couldn't just ship $PSStyle back to PS5), but having color on the formatting objects would be a dream. For examples, I have a format file now in EZTheme that has a bunch of entries that are something like this:

<ListItem>
    <Label>green</Label>
    <ScriptBlock>([PoshCode.Pansies.RgbColor]"Green").ToVtEscapeSequence() + "Green" + "$([char]27)[49;39m"</ScriptBlock>
</ListItem>

I would much rather be able to write something like:

<ListItem>
    <Label>green</Label>
    <PropertyName Foreground="Black" Background="Green">Green</PropertyName>
</ListItem>

Or even

    <Label>green</Label>
    <PropertyName Foreground="Black" Background="00EE00">Green</PropertyName>

Actually, I really want to be able to get the colors with script:

<TableColumnHeader>
  <Alignment>Left</Alignment>
  <Label>Green</Label>
  <Width>5</Width>
  <ForegroundColor>Black</ForegroundColor
  <BackgroundColor>Green</BackgroundColor>
</TableColumnHeader>
...
<TableColumnItem>
  <PropertyName>Green</PropertyName>
  <BackgroundColorScript>$_.Green</BackgroundColorScript> <!-- changes the background of JUST THIS CELL to my custom green -->
</TableColumnItem>

We want more than schemes.

We want to color things simply, maybe even have a <highlight Color="Red">term</highlight> that we could add to automatically do a search and highlight key words in the output field. Maybe be able to script the values and even write something to set the background of the FileInfoObject Length column to something like: Get-Gradient Blue Red -Count 10 -Flatten | Select -Index ([Math]::Min(10, ($this.Length/10gb))) as a way of coloring files relative to their size (ok, that's a super-simplistic gradient algorighm, but you get the point).

Of course, we also need the formatting engine to clear the colors after each cell (or at least, set them back to the colors for the column or row ...

[SteveL-MSFT]

@Jaykul if the desire is to have color be first classes in formatting, then we could add ForegroundColor and BackgroundColor elements to formatting XML (although I'd rather spend time moving away from XML to a DSL which could include attributes...). However, that would be an addition and not mutually exclusive.

With the current proposal, people could use any library they want for decoration as long as the end result is ANSI escape sequences. The main change in the engine is in formatting to use the new substring() method to correctly calculate column widths where ANSI escape sequences may exist.

The point about "cmdlet controlled" is that ANSI is still the de facto language, but specific cmdlets may have options to strip ANSI from received strings or strip ANSI (or not include ANSI) on output. I personally prefer having stripping of ANSI just part of Out-String, but perhaps that's because I'm accusomted to using Out-String in my pester tests.

[iSazonov]

although I'd rather spend time moving away from XML to a DSL which could include attributes

Generalization of formatting and rendering should be our priority.
We pay a lot of attention to output to ANSI terminal, but there are other scenarios with hosting applications. As an example we could think about PowerShell Web Console vNext based on Chrome Engine. In the case HTML/CSS output would be natively expected (maybe HTML5).
What if we will generate HTML/CSS output and then convert to the format expected by target terminal?
@Jaykul's examples resemble this. Anything that can be converted to HTML will render well and simple.
For example, MSFT team implemented Markdown rendering with escapes but we could convert Markdown to HTML and use generic API to send to _any_ console that is more flexible.

[jazzdelightsme]

Generalization of formatting and rendering should be our priority.

I think that sounds good on paper, but it seems like we've already tried that experiment, and it failed (never gained any traction): the current formatting cmdlets don't Console.WriteLine; they output intermediate, generalized objects, allowing for alternative final rendering, should anyone choose to do that. But AFAIK, nobody has chosen to do that.

And after having implemented my own "Formatting+Output" engine, I can see why--somebody else's "generalized" format is usually not going to be quite what you want, anyway. People want the source object itself, or the final rendering; nobody actually wants a generalized, un-rendered representation of formatting. Or put another way, the source objects themselves are much better "intermediate forms" than any other intermediate form you could come up with. If you want to render them in a different way; that's great. But I don't think anybody actually wants or needs any generalized middle step to do that.

[SteveL-MSFT]

I think ANSI escape sequences has already won the war. You already have xterm.js that renders ANSI to HTML. And Jupyter Notebooks with .NET Interactive also renders ANSI to HTML. CI/CD systems also render ANSI to HTML.

[iSazonov]

Won the war? I would say the opposite. People prefer _the Web that won_. But tons of old applications that generate ANSI control codes are forced to work on the Web through extra adapters. Why should we go this crooked path? :-)

Today PowerShell does not issue extra escapes and can output to (1) dumb terminals, (2) web engine like chromium.
If we want to enhance output formatting I'd prefer invest to modern chromium with HTML5 but not old dumb terminals.

[SteveL-MSFT]

@iSazonov The current design would allow for translation of ANSI to HTML if that is required, but for console, ANSI is the de facto standard for authors of tooling

[iSazonov]

By console, you mean a thin dumb teletype (tty) terminal. But at the same time we want coloring, multiple windows, background tasks, progress bar, power command line editor and so on - we actually want a _thick powerful_ terminal.

I am at a loss that we are spending so much effort to overcome the limitations of tty (here and in PSReadline). (Note that these escape sequences are contrary to the object-based nature of PowerShell.) This approach is counterproductive - we are forced to perform complex string conversions at the application level, then at the terminal level, then at the OS level.

_My strong belief is that PowerShell should only offer a generic interface and delegate the rest to the formatting system and host._ The formatting system must understand the target is either a console or a file to apply appropriate formatting.
As for the host, when we run PowerShell on Windows, a graphical window actually opens - why would we need a dumb terminal in it if we can use the full power of chromium engine? From the power console we can connect to any remote system (Core, Nano, Linux) and use all benefits of the console.

[vexx32]

I would be inclined to agree that expending effort to letting users handle this in code itself manually would be massively wasted if it were not easily and readily accessible to formatters (if not outright designed for that first tbh).