Runtime: Console.Clear() doesn't clear the scrollback buffer on Unix-like platforms

[mklement0]

Console.Clear()'s documentation states (emphasis added):

_Clears the console buffer_ and corresponding console window of display information.

Indeed, not just clearing the current terminal _window_ (screen), but also the scrollback buffer is the typical use case, and it is how this method has always worked on Windows.^[1]

By contrast, the Unix implementation currently only clears the _screen_, which makes for an inconsistent cross-platform experience.

While there is no POSIX-compliant way to clear the scrollback buffer (only clearing the _screen_ is mandated by POSIX, via tput clear), xterm-compatible terminal applications do support escape sequence <esc>[3J - see Wikipedia.

In practice, the macOS terminal application and the one on Ubuntu, for instance, do support this escape sequence - generally, terminal emulators based on the X Window System.

I don't know if there are popular terminal emulators out there that do _not_ support it, but even a best-effort implementation would be useful.

Here's a workaround that demonstrates use of the escape sequence:

// Clears the screen and the scrollback buffer in xterm-compatible terminals.
Console.Clear(); Console.WriteLine("\x1b[3J")

---

^{[1] Whether there should be an opt-in for predictably clearing _only_ the screen across platforms, while leaving the scrollback buffer intact, is a separate question.}

[SteveL-MSFT]

Perhaps adding an overload with a bool includeScrollBackBuffer would retain current behavior and allow for the desired behavior

[mklement0]

@SteveL-MSFT: Given that (a) clearing the buffer is documented and (b) doing so is generally the more sensible behavior, my vote is to simply change the existing behavior to align with the documentation / the behavior on Windows.

[danmosemsft]

@stephentoub do you recall whether this was intentional, to not attempt to clear scrollback if supported?

[stephentoub]

do you recall whether this was intentional, to not attempt to clear scrollback if supported?

Not clearing it was not a goal. If there's a "safe" way to do it whenever it's possible (by safe I mean not emitting garbage to stdout when the particular escape code isn't supported by the terminal in use), such as relying on some reliable way to query that a compatible terminal is being used, I'm fine with doing so.

[mklement0]

Thanks, @stephentoub.

relying on some reliable way to query that a compatible terminal is being used

Based on the answers posted at https://unix.stackexchange.com/q/93376/54804, this should be as simple as:

# ... perform screen clearing (as before)
# In compatible terminals, also clear the scrollback buffer.
if (Environment.GetEnvironmentVariable("TERM").StartsWith("xterm")) Console.WriteLine("\x1b[3J");

xterm-compatible terminal emulators set env. var. TERM either to xterm or xterm-<colorCountSpec>, e.g., xterm-256color.

[stephentoub]

And presumably if we did that first and then did the current clear (rather than the opposite order), if we did end up for some reason generating garbage, it'd be mitigated by being immediately cleared? :)

[mklement0]

Exemplary garbage disposal, @stephentoub :)

[danmosemsft]

@mklement0 do you wish to offer a PR?

[khellang]

You should be able to query the terminfo db for the E3 extension, no?

See user_caps(5) under "Recognized capabilities":

E3 _string_, tells how to clear the terminal's scrollback buffer.

When present, the clear(1) program sends this before clearing the terminal.

The command "tput clear" does the same thing.

[mklement0]

Thanks, @khellang, but in practice neither clear nor tput clear clear the scrollback buffer on macOS 10.14 and Ubuntu 18.04.

Consistent with that, the requisite escape sequence, \E[3J, is not present in the output from infocmp when invoked from an xterm-compatible terminal.

I know little about terminal-info databases, so maybe I'm missing something.

@danmosemsft: I'll give it a shot.

[khellang]

Interesting Linux SE answer; https://unix.stackexchange.com/a/375784.

[danmosemsft]

@mklement0 sounds good. I sent you a collaborator invite: it's optional, if you accept it I can formally assign you. Note that accepting it switches on notifications for all of the repo, which you'll likely want to switch off again.

[mklement0]

Thanks, @danmosemsft; I've accepted the invitation.

[danmosemsft]

Assigned.

[mklement0]

Thanks, @khellang; specifically, the passage of interest on the linked page is the following from this answer:

The gnome terminfo entry does not define an E3 capability, and on many systems — still! — neither does the xterm entry as this has not percolated down from Dickey terminfo. So clear just writes out the contents of the clear capability.

It sounds like the right thing to do is:

• (a) query the E3 capability in the terminfo database (based on env. var. TERM) and use the corresponding escape sequence, _if defined_.

• (b) if _not_ defined, and TERM is xterm or starts with xterm- (matches ^xterm(?:-|$)), hard-code "\x1b[3J".

If performance is a concern, we could do (b) first, but I presume it won't matter.

[stephentoub]

If performance is a concern

It shouldn't be. The terminal doesn't change, so what to output for a Clear can be cached (it already is in many cases).

It sounds like the right thing to do is

Sounds reasonable.

[iSazonov]

@mklement0 If you are still working this.
We could implement the fix here
https://github.com/dotnet/corefx/blob/8ff3f535097394f602653a91f41ae4b72bc990eb/src/System.Console/src/System/ConsolePal.Unix.cs#L229

like
```c#
var clearString = TerminalFormatStrings.Instance.Clear;
if (clearString.Equal("\E[H\E[J - a standard value", StringComparison.Ordinal)
{
clearString = "\E3J" + clearString;
}

WriteStdoutAnsiString(clearString);
```

[carlossanlop]

@mkelement0 have you had a chance to continue looking into this?

[janvorli]

I understand that the planned change is going to align the behavior with Windows, however, I am not aware of any text mode application on Unix that would clear the scrollback buffer. I myself would consider such a behavior of any application intrusive and unexpected. What are scenarios when clearing the scrollback buffer is beneficial?

[mklement0]

My apologies for dropping the ball on this, @danmosemsft , @carlossanlop and @iSazonov - can you please unassign me? I hope someone else will take this on.

However, it sounds like maybe there isn't consensus yet as to what to do:

@janvorli, I can see how for _programmatic_ (as opposed to _interactive_ use in a _shell_) clearing the scrollback buffer _by default_ could be considered too invasive.

(On re-reading the documentation, it doesn't actually talk about the _scrollback_ buffer, only about the "console buffer", which can justifiably be interpreted to mean the current screen only.)

Even though also clearing the scroll-back buffer is how it has always worked in regular console windows on Windows, I now see that in Windows Terminal it does _not_.

This takes us back to @SteveL-MSFT's suggestion to make buffer clearing an _opt-in_ via a Boolean parameter.

We could therefore:

• Introduce the proposed overload, public static void Clear (bool includeScrollBackBuffer);

• make that overload work as intended when false is passed in regular console windows on Windows too (which I assume is possible)

• leave the current _default_ behavior (parameterless overload: scrollback-buffer-too clearing in Windows console windows, current-screen clearing only on Unix and in Windows Terminals) and _document_ the inconsistency, recommending use of the overload where you signal the intent _explicitly_ going forward

[danmosemsft]

To add a new API (rather than simply align behavior) we'd want a bit more evidence of need. What sort of scenario would lead someone to pass "true" ?

[khellang]

What sort of scenario would lead someone to pass "true" ?

I don't have a specific scenario in mind, but from researching this a while back, it seems like quite a few people are interested in clearing the buffer as well. Whether that is programmatically or interactively isn't clear (no pun intended) though.

On Windows, there's no need as the existing parameterless overload already does clear the buffer. But that overload isn't consistent across platforms. If you want to to be consistent, call the overload with either true or false and it'll work the same cross-platform 😊

That's the only way I can see this working out and not break everyone relying on the existing behavior.

[mklement0]

Good points, @khellang.

When implementing a _shell_ or _REPL_ the feature is of particular interest: often you want to start with a clean slate in a terminal before submitting a new command, so as not to get confused between the most recent command's output and unrelated output that preceded it.

Indeed, it was the desire for consistent cross-platform behavior of PowerShell's Clear-Host command that brought us here - https://github.com/PowerShell/PowerShell/issues/8606

While PowerShell could certainly implement the behavior without using Console.Clear(), I can see other shells / REPLs benefitting from support in the CLR too - and indeed any type of application looking for consistent cross-platform behavior, notably including applications that on Windows do _not_ want to clear the scrollback buffer.

---

As for native shell / utility behavior:

• On Windows, users of cmd.exe (cls) and PowerShell (Clear-Host, aliased to cls) are used to the scrollback-buffer getting cleared, by default and invariably - though, as stated, Windows Terminal will change that experience.

• On Linux, /usr/bin/clear is _meant_ to clear the scrollback buffer by default - but still doesn't, as of Ubuntu 18.04, due to the terminal-capabilities database still not having been updated - see https://unix.stackexchange.com/a/375784/54804. Option -x is the _opt-out_ that clears the current screen only.

• On macOS, /usr/bin/clear was only ever designed to clear the current screen, but the default terminal offers keyboard shortcut Command-K to clear the scrollback buffer too.