Kitty: Kitty eats one page worth of scrollback on `clear` / Ctrl+L, then scrolling up

[ronjouch]

One GIF speaks a thousand words. Below is:

• On top, gnome-terminal running fish
• On bottom, kitty running the same fish with the same config

I'm running ls -la /bin then clear in both, then I scroll up using the mouse wheel. See how kitty fails to display lines after umount (it "eats" everything from uname to zsh5). Problem happens with both mouse wheel and kitty_mod+page_up.

Am I missing anything? (Fish / Kitty misconfiguration?)

• Fish 2.7.1-1638-gdfcf140e
• Kitty HEAD @ 1f3550b84cb0a113f9b5e7c242de3f3eb4d8ccf9
• kitty.conf keys potentially relevant: scrollback_lines 90000, shell fish

Thanks for Kitty.

[egmontkob]

FYI: gnome-terminal and konsole have nonstandard behavior on "clear", see gnome-terminal/vte 506438 and konsole 384218 for details.

[ronjouch]

"FYI: gnome-terminal and konsole have nonstandard behavior on "clear", see gnome-terminal/vte 506438 and konsole 384218 for details."

@egmontkob thanks for the info! But the bugs you link to got me confused: does that mean Kitty's behavior is intentional / "xterm-compatible"? And if it is, do we agree it's undesirable? By undesirable, I mean that I use Ctrl+L (or clear, which I used in the bug report because it's visible on the GIF) to get a clean slate, e.g. to run unit tests. But that certainly does not mean I want to scrap a partial, arbitrary chunk of scrollback 😄. I do want to preserve this scrollback, e.g. to peek at the previous unit test results. Right?

A succinct sentence for what I expect from Ctrl+L is: give me a clean terminal prompt (by scrolling by a pageful, just enough to have my prompt in the top line), and don't touch the scrollback contents.

Is there a workaround (a config, or maybe a Ctrl+L- overriding map, maybe?) to have this behavior? Am I the only one to feel the current behavior is a bug?

[egmontkob]

the bugs you link to got me confused

If the situation wasn't totally confusing, terminal emulators would have probably fixed it by now so that you wouldn't have reported this bug. :)

does that mean Kitty's behavior is intentional / "xterm-compatible"? And if it is, do we agree it's undesirable?

Yes and yes, the latter in the sense that in my personal opinion, the "other" behavior makes more sense.

But one of the problems is: It shouldn't be the terminal emulators altering their behavior to please the applications running inside. It should be the other way around: apps/utilities being aware of what terminal emulators do, and build a reasonable thing on top of that.

And then if some people believe that the emulator's behavior is unreasonable and change it, or change it to please applications, that's where total messes like this situation arise.

(Disclaimer: I'm not Kitty's author. Let's see what he says.)

[kovidgoyal]

Sorry I am going to be busy for a while, so it will be some time before I can comment.

[egmontkob]

What I accidentally left out of my previous post:

The cleanest way out of this mess would _probably_ be to get bash change its Ctrl+L handling to emit a screenful of newlines first (to effectively scroll out the contets in xterm, kitty, ...) and then change vte and konsole to the standard behavior. I have no clue if this has ever been brought up with bash's developer, and if so, what his response was.

[ronjouch]

"The cleanest way out of this mess would probably be to get bash change its Ctrl+L handling to emit a screenful of newlines first (to effectively scroll out the contets in xterm, kitty, ...) and then change vte and konsole to the standard behavior. I have no clue if this has ever been brought up with bash's developer, and if so, what his response was."

@egmontkob but then you need to convince each shell to do this change 🙂. For example, I use fish.

[egmontkob]

but then you need to convince each shell to do this change

Got any better idea? :)

[kovidgoyal]

Looks to me like kitty is working as it is supposed to. Ctrl+L sends the clear escape code and the standard is very clear about how the clear escape code should work, it must clear the contents of the current screen, not first scroll them up and then clear. I dont know why vte and konsole chose to interpret clear as scroll screen contents+clear instead of just clear.

The solution, IMO is simple. Add another escape code that means "scroll screen contents". Then have your shell map ctrl+l to generate that escape code + clear screen instead of just clear screen.

And since this solution will take a while to be implemented in terminal emulators and shells, here is a workaround you can do in kitty today to get the same behavior:

In kitty.conf

map ctrl+l all kitty +runpy "from kitty.utils import *; cols = screen_size_function()().cols; print('\n' * cols, end='\x0c')"

I haven't fully tested the solution above, so it might need a bit of adjustment, but it works by basically printing screen_height newlines and then the char to clear the screen. If for some reason it does not work, let me know and I will debug it.

@egmontkob if you are interested in implementing such an extra escape code in vte or a similar solution, feel free to continue the discussion. I will be happy to co-operate in coming up with a spec for it.

[kovidgoyal]

And just to note, the solution above is not quite optimal, since if the screen is partially filled it will insert extra blank lines into the scrollback. For the optimal solution, we need a new escape code.

[kovidgoyal]

And i have implemented the optimal solution, whch can be implemented via a mapping in kitty.conf.

If there is interest in creating a standardised escape code for this, as I discussed previously, I'll be happy to participate.

[egmontkob]

I'm not against a standardized (whatever that means) new escape sequence, but how would it solve the current pretty complex problem set?

If you convince shell authors to emit a new escape sequence (subject to feature detection which is another hell), you might as well convince them to emit a screenful of newline characters which is a much simpler and more compatible route to take.

I've taken a quick look at bash, I assume it's lib/readline/display.c _rl_clear_screen() clearing the screen which uses a variable initialized from the cl capability, so why doesn't it clear the scrollback according to terminfo definition? Does readline use its own bundled termcap (there's one in its source tree), or what? (I have no time now to further investigate.)

It's a totally messy situation we are in right now. It's not a new escape sequence that we need, but a complete roadmap to fixing this (which may or may not include a new escape sequence as one of the steps).

[kovidgoyal]

you cant emit a screenfull of newlines, since there could be less than a screenful of contents, which means you'd end up with empty lines in the scrollback. Anyway, I've already solved this issue for kitty, for people that want clear to add screen contents to the scrollback.

[egmontkob]

This is what happens now in VTE and friends. A screenful of empty lines is still better than losing contents (I'm speaking of the user experience here, not whichever technical implementation thereof).

The ideal solution, of course, would be a "smart" escape sequence that scrolls out just as much as required. (How much exactly, what would it depend on? The cursor positioin? If so then what about editing multiline commands (the cursor can be over any of its lines) and pressing Ctrl+L then? Or the onscreen contents? Then how to define "unused lines" vs. "content lines that happen to be empty"?)

The real question is however: if there's such a new escape sequence introduced, would shells really pick it up? How would they know if the emulator supports that (or at least silently ignores, that is, safe to emit)? Can you see the end of the tunnel in this direction, with a higher likelihood than in the direction of shells emitting newlines?

[maximbaz]

Honestly, as a user I always _assumed_ that that's what Ctrl+L does, it puts the current screen into the scrollback. I've never noticed that part of the scrollback is gone after pressing Ctrl+L, that is until this issue was created 🙂

So for me the new config map ctrl+l combine : clear_terminal scroll active : send_text normal,application \x0c is exactly what I intuitively always expected from Ctrl+L, definitely adding that to my kitty.conf.

Maybe worth showing this exact mapping in docs @kovidgoyal, I found it in your commit message 🙂

[kovidgoyal]

@egmontkob yes the escape code would be smart, either based on the cursor position or if you really want to be fancy checking for empty lines from the bottom. See the implementation I just committed for kitty which is based on cursor position. I dont really care if shells pick up the escape code or not. As a terminal developer, providing the escape code is my job, what the rest of the ecosystem does is up to them. In designing the escape code I would simply pick one that is ignored by most existing terminals, then shell can simply send it followed by the codes to clear the screen. So they would continue to work just as tooday in terminals that dont support it and would work as expected in terminals that do. No need for detection.

@maximbaz feel free to send a PR with the addition to the docs. Personally, I've always expected the clear command to not scroll contents into the scrollback :)

[ronjouch]

Thanks Kovid for the fast fix. Added to my config file and tested in a fresh build, works as expected 👍.

"feel free to send a PR with the addition to the docs. Personally, I've always expected the clear command to not scroll contents into the scrollback :)"

@maximbaz rather than adding the command to the docs (or in addition to it), the default configuration file (with commented configuration examples) would be a natural place.

[maximbaz]

rather than adding the command to the docs (or in addition to it), the default configuration file (with commented configuration examples) would be a natural place.

It's the same place, docs and config file are both generated from the same python file 🙂 Created #1121.