kitty 🚀 - Protocol extension: multiple cursors

Personally, I have never needed multiple cursors, but sure, I can see how they might be useful. It isn't particularly hard to design a protocol to do this in kitty, but it should ideally have involvement from somebody who is implementing this feature in some terminal editor, so that the protocol can be designed to meet their needs.

If you can get somebody like that to join the discussion, I will be happy to work with them on designing and implementing such a protocol in kitty.

I'll leave this issue open for a while, lets see if there is buy-in from editor developers.

kovidgoyal on 12 Jul 2018

👍6

Good news! Let me ping a few kakoune's contributors directly, to see if anyone is interested in getting the native support for multiple cursors from terminal emulators: @mawww, @lenormf, @Delapouite, @alexherbo2, @casimir, @danr, @ekie, @occivink.

maximbaz on 12 Jul 2018

Would this be relevant for the vis editor? @martanne

erf on 13 Jul 2018

Hello,

For Kakoune, I believe an easy approach for that would be an attribute, similar to underline/bold/itallic... This attribute would just mean "highlight each character as-if they had the cursor on top of them". This would stay separate from the "real" hardware cursor, which would remain at the hardware cursor position and would be displayed according to the current cursor display mode. The cursor attributes would not respect the current cursor display mode, as it could make sense to hide the real cursor while still display various characters with the cursor attribute, we might even want multiple cursor style attributes (block/vertical line), combined with existing blinking attribute.

The tricky part for Kakoune is that we use ncurses for output, which unfortunately does not give us a way to use arbitrary attributes, so before being able to support this, we would need to rewrite the terminal UI code to replace ncurses (at least for the display parts).

Those are my quick thought on that, but to be clear, I dont consider this a priority in Kakoune at the moment, and it depends on replacing (parts of) ncurses, which is a significant amount of work.

mawww on 17 Jul 2018

That has a few concerns:

1) What happens when you erase the character at a location with a cursor attribute set? Does the cursor remain? Does it get deleted? Similarly what happens when you write a new character into that location?
What happens when you insert a character somewhere before in the line? Does the cursor move?

2) How does it work? Character attributes in terminals typically work by setting the attributes on the cursor, and those attributes are copied into the cell when you output new characters. So to render your cursor you would need to move the hardware cursor to each location and output the character at that location again.

3) From an implementation perspective, it is not ideal, because while it works well for block cursors which are basically rendered by changing the fg/bg colors during normal character rendering, for other types of cursors, it would mean an whole extra scan of every cell to check if it contains a special cursor, in every rendering pass.

I think a better design is to have a dedicated escape code to create/style/position/delete extra cursors. This has the additional advantage that you dont need to change our ncurses based code at all. Simply write the extra escape codes to stdout in between calls to ncurses' API.

Offtopic and a bit ranty: ncurses is a horrible library, I strongly urge you to get rid of it anyway for your own future sanity.

kovidgoyal on 17 Jul 2018

👍1

Regarding your concerns, I was thinking just do exactly as the terminal does for any other attributes, my idea was that displaying characters as-if a cursor was there would work exactly like displaying say an underline attribute on a single character.

In Kakoune display code, we only know of the cursor position in order to tell the terminal at the end of redraw where to place the hardware cursor so that it can use that as a hint for its eventual widgets (such as displaying an emoji picker, or placing the input method window OnTheSpot...). The rendering code itself just paints the full window with ncurses, and cursors have no special semantics, they are drawn exactly as the rest (Kakoune display structure is basically a list of display lines, themselves being a list of display atoms themselves being a tuple of a string, fg/bg colors and attributes).

For the implementation side, I am not sure how kitty paints its cursor, but couldn't it reuse the cursor painting logic and apply it if the cursor attribute is on when it paints a cell ? No need for a second pass.

mawww on 17 Jul 2018

The problem is that different attributes are treated differently in various situations. For example, reverse video has no effect, background may or may not change depending on the bce terminfo property, underline may or may not take effect depending on the character drawn, etc. So the semantics of this "cursor attribute" would need to be defined carefully keeping in mind interactions with all other terminal escape codes.

As for implementation, kitty does not draw individual cells, it is GPU based, which means it draws all cells at once, in parallel on the GPU. So while this is fine for block cursors, which as I said are drawn as normal cells, just with colors changed, for non block cursor there is a separate draw call needed. This separate draw call currently draws only a single cell where the hardware cursor is. With your proposal it would need to draw all cells, since it cannot know which cells have special cursors in them. And worse, it would have to do that all the time, even if no special cursors are being used, which will be the case the vast majority of the time.

With my proposal, just as you tell the terminal where to put the hardware cursor after drawing the lines, you tell it where to put the extra cursors. This is low overhead for you (you dont have to maintain an extra attribute per character which is used only rarely) and low overhead for the terminal, for the same reason. Of course, it does mean that it works differently from how you do special cursors now, which I guess, is why you dont like it.

kovidgoyal on 17 Jul 2018

And I would like to add, semantically, a cursor is not a text attribute. A cursor is instead an element of the UI that is independent of the text under it. Treating it as a text attribute seems wrong to me.

kovidgoyal on 17 Jul 2018

I still dont quite understand why drawing a cursor on top of a cell cannot be done while the cell is drawn, even when done on a GPU, how is that fundamentally different from being able to draw a colored wavy underline on that cell ?

Placing cursors after the fact seems pretty complex to me, we now have some additional state to track between the application and the terminal emulator, which is more potential bugs, and we need a new protocol to control those (whats the lifetime of those cursors ? do they disappear if I repaint the cell ? Does the escape sequence that creates them replaces all the existing ones, or add new ones ? etc...)

Using an attribute is pretty easy, and avoid inventing something new (although it seems there is still a few questions to answer, I was not aware that terminals have different behaviours on cell erase depending on the attributes the cell contains). The alternate implementation would not be difficult to write though.

And finally, the semantics of cursors in a fullscreen terminal application really only concerns the application and the user, I dont really see what the terminal could be doing besides displaying those, which is why I though attributes would make sense (the hardware cursor though has semantic meaning for the terminal, it tells it where to draw, and can act as a hint for where user attention is located, which is why I proposed to keep it as-is).

All that said, I do not care that much, as I already said this is not a priority for Kakoune at the moment.

Cheers.

mawww on 17 Jul 2018

On Mon, Jul 16, 2018 at 11:22:00PM -0700, Maxime Coste wrote:

I still dont quite understand why drawing a cursor on top of a cell cannot be done while the cell is drawn, even when done on a GPU, how is that fundamentally different from being able to draw a colored wavy underline on that cell ?

Underlines are drawn as special characters. I suppose I could draw
cursors as special characters as well, but that comes with a new set of
overheads, essentially on every render call a byte of extra data would
need to be sent to the GPU, and the GPU would need to perform several
extra calculations per pixel on every render. The point is not so much
that this cannot be done, but that it adds extra overhead in the case
of no special cursors, which is by far the common case. I would like a
solution that means the CPU/GPU are doing no significant extra work when there are
no extra cursors.

Placing cursors after the fact seems pretty complex to me, we now have some additional state to track between the application and the terminal emulator, which is more potential bugs, and we need a new protocol to control those (whats the lifetime of those cursors ? do they disappear if I repaint the cell ? Does the escape sequence that creates them replaces all the existing ones, or add new ones ? etc...)

Any text attribute is also state that needs to be tracked between the
application and the emulator, there is no difference there. The lifetime
of the cursor is simple: It exists until deleted by the application or
on transition to/from alternate screen mode or on reset.

Repainting the cell has no effect on the cursors, just as it has no
effect on the hardware cursor. The extra cursors are semantically
indistinguishable from hardware cursors. This seems like the most
natural behavior.

Using an attribute is pretty easy, and avoid inventing something new (although it seems there is still a few questions to answer, I was not aware that terminals have different behaviours on cell erase depending on the attributes the cell contains). The alternate implementation would not be difficult to write though.

See the discussion of background color erase in the ncurses FAQ for some
of the complexity of this.
http://invisible-island.net/ncurses/ncurses.faq.html#bce_mismatches

And finally, the semantics of cursors in a fullscreen terminal application really only concerns the application and the user, I dont really see what the terminal could be doing besides displaying those, which is why I though attributes would make sense (the hardware cursor though has semantic meaning for the terminal, it tells it where to draw, and can act as a hint for where user attention is located, which is why I proposed to keep it as-is).

That's true, however, I dont see what it has to do with the mechanism of
transmission of the location of the extra cursors. Using either my
proposal or yours the terminal still knows where the extra cursors are
and still has to draw them. The only difference is, in one case that data
is tied to text state which is semantically orthogonal to UI state and
cursors are UI not text.

kovidgoyal on 17 Jul 2018

I would like a solution that means the CPU/GPU are doing no significant extra work when there are
no extra cursors.

Aren't we already paying that price to support wavy underlines ? I assume you disable that if there are no wavy underlines to display, which you could do as well if there was no extra cursor to display (the real hardware cursor does not need to go through this path, it could still be handled as it is right now).

Any text attribute is also state that needs to be tracked between the application and the emulator, there is no difference there. The lifetime of the cursor is simple: It exists until deleted by the application or on transition to/from alternate screen mode or on reset.

Text attributes is state we already know how to track both on application and terminal side, tracking additional cursors is more involved, imagine an app with multi-cursor crashes and hence does not remove them, the user now has a shell with multiple cursors displayed on the screen, only one of which is relevant.

Repainting the cell has no effect on the cursors, just as it has no effect on the hardware cursor. The extra cursors are semantically indistinguishable from hardware cursors. This seems like the most natural behavior.

Do you mean that outputting a character to the terminal would insert it at every cursor position, and move those cursors by one cell ? I dont think it would be manageable, I think only the hardware cursor should have those semantic properties, the other cursors are strictly for display (which is why in my view attributes makes a lot of sense, it just means "display those cells as-if there were cursors on them").

mawww on 17 Jul 2018

On Tue, Jul 17, 2018 at 12:15:53AM -0700, Maxime Coste wrote:

I would like a solution that means the CPU/GPU are doing no significant extra work when there are
no extra cursors.

Aren't we already paying that price to support wavy underlines ? I assume you disable that if there are no wavy underlines to display, which you could do as well if there was no extra cursor to display (the real hardware cursor does not need to go through this path, it could still be handled as it is right now).

The thing about GPUs is that they dont like branches (it breaks
parallelism). So underlines are implemented as a blend of colors with an
alpha which is set to zero when no underline is present. In order to do
cursors that way, would imply an extra blend operation per pixel (EDIT: and an extra texture sample per pixel)

Any text attribute is also state that needs to be tracked between the application and the emulator, there is no difference there. The lifetime of the cursor is simple: It exists until deleted by the application or on transition to/from alternate screen mode or on reset.

Text attributes is state we already know how to track both on application and terminal side, tracking additional cursors is more involved, imagine an app with multi-cursor crashes and hence does not remove them, the user now has a shell with multiple cursors displayed on the screen, only one of which is relevant.

Exactly the same thing would happen if the application crashes while
using a cursor text attribute. The screen would still have lots of extra
blinking cursors. In general, terminals are not robust against
application crashes, you pretty much always have to reset the terminal
after a crash.

Repainting the cell has no effect on the cursors, just as it has no effect on the hardware cursor. The extra cursors are semantically indistinguishable from hardware cursors. This seems like the most natural behavior.

Do you mean that outputting a character to the terminal would insert it at every cursor position, and move those cursor by one cell ? I dont think it would be manageable, I think only the hardware cursor should have those semantic properties, the other cursors are strictly for display (which is why in my view attributes makes a lot of sense, it just means "display those cells as-if there were cursors on them").

No, extra cursors are entirely unaffected by text operations in any way.
Repainting a cell, outputting a character, erasing the screen, erasing
lines, words, characters, etc, all have no effect
on the cursor. The one exception is, perhaps, scrolling the screen,
though I can see arguments for either way in that case.

kovidgoyal on 17 Jul 2018

Hi guys,

The issue starts by saying "More and more text editors", yet, surprisingly to me, only one terminal emulator is involved in the discussion. Do you guys intend this to be a Kitty-only feature, or one hopefully adopted by other notable terminal emulators as well? In the former case, let me know and I'm outta here leaving the rest up for you to discuss :-) In the latter case, my (vte/gnome-terminal developer) thoughts so far:

On one hand this sounds like a bit of an overkill to me, on the other hand I can also imagine other useful situations. E.g. even a relatively simple text editor such as "joe" can show multiple viewports to the same file at once, it would be nice if the cursor was shown in all of them (or would it be distracting, since the user couldn't be sure which one was the main one, i.e. the one which will start scrolling to keep the cursor inside the viewport, and where special operation (such as Find) will work?). Tmux-like apps could have synchronized (multiplexed) input sent to all panes, and show the cursor in all.

Saving a few CPU/GPU cycles in one particular terminal emulator should be the very last factor in designing this feature. With all respect, I just cannot care at all which approach saves a few row scans or whatever operatoins in Kitty (or in VTE, or any other emulator). We should go for the design that's simple, clean, builds on top of what we already have, makes it relatively easy for apps to adapt this new feature.

I don't really care about ncurses either, and I don't care about hacks where an app uses a mixture of ncurses and off-ncurses output. Ncurses already has quite a few limitations that make it unsuitable for most "big" projects. E.g. due to its palette approach, it's impossible to display arbitrary (not predefined) colors as they arise (e.g. "mcview" cannot display ANSI colors like "less -R" does, "watch" cannot support 256 colors etc.), and then we haven't even talked about true colors. Then there's the colored and wavy underline, the explicit hyperlinks (VTE and iTerm2; not yet in Kitty), and presumably more. Any application that wishes to use multiple cursors probably sooner or later will also wish to use some of these as well, so will have to switch to its own screen drawing (or some other library) anyway.

I think you guys have already agreed that there would still be only one "main" cursor (where new incoming data is printed etc.), the rest are "extra" cursors. I concur. Usually the user probably wouldn't be able to tell which is the main one, but there might be exceptions. E.g. when you press Ctrl+Shift+U to enter a Unicode character, Kitty brings up its own stuff at the top of the window, so this would be unaffected; however, VTE presents an inline box at the cursor, I think it would still go to the "main" one. Other emulators have visual aids helping you locate the cursor (all the time, or upon a keypress), they might decide to highlight the main one only, not all of them. In VTE we experimented with hiding the cursor on focus out as pretty much every graphical app does; even though we dropped this idea, it might make sense (I'm not sure, we'd need to try it out) to hide the "extra" cursors on focus out.

With this hidden concept of "main" vs. "extra" cursors, the extra cursors carry some semantics in the applications (text editors) towards the users, but they don't have any semantics for the terminal emulator. For the terminal emulator it's just a visual attribute.

As for the overall design (just another attribute, or a totally different kind of escape sequence updating and maintaining the extra cursors' location) I'm firmly in favor of the "just another attribute", that is, mawww's approach.

[kovidgoyal] 1. What happens when you erase the character at a location with a cursor attribute set?

With this approach, this question is already answered. The same should happen that to any other attribute, there's absolutely no point in treating this attribute differently. The background color wrt. bce is a bit special so it deserves a sentence: I don't see any point in applying the new "make it look like as if it was a cursor" attribute to the rest of the line in case of a bce, so still the background color should be the only special one here, the new attribute should behave just like the existing ones (e.g. underline, strikethrough) do and not get applied here.

[kovidgoyal] 2. How does it work? [...] So to render your cursor you would need to move the hardware cursor to each location and output the character at that location again.

That's exactly what happens now with any attribute that an app needs to modify, so I don't see any problem here.

[kovidgoyal] The problem is that different attributes are treated differently in various situations. For example, reverse video has no effect [...], underline may or may not take effect depending on the character drawn, etc. [...]

In VTE there's no special handling of reverse or underline anywhere. Why is there in Kitty? Do we miss anything? I believe all our non-background attributes have the very same handling.

[kovidgoyal] With my proposal, just as you tell the terminal where to put the hardware cursor after drawing the lines, you tell it where to put the extra cursors. This is low overhead for you (you dont have to maintain an extra attribute per character which is used only rarely) and low overhead for the terminal, for the same reason. Of course, it does mean that it works differently from how you do special cursors now, which I guess, is why you dont like it.

It seems to me that for you, "low overhead" means saving a bit, or some CPU cycles. For me, "low overhead" primarly means cleaner code, saving developer time, and hence reusing existing pieces wherever appropriate rather than coming up with something brand new.

[kovidgoyal] The lifetime of the cursor is simple: It exists until deleted by the application or on transition to/from alternate screen mode or on reset.

So we'd need two brand new escape sequences: Placing an extra cursor at a specific position, and deleting. We'd even need a "delete all", for when the application repaints its screen, e.g. to repair a potentially damaged one. Then maybe we don't even need a "delete one". Plus, figure out how these positions are affected by text being output or deleted, by scrolling (and scroll regions), by clear-to-eol, by window resize and so on and so forth. Sounds magnitudes more complex and error-prone (e.g. there's not even a "standard" terminal emulator to follow; what if two emulators implement some corner case differently and it causes troubles later on?) to me than assigning let's say ANSI 66 and 67 to enable/disable "extra cursor" mode and just doing everything the same way we do for other attributes.

A few more points:

What would happen on the normal screen? Would extra cursor be supported at all? Could it scroll out to the scrollback buffer? What would happen to it on rewrap-on-resize?

Repainting the screen after a text editor operation (e.g. page down) can take a few kilobytes of traffic, which might get stuck over ssh. With off-channel communication for the cursor positions, this might easily result in intermittent visual glitches, the extra cursor being shown at the wrong place. With in-channel communication I believe the visual experience would also be better.

[mawww] For Kakoune, I believe an easy approach for that would be an attribute, similar to underline/bold/itallic...

I think it's not just for Kakoune, but for most apps out there. If they have an existing infrastructure of tracking where underline/bold/italic are present, it's way easier to just add yet another attribute to this infrastructure, rather than coming up with a new one for the sake of the cursors. It feels to me that this approach is not only cleaner, but is also easier to implement (replacing ncurses put aside) in most applications, and most terminal emulators (sorry to hear if Kitty is not one of these).

egmontkob on 17 Jul 2018

👍1

The issue starts by saying "More and more text editors", yet, surprisingly to me, only one terminal emulator is involved in the discussion. Do you guys intend this to be a Kitty-only feature, or one hopefully adopted by other notable terminal emulators as well? In the former case, let me know and I'm outta here leaving the rest up for you to discuss :-) In the latter case, my (vte/gnome-terminal developer) thoughts so far:

My bad, I simply didn't know where to open such discussion, and since I'm using kitty and I saw it already implement a few protocol extensions, I decided to propose here yet another one.

Of course this is intended to be adopted by other terminals as well, so your input is very valuable 👍

maximbaz on 17 Jul 2018

👍1

I simply didn't know where to open such discussion, and since I'm using kitty and I saw it already implement a few protocol extensions, I decided to propose here yet another one.

Fair enough, actually there's no right place (other than cross-posting to a few). Kitty is probably the "bravest" emulator now, experimenting with new features, so it's a great place to start. I've notified VTE's main developer, as well as iTerm2's developer. Thanks for posting this idea!

egmontkob on 17 Jul 2018

❤1

On Tue, Jul 17, 2018 at 04:59:07AM -0700, egmontkob wrote:

Hi guys,

The issue starts by saying "More and more text editors", yet, surprisingly to me, only one terminal emulator is involved in the discussion. Do you guys intend this to be a Kitty-only feature, or one hopefully adopted by other notable terminal emulators as well? In the former case, let me know and I'm outta here leaving the rest up for you to discuss :-) In the latter case, my (vte/gnome-terminal developer) thoughts so far:

You are welcome to contribute to the discussion.

Saving a few CPU/GPU cycles in one particular terminal emulator should be the very last factor in designing this feature. With all respect, I just cannot care at all which approach saves a few row scans or whatever operatoins in Kitty (or in VTE, or any other emulator). We should go for the design that's simple, clean, builds on top of what we already have, makes it relatively easy for apps to adapt this new feature.

It's fine that it does not matter to you, it does however matter to me.
So let's not just dismiss it out of hand. Not everyone is happy with
just waving their hands and saying its a few CPU cycles, who cares.
Those few CPU cycles you so casually dismiss for every feature (I've
heard this argument form you before) add up over time. For proof of
that, just compare the CPU usage of kitty+X when scrolling a file in
less: https://sw.kovidgoyal.net/kitty/performance.html

Now to the rest. Rather than responding point-by-point, let me
summarize how the two proposals will work in practice, from the
perspective of the bytes to send to the terminal.

Let's take the use case illustrated by @maww namely having the data
stored in lines which contain runs of formatted text. Suppose we want an
extra cursor at column 2 in the line ABCD

1) Using a text attribute
Bytstream
A attribute>B CD

2) Using an escape code
A
BCD

So, in this case my proposal is actually simpler to implement for an
application developer.

Now suppose that the editor wants to exit multiple cursor mode:

1) using a text attribute
Two possible approaches:
a) Redraw entire screen
b) manually move the cursor to every extra cursor location and output
the character there again without the cursor text attribute.

2) Using a separate escape code
Simply output the escape code to delete all extra cursors

Again, my proposal is much simpler.

Finally, suppose that the editor wants to change the set of cursors
displayed, hide some, move some, etc.

1) Using text attribute
Redraw entire screen

2) Using separate escape code
Hide all cursors and then either redraw entire screen or just redraw the
cursors

In this case it is pretty much a wash.

I think this conclusively demonstrates that my proposal is actually
easier to implement than using text attributes for application
developers (putting aside issues with ncurses, which I agree with
@egmontkob are irrelevant).

Now to address a few points raised by @egmontkob in detail:

P1) there needs to be only a single escape code in my proposal. It can
take parameters to decide what effect it has, i.e. creating a new
cursor, deleting a cursor, deleting all cursors, whatever.

P2) extra cursors in my proposal have no interactions with
erasing/deleting/printing characters lines etc. So there are no
complications there that emulator developers can get wrong.

P3) Regarding normal screen and scrollback
In my proposal you can have a separate set of cursor for the normal and
alternate screens and they obviously dont move into the scrollback
buffer, since they are not text attributes. If you do want to make them
text attributes, you would need to waste even more CPU cycles sanitizing
lines before copying them into the scrollback.

And finally, let me re-iterate my main concern, that no one seems to
want to address:

Cursors are not text attributes. They are part of the UI. In no other
display system is cursor information carried by text attributes.
Nowhere, ever. The very idea of putting cursor information into text is
a horrible hack and requires extra-ordinary justification.

kovidgoyal on 17 Jul 2018

👍2

And let me add, that if there is some use case I missed in my bytestream demonstration above, feel free to point it out, and I will attempt to show how it is easier to do using dedicated escape codes rather than overloading text attributes.

And just to reinforce the point, @mawww my proposal allows you to continue storing extra cursor information as text attributes in you application, if that is what you want to do. The terminal emulator does not care what internal data strucutre you use.

This discussion should be about the protocol between the application and the terminal emulator and the interactions of the extra cursors with other terminal commands such as erase, scroll, print characters, etc.

kovidgoyal on 17 Jul 2018

Sticking to the "just another attribute" philosophy for a while, which I'll get back to later:

1) Using a text attribute [...] So, in this case my proposal is actually simpler to implement for an application developer.

No, it's not easier to implement. It involves fewer bytes of traffic. It would be easier to implement if it was the very first attribute to handle. But assuming that other attributes are already handled the same way, it's easier to stick to that same method, rather than coming up with something new. You miss the fact that the first variant is already implemented and hence doesn't need to be implemented again, while the second isn't yet.

[...] Again, my proposal is much simpler.

Again, it would be simpler if either approach would need to be implemented from scratch. That isn't the case. One is already implemented, while the other one isn't yet.

Finally, suppose that the editor wants to change the set of cursors displayed, hide some, move some, etc. 1) Using text attribute Redraw entire screen [...]

For some mysterious reason here you forgot about the possibility of redrawing only the affected characters. Nevermind.

And finally, let me re-iterate my main concern, that no one seems to want to address: Cursors are not text attributes.

I have to admit that I missed this bit, or more precisely, missed the strong emphasis you put on this approach.

Given that block cursors are a legacy thing, the cursor in all modern apps is a vertical bar _between_ characters, which is reasonably well emulated using the I-beam cursor shape in terminal emulators, I have to stop here for a while and let this approach sink in. I think you might have convinced me (making our previous debate pointless).

While now I agree that this is the right philosophy, I still believe it makes implementation harder (something new needs to be designed and implemented, rather than reusing something that already exists), and hence it's not necessarily the most practical choice, but probably the correct one. I'm not yet explicitly in favor of this approach (give me a few days to ponder about this), but I'm okay with it.

So to move forward from here, I guess we'd need escape sequences to some of these (which ones exactly?) and answers to the questions:

Leave a "trail", place an extra cursor where the real cursor currently is.
Place an extra cursor at an arbitrary (x,y) position.
Would printing data over an extra cursor (or if thinking in I-beam shape: after the extra cursor) leave it there, or would it wipe it out?
Remove the extra cursor from the real cursor's current location.
Remove the extra cursor from an arbitrary (x,y) position.
Remove all extra cursors.
How would an extra cursor be affected if the content underneath moves in any of the four directions? (Vertical scroll, or a previous character in the line inserted/deleted.) Would the cursor follow them, or stay at the same absolute place, or get wiped out?
Would the (x,y) positions affected (and bound) by the scrolling region?
What to do on resize? Is it okay to wipe them out all (as e.g. the scroll region is reset), expecting the app to do a full repaint anyway?
Nit: How would a cursor positioned to the middle of a wide character be handled?
Nit: With I-beam shape in mind, would it be possible to place a cursor after the last column?
Nit: Would all extra cursors automatically share the main one's properties (shape, color, blinking)? Or could they each have their own ones?
Anything that I'm missing or overcomplicating? :)

egmontkob on 17 Jul 2018

On Tue, Jul 17, 2018 at 06:55:10AM -0700, egmontkob wrote:

Sticking to the "just another attribute" philosophy for a while, which I'll get back to later:

1) Using a text attribute [...] So, in this case my proposal is actually simpler to implement for an application developer.

No, it's not easier to implement. It involves fewer bytes of traffic. It would be easier to implement if it was the very first attribute to handle. But assuming that other attributes are already handled the same way, it's easier to stick to that same method, rather than coming up with something new. You miss the fact that the first variant is already implemented and hence doesn't need to be implemented again, while the second isn't yet.

Already implemented by what? Nothing I know of. To take the concrete
example in this thread of kakoune, it uses ncurses, so to implement this
it would either need to ditch ncurses or get ncurses to implement it. In
either case whoever does the implementation, it is easier to output fewer
escape codes than more escape codes.

As I mentioned to @mawww in my follow up post, he can keep using extra
cursors as text attributes in his internal implementation if he likes to
do that, the only difference between the two approaches is in one case
you output more new escape codes than in the other. So when he is
translating his internal text attributes into escape codes for the
terminal he would do less work with a dedicated escape code.

The point is that his total draw cycle appears to be to redraw the
entire line on changes. In this use case, as I demonstrated, using a
dedicated escape code is better or at least no worse than using a new
text attribute.

BTW @mawww apologies if you are female, please read the above he as
she.

[...] Again, my proposal is much simpler.

Again, it would be simpler if either approach would need to be implemented from scratch. That isn't the case. One is already implemented, while the other one isn't yet.

See above.

Finally, suppose that the editor wants to change the set of cursors displayed, hide some, move some, etc. 1) Using text attribute Redraw entire screen [...]

For some mysterious reason here you forgot about the possibility of redrawing only the affected characters. Nevermind.

That is exactly the same as redrawing only the affected cursors with the
escape codes and is why I concluded this use case is wash. Apologies for
neglecting to spell it out.

And finally, let me re-iterate my main concern, that no one seems to want to address: Cursors are not text attributes.

I have to admit that I missed this bit, or more precisely, missed the strong emphasis you put on this approach.

Given that block cursors are a legacy thing, the cursor in all modern apps is a vertical bar _between_ characters, which is reasonably well emulated using the I-beam cursor shape in terminal emulators, I have to stop here for a while and let this approach sink in. I think you might have convinced me (making our previous debate pointless).

While now I agree that this is the right philosophy, I still believe it makes implementation harder (something new needs to be designed and implemented, rather than reusing something that already exists), and hence it's not necessarily the most practical choice, but probably the correct one. I'm not yet explicitly in favor of this approach (give me a few days to ponder about this), but I'm okay with it.

Sure take your time, questions like this deserve due consideration.

So to move forward from here, I guess we'd need escape sequences to some of these (which ones exactly?) and answers to the questions:

Leave a "trail", place an extra cursor where the real cursor currently is.

Not sure what you mean by this? The real cursor will remain where it is.
Are you saying, you would like to distinguish it from other cursors? If
so isn't the point of multiple cursors that they all behave identically,
so that if you insert text, it is inserted at all cursor positions?

Place an extra cursor at an arbitrary (x,y) position.

Easy to do just have the escape code take an extra optional parameter or
require the application to move the real cursor to place the extra
cursor.

Would printing data over an extra cursor (of if thinking in I-beam shape: after the extra cursor) leave it there, or would it wipe it out?

I strongly vote for leave it there, otherwise every time your print a
character you have to check if there is an extra cursor there. And then
if you print characters in "insert mode" you have to check for extra
characters all down the line and potentially cascading all down the
screen thanks to line wrap.

Remove the extra cursor from the real cursor's current location.

Not sure what you mean here? If there is both a real cursor and an extra
cursor at some location the terminal will draw only the real cursor.

Remove the extra cursor from an arbitrary (x,y) position.

Again no problem, simply add a parameter to the escape code. By the way
for how I think this escape code should be designed see the graphics
escape code that kitty uses, it also takes lots of parameters.
https://sw.kovidgoyal.net/kitty/graphics-protocol.html

Remove all extra cursors.

Same, parameter to escape code.

How would an extra cursor be affected if the content underneath moves in any of the four directions? (Vertical scroll, or a previous character in the line inserted/deleted.) Would the cursor follow them, or stay at the same absolute place, or get wiped out?

IMO the extra cursor should not be affected at all. This is the simplest
way and the one least likely to be implemented incorrectly. The one
exception is perhaps vertical scroll, but I dont feel strongly about it
either way.

Would the (x,y) positions affected (and bound) by the scrolling region?

IIRC (and I could be wrong) scrolling regions only affect cursor
movement commands. There are no movement commands as such for extra
cursors, so the question does not arise. (Instead to move a cursor, you
remove and re-create it).

What to do on resize? Is it okay to wipe them out all (as e.g. the scroll region is reset), expecting the app to do a full repaint anyway?

I am OK with either wiping them all or just bounding them in the new
size as you do with the real cursor.

Nit: How would a cursor positioned to the middle of a wide character be handled?

The same way the real cursor is handled in this case, i.e. it would
stretch over two cells if block or replace and one cell if beam (it is
an error to place a cursor in the middle of a wide character).

Nit: With I-beam shape in mind, would it be possible to place a cursor after the last column?

I dont see the point, I think this should be the same as how the real
cursor is handled.

Nit: Would all extra cursors automatically share the main one's properties (shape, color, blinking)? Or could they each have their own ones?

I am in favor of them all being the same. That is the use case needed
for editors and allowing them to be different is a fair bit of overhead
for no actual use case.

Anything that I'm missing or overcomplicating? :)

There's the question of clear screen and reset. However,
let's leave the fine details to the second stage after we have consensus
on the basic question of whether to implement this or not and if so,
whether to use text attributes or extra escape codes.

See my next post.

kovidgoyal on 17 Jul 2018

So for the moment I propose we focus on Stage 1, which is answers to the
following questions:

1) Editor developers: Is this something you care enough that you are
likely to implement this in the reasonably near future if some terminal
emulators add support for it?

2) Terminal emulator developers: Is this something you are willing to
implement in principle? I say yes for kitty.

3) Everybody: Do we want to use a new text attribute or a new escape
code to implement

kovidgoyal on 17 Jul 2018

👍2

Though multicursor support is in the works for Nvim, I don't yet see a need for special terminal features.

kakoune has to hide the terminal cursor completely and "draw" its own cursors by manipulating background color of a cell. This is ugly hack that prevents people from using the much superior native cursor with all of its features (different shape, blinking, proper color).

The way I envision multicursor is a "primary" cursor plus other "marker" (secondary) cursors. The secondary cursors don't need to look identical to the primary cursor, they are just visual clues. I would guess that a human is not greatly aided by all of the cursors looking like the primary cursor, rather it may _increase_ confusion.

Actually, I'd be more interested in new text attributes as @mawww hinted. E.g., ability to display (1) a hollow "box" around a cell, (2) a bar on the left side, and (3) bar on the right side. Entirely orthogonal to cursor(s).

justinmk on 17 Jul 2018

iTerm2 author here: I'm fine with adding this, but I don't want it to be an attribute. I only have a few bits of attributes left before my struct needs to grow. That will increase memory usage and decrease cache efficiency for an attribute that would almost never be set. That's an implementation detail, but it's an important one.

More generally, it's easy to draw an analogy between the "real" cursor and the list of "virtual" cursors. It's a thing that has a coordinate that gets drawn on top of the text and there are some rules for how it's drawn that are configurable both by the user's preferences (e.g., color, shape) and by escape sequences (shape).

Provided we go with a list of cursors, I would be happy to implement it. I think we'd need escape codes to:

Set the number of virtual cursors
Position virtual cursor i at x,y
Set the style (box, line, bar) of virtual cursor i

I would also propose that virtual cursors change position only when told to and have no automatic behavior. This pushes the complexity into the app that wants them, but makes it much more predictable in the face of scrolling regions and double-width characters. However, I'm not quite sure how this would work with an ncurses-based app. Someone who has more experience can perhaps comment here.

gnachman on 17 Jul 2018

👍2

The secondary cursors don't need to look identical to the primary cursor, they are just visual clues. I would guess that a human is not greatly aided by all of the cursors looking like the primary cursor, rather it may increase confusion.

I'm not so sure, I would say it is quite rare when I actually need to know which of the cursors is the primary one. Look in the gif in the first post, I'm manipulating a few lines and I want the same manipulation to happen in all lines, regardless of where is the primary cursor position. I would much prefer to see _all cursors_ turn into identical vertical bar when I'm about to prepend the word "private" on every line, because this would indicate to me that I'm about to insert some text on all of these lines.

Sometimes it is helpful to know where the primary _selection_ is, not the cursor (at least in kakoune), because there are some extra actions you can do with the primary selection (search for the word "main" here), but no actions to do with the primary cursor — and the primary selection is highlighted by using the white text color.

However the above is based on my experience with kakoune, I realize that the situation might be different in other editors.

maximbaz on 17 Jul 2018

[justinmk] E.g., ability to display (1) a hollow "box" around a cell, (2) a bar on the left side, and (3) bar on the right side. Entirely orthogonal to cursor(s).

SGR 51, 60, 62 might be relevant here.

[justinmk] The secondary cursors don't need to look identical to the primary cursor, they are just visual clues.

[maximbaz] I'm not so sure, I would say it is quite rare when I actually need to know which of the cursors is the primary one.

I can easily think of use cases for either approaches. Maybe the best is if we consider supporting both kinds straight from the beginning.

[gnachman] I would also propose that virtual cursors change position only when told to and have no automatic behavior.

I guess I'd rather they scroll along with the content. Otherwise e.g. cat'ing a binary file may result in visual glitches (additional cursors) that don't even scroll out as you tap Enter.

egmontkob on 17 Jul 2018

👍1

[kovidgoyal] Already implemented by what? [...] In either case whoever does the implementation, it is easier to output fewer escape codes than more escape codes.

You still don't understand what I'm speaking about. This will probably be my last attempt in explaining.

Let's assume a non-ncurses based editor (there are plenty of them). Presumably it already handles bold, italic, underline etc. By some internal means it figures out which cells it wants to render with such attributes, and then implements telling this to the terminal emulator. Given this current state, it's much easier to add another attribute that follows this pattern, than an attribute that uses a totally different approach. Easiness doesn't mean fewer bytes to be emitted, easiness doesn't mean something that would be easier to implement from scratch, easiness means easier to implement and maintain given the current codebase, that is, less deviations and less additional code to introduce, reusing whatever's already available (i.e. the way bold/italic/underline are handled by this app).

egmontkob on 17 Jul 2018

I guess I'd rather they scroll along with the content. Otherwise e.g. cat'ing a binary file may result in visual glitches (additional cursors) that don't even scroll out as you tap Enter.

To be sure they should not be allowed to leave the mutable section. I don't think that additional cursors make sense outside an interactive application (i.e., one that would use the alternate screen buffer), so their behavior in the face of cat is not that interesting—no matter what you do it's hopelessly broken. What I want to avoid is having to define their behavior within and without horizontal and vertical scrolling regions. It's a big mess that's hard to test, and we'd likely end up with diverging interpretations of the spec. Interactive apps would rarely scroll the whole screen anyway.

I'm not persuaded that attribute-vs-not-an-attribute makes a meaningful difference to applications, except for backward compatibility with terminals that don't support multiple cursors. The cost to adding an attribute to terminal emulators is significant (assuming others have a similar design to the two I'm familiar with, iTerm2 and tmux).

gnachman on 17 Jul 2018

[gnachman] To be sure they should not be allowed to leave the mutable section.

That's what I had in my mind, too. I guess we agree here.

Indeed my wording wasn't specific enough. I meant I'd prefer these additional cursors to scroll, so that they disappear when they reach the top of the mutable section; rather than stay at the same absolute position, cluttering my work, until a more brutal reset-like operation is issued.

egmontkob on 17 Jul 2018

[kovidgoyal] Are you saying, you would like to distinguish [the real cursor] from other cursors? If so isn't the point of multiple cursors that they all behave identically, so that if you insert text, it is inserted at all cursor positions?

No way!

I mean, that's the behavior that apps are likely to implement, by emitting the proper output that makes the new text appear at all such position. But the terminal emulator shouldn't automatically do it.

I must have misunderstood you when I said:

I think you guys have already agreed that there would still be only one "main" cursor (where new incoming data is printed etc.), the rest are "extra" cursors. I concur.

There's no way multiple equivalent cursors, all processing the incoming data, could work reasonably. There has to be exactly 1 main cursor (as it is now), and extra ones that are display-only, without any cursor functionality.

Processing input at all cursors would lead to at least the following bunch of problems:

In probably most cases, it's not the very same action (and the only action) that needs to be performed by applications when multiple cursors are present and the user hits a key. E.g. take a look at the opening request's video, when the word "private" is being typed. At each character of this word, all three cursor lines can probably be updated using the same sequence (e.g. ICH followed by the new letter), however, notice that there's also a bottom line showing some data that also changes. In order to update the bottom line, you'd need to remove two cursors so that you're left with only one, then update at the bottom, then recreate the two additional ones. Anything similar, e.g. a coloring change due to syntax highlight in only some of the lines being modified, or if your app is tmux then input arriving in the other pane would require the same steps. Also what if the app's drawing routines are not clever enough to use ICH but rather print out the whole lines again, then also the cursors would need to be unified and then split again.
Even if the very same operation at each cursor would be sufficient (e.g. ICH followed by the new letter, if the example video didn't have a bottom status line), how would the applilcation know to rely on this feature of the terminal? A change to the internal buffer (the text file being edited) probably doesn't directly result in escape sequences being emitted. Rather, most likely it causes changes to its in-memory representation of what should be on the screen (including updates to the syntax highlighting and such), and then a subsequent step compares this buffer with what's on the screen and does the update. Even with a single cursor, finding optimal steps to update the screen is a hard problem. This is where ncurses optimizes heavily, while other implementations are typically more lazy, and simply update everything that's changed without too much optimization. With multiple cursors, an already hard problem becomes even harder. How would the screen drawing code of an app figure out that it can reach the desired result by applying some clever steps at all the cursors simultaneously? Needs way too much engineering work that probably hardly any (if at all) application developer is willing to put in, plus the runtime cost is high too.
In what order would the cursors handle incoming data? It's not hard to come up with examples where the order does matter (e.g. ICH at two cursors within the same line, if the cursor stays at the same physical position). Would it be the display order of cursors (and then isn't it an expensive operation to keep the cursors sorted all the time)? Or would it be the creation order? Or some random unspecified, resulting in undeterministic behavior?
Would it be allowed to have two or more cursors at the same offset? If so then e.g. an ICH would insert two or more spaces there, correct? If not then at every step the cursors need to be deduped, again a potentially expensive operation.
What if there's a cursor at the bottom right, and one in the top row? New input at the bottom right causes a line wrap and hence the contents scroll, but then the cursor at the top will be scrolled out. Is it lost then, or what?
How to clean up additional cursors so that only one remains? I mean, which one would stay?
Compatibility: If an application incorrectly believes that the terminal emulator supports multiple cursors, while in fact it doesn't, ideally the worst that should happen is that these escape sequences are silently ignored and hence the additional cursors don't show up. Instead, the entire screen could fall apart big time, since things that should be updated at multiple places will be updated at only one.
Protection against cat'ing a random binary file with random escape sequences: I'm okay with seeing garbage as a result, but I heck don't want to end up with multiple "active" shell prompts, typing into all of them at once.

egmontkob on 17 Jul 2018

On Tue, Jul 17, 2018 at 01:52:13PM -0700, egmontkob wrote:

[kovidgoyal] Already implemented by what? [...] In either case whoever does the implementation, it is easier to output fewer escape codes than more escape codes.

You still don't understand what I'm speaking about. This will probably be my last attempt in explaining.

Let's assume a non-ncurses based editor (there are plenty of them). Presumably it already handles bold, italic, underline etc. By some internal means it figures out which cells it wants to render with such attributes, and then implements telling this to the terminal emulator. Given this current state, it's much easier to add another attribute that follows this pattern, than an attribute that uses a totally different approach. Easiness doesn't mean fewer bytes to be emitted, easiness doesn't mean something that would be easier to implement from scratch, easiness means easier to implement and maintain given the current codebase, that is, less deviations and less additional code to introduce, reusing whatever's already available (i.e. the way bold/italic/underline are handled by this app).

And you still dont understand what I am saying. The application can
continue to represent cursors as text attributes if it wants to. In
either case it will still need to implement new escape codes to have the
attribute recognized by the terminal emulator. In my proposal, those
escape codes are simpler, more flexible, semantically correct and
therefore easier to use.

kovidgoyal on 18 Jul 2018

👍1

On Tue, Jul 17, 2018 at 02:47:10PM -0700, egmontkob wrote:

[kovidgoyal] Are you saying, you would like to distinguish [the real cursor] from other cursors? If so isn't the point of multiple cursors that they all behave identically, so that if you insert text, it is inserted at all cursor positions?

No way!

I mean, that's the behavior that apps are likely to implement, by emitting the proper output that makes the new text appear at all such position. But the terminal emulator shouldn't automatically do it.

Dont worry, I'm not suggesting the terminal emulator do it. I'm just
saying that is the way multiple cursors are usually used so in the
absence of any other use case I dont see a compelling need for
distinguishing one cursor from the others.

kovidgoyal on 18 Jul 2018

On Tue, Jul 17, 2018 at 09:06:20AM -0700, George Nachman wrote:

iTerm2 author here: I'm fine with adding this, but I don't want it to be an attribute. I only have a few bits of attributes left before my struct needs to grow. That will increase memory usage and decrease cache efficiency for an attribute that would almost never be set. That's an implementation detail, but it's an important one.

Good to hear. I agree with you that attributes are a bad implementation
strategy performance wide. kitty currently does have enough space in its
struct, but it would add overhead for other reasons, I have detailed
above.

Set the number of virtual cursors

Position virtual cursor i at x,y

Set the style (box, line, bar) of virtual cursor i

We'd also need escape codes to delete cursors either specific ones or
all of them. But, lets hold off on designing the actual protocol for a
bit longer until we have buy in from editor developers. I dont want to
spend time on designing/implementing this only to have it not be used by anybody.

I would also propose that virtual cursors change position only when told to and have no automatic behavior. This pushes the complexity into the app that wants them, but makes it much more predictable in the face of scrolling regions and double-width characters.

Agreed.

However, I'm not quite sure how this would work with an ncurses-based app. Someone who has more experience can perhaps comment here.

It wouldn't. But neither would any other approach. ncurses needs new API
for this either way.

kovidgoyal on 18 Jul 2018

On Tue, Jul 17, 2018 at 09:06:07AM -0700, Justin M. Keyes wrote:

The way I envision multicursor is a "primary" cursor plus other "marker" (secondary) cursors. The secondary cursors don't need to look identical to the primary cursor, they are just visual clues. I would guess that a human is not greatly aided by all of the cursors looking like the primary cursor, rather it may _increase_ confusion.

Are you saying that if terminals were to implement it, you would not be
interested in using it -- i.e. you think whatever multi-cursor features
you intend to implement in nvim would be better off not using terminal
rendered cursors?

kovidgoyal on 18 Jul 2018

👍1

As an additional data point, its likely we would need to display different cursors with different "style", at least in the case where there are multiple cursors on the buffer, and we are in prompt mode (having the buffer cursor change shape due to the interactions with the prompt would be pretty strange).

And you still dont understand what I am saying. The application can continue to represent cursors as text attributes if it wants to. In either case it will still need to implement new escape codes to have the
attribute recognized by the terminal emulator. In my proposal, those escape codes are simpler, more flexible, semantically correct and therefore easier to use.

I still think an attribute makes the most sense, and is the semantically correct thing, it seems established that the only cursor that has semantic meaning in the terminal is the main one, the others do not have any semantic meaning for the terminal, they are literally a display attribute of the character they lie on. I believe most full screen terminal applications will find this model much simpler than having to clear and replace all cursors on each redraw (or track where they are to move them around).

Representing the cursor as text attributes inside the application only to extract those attributes to generate a list of cursors is not really practical, in Kakoune it will be far simpler to pass the cursors position as a separate list to the display backend if that was the way the terminal wanted it.

In my view, the only argument in favor of non-attribute cursors is that it is easier/cleaner for terminals to implement (which is a good reason, dont get me wrong).

mawww on 18 Jul 2018

👍1

@gnachman Just FYI, there are escape sequences to set both the cursor color and the shape, not just the shape. See http://invisible-island.net/xterm/ctlseqs/ctlseqs.html (search for cursor color) There are fairly widely implemented (IIRC kitty, VTE based terminals, xterm, and konosle, though as usual konsole will be using escape codes different from everybody else).

kovidgoyal on 18 Jul 2018

😄1

Let me summarize where I think we are, please correct me if I misrepresent any of your positions. Regarding the three questions of stage 1:

1) Editor developers: Are you willing to write code to use this feature if terminals implement it.
kakoune says maybe in the distant future. nvim at least for the moment seems to say no.

2) Terminal developers: Are you willing to implement this feature. kitty and iterm2 say yes. VTE hasn't said yes or no yet.

3) Text attribute or new escape code:
kitty - new escape code
iterm2 - new escape code
VTE - probably new escape code
kakoune - relunctantly OK with new escape code?
nvim - no opinion?

kovidgoyal on 18 Jul 2018

👍1

The application can continue to represent cursors as text attributes if it wants to. In either case it will still need to implement new escape codes to have the attribute recognized by the terminal emulator.

Which, if the exitsing pattern is followed, might be as easy as adding a new entry into an array (let's say an array mapping positions of a bitmap into ANSI codes), or as easy as copy-pasting an existing few lines and just altering the internal attribute ID and the ANSI numbers. Whereas, with a different approach, brand new code has to be written (even it's just a few lines) and thoroughly tested.

In my proposal, those escape codes are simpler, more flexible, semantically correct and therefore easier to use.

Simpler, easier to use: no. Semantically correct: yes; let's remember that all this debate occurred assuming we take the "just another attribute" approach, which no longer seems to be the case, and the new approach makes this debate pointless.

Dont worry, I'm not suggesting the terminal emulator do it. [...]

In that case it's unclear to me why you had all those question to the list of suggested actions I posted. E.g.

Leave a "trail", place an extra cursor where the real cursor currently is.

We may or may not need such an escape sequence, but it's unclear to me what is unclear in this proposal :) There's always exactly 1 "real cursor"; do we need an escape sequence that sets an "extra cursor" at the very place where the "real cursor" is right now? That is, if the real cursor is subsequently moved away, that position still visually shows up as a cursor.

Remove the extra cursor from the real cursor's current location.

I meant to make that position no longer an extra cursor position, that is, if the real cursor is moved away, it no longer shows up as a cursor.

egmontkob on 18 Jul 2018

I'm just saying that is the way multiple cursors are usually used so in the absence of any other use case I dont see a compelling need for distinguishing one cursor from the others.

I do, and have vaguely mentioned one, so let's look at a concrete example in depths.

Take the joe text editor as example, open a writable file with joe filename. Split the view with Ctrl+K O. Now you have two viewports showing the same file. If you edit one, the changes are reflected in the other one as well. You may jump to the other viewport at any time using Ctrl+K N.

Since characters typed are being inserted in both viewports, it makes sense to have an additional cursor shown in the other viewport as well (assuming the viewport does contain the cursor position, of course), using our brand new feature, correct?

Now let's imagine we've implemented this, we use joe with this new feature, and stand up for a coffee break and then return. There's no visual distinction telling which of the two viewports is the "active" one, and we forgot it during the break. Although, this is an important piece of information for using the software.

Just move the cursor Down, expecting to walk downwards in the file. In one of the viewports, the file contents will scroll to keep the cursor visible. In the other one, the cursor will disappear from the viewport. You want to know it unconsciously in advance, rather than being a surprise when the first viewport's cursor reaches the bottom.

Operations like Find (Ctrl+K F) work in the "active" viewport. (Cancel with Ctrl+C.) Closing the window (Ctrl+C) works on the "active" viewport etc.

If this feature was implemented in terminal emulators and in turn the joe text editor, I'd definitely want to see "extra cursors" marked differently, somewhat weaker than the normal one.

I'd like to design this feature in a way that "extra cursors" can either look the same as the normal one, or can look weaker, according to the application's request.

egmontkob on 18 Jul 2018

On Wed, Jul 18, 2018 at 02:04:59AM -0700, egmontkob wrote:

The application can continue to represent cursors as text attributes if it wants to. In either case it will still need to implement new escape codes to have the attribute recognized by the terminal emulator.

Which, if the exitsing pattern is followed, might be as easy as adding a new entry into an array (let's say an array mapping positions of a bitmap into ANSI codes), or as easy as copy-pasting an existing few lines and just altering the internal attribute ID and the ANSI numbers. Whereas, with a different approach, brand new code has to be written (even it's just a few lines) and thoroughly tested.

In either case you will need to write brand new code to convert the
extra attribute into something a terminal emulator will understand. In
one case that brand new code is simpler than the other and will require
less testing and less validation.

Anyway, I am tired of this argument. It's pretty clear that the
consensus is to not use an attribute. Further verbiage on this issue is
a waste of everyone's time.

kovidgoyal on 18 Jul 2018

On Wed, Jul 18, 2018 at 09:15:52AM +0000, egmontkob wrote:

I'd like to design this feature in a way that "extra cursors" can either look the same as the normal one, or can look weaker, according to the application's request.

That's fine, if you have a use case for it, I have no objection.
Although ideally, I'd like to see that use case backed up by some editor
developer that actually intends to use it.

In any case it can be discussed in detail when we nail down the protcol
for multiple features.

kovidgoyal on 18 Jul 2018

I'm fine if we just keep it in mind, and design the protocol to be easily extendible in this direction in the future.

egmontkob on 18 Jul 2018

Sounds good to me.

kovidgoyal on 18 Jul 2018

I had half an hour so I wrote a first draft of the protocol: https://github.com/kovidgoyal/kitty/commit/67324efae5f68f194df13b134666e394602215e6

Comments are welcome. I have deliberately done the minimum needed, but the design is highly extensible.

kovidgoyal on 19 Jul 2018

👍1

Easier to read link: https://github.com/kovidgoyal/kitty/blob/67324efae5f68f194df13b134666e394602215e6/docs/protocol-extensions.rst#multiple-cursors

kovidgoyal on 19 Jul 2018

Also ccing @leonerd as he might be interested

kovidgoyal on 19 Jul 2018

❤1

It is now crystal clear how a single new escape code can achieve all mentioned scenarios, and I personally like this approach. Thanks for drafting the protocol!

maximbaz on 19 Jul 2018

Thanks for working on this! Here's my first few thoughts:

I haven't met APC so far, so I cannot comment on choosing it. Read: I'm fine with it.

My biggest concern is the use of indices, at first glimpse I dislike it. IMO it imposes an additional unnecessary complexity both for terminal emulators and apps.

If I were to write a text editor, I don't think I would want to assign IDs to the cursors, just as I don't want to assign IDs to bold or italic or green words etc. I just don't see the point in them. Creating IDs if I don't have them is an additional burden. (The other way around, i.e. not telling them to the terminal emulator even when I have them, is no problem.)
If I really have to have IDs, and assuming that I use double buffering in my editor, i.e. I have an internal representation how I believe the screen currently looks like, and another one about how I want the screen to look like, and every once in a while I call the method that updates the terminal emulator accordingly, then the concept of IDs make me require either a separate data structure for them; or (as you told mawww he'd still be able to store these as if they were text attributes) 32 bits per cell (to be able to remove them by ID), rather than 1 bit per cell (and remove them at the positions where they are no longer present).
IDs are also inconvenient for tmux and friends: they'd have to dynamically maintain a mapping from each pane's cursor IDs (as specified by the app running inside) to the IDs use towards the host emulator, to avoid clashes.

I think I'd rather see IDs gone, and cursors just identified by their locations.

A few more corner cases or nitpicks:

If we stick to IDs, is it allowed to have multiple cursors (with different IDs) at the same position? If so, deleting/moving one retains the other, correct?
Reset, Clear screen: As the draft evolves, we should become more specific on the escape sequences. E.g. soft reset (<ESC>[!p) just resets the cursor's attributes, I don't think it should influence extra cursors. What about "clear to the right and bottom from the cursor" and similar ones?
Switching between primary and alternate screen: I'm unsure, e.g. if mc were to use such cursors, it would need to reinstall them if you press <Ctrl>+O twice. (Not sure if it repaints its alternate screen anyway, or just switches back and forth. But what if the normal screen underneath has multiple cursors for a good reason?) I'd probably say each screen should keep their cursors. If this is rejected, then at least it should be emphasized that switching "to" the currently active screen (a no-op) shouldn't wipe out the cursors, I vaguely recall mc emitting unnecessary switches.
Vertical scrolling: Should, for whatever reason, an application that does partial screen handling (like a shell prompt, or a multiline wget progress bar, etc.) decide to have multiple cursors, they would need them to scroll along with the text, as it's completely out of their control (and their knowledge) when the terminal scrolls. Also, if cat'ing a binary file sets some additional cursors, I'd expect them not to clutter my screen until a hard reset, but move upwards, eventually out of the screen, as I simply tap on Enter. So I vote for scrolling.
The overhead of the escape sequence is 5 bytes (<ESC>_C, and then <ESC>\), I'm wondering if there should be a shortcut for operations on multiple cursors, with smaller additional overhead than repeating these 5 bytes? (I'm fine without, just asking.)
I don't get how r=10,c=3 becomes "row 10 and column 2", I guess it's just a typo. In terminal emulators, traditionally rows and columns are numbered from 1, and so shall it be in our new escape codes as well for consistency. If you prefer to use 0-based in the English explanation, then it should go for both axes (i.e. "0-based row 9 and column 2").
What should "save/restore cursor" do?

egmontkob on 19 Jul 2018

On Thu, Jul 19, 2018 at 09:28:08AM -0700, egmontkob wrote:

Thanks for working on this! Here's my first few thoughts:

I haven't met APC so far, so I cannot comment on choosing it. Read: I'm fine with it.

It is used for kitty's graphics protocol.

My biggest concern is the use of indices, at first glimpse I dislike it. IMO it imposes an additional unnecessary complexity both for terminal emulators and apps.

If I were to write a text editor, I don't think I would want to assign IDs to the cursors, just as I don't want to assign IDs to bold or italic or green words etc. I just don't see the point in them. Creating IDs if I don't have them is an additional burden. (The other way around, i.e. not telling them to the terminal emulator even when I have them, is no problem.)

If I really have to have IDs, and assuming that I use double buffering in my editor, i.e. I have an internal representation how I believe the screen currently looks like, and another one about how I want the screen to look like, and every once in a while I call the method that updates the terminal emulator accordingly, then the concept of IDs make me require either a separate data structure for them; or (as you told mawww he'd still be able to store these as if they were text attributes) 32 bits per cell (to be able to remove them by ID), rather than 1 bit per cell (and remove them at the positions where they are no longer present).

If the application is maintaining a list of extra cursors anyway, the id
can simply be the number in that list. I dont see how it adds
significant extra burden. The alternative is, the application has to
remember the location of each extra cursor. ids are more robust. They
support multiple cursors at the same location. Not to mention that
knowing the location is not always easy (think of wide characters).
As it is I see a lot of bug reports caused by the fact that most editors
use out of date wcwidth() implementations that dont agree with kitty
which uses one derived from the unicode 11 standard. This is the main
reason I chose to use ids.

IDs are also inconvenient for tmux and friends: they'd have to dynamically maintain a mapping from each pane's cursor IDs (as specified by the app running inside) to the IDs use towards the host emulator, to avoid clashes.

tmux would have to rewrite the escape codes anyway, to map locations.
Rewriting ids shouldn't be a particularly burdensome additional task.
Although, given my recent experiences with tmux upstream, I am pretty
convinced it should die in a fire. See #413

If we stick to IDs, is it allowed to have multiple cursors (with different IDs) at the same position? If so, deleting/moving one retains the other, correct?

Yes and yes.

Reset, Clear screen: As the draft evolves, we should become more specific on the escape sequences. E.g. soft reset (<ESC>[!p) just resets the cursor's attributes, I don't think it should influence extra cursors. What about "clear to the right and bottom from the cursor" and similar ones?

Reset is rs1 from terminfo and clear screen is clear from terminfo. No
other escape codes should have any effect.

Switching between primary and alternate screen: I'm unsure, e.g. if mc were to use such cursors, it would need to reinstall them if you press <Ctrl>+O twice. (Not sure if it repaints its alternate screen anyway, or just switches back and forth. But what if the normal screen underneath has multiple cursors for a good reason?) I'd probably say each screen should keep their cursors. If this is rejected, then at least it should be emphasized that switching "to" the currently active screen (a no-op) shouldn't wipe out the cursors, I vaguely recall mc emitting unnecessary switches.

I'm OK with changing that, let's wait to hear if anyone else has any
thoughts on it. And yes, changing to the existing screen is a
no-op.

Vertical scrolling: Should, for whatever reason, an application that does partial screen handling (like a shell prompt, or a multiline wget progress bar, etc.) decide to have multiple cursors, they would need them to scroll along with the text, as it's completely out of their control (and their knowledge) when the terminal scrolls. Also, if cat'ing a binary file sets some additional cursors, I'd expect them not to clutter my screen until a hard reset, but move upwards, eventually out of the screen, as I simply tap on Enter. So I vote for scrolling.

If you allow this then you have to worry about DECOM (i.e. margins).
Since index operations are bounded by margins, you would also have to
specify the behavior of extra cursors in the presence of margins.

Still it's not an absolute no-no from me. Let's see if other people want
it.

The overhead of the escape sequence is 5 bytes (<ESC>_C, and then <ESC>\), I'm wondering if there should be a shortcut for operations on multiple cursors, with smaller additional overhead than repeating these 5 bytes? (I'm fine without, just asking.)

I dont think this is critical enough to need that. Also what operation
would you do in bulk? Currently the only two operations are delete
(which you can do in bulk for all anyway) and move which makes not much
sense to do in bulk.

In the future if other operations are added, we can revisit.

I don't get how r=10,c=3 becomes "row 10 and column 2", I guess it's just a typo. In terminal emulators, traditionally rows and columns are numbered from 1, and so shall it be in our new escape codes as well for consistency. If you prefer to use 0-based in the English explanation, then it should go for both axes (i.e. "0-based row 9 and column 2").

Typo. As noted indexing is 1-based.

What should "save/restore cursor" do?

Nothing, as far as I am concerned. I dont want terminal emulators
to have to maintain stacks of large numbers of cursor objects.

kovidgoyal on 20 Jul 2018

I have made some modifications to the protocol specification based on the feedback. See the cursors branch of this repo for diffs.

kovidgoyal on 20 Jul 2018

I like having one escape sequence to control everything. Makes it easy to reason about.

Are row and column numbers relative in origin mode?

APC is not ignored by Terminal.app. How does VTE treat it? I generally prefer OSC for this reason. I believe VTE now ignores OSC.

I agree that treating it as a list of cursors makes sense. I can't imagine an editor storing its cursors any other way, but perhaps I lack imagination :)

I think sequences for changing shape and color should be in the first draft. How about <ESC>_Ci=1,s=N,c=#rgb<<ESC>\\ where N corresponds to the Ps of DECSCUSR and #rgb is an SRGB value like #ff8000 or #f80. If you want to add more color spaces later, you can always stick something before the #.

gnachman on 20 Jul 2018

On Thu, Jul 19, 2018 at 10:13:27PM -0700, George Nachman wrote:

Are row and column numbers relative in origin mode?

No, as noted in the spec DECOM is ignored. Do you think it should not
be? An argument can be made either way.

APC is not ignored by Terminal.app. How does VTE treat it? I generally prefer OSC for this reason. I believe VTE now ignores OSC.

What does not ignored mean? What does iterm2 do if it encounters an APC
it does not understand?

I think sequences for changing shape and color should be in the first draft. How about <ESC>_Ci=1,s=N,c=#rgb<<ESC>\\ where N corresponds to the Ps of DECSCUSR and #rgb is an SRGB value like #ff8000 or #f80. If you want to add more color spaces later, you can always stick something before the #.

The reason I am reluctant to add shape and color to the first draft is
that it adds overhead to the implementation for a feature I have yet to
see a use case for, from someone actually intending to use it, as
opposed to theoretical usage scenarios.

However, I am open to being convinced otherwise.

kovidgoyal on 20 Jul 2018

I have added a "cursors" terminfo property to the spec that applications can use to detect support for this feature.

kovidgoyal on 20 Jul 2018

👍1

To give a bit of input, its likely on Kakoune side we would just clear and resend cursors on every redraw. This is because the display backend (which uses ncurses at the moment) does not know anything about cursors (so far) except for the main cursor location (which I added only to let the terminal know a sane place to locate their insert related widgets). It would not know how to relate one cursor from one draw to another cursor from the next draw.

I wonder if other editors would have other needs, but for Kakoune, a single sequence defining all cursors (and implicitely removing any previous ones) would be good enough, which means we would never need to identify a specific cursor.

mawww on 20 Jul 2018

On Thu, Jul 19, 2018 at 11:29:36PM -0700, Maxime Coste wrote:

To give a bit of input, its likely on Kakoune side we would just clear and resend cursors on every redraw. This is because the display backend (which uses ncurses at the moment) does not know anything about cursors (so far) except for the main cursor location (which I added only to let the terminal know a sane place to locate their insert related widgets). It would not know how to relate one cursor from one draw to another cursor from the next draw.

Thanks for the input. So the current proposal meets your needs? In
pseudo-code, I image you would do something like this, to display the
cursors:

for cursor in cursors:
write(_Ci=counter++,r=cursor.x,c=cursor.y)

And to delete:
write(_Ca=d)

You simply output the delete sequence at the start of your display loop
and the creation sequences at the end.

kovidgoyal on 20 Jul 2018

APC is not ignored by Terminal.app. How does VTE treat it?

As of the current development series (forthcoming 0.54), thanks to Christian's parser rewrite, it ignores them.

Current stable 0.52 prints garbage. Given that it'll take a few years for this feature to get widely adapted, I don't mind this.

egmontkob on 20 Jul 2018

If the application is maintaining a list of extra cursors anyway, the id can simply be the number in that list. I dont see how it adds significant extra burden.

_If_ the application is maintaining a _list_ of extra cursors anyway. What if not? What if it uses just another bit per cell in its already existing internal WxH matrix denoting what's already on the screen (i.e. treats them as if they were text attributes, as you told mawww it'd be fine)?

ids are more robust.

I don't see it that way, I see them as an additional burden for no real use.

They support multiple cursors at the same location.

A feature you would use for... for what? (I don't mind it as a by-product of any design, but I don't see any use for it that could make it a pro argument.)

Not to mention that knowing the location is not always easy (think of wide characters).

I'm not sure I get it. I can't imagine a text editor that doesn't have a consistent internal representation (including wide characters) of what it believes is on the screen. If the code places a cursor at (x,y) then removing a cursor specifying the same (x,y) coordinates will remove that. If an app places an extra cursor at the real cursor's current location, then later repositioning the real cursor there and removing the extra cursor _should_ (see below) also remove it.

As it is I see a lot of bug reports caused by the fact that most editors use out of date wcwidth() implementations that dont agree with kitty which uses one derived from the unicode 11 standard. This is the main reason I chose to use ids.

If there's a misagreement (which unfortunately does happen every once in a while, just as well as another app (e.g. write(1)) can pollute the display) then contents fall apart anyway. Hence most editors offer a shortcut to redisplay their contents, which, depending on the nature of the breakage, may or may not fix it. I understand that using IDs could make the display of extra cursors more robust against these kinds of scenarios, but I honestly don't see any point in having this as a goal, if it's broken then it's broken anyway in ways that make it hard or practically impossible to use the editor at those lines of the file. A few leftover cursors IMHO doesn't make it worse then, it's just as bad.

Thanks for your explanation, though, I can now see what's your rationale behind this design. I just disagree with the priority order of the factors behind this decision :-) I think ease of use, even from apps that don't maintain the cursors as a _list_, or from terminal multiplexers that would have to maintain a mapping with your proposal, is a more important factor than damage control.

[vertical scrolling] Still it's not an absolute no-no from me.

Another use case that occurred to me is if the editor crashes. This is not something that has never happened to me. The prompt might be displayed amidst the leftovers from the file's contents, it might be colored (but _my_ prompt does reset that), there's no scrollback buffer since we're still on the alternate screen, but I just press a couple of Enters to get to the bottom of the screen and it's usable. Repeating myself: I wouldn't want cursors to remain there cluttering my command line.

[one escape sequence for multiple operations]

We've heard about at least one app that would always clear all existing extra cursors and place them again (rather than keeping track of IDs, moving them etc.). Being able to specify multiple actions in a single sequence has the small advantage that the terminal emulator can handle all of them as an atomic step. That is, even if the connection stalls while receiving them, it could avoid the visual flickering of removing and later at the same place reinstalling some extra cursors. My bad for not mentioning this the last time. It's by far not the most important issue here, but one that we might consider nevertheless.

egmontkob on 20 Jul 2018

On Fri, Jul 20, 2018 at 09:16:58AM +0000, egmontkob wrote:

If the application is maintaining a list of extra cursors anyway, the id can simply be the number in that list. I dont see how it adds significant extra burden.

_If_ the application is maintaining a _list_ of extra cursors anyway. What if not? What if it uses just another bit per cell in its already existing internal WxH matrix denoting what's already on the screen (i.e. treats them as if they were text attributes, as you told mawww it'd be fine)?

It doesn't matter how it stores them. In either case you have to output
the escape codes for them, when you do simply output an incrementing
counter. It's two extra lines of code.

Not to mention that knowing the location is not always easy (think of wide characters).

I'm not sure I get it. I can't imagine a text editor that doesn't have a consistent internal representation (including wide characters) of what it believes is on the screen. If the code places a cursor at (x,y) then removing a cursor specifying the same (x,y) coordinates will remove that. If an app places an extra cursor at the real cursor's current location, then later repositioning the real cursor there and removing the extra cursor _should_ (see below) also remove it.

Remember that the escape code places the extra cursor at the current
cursor position, bu default, which apps often dont know, and cannot in
general know unless they either query the emulator or have a wcwidth()
implementation that exactly matches that of the emulator. Obviously when
the app specifies the location (x, y) in the escape code, it knows the
location. When it does not, it may or may not. ids on the another hand
it always knows.

And you forgot about scrolling and resizing of the screen. If we make
the extra cursors move with scrolling then the app has to
update their locations just to address them.
Similarly after a resize it cannot know the new locations. And given
that screen scroll all the time, for example when wrappiing is enabled,
this is not a small issue.

ids are just better :) And sorry, but needing to add two extra
lines of code in applications that dont care about ids is not an undue
burden, to make the implementation robust for applications that do care
about addressing individual cursors.

[vertical scrolling] Still it's not an absolute no-no from me.

Another use case that occurred to me is if the editor crashes. This is not something that has never happened to me. The prompt might be displayed amidst the leftovers from the file's contents, it might be colored (but _my_ prompt does reset that), there's no scrollback buffer since we're still on the alternate screen, but I just press a couple of Enters to get to the bottom of the screen and it's usable. Repeating myself: I wouldn't want cursors to remain there cluttering my command line.

If the editor crashes, I dont see how you can possibly hope to recover
other than by doing a reset. Everything from colors, line discipline,
echo mode, all the dozens of bbits of state in termios and the emulator,
the keyboard mode, etc, etc, etc could be wrong. And reset or
clear removes extra cursors. Terminals are not robust against program
crashes and never will be. It's pointless worrying about it.

[one escape sequence for multiple operations]

We've heard about at least one app that would always clear all existing extra cursors and place them again (rather than keeping track of IDs, moving them etc.). Being able to specify multiple actions in a single sequence has the small advantage that the terminal emulator can handle all of them as an atomic step. That is, even if the connection stalls while receiving them, it could avoid the visual flickering of removing and later at the same place reinstalling some extra cursors. My bad for not mentioning this the last time. It's by far not the most important issue here, but one that we might consider nevertheless.

If the connection stalls in the middle there is no visual flickering,
you get some cursors that appear before and some cursors that appear
after. Just as you get some text that appears before and some text that
appears after. Terminals do not paint the screen in an atomic manner,
they never have and I doubt they ever will. But if they ever do,
whatever mechanism they use for it, can just as well apply to this
protocol just as it would have to apply to all the rest.

kovidgoyal on 20 Jul 2018

And just because I'm bored, let's do a bit of simple maths to calculate the probability of their being one extra cursor on the screen after catting a 10MB binary file.

You need a minimum of an 8-byte sequence to create an extra cursor. The probability of such a sequence is ((1/255)**7) * (1/245) which is 1e-20 (technically its a bit higher since you can have longer than 8 byte sequences as well that create extra cursors, but I doubt the contribution from them will be significant compared to the shortest sequence). There are approx 1e-6 8 byte sequences in a 10 MB file. Which leaves you with a first order probability of 1e-14 of seeing a one single extra cursor on your screen after catting a 10MB file. I can live with that.

kovidgoyal on 20 Jul 2018

In either case you have to output the escape codes for them, when you do simply output an incrementing counter. It's two extra lines of code.

_If_ the app always removes and reinstalls all the cursors at once. If that was the only intended way of using these escape sequences, there'd clearly be no need for IDs.

If an app decides to use the more fine grained sequences, that is only remove the cursors that are no longer necessary, only add the newly appearing ones, and leave untouoched the ones that don't move, then having to have IDs _is_ an additional, nontrivial burden.

Similarly after a resize it cannot know the new locations.

Why can't? Won't it be part of the specification?

Terminals do not paint the screen in an atomic manner, they never have and I doubt they ever will.

Not updating the entire screen as an atomic step is fine. What I'm mentioning is when something _shouldn't change_, but due to a lag in traffic, ends up changing back and forth on the UI. This can already happen to the real cursor, but would be nice not to extend to the extra cursors, if reasonably possible.

egmontkob on 20 Jul 2018

On Fri, Jul 20, 2018 at 03:06:00AM -0700, egmontkob wrote:

In either case you have to output the escape codes for them, when you do simply output an incrementing counter. It's two extra lines of code.

_If_ the app always removes and reinstalls all the cursors at once. If that was the only intended way of using these escape sequences, there'd clearly be no need for IDs.

If an app decides to use the more fine grained sequences, that is only remove the cursors that are no longer necessary, only add the newly appearing ones, and leave untouoched the ones that don't move, then having to have IDs _is_ an additional, nontrivial burden.

Huh? It is precisely in that case that having ids is superior to not
having them. If the application wants to address individual cursors in
the terminal, then it must be able to address them internally as well,
which means they already have an internal id (or at least some kind of
internal data structure to represent an individual cursor). Which means
there is no extra burden giving them an id in the escape code, simply
add an id field to the internal structure which uses a globally
incrementing counter, again two lines of code.

Similarly after a resize it cannot know the new locations.

Why can't? Won't it be part of the specification?

No, at least I dont have any interest in adding it to the specification,
but if you do, feel free to send a PR.

Terminals do not paint the screen in an atomic manner, they never have and I doubt they ever will.

Not updating the entire screen as an atomic step is fine. What I'm mentioning is when something _shouldn't change_, but due to a lag in traffic, ends up changing back and forth on the UI. This can already happen to the real cursor, but would be nice not to extend to the extra cursors, if reasonably possible.

What shouldn't change? You are sending n cursor creation escape codes
one after the other. The only thing that can happen is that they might
not appear on the screen at the exact same moment. Nothing can appear
that later should not appear since you are transmitting the escape codes
immediately after each other, regardless of how many pauses there are in
transmission.

kovidgoyal on 20 Jul 2018

it must be able to address them internally as well, which means they already have an internal id (or at least some kind of internal data structure to represent an individual cursor)

This can be just 1 bit per cell in its already existing matrix denoting what's on the screen, next to the bold, italic etc. attributes. With no IDs. This is what you said to mawww that he'd still be able to do. This is what I believe is the simplest approach to implement this feature in any non-ncurses based app, because it reuses what the app already does for bold etc., without having to come up with something new as well.

What shouldn't change? You are sending n cursor creation escape codes one after the other. [...]

Imagine that there are three extra cursors at positions A, B and C. The app wants to change it to four positions: A, B, D and E. The app's internal logic doesn't wish to fiddle with IDs, so instead clears and reinstalls all the cursors. There's a potential for a flicker at positions A and B, the cursor disappearing for a short time. Being able to specify multiple cursor operations in a single escape sequence could avoid this flicker.

egmontkob on 20 Jul 2018

On Fri, Jul 20, 2018 at 03:24:14AM -0700, egmontkob wrote:

it must be able to address them internally as well, which means they already have an internal id (or at least some kind of internal data structure to represent an individual cursor)

This can be just 1 bit per cell in its already existing matrix denoting what's on the screen, next to the bold, italic etc. attributes. With no IDs. This is what you said to mawww that he'd still be able to do. This is what I believe is the simplest approach to implement this feature in any non-ncurses based app, because it reuses what the app already does for bold etc., without having to come with something new as well.

Sigh, this gain? If it is 1-bit per cell, then individual cursors are not
addressable by the application. The application does not care about ids
and must delete all cursors on each round anyway and so can use a simple global
counter for the ids.

Let me spell it out for you in pseudo code.

struct {
   bool has_extra_cursor;
   SGRAttributes text_formatting_attributes;
   char *ch;
} Cell;

struct {
    Cell *cells;
} Line;

counter = 1
write(<delete all cursors>);
for line in lines:
   for cell in cells:
      write(cell.text_formatting_attributes)
      if cell.has_extra_cursor:
        counter += 1;
        write(<esc>_Cid=counter<esc>\)
      write(cell.ch);

As I said, literally two extra lines of code. Initialize the counter and
increment it.

What shouldn't change? You are sending n cursor creation escape codes one after the other. [...]

Imagine that there are three extra cursors at positions A, B and C. The app wants to change it to four positions: A, B, D and E. The app's internal logic doesn't wish to fiddle with IDs, so instead clears and reinstalls all the cursors. There's a potential for a flicker at positions A and B, the cursor disappearing for a short time. Being able to specify multiple cursor operations in a single escape sequence could avoid this flicker.

You mean between clearing the cursors and re-creating them? That would
be present regardless, even if the protocol allowed you to create
multiple cursors in one escape code.

If you mean you want to be able to both delete and create cursors in the
same escape code, I have no idea how that would work while still
remaining small and relatively easily parseable. I supose one culd do
something like this

_C;;..._

Doesn't seem important enough to me to bother.

kovidgoyal on 20 Jul 2018

As I said, literally two extra lines of code.

Two extra lines of code, with a design that increases by magnitudes the chance of flickering (an extra cursor disappearing and a few draw cycles later reappearing at the same position).

You mean between clearing the cursors and re-creating them? That would be present regardless, even if the protocol allowed you to create multiple cursors in one escape code.

The terminal emulator could decide whether to process individual requests as it encounters them, or batch them up until the terminating <ESC>\ sequence and then process them all, atomically. In case of VTE, due to the way its parser used to work (and I presume still works as of the rewrite, although I haven't checked), we'd do the second approach anyway, it'd be atomic.

egmontkob on 20 Jul 2018

Two extra lines of code

Sorry, I have to correct myself, I responded too quickly.

Screen drawing is probably more sophisticated than the pseudo-code you outlined. I mean they probably do some minimal diff-ing between the old and new buffers, e.g. update only the lines that changed, and maybe even within the line they omit the beginning and trailing portion that remained the same.

With your approach, they'd need a bit more than those 2 lines to re-process the extra cursors in lines that didn't change.

I'm not saying it's hard to do, but I'm still saying it's an unnecessary additional complexity.

egmontkob on 20 Jul 2018

On Fri, Jul 20, 2018 at 03:47:16AM -0700, egmontkob wrote:

As I said, literally two extra lines of code.

Two extra lines of code, with a design that increases by magnitudes the chance of flickering (an extra cursor disappearing and a few draw cycles later reappearing at the same position).

What on earth have ids to do with flickering? At this point you are just
arguing for the sake of arguing.

And I'll just note in passing that cursors blink, which means that
flickering for them is meaningless.

Hmm, I think I have had it with this discussion. I'm afraid I no longer
have the motivation to implement it if it involves this level of
argumentation.

@everyone: sorry for wasting your time. If somebody else wishes to
implement this in their terminal emulator feel free to use my work as a
baseline, or not, as you wish.

kovidgoyal on 20 Jul 2018

The maintainer of tmux had a few comments about the design, hope this helps:

Their choice of escape sequence for this seems unnecessarily different from everything else. Why make it delimited rather than using something with arguments? Why make it comma-separated when ; or : are the usual separators? If you are going to use key-value pairs, why not make them comprehensible rather than single letters?
If you want my opinion on the design then I would personally forget all this i=x,c=y,r=z stuff and just have a cursor set and cursor clear sequence. There seems no harm in requiring the existing cursor to move first like for tab stops. I'd just do something like add some extended WINOPS:

\e[99999;0;10t <-- add cursor ID 10 at current main cursor position
\e[99999;1;10t <-- delete cursor ID 10
\e[99999;1t <-- delete all extra cursors
\e[99999;2;10;x;y;zt <-- and so on, whatever else you want, cursor colour, etc

You are much more likely to get support if the escape sequence is similar to or extends an existing sequence rather than making everyone add a parser for yet another form.

lenormf on 20 Jul 2018

👍3

What on earth have ids to do with flickering?

With the code snippet you showed, the removal of the cursor is followed by potentially up to a few kilobytes of text data (and attribute changing escape sequences), then re-installing the cursor at the same position. The amount of data inserted in between could easily make them no longer fit in the same ethernet or similar packet, eventually noticeably increasing the chance of a network lag kicking in.

And I'll just note in passing that cursors blink

or not (oh, wait, it seems to me that _in Kitty_ it always blinks)

which means that flickering for them is meaningless.

A flickering during the "on" state of a blinking cursor is IMHO still a bit annoying.

At this point you are just arguing for the sake of arguing.

No. I'm trying to argue (although I'm really lost) because I have a bad feeling that no matter how I phrase my thoughts, I cannot make them come across to you. I mean, if _I felt_ you understood what I'm saying and still disagreed, that'd be fine. But unfortunately I _feel_ like you don't understand what I'm saying (maybe don't take the time or don't have the motivation to _try_ to understand what I'm saying?? I'm really unsure) then it's very hard. I sincerely hope that my feelings are incorrect. I really don't know what _I am_ doing wrong if my thoughts don't come across clearly.

I was just about to quit this conversation and let you continue in your preferred way. I'm sorry you did it first and not me, and sorry from others too. This enhancmenet request was filed against Kitty, it's your project, do as you wish. If you reopen and continue the work, I promise I'll stay out of the way.

egmontkob on 20 Jul 2018

Sorry, but this discussion has made it clear to me that I lack the patience to steward something like this. As I said, feel free to implement it in your own terminals using my existing proposal as a base or not. If and when some proposal becomes widely adopted/used by editors, I will be happy to implement it in kitty.

But I am done spending my valuable time creating proposals just to engage in endless meaningless arguments about them.

kovidgoyal on 20 Jul 2018

@justinmk wrote:

Actually, I'd be more interested in new text attributes as @mawww hinted. E.g., ability to display (1) a hollow "box" around a cell, (2) a bar on the left side, and (3) bar on the right side. Entirely orthogonal to cursor(s).

This would be nice. Totally independently of extra cursors, I could see this being useful for other cases besides. They'd be very easy to add as new numbered SGR attributes

leonerd on 20 Jul 2018

@egmontkob wrote:

SGR 51, 60, 62 might be relevant here.

I hadn't heard of those before. Do you have some relevant documentation? I'd like to add them to my spreadsheet, where I'm trying to keep things like this organised:

https://docs.google.com/spreadsheets/d/19W-lXWS9jYwqCK-LwgYo31GucPPxYVld_hVEcfpNpXg/edit#gid=836709407

(I'm happy to add/edit things in there for other terminal emulators, also)

leonerd on 20 Jul 2018

@leonerd it's in wikipedia https://en.wikipedia.org/wiki/ANSI_escape_code#SGR_(Select_Graphic_Rendition)_parameters

kovidgoyal on 20 Jul 2018

❤2

So the "hardly ever supported" (51-65) SGR parameters most likely will continue to lack support in kitty, iterm2, VTE, because of struct space?

justinmk on 20 Jul 2018

If you want them supported, I am not adamantly opposed, but you will need to make a case for it to convince us :) kitty actually has space in its struct at the moment. The downside of supporting this in kitty will be extra per pixel operations in the shader. There are also some subtleties to be aware of, namely those are meant for ideographic characters, I dont know if it is appropriate to simply use them for ordinary characters.And one has to consider what happens with them on wide characters.

kovidgoyal on 20 Jul 2018

And if you do want to make a case, I suggest opening a new enhancement report for it, this one is already quite long :)

kovidgoyal on 20 Jul 2018

Do you have some relevant documentation?

No, I don't. All I know is what's written in ECMA-48 and the Wiki page linked by Kovid. I don't even know what "ideogram" means, nor have an idea what to do with "underline _or_ right side like", I mean which one?

The feature could be useful for nice and compact representation of tables. Double width lines could be achieved by placing e.g. a character with a right side line and a next character with a left side line next to each other. Not sure if it's worth 4 new bits per cell, though. Not sure how we are with free bits in VTE, the main developer recently reorganized the bits and I haven't checked the code since. Just as Kovid, I'd also prefer to see actual (not hypothetical) use cases, utilities willing to add support.

If we'd like to move forward, I'd like to propose 4 new attributes for the 4 sides. SGR 51 ("boxed" or "framed") might be a shortcut for enabling all 4 at a time. The wording of SGR 60 and 62 are so unclear to me that I wouldn't want to use them, unless someone has a clarification on them. SGR 53 (overlined, hardly ever used) may be used for the top one, or may be obsoleted by the top one.

I'd like the bottom one to be separate from the underline. The underline goes slightly below the baseline of the font, and might be subject to "text-decoration-skip-ink" rendering, curliness, different color (or would we want that for the frame too?) etc. The border bits would I imagine go to the very edge of the character cell, so that adjacent ones are joined even in case of increased line spacing or character spacing.

egmontkob on 20 Jul 2018

👍1

As for the very concrete escape sequences, we might be thinking along the lines of 51:n (a subparameter just as with the underline style), the second parameter going from 0 to 15, being a 4-bit bitmap for the four sides. That way we don't pollute the "global" SGR space with 4-5 new entries.

egmontkob on 20 Jul 2018

I agree with Egmont about 51:n and these being independent of underlines. Those were the choices I would have made myself. The only question is whether to use 51 or some other number. After all some terminal emulators out there may already use 51.

I am curious as to what the use case for this is though. Since the border lines could potentially overlap with the rendered character, I dont know how nice it would look (think of wide characters like W that touch the edges off the cell in most fonts. But, as I said, feel free to make your case, preferably in a separate bug report.

kovidgoyal on 20 Jul 2018

51 could remain an alias for 51:15, that is, turning on all four sides. Existing implementations, however, may not put the frame at the very border of the cell, which – as far as I understand – would be a requirement for the new one. So I don't mind introducing a brand new number. We've recently assigned 58 and 59 to colored underline, but 56 and 57 are still free, aren't they? And we'd only need one.

I agree that this whole discussion should continue in a separate issue.

egmontkob on 20 Jul 2018

👍1

As long as we're bikeshedding, I'd like to add an extension to enable atomic drawing so that when the entire screen gets redrawn it doesn't flash a half-drawn screen for a fraction of a second before the rest of it appears. I have a hack in iTerm2 that tries to use whether the cursor is visible to provide synchronization but it is far from perfect. Since that came up in this discussion, I wonder if anyone would be open to the idea? If so I'll open a new issue and write up a proposal.

gnachman on 21 Jul 2018

Sure, I know this bother's Egmont a lot, so it's worth discussing. It should be trivial to have an escape code that turn on and off "pending mode". In pending mode all received bytes would be buffered. You'd probably also need a max pending timeout and a max buffer size.

kovidgoyal on 21 Jul 2018

Sounds good.

egmontkob on 21 Jul 2018

Link to spec and issue to host the discussion are here: https://gitlab.com/gnachman/iterm2/issues/6873

gnachman on 22 Jul 2018

Kitty: Protocol extension: multiple cursors

Most helpful comment

All 81 comments

Related issues