Kitty: rich text

[zokrezyl]

Great project. Did you ever consider extending the terminal parlance with something like "basic rich text". I could imagine lines with different font heights very useful when reading logs containing lines with different relevance. Nothing complicated just some ansi codes to change the terminal height, define the height of the line as max of the heights and render accordingly... Just an idea before sleeping :)

[Luflosi]

Nice idea but I very much doubt that this will ever be introduced into kitty. The current way kitty renders text in cells would require heavy modifications to the render code. It would probably not be worth the large amount of effort.
This would also introduce problems like non-linear mouse scrolling speed, since one mouse wheel rotation "click" corresponds to a certain number of lines and not a certain number of pixels.
Changing the font size of certain lines while keeping the line height the same would probably be a little easier but it would still not be a simple modification by any means.
@kovidgoyal would you like to add your opinion or comment on or correct my explanation?

[kovidgoyal]

I am happy to add support for rendering various kinds of text
formatting, however, multiple sized text is not one of them. This is
because of the way kitty achieves its performance, rendering text as a
cell of grids, multiple sized text means it cannot do that anymore.

[zokrezyl]

@kovidgoyal, I understand that adding multiple text sizes would need lot of refactoring, but what about implementing a generic protocol with escape sequences to pass data to kitty something similar to how graphics data is passed. I could imagine writing some kitty plugin's and pass data to the plugins through this generic protocol. The plugins would draw to opengl whatever one can imagine. Imagine VisPy graphics displayed in Kitty terminal, even live.

[kovidgoyal]

It's not a question of refactoring, multiple text sizes are
fundamentally incompatible with kitty's design, and as such are never
going to be supported.

And kitty already has a pretty powerful graphics protocol, thaat
supports arbitrary graphics, so you can
show whatever you want on kittys screen, already.

[zokrezyl]

I understood that you won't support the original proposal and I respect that.

What I was proposing was something similar to the existing graphics protocol. Displaying a picture through the graphic protocol is a one shot action. What I am talking about is designing/implementing a plugin mechanism that would allow individual plugins to:
-> access the opengl visual and draw there whatever they want based on commands received from the application running on the terminal
-> extend the existing graphical protocol (using the same structure/escape codes) to allow this

Example: Imagine a python application that would use the graphic protocol as a channel for rpyc (remote python lib) and talk to a kitty plugin to draw real time sound samples sent by the application. The plugin would use the opengl visual of the running terminal.

I used terms like "opengl visual" as I am not sure how one could hook into the existing design, etc. Obviously there are bunch of details like how these plugins would handle scrolling etc.

I did not want to spend to much time to understand the code, it is just a proposal, personally I would find it very cool and very straightforward to add new features to the terminal. I am aware lot of people would find it silly but well, we all have different imagination :)

Hope this will not annoy you

[Luflosi]

I don't quite understand what kitty would be able to do with the proposal implemented, that it can't already do. An application could use OpenGL to render to an offscreen buffer and then send the image to kitty, right?

[kovidgoyal]

I dont think OpenGL is the right API to expose. For instance kitty will
probably ditch OpenGL in favor of Vulkan at some time. The display
technology kitty uses is an implementation detail. Not to mention that
allowing external applications direct access to video memory is huge
security risk.

And the graphics protocol is most definitely not one-shot, you can place
and update images using it. So as long as the client generating the
images is running locally, it should work fine with reasonable
performance.

[zokrezyl]

Well, exposing whatever can be exposed if it is easy to expose. The plugin developer is taking the risks of his plugin being soon broken, leave it to him to take the risk.

I did not say to allow direct access to the video memory. Even if you would give access you would not give access by default. Running even compiled code is equally risky business. But again I did not propose to give access to the video memory: what I proposed is:
-> plugin framework that can have access to (some) of the internals
-> simple design of the extension of the graphic protocol. (It is mainly about reusing the ANSI escape keys in a clever way)
(not sure how to describe briefly without repeating myself)

The application that would talk to the plugin would have the same rights as the terminal. Thus not sure what you mean about "security risk".

About the one-shot: the point I tried to make is exactly the reason you guys wrote kitty using opengl: Performance. You don't want to send hundreds of bytes of pre-rendered data through the graphics protocol, you want to send commands to the plugin to do whatever it has to do. Think it raster vs vector graphics. Think it as having a plugin that plays video and you just send a command with the filename. Or think it as a plugin that displays the chords of the song you are listening :)
Think about your love for terminals :)

[zokrezyl]

What about simplifying:
-> Is it easy to expose the opengl to python (kitty)
-> Is it easy to design a plugin mechanism in python that would:
--> have access to the buffers/filoIO etc
--> could hook someway into the escape sequences?

If the answer is yes, let others to do the rest and enjoy the freedom to build on what you guys did :) The benefits would also come in the opposite direction

[kovidgoyal]

I suggest you actually read the graphics protocol and understand it and then tell us what it is you want to do that isn't covered by it. You dont seme to understnd how terminal escape codes work. There is no question of aplugin mechanism here. You write an independent program with full access to every resource on your computer. That program communicates with the terminal emulator using escape codes. The graphics protocol in kitty allows the program to draw arbitrary, pixel-perfect graphics at any pixel based or cell based location in the kitty window.

[zokrezyl]

I am not sure how to explain. In all the story the less import is how the escape code works, but you seem to conclude that I do not understand how they work.

You say there is no question of plugin mechanism here. Well, there is. And that is why I am writing the nTH post already and asking that question:

And the question is:
"Do you intend to support some kind of plugin mechanism"?

Can you please answer the question without moving focus to something else?

I completely understand that the program communicates with the terminal emulator with escape codes.

I also understood that it passes graphic data to the terminal emulator through the escape sequences.

More: I wrote a simple python script that reads ffmpeg 1920x720 video and sends every frame through the escape sequences. Result: with two processes running at 100% I get only 1 frame per second, as passing that huge raw data through the buffer is slow.

So whatever data I would pass through that mechanism pixel by pixel would be inefficient. If I would pass a dynamic function drawing instead of passing 100byte/second, I will pass 1920x720x3 bytes. Do you see the loss in compression rate?

Therefore I was proposing the idea to be able to pass arbitrary data through the same protocol and let plugins interpret them and draw to opengl surface. Blind simple, am really confused why is it so difficult to grasp the idea.

If it is feasible or not, another question. We cannot talk feasibility if we cannot understand what we are talking about.

Hopefully now is more clear.

[zokrezyl]

https://gist.github.com/zokrezyl/e70e60511c718fe26c0ddc106e08d4bf

[kovidgoyal]

And that just shows you have not bothered to actually read and
understand the protocol. If you had you would know that the graphics
protocol does not need to pass data through escape codes, it can do it
directly via shared memory. Whatever process is going to draw images to
the screen has to get the pixels into memory before doing that, there
there is no inefficiency in using the graphics protocol to do so. It's
not as efficient as hardware accelerated video decoding, for obvious
reasons, but if you want to be doing something like that, a terminal
emulator is not the right place to be doing it in.

And kitty supports "plugins" whatever that means. They are called
kittens. They are not going to be drawing to opengl surfaces, because
again, that is the wrong abstraction to be exposing an API surface on.

[zokrezyl]

You seem to got the point " It's not as efficient as hardware accelerated video decoding", but for some reason you have to be offensive by writing "you have not bothered to actually read and understand the protocol." No further comment on that....

I will end myself this annoying chat here, but just following your reasoning " terminal emulator is not the right place to be doing it in", I can only conclude that you did useless job by implementing graphics as terminals are not the right place to do that :)

Good luck with your kitties

[Luflosi]

This discussion got a little out of hand.
@zokrezyl Could you please try to use the shared memory feature of the graphics protocol and see if the performance is acceptable for your application? I would also be interested to know how well it performs.
But if you need the full native OpenGL performance, you would be better off just writing your own application that displays OpenGL graphics directly.
The graphics protocol is really meant for images and for this application it is perfectly adequate and useful. It's just not optimised for streaming video but it might still have acceptable performance when using shared memory.

[zokrezyl]

https://gist.github.com/zokrezyl/e70e60511c718fe26c0ddc106e08d4bf#file-kittyvplayer-py

Please feel free to include it in your code...

Works bit better with shared memory, but annoying that the shared memory object is deleted and you have to re-open file, remap etc. That is bad coding. There should be at least an option to keep the file open. Doing more than needed is very bad design decision.

And still, it is a bad hack. My original intention is not to watch video (though that is not bad either) on the terminal, but display real-time graphics with openGL. I do not need an opengl application, I need something like ipython where you can edit both text and display graphics without the monster of a browser.

At the end, honestly I do not need anything. It is just hobby work playing with imagination, trying to take the idea of terminal to the next level. I thought kitty would have that philosophy but with great regret it was a big disappointment to understand that IT IS NOT. I strongly ignore childish and self-limiting reactions: "this is not what you do in a terminal emulator". My imagination is not yours...

I find it so sad to bump into "Linus Torvalds" like behavior that is sucking energy and definitively creating more harm than added value. Projects come and go for that reason. One creates a project and thinks he is UBER on the possibilities can decide what is a terminal supposed to do etc.

At the end I have other potential continuation points
https://github.com/glumpy/glumpy/blob/master/examples/gloo-terminal.py

I do not care about the speed of scrolling that kitty authors are obsessed about, I care about freedom of building and combining ideas.

Just as another idea to extending the graphics protocol. You could have a plugin that is displaying rich text and make something like this possible inside a terminal at a reasonable speed:
-> https://github.com/rhysd/NyaoVim

[SolitudeSF]

One creates a project and thinks he is UBER on the possibilities can decide what is a terminal supposed to do etc.

creator of the project most certainly can decide what his project supposed to do.

I do not care about the speed of scrolling that kitty authors are obsessed about, I care about freedom of building and combining ideas.

why are you writing this in kitty issue tracker then?.

also, its open source. if your vision of how kitty should function differs, you are free to fork it.

[zokrezyl]

@SolitudeSF lot's of things do not make sense in this conversation. The obvious in form of noise you just wrote makes even less sense. Please stay on topic

[Luflosi]

@zokrezyl I don't think @SolitudeSF's comment was noise, his last sentence is especially relevant.

I changed your code a bit: https://gist.github.com/Luflosi/365b9db2e88de821ba114aa792e8f6f5
It relies on my kitty fork, which does not delete shared memory objects. On my 7 year old MacBook Air, I get a pretty respectable frame rate of 30 fps when removing the delay in the code and when using the full screen with a resolution of 1440x900. With a lower resolution the frame rate can get much higher. I think the bottleneck is my integrated Intel graphics chip.

[zokrezyl]

@SolitudeSF comment is noise as he is telling the obvious "The obvious in form of noise". Sounds like a parrot at a party. I am trying to prove it by asking: Do you think there is someone around here on GitHub that would ask for allowance or "free"-dom to fork a project? Unlikely...

Combining https://github.com/neovim/libvterm and https://github.com/glumpy/glumpy/blob/master/examples/gloo-terminal.py are for sure way better choices than fork this one for what I would like to see...

But let's stay on the topic...

I have over 100fps on my 10 years old workstation with 1920x720 (original size of the video I was testing), that is not a problem. I introduced the delay to see the video at normal speed.

But as said find it a hack even using it for the video. When you could create a plugin for playing the video and you just pass the filename like you do it with the image, or stream encoded video data. But am afraid we already stepped into the realm of https://en.wikipedia.org/wiki/Confirmation_bias

Maybe you can ask the author at least to fix that ugly behavior of deleting the shared memory object.