Neovim: :terminal reflow

[trusktr]

Terminal reflow (line wrapping) would be nice. Here's a video of what I mean in OS X Terminal and iTerm2: https://youtu.be/sFJtCBM00dg

[curvi]

Thank you for taking my suggestion seriously.
I don't really know how the terminal emulation works in nvim, but it should be possible to redraw the screen periodically, when a resizing event is detected. The size calculation is done at the moment only once, but the logic should stay the same as much as possible.

[tarruda]

I'm not sure we can have this unless libvterm adds support itself. cc @leonerd

[critiqjo]

If you run tmux inside, then tmux will handle all reflows. But to scroll up, you'll now have to use tmux's copy-mode.

And further, if you're running neovim itself inside tmux, it'll start getting messy (unless you remap the tmux prefix for the session inside neovim, and can keep up with all the key-bindings and modes -- neovim's normal/terminal modes, copy-modes of tmux-es both inside and outside!).

Just a suggestion!! :) Since there haven't been any update on this... :(

[curvi]

Thanks, that's actually a great idea. However I was asking, becuase I use nvim inside tmux at the moment. And I'd spare myself 1 program if I could have a simple but sleek terminal inside nvim alone (no reflow is actually a dealbreaker in tabbed views).
Hopefully the project nvim chose was a good choice and they improve their base... (I don't have much hope for that).

[tarruda]

Hopefully the project nvim chose was a good choice and they improve their base... (I don't have much hope for that).

Libvterm code is actually easy to follow, so it is mostly a matter of sending a patch to implement the feature. When more urgent issues are dealt with, I will see if I can port an existing implementation(gnome terminal, tmux...).

[vito-c]

+1

[josephholsten]

@tarruda Should this patch be made against https://github.com/neovim/libvterm/? Or where should that go?

[tarruda]

@tarruda Should this patch be made against https://github.com/neovim/libvterm/? Or where should that go?

it should go to libvterm

[justinmk]

Libvterm already fixed it. I'm sending a PR now.

[trusktr]

This is closed due to #4412, but that's just an upgrade to libvterm. Does that automatically add the terminal reflow? Or does that need to be separate enabled? Just making sure this issue shouldn't still be open.

[trusktr]

I'm in NVIM 0.2.0-dev, but I don't see terminal lines being kept on a single line. Is there an option?

Is it in my version?

 0 withme git:jpea /usr/bin/nvim --version                                                                                                                                                               ✹
 1 NVIM 0.2.0-dev
 2 Build type: RelWithDebInfo
 3 Compilation: /usr/bin/gcc -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Wformat-security -Werror=format-security -Wconversion -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -O2 -g -DDISABLE_LOG -
 4 l -Wextra -pedantic -Wno-unused-parameter -Wstrict-prototypes -std=gnu99 -Wvla -fstack-protector --param ssp-buffer-size=4 -DINCLUDE_GENERATED_DECLARATIONS -DHAVE_CONFIG_H -D_GNU_SOURCE -I/build/neovim
 5 WnFK/neovim-0.1.5ubuntu1+git201612051201+2941+22~ubuntu12.04.1/build/config -I/build/neovim-shWnFK/neovim-0.1.5ubuntu1+git201612051201+2941+22~ubuntu12.04.1/src -I/build/neovim-shWnFK/neovim-0.1.5ubunt
 6 git201612051201+2941+22~ubuntu12.04.1/.deps/usr/include -I/usr/include -I/build/neovim-shWnFK/neovim-0.1.5ubuntu1+git201612051201+2941+22~ubuntu12.04.1/build/src/nvim/auto -I/build/neovim-shWnFK/neovim
 7 1.5ubuntu1+git201612051201+2941+22~ubuntu12.04.1/build/include
 8 Compiled by root@lgw01-42
 9
10 Optional features included (+) or not (-): +acl   +iconv    +jemalloc +tui
11 For differences from Vim, see :help vim-differences
12
13    system vimrc file: "$VIM/sysinit.vim"
14   fall-back for $VIM: "/usr/share/nvim"

[trusktr]

@justinmk ^ Do I have the right version? or do I need to set an option?

[joshuarubin]

It seems that this works for the "current line" (where the terminal cursor is), but not the lines above it.

[kennylevinsen]

Yeah, I'm definitely not seeing reflow in 0.2.0.

[tbo]

The issue persists for me as well.

@justinmk Shouldn't we reopen this issue?

[justinmk]

Reflow works for interactive programs such as top. Libvterm doesn't support anything else.

[josephholsten]

it's good to know that libvterm supports interactive programs, but doesn't actually speak to the issue as originally described.

I think we all agree that this is a new feature request for libvterm, not a report of functionality regression. For those interested in actual implementation, a git mirror lives at https://github.com/neovim/libvterm/ and the official launchpad project lives at https://launchpad.net/libvterm/.

[josephholsten]

In fact, this seems to be the same issue as https://bugs.launchpad.net/libvterm/+bug/1440605

[josephholsten]

@tarruda you mentioned copying an existing impl, the launchpad bug above includes a link to iTerm2's maintainer documenting how they solve the problem: https://gitlab.com/gnachman/iterm2/issues/3480

interestingly, the need for that implementation is "was what prompted [gnachman] to fork the project" originally

[leonerd]

For the record: I'm not against the idea, I just think it will be nontrivial to implement with some tricky edge cases and needs quite a bit of testing. I'm happy to assist people with it.

[i5513]

Hello,

Why is this issue closed (sorry I have just read https://github.com/neovim/neovim/issues/2514#issuecomment-267498423 and understand)? I would like have wrapped lines, without any newline added. When I use $ it should go to the end of the line, not end of the "row"

Is it not posible to cat a file and copy - paste to another buffer. I think it should be a very important feature. Is there already an issue to track this ?

Thank you !

[devth]

This is one of the last things I'm trying to figure out on my journey to replace tmux with nvim. Hope this can get implemented!

[leonerd]

@devth Then do help me with it. There's a whole tonne of "How on earth is this supposed to work?" design questions currently blocking it.

[devth]

Without having any context my initial uninformed naive thought is:

Do whatever an existing implementation already does. tmux, iTerm2, macOS Terminal seem to do the right thing.

[justinmk]

I'm not sure we actually need libvterm to implement this. We already get the desired behavior in all of the "scrollback" lines. So a PoC hack would be to temporarily put the buffer viewport into "scrollback" then bring it back into view.

[leonerd]

@devth When to and when not to break lines.

The fundamental question is: When the terminal is made 1 column wider, how do I know, for each line in the display, whether to fill in the new column on the right by pulling the leftmost cell from the next line down, and when to leave it alone and fill in blank? Or alternatively, when being made 1 column narrower, which lines do I push down to the next and which not?

If other terminals do the right thing, can you do some research on those into various rules that they take? I want to know how I can decide when to do that and when not. It seems to me there'd be many many variations and special cases.

[josephholsten]

@leonerd can we start with an answer of "always"? Assuming the data being shown is stored as a list of lines, when is "always" the wrong answer? I don't think we want magical word-aware line wrapping, just to have the characters flow back when a narrowing operation is followed by a widening operation.

[leonerd]

@josephholsten It won't be that simple. I presume when you make the window wider, you want the beginning-of-line prompts from previous lines to stay at the beginning of the lines, rather than be wrapped around to fill the expanding width?

Basically, the problem is: print 1000 digits of pi, then resize wider - you want them to reflow. print an 8x8 chess board, then resize wider - you don't want them to reflow. How do you tell the difference?

[jamessan]

Basically, the problem is: print 1000 digits of pi, then resize wider - you want them to reflow. print an 8x8 chess board, then resize wider - you don't want them to reflow. How do you tell the difference?

That was already implied in the previous post:

Assuming the data being shown is stored as a list of lines.

In your example, the 8x8 grid would be stored as 8 distinct pieces of data while the digits of pi would be one piece of data. The data and how it is displayed are orthogonal, but dependent.

[jamessan]

There's also the previously linked discussion of how iTerm2 actually implemented its support.

[josephholsten]

@leonerd I'm sorry, I honestly can't tell if you're joking with me. Write a script to print the digits of pi. run it under any of the other programs mentioned. Observe what happens when you narrow the width of the terminal. Here's something similar I have lying around:

#!/bin/sh
# ruler: because 72 characters is supposed to be enough for anyone
echo "         1         2         3         4         5         6         7  "
echo "123456789012345678901234567890123456789012345678901234567890123456789012"

Then, write a script to print an 8x8 chess board in a terminal. Then post that code because I haven't got any clue what you're talking about. It doesn't make sense to try to fit a curses interface into a space that's narrower than its minimum width, so I assume that isn't what you mean. And if you're talking about a table of descriptive notation, then that's just 8 lines of text which should behave as the previous example, so I assume that isn't what you mean either.

Here's another way to think about it: say I run some commands via :.!my-command so the output ends up in the buffer. What do you expect to happen when the window is resized? /That/ is what is desired. Honestly, that's pretty much all that acme used to do, and all I really want is for neovim to be as powerful as acme was in 1995. For bonus points, imagine :set nowrap.

[leonerd]

@josephholsten Yes; I know what these terminals do in those two simple examples. I don't know how they decide which things to do.

Lets pick a really simple case:

echo "A"x100
echo "B"x100

we're expecting to see 100 "A"s, then 100 "B"s. On a terminal only 80 columns wide, this will spill over 4 lines in total - a line of 80 "A"s, then 20 more "A"s then the rest is blank. Then a line of 80 "B"s then 20 more "B"s and a remaining blank line. In total, 4 lines.

Now make the terminal wider to, say, 90 columns. Lets guess what happens.

You and I both know (and can expect) that this means now we've got two lines of 90 characters each, followed by two lines of 10 remaining characters.

But how does the code know that? How does the code know, in the first of each pair of lines, that it has to steal those 10 filled characters from the start of the second of each pair of lines, and generally shuffle them around, but not to "borrow" 10 of the "B"s to fill in the gap at the end of the second line, where there used to be 20 "A"s and 60 blanks? It seems that for some lines of the terminal content, they have to steal the leading few cells of the following line in order to make them wider, but for other lines of the terminal they don't.

This clearly has something to do with whether 80->81st column spillover happened as a result of overstuffing of line line.

What if instead we did

echo "A"x80; echo "A"x20
echo "B"x80; echo "B"x20

Visually, the screen looks now identical to how it looked after the first script, but now when we expand this from 80 columns to 90, none of the lines have wrapped around at all; we keep all of them exactly as they were. Do we? Or do we reflow them anyway?

All of these examples have been the simplest possible things - no cursor moves, no edits, all entirely 100% just character prints. Imagine some much more complicated examples involving lots of cursor moves, deletes, inserts, scrolls etc... to create the history of what the content was. And now after all of that imagine trying to decide, per line of the screen, whether when making it wider it should steal characters from the start of the next line, or just be extended with blanks.

Can you now write out those rules for me in English? Because I for one don't really have a clear idea of when and when not to.

[leonerd]

@jamessan The iTerm2 link talks about "continuation marks". OK. that's a good discussion point.

On resize, we reflow or not any given line from its subsequent, depending on whether it has a continuation mark. That's an easy statement.

Now can we list the rules for when we add or delete a continuation mark from a line?

Attempting to demonstrate how this isn't so simple: lets imagine a sequence in which we move to a line (lets pick 10), clear the entire line, move to 79th column, print 3 letters (thus spilling around). Move back to 1st column of 10th line. Print three more. Now make wider - do we reflow this line? Did it still keep its continuation marker?

OK what about: move to line 15. Print 100 characters (e.g. like the the "A"x100 example above). Move to start of line 15 again. Erase to end of line. Print 30 characters then a linefeed. Now resize wider. What happens to those final 20 "A"s that ended up on line 16 - do they get reflowed into the end of line 15, as we would have done had we not done EL/30 more/LF? Or do they now not reflow?

It seems to me at least, the choice of whether-or-not to reflow on a given line is quite a complex and subtle question involving the previous edit history of both that line and those around it. I again invite someone to write down for me in English, the rules for deciding when to do this. Once we have some rules, it should be easy enough to go about implementing them.

[jsbronder]

I don't know if it'll help but the discussion around implementing this in libvte can be seen here, https://bugzilla.gnome.org/show_bug.cgi?id=336238. The comments (and attached documents) are pretty in-depth.

[zwegner]

I've been annoyed by this behavior for years now, so for the last few days I've been working on fixing it. I've lost important data that got truncated when I accidentally did a window split, and I've had to manually join together countless lines yanked from terminal windows. Well, at least things are a lot better than the old ConqueTerm days, as noble of an effort as that was...

Anyways, my forks are here:
https://github.com/zwegner/libvterm
https://github.com/zwegner/neovim

For now, I'm working directly off the neovim fork of libvterm, but obviously it would be cool to get this merged in upstream libvterm eventually.

I'm maybe about halfway done now. I have most of the basic reflowing working in libvterm, with only some minor modifications to neovim itself. There's still a few glitches, weird behavior, and unfinished parts (I'm completely ignoring double-width characters for now), so it's definitely not ready to be used yet. The whole libvterm test suite is passing (except for one single test, with a known fix), but plenty of new tests need to be written.

Implementation notes for the nerdy:

Each screen cell gets a "wraparound" attribute. This is only used for the first column in each row, but it's just one bit, and it fit nicely into the VTermScreenCellAttrs/ScreenPen bitfield (which was previously 15 bits), so it's stored for every screen cell. The bit is set when writing text causes a wraparound from the last to first column, and not from any CR/LF/etc characters. The bit is cleared when erasing the screen (either from internal scrolling activity, or from the erase ANSI codes). Printing normal characters will not overwrite the bit though, so if you have a long line wrap around, followed by a CR and some more text, it will remember the fact that the second is part of the first line. The rules are pretty simple now, but it's possible that some more logic will need to be added here to account for all the edge cases. When resizing the terminal, we check for this bit in the first column of each row of the old screen and join together lines, to be reflowed into the new screen geometry.

I've had to modify the scrollback API for libvterm. Instead of passing one row's worth of cells each to neovim (or whatever library client), libvterm will now pass one line's worth of cells, or possibly a partial line if some of that line is visible on the screen.

The main important change in the libvterm API is that ownership of scrollback memory is now traded back and forth between libvterm and neovim. Previously, there was no sharing: libvterm would allocate a scrollback line, and would expect neovim to copy it into it's internal memory in the sb_pushline callback. Likewise, for a pop, libvterm would allocate one row's worth of memory, and expect neovim to fill it in. Since scrollback lines are arbitrarily sized now, libvterm can't allocate the memory up front for neovim to copy a line into (at least without providing another callback or something silly like that). So now, when scrolling, libvterm will allocate a scrollback line, and pass ownership of it to neovim through sb_pushline. After this, libvterm won't touch it anymore, and Neovim can do whatever with the line, until the next sb_popline. There, neovim passes control of the line back to libvterm, and can't touch it afterwards. One of the neovim changes I made for this was to use libvterm's allocator hooks, so that libvterm allocates all memory through neovim's APIs (xmalloc etc). Right now neovim and libvterm just use small wrappers around malloc(), but this ensures that the memory sharing will work if someone changes neovim's allocator.

What's left to do:

• Write a bunch more tests. There's some hairy logic here, and I want this code to be 100% rock solid (as bad as losing some truncated terminal lines is now, having vim crash is way worse).
• Support wide unicode characters properly (that is, don't split them across lines)
• Make neovim aware of the new scrollback line structure when rendering: right now, if you print out long lines in a terminal, exit terminal mode, and scroll up, neovim will silently truncate each line at the rightmost column. This will involve some of the same problems as reflowing within libvterm, but presumably this is already solved somewhere else in vim (basically, how does vim flow text from a file?). I haven't yet looked into neovim enough to see if there's a clean way to use pre-existing text flow code here.
• Optimize neovim's scrollback handling. At the very least, we should use a circular buffer for scrollback lines. It would probably make sense to adopt a block scheme like the one described in George Nachman's writeup on iTerm.
• My libvterm git history has several revisions of ideas as I've fleshed out the implementation. It would probably make sense to squash some of them in a rebase to make reviewing easier.

[leonerd]

@zwegner I'll take a look, though parts of it don't sound quite the same as what I had in mind. I'll see if we can converge them

[leonerd]

@zwegner I've had a quick look at the code, and yes this is quite a larger change than I felt was really necessary. It should be doable without quite so much change to the scrollback logic. Perhaps we can discuss that one?

I have already started thinking on this and I just had some questions relating to what the required behaviour ought to be in a few edit cases. The rest of the logic isn't hard to write but until I have some clarity on the test cases, I can't write a full implementation to cover all of them.

I started work at:

https://blueprints.launchpad.net/libvterm/+spec/reflow-cases

if you could take a look at those and run them through a few other example terminals you might have, and also provide any comment on whether they should reflow or not - as compared to whether they actually do. That's basically all I need - once I know what the behaviour ought to be then this is a simple enough change to write.

Once again in summary: the required behaviour is the complicated part here, not the code. Just describe to me the behaviour and I will write the code.

[zwegner]

Which branch? Note that we're stuck on an older version of libvterm master, blocked by #9494 (if you have bandwidth to take over that PR, that would be massively helpful).

@justinmk: Right now the nvim branch. I didn't realize it was behind the upstream master. From a quick look at that PR, I'm not entirely sure what the problem is, but I can look more later.

[zwegner]

@leonerd From those test cases, I think the behavior described for libvte/uxvrt looks correct, and OS X's Terminal.app behaves the same. Right now I'm reflowing D as well, but as I said that logic might need some tweaking.

If these test cases are the main things you're concerned about, though, I think you might be underestimating the complication of properly dealing with wrapping and scrollback, especially when a scrollback line is partially visible.

Consider the case of printing a 10000 character line in a 80x24 terminal, with the cursor on the next position after the end of the line. The cursor should be on column 0, as 10000 % 80 is 0. After resizing to 79x24, the cursor should be on column 10000%79=46, and so on. With the current scrollback scheme of fixed-length rows, handling this case correctly would be quite complicated, and probably very slow.

Here are some properties that I think a reflowing solution must have:

• The transition from the screen controlled by libvterm and the scrollback controlled by neovim should be absolutely seamless
• After printing out any amount of pure text, with no control sequences, and any subsequent resizing to arbitrary geometries (assuming infinite scrollback), yanking the text in vim should give you the exact same sequence of bytes

If there's a way to accomplish this with significantly less complexity than what I have in the code, I'm all ears. But I doubt it. I will say again that my commits so far have several iterations of ideas that could be squashed together, and there's a few simplifications I can make as well. Overall though, the changes are really not that big. Here's a diffstat of all my commits, looking only at src/ and include/:
6 files changed, 383 insertions(+), 126 deletions(-)
That's less than 300 lines added, many of which are comments or debugging code.

[zwegner]

Made some good progress, at least enough for a quick demo. Still lots more to do!

[leonerd]

I have been working on this too the past couple of days and I now have something quite similar in fact. It might be best for you to pause here for now and see if we can converge the two in a better way.

[leonerd]

Additionally, I realised I had forgotten to add a CONTRIBUTING file to libvterm; but I've added one now:

https://bazaar.launchpad.net/~libvterm/libvterm/trunk/view/head:/CONTRIBUTING

@zwegner - perhaps you could take a look there and then come find me at one of those locations and we can work on this feature there, rather than adding further noise on neovim :)

[washtubs]

Curious what the status of this is since it's been about 6 months. Just looked into using neovim as a tmux replacement and realized this issue was pretty much a deal breaker

[zwegner]

Well, to make a long story short, I gave up on it. I greatly underestimated the complexity of having the "correct" behavior without weird edge cases and terrible performance in some situations.

A rundown with (too much) more detail, in case anyone is interested:

• Terminal reflow, as I was implementing it, requires row-wise scrolling. Otherwise when terminal lines that wrap around are scrolled off screen, the bottom row of the terminal will jump up leaving blank rows at the bottom, since vim/neovim can only show a partial line at the top if it's the only line on screen. In the worst case, after you run a shell command like cat big_file_with_no_newlines.txt, the next shell prompt will be the top line, and the entire output of cat will be scrolled off screen.
• So, I spent a long time on row-wise scrolling (#11014). Like, a really long time, over a month's worth of full-time work. And this was really annoying, frustrating work. The vim codebase, with regards to the low-level text handling, is just... awful. So many years of cruft built up, with so little attention paid to having a sane architecture. There's so many options that affect rendering, and these options have influence over many of the lowest-level string handling operations. (Neovim is not much of an improvement in these respects, having mostly not touched this code, aside from removing the FEAT_* macros). The most basic version of row-wise scrolling was pretty easy to get going, but after that development got increasingly tedious, trying to debug all the weird corner cases.
• And even if I could manage to untangle all the years of hacked together code and refactor it into a state that generally works (which I believe is still far away), the performance would be a major problem. Neo/vim's data structures are fundamentally tied to lines, always stored as contiguous bytes (I had plans to change this, and replace the data structures, but this is a massive amount of work). This has implications all over the place in the rendering system. Basically, when a line is scrolled off screen, the size of the line in screen columns needs to be computed, which invariably involves a linear scan of the entire line (and this scan itself is quite non-trivial, consisting of several functions, and lots of intertwined options). Even adding a cache to not run the same slow scan repeatedly on the same line would be quite non-trivial to get right.
• Tying this back to terminal reflow, while slow performance with long lines would be acceptable for an opt-in standalone feature, it would be unacceptable IMO to have a huge performance regression for terminal use when the terminal program emits long lines. Likewise it would be unacceptable to force the non-row-wise scrolling jumping-prompt behavior described above on everyone. Making terminal reflow an option, so that reflow-but-with-bad-performance is opt-in, would be extremely complicated.
• On a related note, the API between libvterm and neovim would get much more complicated. I thought about this a good amount, and it's still not clear to me how to make a clean interface to pass lines (not screen rows) back and forth between them, properly handling screen damage. My latest versions of both libvterm and neovim had a very ugly and inefficient API hacked together to get the right behavior.
• So all in all, when finishing terminal reflow and row-wise scrolling meant lots more of my least-favorite kind of coding, all done for free, with questionable interest from the community, along with the prospect of trying to merge patches upstream to vim, and dealing with all the icky tradeoffs involved in these features, etc., it was really just not worth my time.
• Looking back at my repo, I have a decent amount of unpublished work. I'm not sure what the exact status of it is, and it looks like a lot of my WIP is not in a publishable state. Honestly I don't even think it's worth anyone's time trying to continue this work. Vim is just broken.
• Before completely giving up, when I still had plans to replace the neovim data structures, I started an experimental text editor backend using a new data structure that I believe is novel. I began building a basic text editor UI in Lua to test it, and quickly found that I was having much more fun without dealing with all the years of technical debt in neovim. Within a couple weeks, I had quite a few features, including row-wise scrolling using only O(log n) algorithms. I have the source here: https://github.com/zwegner/zmt. I've been sidetracked on several other things and haven't worked on it in a while, and it's a long way away from a usable text editor, but I have plans for several "killer features" that would be very difficult to implement in vim.

[bfredl]

@zwegner I understand. Good luck with you new editor work.

Looking back at my repo, I have a decent amount of unpublished work. I'm not sure what the exact status of it is, and it looks like a lot of my WIP is not in a publishable state. Honestly I don't even think it's worth anyone's time trying to continue this work. Vim is just broken.

I would be happy to take a look at your work. Even if we probably won't use much of it as-is, there likely will be ideas and design considerations we could derive from it. (if you don't want to be bothered with questions about it that is fine. Just us looking at the code is much better than throwing it all in the garbage bin, after all this time)

, along with the prospect of trying to merge patches upstream to vim,

To be clear (to any one else potentially interested) interaction with upstream vim in any direction is _never_ any sort of requirement when doing feature work on neovim. We take the opportunities to collaborate with vim when such opportunities clearly exist, but uncertainty of what vim wants is _irrelevant_ when considering new features or heavy refactors. If a (future proposed) heavy refactor of the memline or displayline logic makes vim patches hard to merge, then so be it. Merging vim patches for features and bugfixes is surely useful, but If neovim dev is reduced to an exercise in constantly considering vim patch mergability, to the impediment of feature work, there is no point in having a separate codebase in the first place.

[leonerd]

I'm still intending to look into it. If folks have test cases/examples/etc that would help me a lot.

[bfredl]

@leonerd Yes, so are we. As a fallback plan we should consider "physically" reflowing the actual buffer, instead of storing the scrollback as long lines in the buffer. I'm not sure if that makes any difference to the interface between libvterm and neovim.

Do you need more input before making a first prototype i e using pangoterm? Otherwise I think the best approach is to make something that people can try (and then complain if it really is wrong, but most people will be happy with the common cases working).

[leonerd]

@bfredl I have some rough ideas. What I'd really really really like is for someone to write me a bunch of .t files to go in the ilbvterm unit tests to demo their expectations. So far I haven't seen any

[leonerd]

With regard to what makes things simple or complicated: I think a massively simpler cut-down version would be to only reflow what's "live" on the onscreen buffer and ignore scrollback lines. Doing it that way wouldn't change the libvterm<->embedding API at all; and thus pangoterm, nvim, et.al, wouldn't need to worry. That would at least give a first step to letting people see what the reflow behaviour works like; and about 98% of the "it's awkward" interaction cases happen there, not in scrollback.

[leonerd]

If anyone wants to try out a live-only (without scrollback) reflow version of libvterm there's a bzr branch at

http://bazaar.leonerd.org.uk/c/libvterm.WIP-REFLOW/

[artem-nefedov]

Big thanks for the work done so far, and (possibly) in the future.
Just wanted to point out that this is probably the last major thing that prevents neovim from being a fully fledged terminal multiplexer (everything else is customisable).

[zwegner]

@zwegner I understand. Good luck with you new editor work.

Thanks! BTW the backend is not quite suitable for a memline replacement yet, but that was the original plan. Might be worth looking into down the road if somebody wanted to take on that huge project...

I would be happy to take a look at your work. Even if we probably won't use much of it as-is, there likely will be ideas and design considerations we could derive from it. (if you don't want to be bothered with questions about it that is fine. Just us looking at the code is much better than throwing it all in the garbage bin, after all this time)

After a night's rest, I think I was probably being a bit over-dramatic there :) I'm sure there's salvageable pieces in it, and it's probably worth having row-wise scrolling as an experimental feature, with possible glitches and bad performance.

I'll try and clean things up enough to be presentable sometime soon. There's lots of small random changes I have that I can't remember the purpose of, which could be confusing...

To be clear (to any one else potentially interested) interaction with upstream vim in any direction is _never_ any sort of requirement when doing feature work on neovim. We take the opportunities to collaborate with vim when such opportunities clearly exist, but uncertainty of what vim wants is _irrelevant_ when considering new features or heavy refactors. If a (future proposed) heavy refactor of the memline or displayline logic makes vim patches hard to merge, then so be it. Merging vim patches for features and bugfixes is surely useful, but If neovim dev is reduced to an exercise in constantly considering vim patch mergability, to the impediment of feature work, there is no point in having a separate codebase in the first place.

Right--I shouldn't have implied otherwise. You and Justin were both clear that diverging is OK. Even so, it still weighed on my mind, mostly in trying to structure commits/branches to make them more easily upstream-able, or thinking that somebody would want help integrating them into vim.

[zwegner]

@leonerd Yes, so are we. As a fallback plan we should consider "physically" reflowing the actual buffer, instead of storing the scrollback as long lines in the buffer. I'm not sure if that makes any difference to the interface between libvterm and neovim.

It occurs to me that one easy shortcut would be to do things pretty much as they are in my branches of libvterm/neovim, but forcibly break lines at 500 rows (or whatever, could be configurable). This gets around most of my objections, though the libvterm/neovim interface would still be weird. Row-wise scrolling works fine with short-ish lines and for basic options, like you'd have in a terminal.

[justinmk]

it was really just not worth my time. ... with questionable interest from the community

@zwegner Neovim project has funding for exactly this purpose: to support important but perhaps unpleasant work. Email or message me on https://gitter.im if this would help you.

• Reaching just the first step (row-wise scrolling) would be an excellent use of project funds.
  
  
  
  • This is a separate step from memline refactor.
  
  
• Community interest: https://github.com/neovim/neovim/issues/8715 has relatively high number of +1 votes. And it is a goal of Neovim in general, though it lacked an owner until your PR arrived :)

forcibly break lines at 500 rows (or whatever, could be configurable)

Perfect. Completely fine tradeoff. Partial "live" solution is better than forever dead.

I think a massively simpler cut-down version would be to only reflow what's "live" on the onscreen buffer and ignore scrollback lines.

@leonerd yes, please!

[skt041959]

What's the status of this issue? I just found that run :terminal powershell.exe in WSL, the powershell will offer auto reflow for the stdout.
Might give some reference for this issue.

[leonerd]

The status is I have a partial work-in-progress branch of libvterm which handles a tiny fraction of the cases it could handle, but maybe you'll find it sufficient. Try it out and let me know all the things it doesn't handle.

[skt041959]

@leonerd That's great! Thanks for your effort!

[zwegner]

I had talked with @justinmk a bit back in January about the possibility of getting funding to complete my work here and on row-wise scrolling--perhaps that's still an option.

IMO, for work to go forward with my initial design (allowing full reflow of all scrollback lines), it's essential that @leonerd and I can work out a good interface between libvterm and neovim. This isn't trivial, since a grid-based API (like libvterm trunk) vs. line-based API (my fork) are quite different.

My libvterm fork has already diverged from upstream a good deal, and if any of my reflow work is to survive, it'd be nice to nail this down and get the forks merged.