Terminal: Add support for VT102 Insertion-Replace Mode (IRM)

[markmi]

Insertion-Replacement Mode (IRM)

ESC  [   4   h
033 133 064 150

Set selects insert mode and turns INSERT on. New display characters move old display characters to the right. Characters moved past the right margin are lost.

ESC  [   4   l
033 133 064 154

Reset selects replace mode and turns INSERT off. New display characters replace old display characters at cursor position. The old character is erased.

---

original issue body follows.

The following is from recording command line editing in an ssh session to a freebsd environment (via FreeBSD's script). The displayed text is odd in Windows Terminal. First I show a textual form that shows the escape sequences and such in a readable form (I eliminated bell-ring characters from the file and other lines of text):

more ~/odd_commnd_line_editing_display.txt
ls ~/sys_typescriptsESC[1ESC[4h-ESC[4lESC[4hlESC[4lESC[4hTESC[4lESC[4hdESC[4lESC[4htESC[4lESC[4h ESC[4

A normal display in other environments (for example macOS terminal doing an ssh into FreeBSD) looks like:

cat ~/odd_commnd_line_editing_display.txt
ls -lTdt ~/sys_typescripts/

(The above shows the actual ls command that was executed.)

But under Windows Terminal (Preview) it shows:

cat ~/odd_commnd_line_editing_display.txt
ls -lTdt ~ypescripts/

This is also how it ends up looking during the command line editing,
not showing the actual text of the command to be executed: missing
some text.

The sequence is from deciding to add "-lTdt" after having typed the
path. (It was a "set -o vi" type of command line editing context.)

I was using TERM being xterm-256color on FreeBSD .

[j4james]

I suspect this is because the Windows terminal doesn't yet support Insertion-Replacement Mode (IRM).

[DHowett-MSFT]

Interesting! Didn't know IRM was a thing. I'm going to repurpose this as a task to implement IRM.

Notes:

Insertion-Replacement Mode (IRM)

ESC  [   4   h
033 133 064 150

Set selects insert mode and turns INSERT on. New display characters move old display characters to the right. Characters moved past the right margin are lost.

ESC  [   4   l
033 133 064 154

Reset selects replace mode and turns INSERT off. New display characters replace old display characters at cursor position. The old character is erased.

[egmontkob]

Yet another symptom of the same bug:

Resize the window to 63 columns. Start mc (Midnight Commander) on Ubuntu, compiled against the S-Lang screen drawing library (as it's done in Ubuntu).

Expected: The bottom row should display 9 buttons, each of them being 2 characters for the function key number + 5 for the action. The rightmost should be <space>9Pu~Dn, reaching all the way to the very end.

Actual: <space>9Pu~D is printed with the expected colors, and the last column remains empty, showing the terminal's background color.

What the S-Lang library does is an ancient and obsolete hack for some terminals that don't have "pending wrap" (@j4james do you know more about them?), to print in the lower right corner without the cursor automatically wrapping to the next line and shifting everything up. It prints the intended last letter (n in this case) in the penultimate (62nd) column, and then using IRM, prints the penultimate letter (D) again in the penultimate (62nd) column, expecting it to push the previously printed n to the 63rd.

Even at bigger window sizes, such as the normal 80 columns, mc misses to paint the bottom right corner using _some_ of its skins, e.g. F9 -> Options -> Appearance -> sand256 is one of these. For some mysterious reason, S-Lang decides to use different drawing method depending on some property of the skin. In this case, it repeats the aforementioned trick with space characters, whereas with most other themes, it just emits the escape sequence that wipes to the end.

This is a tiny inconsistency (almost-bug) somewhere in S-Lang, my wild guess is that it could be related to non-color attributes (in this case italic) being used there, but I don't think it's worth investigating S-Lang's behavior any further. Also, for the record: I couldn't reproduce the issue with mc compiled against ncurses, apparently it knows about "pending wrap" and knows that this hack is not necessary.

[j4james]

What the S-Lang library does is an ancient and obsolete hack for some terminals that don't have "pending wrap" (@j4james do you know more about them?)

I know DOS and the original Windows shell didn't support "pending wrap", but then they wouldn't have supported IRM mode either. However, it wouldn't surprise me if there were other third-party terminal emulators for Windows that shared that wrapping behaviour, but were still able to handle IRM, so maybe that's the sort of thing that S-Lang is targetting.

By the way, IRM is on my TODO list (assuming nobody else takes it on first). It's just that we don't yet have a handler for ANSI modes, and the existing dispatch mechanism makes that a bit of a pain to add, so that was something I was hoping to address first. There are various other issues related to parameter handling (e.g. #2101 and possibly #942), and I wanted to make sure we had something that could satisfy all of those requirements if possible, so it's probably not going to be a quick fix.

[zaphod77]

this is a a real issue. pico editor seems to use this when ran under emulated xterm. and the characters do not move to the right like they are supposed to.

still an issue as of this day.

[henrikj242]

I'm also affected by this. Is there any known way to work around it?

[j4james]

I was actually looking into this recently, and I think it's going to require some architectural decisions before we can consider implementing anything.

I'd initially thought that the AdaptDispatch class could handle the inserting prior to writing anything, by simply scrolling the line it was about to write into by the appropriate amount. But that assumes it knows how much space the text is going to take (which it doesn't), and if the text is going to wrap onto another line, then potentially handle inserting on later lines (which may not even exist yet if vertical scrolling is required).

As things currently stand, I don't think this is possible. So either the inserting must be handled in the PrintString implementation itself (which is essentially the WriteCharsLegacy function that we don't want to touch). Or the AdaptDispatch class needs to take over the majority of the text layout, and have PrintString be a very basic operation that only writes to a single line (no inserting, wrapping, scrolling, or margin checks).

Considering the additional complications that are going to be introduced to the layout as we expand the VT functionality, I think the latter option may well be the best long term approach. But that is going to require some additional APIs, e.g. to query how much space a string of text will occupy and/or where it would be clipped.

[philkernick]

I also have the same problem, but I have a workaround.

FreeBSD uses termcap (rather than terminfo) and the termcap entries for xterm include the 'im' and 'ei' capabilities which invoke the VT102 IRM. In the Linux terminfo database, these capabilities are not implemented.

The workaround is to create a .termcap file in your home directory with this one entry in it:
xterm-256color:mi@:IC=\E[%d@:ei@:ic=\E[@:im@:tc=xterm-new:

This will use this instead for the xterm-256color terminal (which is what Windows Terminal is), and delete the 'im' and 'ei' capabilities.

Until the IRM capabilities are implemented, this works perfectly.

[DHowett]

It's wild how IRM works with line wrapping in xterm (351). Text is pushed off the right margin, as expected, but when the inserted region is wider than will fit _it_ wraps and pushes text on the _next_ line to the right.

This is a terribly specific feature. I wonder why it was introduced!

[vixie]

no version of xterm has ever not supported IRM. if you're not going to implement it, please identify yourself as a VT100 instead of xterm256-color. the pending-wrap thing is a side show, don't worry about it. just please make :ei=\E[4l: and :im=\E[4h: work. i've gone back to konsole as a result of this bug.

[zaphod77]

I have to use putty because of this STILL, even though putty doesn't have perfect xterm emulation either. For those of you who can't seem to reproduce this, your pico is actually nano, which has a workaround for this bug.

If you actually use the pico text editor, you can very easily see the behavior. it's the simplest test case.

[mreymann]

The workaround is to create a .termcap file in your home directory with this one entry in it:
xterm-256color:mi@:IC=\E[%d@:ei@:ic=\E[@:im@:tc=xterm-new:
Until the IRM capabilities are implemented, this works perfectly.

Thanks, your workaround fixes some weird behaviour when connecting to FreeBSD boxes. Waiting for a real fix ;-)

[vixie]

Marc Reymann writes:

...

Thanks, your workaround fixes some weird behaviour when connecting to
FreeBSD boxes. Waiting for a real fix ;-)

freebsd is not the only place this matters. other systems also expect that
if the offered terminal type is xterm256-color, then IRM will work. my
workaround is to use TERM=vt100, which means, my workaround is to use
konsole instead.

[zatarain]

Great!! I was dealing with this bug since April, it was really annoying. I just apply the workaround with xterm-clear:te=\E[?1049l:ti=\E[?1049h:mi@:IC=\E[%d@:ei@:ic=\E[@:im@:tc=xterm-new: and it worked!! I hope Microsoft implements the real fix soon!!

[DHowett]

I'm taking a crack at this.

It looks like we don't even have SM/RM, so there's a lot of plumbing to do.

[DHowett]

@j4james the approach I've gone with -- which is of course subject to discussion and eventual change :smile: -- is to make TextBuffer do an inserting write at the cursor position. I've not made any architectural changes to Adapt or change whose job it is to do text layout. I tested gnome-terminal here & found that they don't do any re-wrapping, _even if_ the line was emitted wrapped. Insert mode there impacts only the physical display & not the logical paragraph/wrap state.

[vixie]

happy to review or test.