Wezterm: on Windows: vim (in WSL or remote Linux system) replaces a character to `g` on file open

[hgkamath]

Describe the bug

This is vague bug. so requesting other users to report/comment if they have encountered this.

Upon opening, vim replaces a character to "g" on file open. The character replaced is at the cursor position of when the file was last closed. present current position is 1 word position back.
Happens spuriously, and many components are involved. I was hesitant to file bug for some time on account of not being able to acertain the component, So apologies if not on account of wezterm . It happens when typing fast
For example, executing ls(enter) ls(enter) and then vim \

If one closes the file (colon-q) without editting, then vim will warn the user that the file has changed, and this easy to catch.
If one makes changes and saves, it can be very sneaky. May result in error in program or typo in file.

I am unsure if the following is related which is also troublesome when it happens, just as often, but easier to spot.
Sometimes, pressing enter in the shell prompt, barfs up 'term escape seq'-like characters "[55;43R" as input to next readline, requiring clearing (end ctrl-u) before typing next command, in order to avoid command not found errors. I am unsure what they are, what they do or where they come from.

So, I wonder if there is something in the term input buffer, that vim is reading from the term upon switching to curses mode, which causes vim to execute vim commands like character replace. That is, term escape sequences are slipping into input characters.

workaround:
Habituate oneself to press 'u' undo on vim file open, to undo that chance automatic character replace on file open.

Other variants:

• Sometimes, vim cursor will be at the bottom right corner of screen. transient which fixes itself upon arrow key press
• Sometimes, vim edit cursor will have jumped around but close to last save position, even when not having done anything inside vim
• Sometimes, vim will already be in 'R' overwrite edit mode
• Sometimes, vim will have overwritten some text like "rgb:6565/7b7b/8383rgb:6565/7b7b/8383", and be in edit mode. the text does not come from any content. sometimes different, perhaps could be term color sequences.

fresh vim, no fancy vim configuration

Most likely explanation is that the key sequence r-g-b in vim command-mode is "replace character with 'g' and move one word back"

The TERM environment variable is "xterm-256-color", seems to be auto-set from pwsh

The nature of this corruption changes after changing TERM variable to "ansi" or issuing commands like 'stty sane'

Environment (please complete the following information):

• OS: Windows-10 (2004) 19041.329
• Version: Wezterm version testbuild runs/820655363 made for issue #235
  .\WezTerm\wezterm.exe -V
  wezterm 20200620-160318-e00b076c-13-gab1e83c7
  also happens on release build
  wezterm 20200620-160318-e00b076c

To Reproduce

Happens spuriously, when typing fast, opening, editing, closing, doing shell commands in between.
For example, executing ls(enter) ls(enter) and then vim might do it, but not always.
Does not happen when typing slow
Does not seem to be because of any typing mis-presses

Have seen it happen in 2 ways. I have 2 VMs (Virtualbox 6.1.0) on my machine, alpine-linux, fedora. In a wezterm window, I have 4 wezterm tabs open. The tabs are fresh. start new tab, start ssh session, navigate to folder, do vim-open-close cycle.

• win/wezterm-tab -> pwsh-core 7.0.2
• win/wezterm-tab -> pwsh-core 7.0.2
• win/wezterm-tab -> pwsh-core 7.0.2 -> sshclient (OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5) -> alpine-3.12: openssh-server (8.3_p1-r0) -> ash-shell (busybox 1.31) -> su root -> vim 8.2.0735
• win/wezterm-tab -> pwsh-core 7.0.2 -> sshclient (OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5) -> fedora-32: openssh-server
  (8.3p1-3) -> su root -> bash-shell(5.017) -> vim 8.2.1081

The bug happens with fedora /usr/bin/vim, fedora /usr/bin/vi and alpine /usr/bin/vim which are all 'Bram Moolenar' vim
alpine /usr/bin/vi is a pointer to busybox, has limited vi commands, does not do colors, and so this is not seen there

Configuration

nothing unusual. same as issue #235

Expected behavior

no sneaky character replaces.

Screenshots

NA

Session Recording

wt-record.sh does not have script command for windows

If the issue is with the way that escape sequences are processed it can be helpful
to capture the terminal output using the wt-record
script to run wezterm and record a transcript. This requires the script utility
to be installed on your system (this is part of macOS and available in the util-linux
package on linux systems).

In the example below a file named 20180225161026.tgz is produced. Please attach that
file to this issue, or if it contains private or sensitive issue that you don't want the
public to see on GitHub, please find some other way to get that file to a project
contributor (perhaps Dropbox or email?).

$ ./wt-record
Transcript recorded in 20180225161026.tgz

You can use wt-replay 20180225161026.tgz to replay that file.

wt-record can only record the terminal output; it cannot record the input events going
in to the terminal, so if you are having an issue with input, please be sure to describe
it below!

Additional context

Add any other context about the problem here.

[wez]

Thanks for reporting this; I have noticed this happen twice but didn't see it repro 100%.
Notably, you can see the effect of this in the commit summary of this commit when I quickly ran git commit --amend && git push and didn't notice that version changed to gersion:
https://github.com/alexcrichton/ssh2-rs/commit/5fe0a8cd26aa001209e1fbc99e1ecc48a9b18d61

This sounds like an issue in the shape of a response to a query issued by vim; there were a number of changes to improve esctest conformance in 20200620-160318-e00b076c and I suspect that some of those uncovered an edge case somewhere, or is tickling a compatibility issue with the Windows PTY layer and its interpretation of these sequences too.

If you have a chance to look at this a bit more before I next have an opportunity to fire up my Windows environment, running something like script vim in one of your remote sessions should capture the output escape sequences and might help focus the next steps.

[hgkamath]

testfile.txt
testfile.txt

script vr01.log
vr01.log

script -T vr01_t.log vr01_log
vr02.log
vr02_t.log

script -T vr03_t.log vr03.log -m advanced
vr03.log
vr03_t.log

script was executed inside vbox-6.1.10 fedora-32 VM
win/wezterm-tab -> pwsh-core 7.0.2 -> sshclient (OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5) -> fedora-32: openssh-server
(8.3p1-3) -> su root -> bash-shell(5.017) -> script (util-linux 2.35.2) -> vim 8.2.1081

In each attempt capture,

• execute script
• type 'vim testfile.txt \
• visually confirm that a 'g' was present (those attempts that did not exhibit aberration, the logs were discarded)
• then 'esc q' or 'esc colon q' maybe after a little arrow movement
• type 'exit\

cursor in file was located at line 13 or 14 around words attributes/configuration, which is where the 'g' was seen

• if you script-replay it in wezterm again and play it, as it is spurious it might do additional aberrations, may not do the same aberration ie replace a char with 'g'
• if you script-replay it in a different mature term-software, the spurious effect might not happen.

# scriptreplay -t vr02_t.log  vr02.log
# scriptreplay -t vr03_t.log  vr03.log

In hyper.js, in the script-replay of the logs, the words 'attributes/configuration' were corrupted a little differently. (nb redoing the whole ssh/attempt from hyper.js does not exhibit any abberations)

Also your term size may need to be exact

# tput cols
119
# tput lines
55

[wez]

Thanks for being so thorough with the data you've collected! I'm looking into this now.

[wez]

(My Windows machine is unplugged at the moment due to some re-organizing in my study, so I'm speculating from my linux machine)

I'm wondering if this might be a confluence of this change in wezterm:
https://github.com/wez/wezterm/commit/5ef357bb952e6f83bbea49c4b5329affe71057ee#diff-9ed8045486fc4b6c21417acd3141788d
(where we used to respond to color queries with #RRGGBB style colors but fixed our response to be rgb:rrrr/gggg/bbbb)

interacting with these issues in the windows pty layer:
https://github.com/microsoft/terminal/issues/3715
https://github.com/microsoft/terminal/issues/3718

I haven't been able to reproduce the same effect on linux (the scripts you provided do demonstrate what you saw; I just can't trigger the same issue when run natively on linux) so it does make me suspect something in the conpty layer.

I'll get my Windows system reconnected a bit later and play around a bit. I suspect that there are a couple of directions to go:

1. Updating our bundled conhost.exe to a more recent build from microsoft/terminal may help (or at least change the nature of the set of bugs!)
2. It may be possible to nerf color reporting if we know that we are going through conpty, but that feels a bit nasty. (Another option is to revert back to #RGB in the windows build, but that is differently nasty)

[wez]

I didn't have a super satisfactory time reproducing this issue on my windows machine; I saw it only once!

The non-determinism makes me suspect a timing/buffering issue so I pushed a speculative commit to buffer the writes around these responses and flush them explicitly when we have a fully formed escape sequence to send.

I haven't seen the issue with that new build, but I also haven't seen the issue with the current release after the initial occurrence, so I don't want to draw a hasty conclusion.

[hgkamath]

I downloaded the latest continuous build from 20200712 run 166557176

wezterm.exe -V
E:\apps_win\wezterm>wezterm 20200620-160318-e00b076c-25-ga9ee544d

In this testbuild, confirming that g-replacement is not reproducible, However, the following variation exists
Open vim testfile.txt, and first time around before closing file, position edit-cursor at letter b of the word attributes
Repeat the below key-presses mechanically in a non-stop loop

1. # script -T vr04_t.log vr04.log -m advanced
2. <up arrow> <enter> / vim testfile.txt <enter>
3. :q
4. # exit
   While g-replacement and word back does not happen, there are times, roughly 1 in 10 attempts, where vim has opened in vim-replace-mode as if 'R' was pressed already, breaking the non-stop loop typing action, requiring one to then undo the subsequent over-edits of : and q or press <escape>:q! ( exit vim by returning to vim-command mode and then quitting without saving)

yes, the non-determinism cold be either processor-clockspeed/time dependent or the memory leak could be build-binary/memory -placement dependent.
# script -T vr04_t.log vr04.log -m advanced
vr04_t.log
vr04.log

# scriptreplay -t vr04_t.log vr04.log

Another thing I had tried with the previous test-build, was creating a ~/.vimrc with the following line

set t_RB=
set t_RF=

From what I understood from the documentation (:help t_RB), those vim_term variables control whether vim does the background/foreground detection. unsetting them ought to disable it. What I had also noticed, is that very presence of an empty .vimrc, itself affected reproducibility. When an empty .vimrc is present, vim cursor would not be at last saved position. It seemed like I had to set the viminfo variable, as vim would not remember and recover last saved position mark. Without a .vimrc, a default viminfo='20,"50 is present by default. But my understanding/surity on these is not dependable. When an empty .vimrc is present other random overwrites could happen, maybe other text sequences are being read as commands, like 'gg' to goto start of file, 'I' to insert at start of line, '0' to goto start of line, '^' to goto first char of line.

The weird effects still exist in

• testbuild of 2020-07-17 wezterm 20200620-160318-e00b076c-30-gf64b268e
  The random effects include, replace mode, cursor to first char of line ('0'), content being overwritten with characters "[2;2R"
  vr05.log, vr05.log After entering vim, garbled text exits, 5 seconds later, pressing esc does it again, then esc-q! exits vim

[wez]

I'm curious about whether the variant bad behavior is still an issue now in master after updating openconsole.exe; if you have a few moments to run through your test scenario, please let me know!

[hgkamath]

yes I did, there are random effects that happen as often as 1 in 3 attempts. If at first they don't show up on their own, they can be made to happen by making changes to the .vimrc. It was not a viminfo/marker problem, commands like 'gg' can cause the effect of making cursor goto start of file and "I" to insert at start of file, '0' to start of line, '^' to first char of line.
Since the "rgb:6565/7b7b/8383rgb:6565/7b7b/8383" like sequences are not leaking, there is no "replace to g and back one word" effect, or at least I have not come across the effect yet.
A better title for this bug might be "vim terminal escape sequence query responses leaking into command/input buffer", so there could be more than one way these random effects happen.
This will be something I encounter, so I have been testing, and will test often until it doesn't happen. I will be appending the test result on a tested build to the testbuilds/releases list to the end of comment-6 and will attempt a script capture if possible.
With syntax coloring, I wonder if vim may make a random modification while scrolling editting the a file in between, that has not happened so far, but its also true I have not done a long duration editting, just small edits open close things.
This is a serious bug that effects functionality as it screws up trust in the editor. As mentioned it perhaps shows up in only remote-linux/WSL shells.

[wez]

How large is your terminal when you encounter this?
I'm wondering if this is a bad interaction with the output rate limiter that might be introducing a timing issue.

Could you try setting:

return {
   ratelimit_output_bytes_per_second = 4289999998,
}

to increase the ratelimit to just about as large as it can be set and see if that helps?

[hgkamath]

tried setting the large rate limit. no effect, random effects happen once in 3 attempts.

May not be because of excess rates because,its a VM on a laptop, VM being slower, and text comes over an ssh connection.

[hgkamath]

Some updates,
As I googled, I came to know that the "VIM starting in replace mode in WSL sessions" was a issue by its own.
VIM over SSH starts in REPLACE #4463

" FIX/Workaround: ssh from wsl starting with REPLACE mode
" https://stackoverflow.com/a/11940894
" https://stackoverflow.com/questions/51388353
if $TERM =~ 'xterm-256color'
  set noek
endif

While the above vimrc snippet, offered temporary relief, it made movement inside edit mode difficult as arrow keys can no-longer be used. The info hints that there may be some encoding-bugs or feature-deficiency with older openssh. All this while, in Windows-10, I have been using the default OS-bundled ssh-client

> C:\WINDOWS\System32\OpenSSH\ssh.exe -V
OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5 

I updated an instance of a local windows openssh server, to a pre-release version 8.1. The ssh-server also comes with its own ssh-client.

> 'C:\Program Files\OpenSSH-Win64\ssh' -V
OpenSSH_for_Windows_8.1p1, LibreSSL 2.9.2 

For the moment, with this ssh-client, no random effects seem to happen, I will wait a while before confirming none are happening.

Possibly, 'vim losing column position of last file close in the presence of vimrc' is another issue. I am looking into that at the moment.

[hgkamath]

It may be that the wezterm bug is resolved, if nothing else is found in two weeks.

I am 99.99% confident that the 'vim losing column position' is neither a wezterm issue nor an ssh issue. It's possibly a vim bug, but I could not find bug filed. The bug exists even when using the virtualbox direct console, and without using ssh. I am unsure why something like this is not caught by anyone elsewhere.

Documenting the issue here, as it has been mentioned in comments above.

Description:
vim, upon opening a file, positions the cursor only on the line, at the column of 1st indent` and not at the column of last closed.
The very presence of a .vimrc file, even an empty one, causes this to happen.
Resolution:
none found

Troubleshooting:
The commands `" and :call setpos('.', getpos("'\"")) work as expected and goto the mark correctly.

:autocmd BufRead
--- Autocommands ---
fedora  BufRead
    *         if line("'\"") > 0 && line ("'\"") <= line("$") |   exe "normal! g'\"" | endif
:

The autocommand on bufread does execute. The normal mode command g<singlequote><doublequote> means goto the mark doublequote, which is the last position of previous closing. The mark is correct, and is also recorded and loaded correctly form the viminfo file. But, on account of this bug, the g command always positions the cursor at the position of first indent, even when invoked on colon-command-line.

vim is the latest on fedora-32

# rpm -qa | egrep -i vim
vim-minimal-8.2.1224-2.fc32.x86_64
vim-filesystem-8.2.1224-2.fc32.noarch
vim-enhanced-8.2.1224-2.fc32.x86_64
vim-common-8.2.1224-2.fc32.x86_64

There have been similar reports, stackoverflow-31094647 and stackexchange-q2031 but they may be different issues.

[wez]

Since it seems like we fixed the wezterm side (improving buffering/flushing) and the remaining issue sounds like a vim issue, I'll close this one out.