Terminal: Vim background color renders incorrectly

[caksoylar]

• Your Windows build number: (Type ver at a Windows Command Prompt)
  Microsoft Windows [Version 10.0.18362.86]

• What you're doing and what's happening: (Copy & paste specific commands and their output, or include screen shots)
  In conhost or Windows Terminal, using both Vim.exe and WSL Vim, background color rendering becomes broken if a background color >16 or true colors is used.

With an example file, e.g. https://github.com/GilbertsHub/web-utils/blob/master/cgibinIntercept.sh:

vim -u NONE cgibinIntercept.sh +"hi Normal ctermbg=17" # for Windows: +"set t_Co=256"
# scroll down with CTRL-D

In general it is much easier to reproduce in WSL than Vim.exe, because it doesn't seem to occur after the initial scroll on Vim.exe and the effects are a bit different as described below. I can reproduce with any combination of conhost/terminal and Vim 8.1.883 on Ubuntu/8.0 on Debian/8.1 on Win32. Does not reproduce with neovim (v0.4.0-430-g8698830cb).

A few cases of example behavior:

• Debian with conhost, Vim from official repository

$ sudo apt install vim
$ vim --version
VIM - Vi IMproved 8.0 (2016 Sep 12, compiled Sep 30 2017 18:21:38)
Included patches: 1-197, 322, 377-378, 550, 703, 706-707

Same behavior in the new terminal.

• Vim.exe on conhost cmd.exe installed from https://github.com/vim/vim-win32-installer/releases/download/v8.1.1099/gvim_8.1.1099_x64.exe
  
  Background color is wrong on the text part but correct otherwise (purple, although this seems to depend on active colorscheme) but is fixed after a Ctrl-L redraw.

• Vim.exe on Terminal cmd.exe, same as above
  
  Background color is again wrong, but the background highlight completely disappears after a Ctrl-L redraw.

  • What's wrong / what should be happening instead:
    Background rendered the whole width of the window, e.g. in wsltty:
    

The effect is very similar to the issue https://github.com/microsoft/terminal/issues/375, however that one happens with bg color set to <16 (darkBlue). However I cannot reproduce that issue anymore, including the terminal bg change.

The issues for Win32 and Linux versions might be separate, since their effects are pretty different as described above.

I also haven't been able to reproduce this using VT sequences, but I hope the bug report is still useful.

[xtremeperf]

The common fix is to disable Vim's "Background Color Erase (BCE)" option by setting t_ut to an empty string, like this:

:set t_ut=""
(then CTRL-L to re-draw)

This tells Vim to erase areas of the screen, as it does when back-scrolling, using the current background color (from the Vim color scheme) instead of relying on that specific Terminal's implementation of Background Color Erase (BCE) mode. Otherwise, areas are simply erased without regard for background color, which means the Terminal's DEFAULT background value would be used.

[caksoylar]

Thanks for the pointer @xtremeperf, that does work around the issue. Although I still see flashes of the wrong background color during scrolling with something like vim-smooth-scroll which scrolls line-by-line rapidly.

As I understand it, Vim expects the terminal to support BCE (thus setting t_ut=y) when it reports itself as xterm-256color, but the terminal does not support it. Interestingly this implies that it might be an issue that was fixed before. Another possibly related issue I found is #70.

[xtremeperf]

Exactly. Well, it can also happen if t_Co=256 and/or term=xterm

Also, to prevent those flashes of the wrong bg color during rapid scrolling when bce mode is disabled, you'll want to additionally configure the maximum number of lines that vim will scroll before redrawing the screen. Set it to 1 line. The following works well placed in a vimrc file:

if (&term =~ '^xterm' && &t_Co == 256)
  set t_ut= | set ttyscroll=1
endif

I'm think Microsoft/WSL #1706 was maybe an end-of-line/newline erase bug or something like that, but Microsoft/Terminal #70 sounds like the same Vim bce issue we're discussing here.

[brentpfister]

On Windows 10 1903 (18362.175) with WSL Ubuntu 18.04, the @xtremeperf workarounds involving set t_ut did not work for me. I found another workaround in an answer to a stack overflow question.
In ~/.vimrc put:

highlight Normal ctermfg=black ctermbg=lightgrey

When in vim, this sets the text color to black and the background color to lightgrey. Other colors are listed in /usr/share/vim/vim80/rgb.txt Not all those colors are supported on a 256 color xterm.

[caksoylar]

Note that you are setting the background to color 7 (and not to a true-color or 16-256 terminal colors) using lightgrey, which you can see in :help cterm-colors in Vim. As I noted above, this problem only occurs if the background color is set to either >=16 or an RGB value.

[j4james]

This looks very similar to issue #70 - might be a duplicate. What is happening when you scroll, is you get a delete lines (DL) or insert lines (IL) escape sequence, which should fill the new space with the current background color, but it doesn't work correctly when using colors from the 256-color palette or RGB mode. A simple test case would be something like this:

printf "\ec\e[1;10r\e[H\e[48;5;2m\e[10M\nBG COLOR\e[r\e[m"

That sets a scroll region of 10 lines, and then erases those 10 lines with a green background color from the 256-color palette. While the BG COLOR text shows what the color should have been, the background is actually filled with black. If you replace the \e[48;5;2m sequence with the simpler 16-color sequence \e[42m it'll work.

printf "\ec\e[1;10r\e[H\e[42m\e[10M\nBG COLOR\e[r\e[m"

The reason for the failure can be found in the DoSrvPrivateModifyLinesImpl method, which calls the ScrollConsoleScreenBufferWImpl method to handle the scrolling. The latter only accepts legacy 16-color attributes, so it tries to derive an appropriate value using the TextAttribute::GetLegacyAttributes method. Unfortunately that method is essentially useless for anything that isn't already a legacy color.

That said, the ScrollConsoleScreenBufferWImpl does try to make up for the limitations of the API, and map the 16-color attributes back to their original RGB values. It does this by comparing the legacy value against the legacy equivalent of the current colors - if they match then it will use the full RGB values of the current colors. Unfortunately this doesn't work because it uses a different calculation for the legacy attributes (the Settings::GenerateLegacyAttributes method).

Assuming the ScrollConsoleScreenBufferWImpl can't be updated to handle full RGB attributes, the next best thing would be to use a better legacy attribute calculation in the DoSrvPrivateModifyLinesImpl method. If we were to use the same Settings::GenerateLegacyAttributes on both ends then the mapping back to RGB should work correctly. And in the event it can't map, because it really doesn't match the current color, then it should at least give a better approximation of the requested color.

[kshji]

I installed Windows Terminal 0.3 from Store and start bash shell using default profile wsl.exe -d Ubuntu-18.04 , no problem using color schema One Half Light which is reverse setup.

If I setup CMD reverse (fg 242, bg 12) and start wsl.exe -d Ubuntu-18.04 , no problem.

But when you set app Ubuntu 18.04 Ubuntu window/command properties same values, vim insert mode add black background.

[msftbot[bot]]

:tada:This issue was addressed in #2668, which has now been successfully released as Windows Terminal Preview v0.5.2661.0.:tada:

Handy links:

• Release Notes
• Store Download

[kazucmpt]

Also, to prevent those flashes of the wrong bg color during rapid scrolling when bce mode is disabled, you'll want to additionally configure the maximum number of lines that vim will scroll before redrawing the screen. Set it to 1 line. The following works well placed in a vimrc file:

if (&term =~ '^xterm' && &t_Co == 256)
  set t_ut= | set ttyscroll=1
endif

I'm think Microsoft/WSL #1706 was maybe an end-of-line/newline erase bug or something like that, but Microsoft/Terminal #70 sounds like the same Vim bce issue we're discussing here.

Thanks to @xtremeperf, those flashes has been disappeared by adding

if (&term =~ '^xterm' && &t_Co == 256)
   set t_ut= | set ttyscroll=1
endif

on .vimrc.

But when I use vim on tmux, the flashes are appeard again.

Does somebody know how should I fix it?

[jasper-zanjani]

I am experiencing similar issues using vim in Windows Terminal 1.2.2381.0.
As you can see, guifg and guibg are set to NONE, but the background is being painted black. This is definitely not Windows Terminal's background color, which was explicitly set to match the chosen vim theme.

Notably, this problem appears to be specific to Windows and does not occur in WSL: