Neovim: Weird characters in terminal after 0.2.1

[htfy96]

nvim --version:

NVIM v0.2.1-511-g105d680ae
Build type: RelWithDebInfo
Compilation: /usr/lib/ccache/bin/cc -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -g -fvar-tracking-assignments -Wconversion -DNVIM_MSGPACK_HAS_FLOAT32 -Og -g -Wall -Wextra -pedantic -Wno-unused-parameter -Wstrict-prototypes -std=gnu99 -Wimplicit-fallthrough -Wvla -fstack-protector-strong -fdiagnostics-color=auto -DINCLUDE_GENERATED_DECLARATIONS -D_GNU_SOURCE -I/tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/build/config -I/tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/src -I/usr/include -I/usr/include -I/usr/include -I/usr/include -I/usr/include -I/usr/include -I/usr/include -I/tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/build/src/nvim/auto -I/tmp/yaourt-tmp-lz/aur-neovim-git/src/neovim-git/build/include
Compiled by lz@lzPC

Optional features included (+) or not (-): +acl   +iconv    +jemalloc +tui      
For differences from Vim, see :help vim-differences

   system vimrc file: "$VIM/sysinit.vim"
  fall-back for $VIM: "/usr/share/nvim"

Vim (version: ) behaves differently?

Vim 8.0.628 with vim -u NONE behaves properly.

Operating system/version:

Arch Linux x86-64

Terminal name/version:

Guake Terminal 0.8.10
Cannot reproduce on gnome-terminals

$TERM:

xterm-256color

Steps to reproduce using nvim -u NORC

nvim -u NORC
// press i
// input some characters
// Observe that weird characters appears

Actual behaviour

Expected behaviour

No weird characters.

Addition

Ran valgrind --track-origins=yes --leak-check=full nvim -u NONE 2>/tmp/record.txt. Here is the log:
record.txt. Not sure whether it is related to #4796 since the same error is reported on gnome-terminal which works well.

[Shougo]

You can test it.
https://github.com/neovim/neovim/pull/6997

[kunalg]

Same here. I'm on a nightly build on ubuntu 16.04 LTS (using neovim-ppa/unstable).

kunal@hydrogen:~$ nvim -v
NVIM v0.2.1-dev
Build type: RelWithDebInfo
Compilation: /usr/bin/x86_64-linux-gnu-gcc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wconversion -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -DNVIM_MSGPACK_HAS_FLOAT32 -Og -g -Wall -Wextra -pedantic -Wno-unused-parameter -Wstrict-prototypes -std=gnu99 -Wvla -fstack-protector-strong -fdiagnostics-color=auto -DINCLUDE_GENERATED_DECLARATIONS -D_GNU_SOURCE -I/build/neovim-45NS1U/neovim-0.2.0ubuntu1+git201707091951+3517+26~ubuntu16.04.1/build/config -I/build/neovim-45NS1U/neovim-0.2.0ubuntu1+git201707091951+3517+26~ubuntu16.04.1/src -I/build/neovim-45NS1U/neovim-0.2.0ubuntu1+git201707091951+3517+26~ubuntu16.04.1/.deps/usr/include -I/build/neovim-45NS1U/neovim-0.2.0ubuntu1+git201707091951+3517+26~ubuntu16.04.1/.deps/usr/include -I/build/neovim-45NS1U/neovim-0.2.0ubuntu1+git201707091951+3517+26~ubuntu16.04.1/.deps/usr/include -I/build/neovim-45NS1U/neovim-0.2.0ubuntu1+git201707091951+3517+26~ubuntu16.04.1/.deps/usr/include -I/build/neovim-45NS1U/neovim-0.2.0ubuntu1+git201707091951+3517+26~ubuntu16.04.1/.deps/usr/include -I/build/neovim-45NS1U/neovim-0.2.0ubuntu1+git201707091951+3517+26~ubuntu16.04.1/.deps/usr/include -I/usr/include -I/build/neovim-45NS1U/neovim-0.2.0ubuntu1+git201707091951+3517+26~ubuntu16.04.1/build/src/nvim/auto -I/build/neovim-45NS1U/neovim-0.2.0ubuntu1+git201707091951+3517+26~ubuntu16.04.1/build/include
Compiled by root@lgw01-23

Optional features included (+) or not (-): +acl   +iconv    +jemalloc +tui      
For differences from Vim, see :help vim-differences

   system vimrc file: "$VIM/sysinit.vim"
  fall-back for $VIM: "/usr/share/nvim"

System details:

kunal@hydrogen:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.2 LTS
Release:        16.04
Codename:       xenial

Typically use Konsole (KDE). Also tried in a tty - the issue remains.

[Shougo]

Have you tested https://github.com/neovim/neovim/pull/6997?

[DarkDefender]

If anyone still has problem with this after you have tired #6997 , let me know in that pull request.

[kunalg]

6997 CI builds seem to have failed.

Anyway, will try to build it locally later.

[FabioAntunes]

@Shougo The builds have failed, it's safe to compile it anyway?

[DarkDefender]

@FabioAntunes I might be based, but it works fine on my system. I would really like if you could test it out.

I don't know why the builds fail. I can only see that travis complains about lint stuff. But the other two services I have a hard time to find what is going wrong.

[FabioAntunes]

@DarkDefender tried out your pull request, unfortunately still having issues:

NVIM v0.2.1-534-g78869f239

• Elementary OS (Ubuntu 14.04)
• Pantheon Terminal

[DarkDefender]

@FabioAntunes I guess this does not chage cursor shape if you run it outside of neovim?
printf "\x1b[4 q"

I'm also guessing TERM is set to xterm?
What is VTE_VERSION set to?

It sadly seems like Pantheon Terminal uses an old version of libvte. The problem seems to be that VTE versions this old do not support cursor shape changes with terminal codes. However by saying that they are xterm, terminal programs that uses terminfo or just checks TERM will think they do.

Edit: If this is the case I will add a workaround BTW.

[FabioAntunes]

@DarkDefender sorry for the late reply

I guess this does not chage cursor shape if you run it outside of neovim?
printf "x1b[4 q"

Correct!

Also:

• TERM: xterm
• VTE_VERSION: 3409

I have rolled back to the 0.2.0 for now.

[DarkDefender]

@FabioAntunes I've pushed an update that should fix it. Can you try it out?

[FabioAntunes]

@DarkDefender still having issues.. Is there something I can do to help you out?

[DarkDefender]

@FabioAntunes Ah, my bad. I did not think through my previous fix. It didn't really fix anything. Can you try again now? :P

[FabioAntunes]

@DarkDefender it's working now, so far no strange characters

I will be using this build during the day I will let you know if I find something odd.

[tecywiz121]

For anyone using ppa:neovim-ppa/unstable, you can work around it with :set guicursor=

[michaelmior]

@DarkDefender I'm seeing the same issue as in this comment, but unfortunately #6997 doesn't solve it for me.

[DarkDefender]

@michaelmior what is VTE_VERSION set to?

[michaelmior]

@DarkDefender It's not set.

[DarkDefender]

So echo $VTE_VERSION returns nothing? Mate terminal is based on libvte so I would think that it would still be there.

If it is not, then I can sadly only suggest manually setting :set guicursor= in your init.vim or upgrading your Mate terminal version. 1.18 seems to be the latest one and I suspect that it should have a new enought version of libvte that this will not be a problem.

[DarkDefender]

Note that VTE_VERSION might not be there if you are using ssh

[michaelmior]

@DarkDefender Correct, I get nothing (and not using SSH). I have been considering upgrading to a newer version of MATE, although I may also just try a different terminal emulator to see if that helps.

[michaelmior]

As expected, :set guicursor= fixes the problem. This was previously solved with NVIM_TUI_ENABLE_CURSOR_SHAPE which is I see is now ignored. Unfortunately setting guicursor also messes up the colors.

[DarkDefender]

Messes up the colors how?

[michaelmior]

I'll try to remember to post a screenshot in the morning when I'm back at my PC but the colours of everything in the UI is different and not matching my defined colour scheme.

[michaelmior]

I can't seem to get the "bad" behaviour consistently, but here are before and after screenshots.

Before set guicursor=:

After:

[DarkDefender]

Does this happen outside of tmux also?

What is TERM set to outside of tmux?

[michaelmior]

No, I can't seem to reproduce this outside of tmux. Outside of tmux TERM is xterm-256color (and screen-256color inside tmux).

[DarkDefender]

Do you have true color switched on with termguicolors?

[michaelmior]

Nope, that's turned off.

[DarkDefender]

Does it still happen if you build neovim from this commit:
https://github.com/neovim/neovim/commit/69f0847ccc79acabb1cad5b1b54c906973946d81

I would also like if you could join the neovim IRC channel (freenode #neovim) so we can figure this out quickly.

[michaelmior]

It doesn't seem to, although I'm not convinced I know the steps to reproduce accurately. I'm on #neovim now.

[egmontkob]

Just for the record: the $VTE_VERSION variable was introduced in 3405 (well, vte-0.34.5 I mean). Whoever does not have this set in mate-terminal probably uses a gtk2-based version thereof.

[egmontkob]

... Oh, and vte's handling of unrecognized escape sequences sucks.

[htfy96]

Guake depends on buggy vte which cannot handle cursor control sequence well even with the version in PR.

Finally I switched to Tilda which depends on vte3.

[michaelmior]

@DarkDefender FYI, I haven't seen the colour issue again that I previously mentioned. However, I'm still seeing the unrecognized escape sequences being displayed even with guicursor=.

[DarkDefender]

@michaelmior Do you see any while inside neovim?

[michaelmior]

Yes. If I switch to a different tmux window and then come back.

[DarkDefender]

Does that happen if you are switching tmux windows regardless (without nvim)? Tmux also can try to change cursor shape.

If you do not get this when outside of tmux, then I don't think we can fix this from our side (but I could be wrong).

[michaelmior]

Unfortunately, I can't nail down under what circumstances this happens. I will note that if I create a new tmux window within a session that has this problem, switching to that window does not immediately produce this issue.

[DarkDefender]

I'm just taking a wild guess here. If it only triggers in tmux, then it might be because tmux tries to "restore" the cursor shape when switching between tmux windows. This is a tmux feature so that you can have different cursor shapes in different windows. You can also do the same with cursor colors.

This was introduced so that if you changed shape/color of the cursor it would be limited to that specific tmux window.

[michaelmior]

I have the following in ~/.tmux.conf which suggested to me previously

set -g terminal-overrides "xterm*:XT:smcup@:rmcup@"                             
set -ga terminal-overrides ',*:Ss=\E[%p1%d q:Se=\E[2 q'

To be honest, I'm not entirely sure of the purpose of this but I assume it may be related.

[DarkDefender]

Ah, does it happen if you remove the "Ss and Se" line entirely?
Or if you do:

set -ga terminal-overrides ',*:Ss@:Se@'

?

The Ss and Se stuff is to a variable that says that your terminal supports changing cursor shapes using the specific codes that they are set to.

IIRC the @ should unset them

[michaelmior]

Setting it and reloading my tmux config in an existing session didn't change anything, but I'll keep tabs on this and see if the behaviour changes. Thanks for the help unrelated to neovim :)

[thuandt]

@michaelmior did you solved this issue?

I use byobu with tmux backend and it happened since last month, I update nvim everyday but seem problem isn't fixed yet.

FYI, I use Ubuntu 16.04, nvim v0.2.1-dev

NVIM v0.2.1-dev
Build type: RelWithDebInfo
Compilation: /usr/bin/x86_64-linux-gnu-gcc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wconversion -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -DNVIM_MSGPACK_HAS_FLOAT32 -Og -g -Wall -Wextra -pedantic -Wno-unused-parameter -Wstrict-prototypes -std=gnu99 -Wvla -fstack-protector-strong -fdiagnostics-color=auto -DINCLUDE_GENERATED_DECLARATIONS -D_GNU_SOURCE -I/build/neovim-iaPgIb/neovim-0.2.0ubuntu1+git201707221704+3540+26~ubuntu16.04.1/build/config -I/build/neovim-iaPgIb/neovim-0.2.0ubuntu1+git201707221704+3540+26~ubuntu16.04.1/src -I/build/neovim-iaPgIb/neovim-0.2.0ubuntu1+git201707221704+3540+26~ubuntu16.04.1/.deps/usr/include -I/build/neovim-iaPgIb/neovim-0.2.0ubuntu1+git201707221704+3540+26~ubuntu16.04.1/.deps/usr/include -I/build/neovim-iaPgIb/neovim-0.2.0ubuntu1+git201707221704+3540+26~ubuntu16.04.1/.deps/usr/include -I/build/neovim-iaPgIb/neovim-0.2.0ubuntu1+git201707221704+3540+26~ubuntu16.04.1/.deps/usr/include -I/build/neovim-iaPgIb/neovim-0.2.0ubuntu1+git201707221704+3540+26~ubuntu16.04.1/.deps/usr/include -I/build/neovim-iaPgIb/neovim-0.2.0ubuntu1+git201707221704+3540+26~ubuntu16.04.1/.deps/usr/include -I/usr/include -I/build/neovim-iaPgIb/neovim-0.2.0ubuntu1+git201707221704+3540+26~ubuntu16.04.1/build/src/nvim/auto -I/build/neovim-iaPgIb/neovim-0.2.0ubuntu1+git201707221704+3540+26~ubuntu16.04.1/build/include
Compiled by root@lcy01-09

Optional features included (+) or not (-): +acl   +iconv    +jemalloc +tui
For differences from Vim, see :help vim-differences

   system vimrc file: "$VIM/sysinit.vim"
  fall-back for $VIM: "/usr/share/nvim"

[michaelmior]

@thuandt Nope, no consistent solution yet for me.

[riddochc]

I'm having this issue in konsole 17.0.4.3, but not xterm 308. My $TERM is xterm-256color.

Running ttyrec, the only place (besides when I exit nvim) that a 'q' shows up in the log is when the following escape sequence is written:

e [ 2 space q

I could be wrong about this, but i think the space causes the terminal to decide that the command has finished, and the following 'q' is treated as a character to display instead of part of the escape sequence.

I haven't bisected this, but I know the issue was introduced sometime since commit 26124b48.

[riddochc]

I just tried bisecting this issue, I was wrong in thinking the issue started at commit 26124b4, I had to go back further. Git bisect reported this:

There are only 'skip'ped commits left to test.
The first bad commit could be any of:
1f3b5e1a826fbff8ce2fa84e0e50d082d65a08bd
6910bfee0fb9470b0b30e5a46efeebfe760e6a2c
We cannot bisect more!

I had to skip 1f3b5e1a because it segfaulted on startup, but given the nature of that change, I think we can conclude the issue was introduced in 6910bfee0. Checking my versions again, it looks like I've been using neovim with unibilium-v1.2.0-7-g0dae863, which I installed last december, but there's only been 5 commits on unibilium since then, none of which touch code, so we can rule out the possibility of version differences there affecting this.

Github's not letting me paste the whole bisect log as a text file for some reason, and otherwise thinks it's markdown full of headers, so I'll just put it up on my personal website and provide a link: https://syntacticsugar.org/neovim-bisect.txt

[riddochc]

I should have tried this sooner... the :set guicursor= workaround works for me. Not ideal, I suppose, but I haven't really expected the cursor to be changing much, either. I can try looking into this further and actually read some code instead of just poking at git, or test patches if someone else gets around to it before I do.

[wikimatze]

@riddochc thanks for the trick with set guicursors= that fixed the issue for me in normal terminal mode. When using tmux, I don't need this setting, but anyways, it is working now.

[frdeso]

I am having this issue on nvim v0.2.2 on Terminator but not on Gnome-Terminal. set guicursor= fixes it.

[egmontkob]

@frdeso Probably you're using an old Terminator (version 1.0 or older) and hence an ancient VTE. You should upgrade to Terminator 1.91 or so.

[frdeso]

@egmontkob You are right. The Ubuntu Xenial LTS package is on version 0.98.

[ghost]

I think this issue is related.
Some charaters appear out of nowhere, most of them are "q"s

Using nvim v0.2.2 and Arch Linux
set guicursor= doesn't seem to work.

[justinmk]

https://github.com/neovim/neovim/wiki/FAQ#nvim-shows-weird-symbols-2-q-when-changing-modes