Vimtex: (Neo)Vim 'w' behaves like 'W' in a certain scenario

[Zeta611]

Issue
Vim word movements like 'w', 'e', 'b', etc. suddenly behave like their capital counterparts 'W', 'E', 'B', etc., when a quickfix list pops out and if the cursor is on the end-of-a-line-}.
This issue has been nagging me for a few years, and I finally had a time to investigate it.

iskeyword seems to be fine:

 iskeyword=@,48-57,_,192-255,:

This issue is always reproducible with the following minimal.vim & minimal.tex on my Macs.

minimal.vim
vim let &runtimepath = '~/.local/share/nvim/plugged/vimtex,' . &runtimepath let &runtimepath .= ',~/.local/share/nvim/plugged/vimtex/after'

minimal.tex
tex % Below 'articl' is an intended typo \documentclass{articl} \begin{document} \TeX \end{document}

Commands/Input
Put the cursor on the closing curly bracket } in {acticl}.
Then start a continuous compile mode via \ll.

Observed Behaviour
The missing 'e' opens the quickfix list, and if you try to move using w, it would behave like W.

This also breaks a 3rd party plugin like easymotion, only showing the starts of Words (delimited by spaces).

Expected Behaviour
It should still act like a normal w(, e, b, etc.).

Output from VimtexInfo

System info
  OS: Mac OS X 10.15.3 (19D76)
  Vim version: NVIM v0.4.3
  Has clientserver: true
  Servername: /var/folders/cs/7bfzsfzx7bn67bhxn0pgzk1w0000gn/T/nvim72gwdA/0

vimtex project: minimal
  base: minimal.tex
  root: /Users/jay/vimtex
  tex: /Users/jay/vimtex/minimal.tex
  out: /Users/jay/vimtex/minimal.pdf
  log: /Users/jay/vimtex/minimal.log
  aux: /Users/jay/vimtex/minimal.aux
  fls: /Users/jay/vimtex/minimal.fls
  compiler: latexmk
    backend: nvim
    output: /var/folders/cs/7bfzsfzx7bn67bhxn0pgzk1w0000gn/T/nvim72gwdA/1
    configuration: 
      continuous: 1
      callback: 1
      latexmk options:
        -verbose
        -file-line-error
        -synctex=1
        -interaction=nonstopmode
      latexmk engine: -pdf
  viewer: General
  qf: LaTeX logfile
    config: 
      packages: 
        default: 1
      default: 1
  document class: article

Output from chechhealth

health#vimtex#check
========================================================================
## vimtex
  - OK: Vim version should have full support!
  - OK: General viewer should work properly!
  - WARNING: pstree is not available
    - ADVICE:
      - vimtex#view#reverse_goto is better if pstree is available.
  - WARNING: Compiler callbacks will not work!
    - ADVICE:
      - `neovim-remote` / `nvr` is required for callbacks to work with neovim
      - Please also set |g:vimtex_compiler_progname| = 'nvr'
      - For more info, see :help |vimtex-faq-neovim|

health#nvim#check
========================================================================
## Configuration
  - OK: no issues found

## Performance
  - OK: Build type: Release

## Remote Plugins
  - OK: Up to date

## terminal
  - INFO: key_backspace (kbs) terminfo entry: key_backspace=^H
  - INFO: key_dc (kdch1) terminfo entry: key_dc=\E[3~
  - INFO: $TERM_PROGRAM='iTerm.app'
  - INFO: $COLORTERM='truecolor'

## tmux
  - OK: escape-time: 0ms
  - INFO: $TERM: tmux-256color

health#provider#check
========================================================================
## Clipboard (optional)
  - OK: Clipboard tool found: pbcopy

## Python 2 provider (optional)
  - INFO: pyenv: Path: /usr/local/Cellar/pyenv/1.2.16/libexec/pyenv
  - INFO: pyenv: $PYENV_ROOT is not set. Infer from `pyenv root`.
  - INFO: pyenv: Root: /Users/jay/.pyenv
  - WARNING: No Python executable found that can `import neovim`. Using the first available executable for diagnostics.
  - ERROR: Python provider error:
    - ADVICE:
      - provider/pythonx: Could not load Python 2:
          /Users/jay/.pyenv/shims/python2 does not exist: pyenv: python2: command not found

          The `python2' command exists in these Python versions:
          2.7.17
          2.7.17/envs/neovim-python2
          neovim-python2


          /Users/jay/.pyenv/shims/python2.7 does not exist: pyenv: python2.7: command not found

          The `python2.7' command exists in these Python versions:
          2.7.17
          2.7.17/envs/neovim-python2
          neovim-python2


          python2.6 not found in search path or not executable.
          /Users/jay/.pyenv/shims/python is Python 3.8 and cannot provide Python 2.
  - INFO: Executable: Not found

## Python 3 provider (optional)
  - INFO: pyenv: Path: /usr/local/Cellar/pyenv/1.2.16/libexec/pyenv
  - INFO: pyenv: $PYENV_ROOT is not set. Infer from `pyenv root`.
  - INFO: pyenv: Root: /Users/jay/.pyenv
  - WARNING: No Python executable found that can `import neovim`. Using the first available executable for diagnostics.
  - ERROR: Python provider error:
    - ADVICE:
      - provider/pythonx: Could not load Python 3:
          /Users/jay/.pyenv/shims/python3 does not have the "neovim" module. :help |provider-python|
          /usr/local/bin/python3.7 does not have the "neovim" module. :help |provider-python|
          python3.6 not found in search path or not executable.
          python3.5 not found in search path or not executable.
          python3.4 not found in search path or not executable.
          python3.3 not found in search path or not executable.
          /Users/jay/.pyenv/shims/python does not have the "neovim" module. :help |provider-python|
  - INFO: Executable: Not found

## Ruby provider (optional)
  - INFO: Ruby: ruby 2.7.0p0 (2019-12-25 revision 647ee6f091) [x86_64-darwin19]
  - INFO: Host: /Users/jay/.rbenv/shims/neovim-ruby-host
  - OK: Latest "neovim" gem is installed: 0.8.1

## Node.js provider (optional)
  - INFO: Node.js: v13.7.0
  - INFO: Neovim node.js host: /usr/local/lib/node_modules/neovim/bin/cli.js
  - OK: Latest "neovim" npm/yarn package is installed: 4.8.0

[lervag]

I'm sorry, but I can't reproduce. Could you make it more clear what you mean by:

Commands/Input
Put the cursor on the closing curly bracket } in {acticl}.
Then start a continuous compile mode via \ll.

Observed Behaviour
The missing 'e' opens the quickfix list, and if you try to move using w, it would behave like W.

OK, I make a minimal vimrc file like this:

set nocompatible
let &rtp = '/path/to/vimtex,' . &rtp
let &rtp .= ',/path/to/vimtex/after'
filetype plugin indent on
syntax enable

Then I open the test file with vim --servername VIM -u test.vim test.tex or nvim --servername -u test.vim test.tex. I move to the closing bracket, then do \ll. I observe that the quickfix window opens due to the error. Then I type w, and the cursor moves to the next line as expected. The next w moves to b in \begin, also as expected.

Am I missing the point here?

[Zeta611]

That's odd, since I did exactly the same on my colleague's Mac and could reproduce the bug.
If I type w, the cursor moves to the next line's \, and another w moves me to the next line's \ before TeX, instead of bringing me to the b of begin.

My colleague's Mac didn't have Neovim/Vimtex installed, so I freshed installed it using brew install neovim & installed Vimtex using vim-plug (which was also fresh-installed).
I placed minimal.vim and minimal.tex under /tmp/vimtex and opened it using nvim -u minimal.vim minimal.tex.

If relevant, I'm on macOS Catalina 10.15.2 using zsh 5.7.1.

+The following shows what I did on the new Mac:

[Bancini]

I have tried in Debian 10 with default applications (Bash shell, gnome-terminal, ...), but also I have tried with tmux and zhs in Ubuntu 18.04, and I can't reproduce it. w and W worked as expected, following all your instructions.
Can it be something related to macOS or localization?
Also, although it may have nothing to do with this strange motion behavior, it seems the :VimtexInfo does not correspond to the file minimal.tex (document class: article; packages: expl3, ... ,xparse).

[Zeta611]

@Bancini Regarding the :VimtexInfo, it was my mistake. It was done using my custom vim settings. I updated the info.

[Zeta611]

• I also noticed that ctrl-n and ctrl-p no longer autocompletes normally, but it prompts "Scanning tags." The autocompletion that pops up blindly shows all the words used in a document, not a normal 'smart' suggestions..

[lervag]

Yes, ctrl-n and ctrl-p are not mapped in vimtex and is not related to vimtex. By default, they do keyword completion, please read :help i_CTRL-N. So this is as expected.

I am sorry to say I can't really help here because I can't reproduce. Could you try to either upgrade or downgrade your neovim version?

Perhaps @clason might have time to test this, I know he uses a Mac.

[clason]

Indeed, I can reproduce this (neovim nightly on Catalina) -- the w issue, I mean. Seeing as this is likely a neovim bug, it would be good to bisect it.

[lervag]

Ok, good! That's progress! Can you reproduce the issue in Vim as well?

[clason]

No, because I apparently forgot how to use vim (the minimal vimrc doesn't work on vim for me), but at least I could reproduce on Linux (openSUSE Tumbleweed, again neovim head).

[Zeta611]

@lervag Vim does work fine! So apparently this is a Neovim bug..

[lervag]

No, because I apparently forgot how to use vim (the minimal vimrc doesn't work on vim for me), ...

Haha :)

... but at least I could reproduce on Linux (openSUSE Tumbleweed, again neovim head).

That's interesting. I've tested with NVIM v0.5.0-132-g83b6d9f19, which I guess is not the nightly build. But it is quite recent. Which version are you on that fails? At least this should give us some bounds for a bisection, right?

[clason]

It's NVIM v0.5.0-d9657b3ae, but OP is on the last release (0.4.3), so it has to be quite back. But maybe it's tied to a system library.

@lervag How did you install nvim? Built yourself from the repo or via nightly appimage / distropackage?

[Zeta611]

@clason I installed it using Homebrew package manger for Mac. I've been having this issue for quite a long time--at least a year.

[lervag]

I'm on Arch Linux and use the neovim-git package from AUR. It should be a recent version. So I agree, it seems it may be related to a system library. Not quite sure how to debug this...

[clason]

Well, if you're up for it, you could clone and build the repo (or maybe easier, download the appimage) and check if you can repro it with that?

The point is that distro packages tend to be aggressive about using system libraries rather than the bundled dependencies (libvterm etc.), so that would make a difference.

@Zeta611, is that with terminal nvim or VimR? If the former, could you also try the latter?

[lervag]

Ok, first: I've reproduced the issue as well. I don't know why I didn't reproduce it before, but now it is reproduced when I follow the steps.

I've tried to do bisecting, but I fail. I did something like this:

# in test folder

git clone https://github.com/neovim/neovim.git
cd neovim
make
cd ..

VIMRUNTIME=neovim/runtime ./neovim/build/bin/nvim -u test.vim test.tex

The first time I can reproduce the fail. Next, I do git bisect start and git bisect bad, then I do git checkout $(git rev-list -n 1 --before='2019-09-18' master) to checkout an older commit. However, now neovim won't build. I tried several different older commits, none of them built without error. :\

[clason]

This probably means that you checkout out past a big dependency update -- did you try make distclean; make build?

[lervag]

Yes, good point. I now found that make clean seems to help; can verify that the bug is still present in october so far...

[clason]

My intuition would be to check the big dependency bumps (libvterm and unibilium); the commits should be easy to find by searching in github for those terms.

[lervag]

Bisection is now running, I'm getting close...

[clason]

I'd love to help you; unfortunately, a recent commit is required for neovim to build at all under recent macOS...

[lervag]

Well, I found the commit that introduced the bug:
https://github.com/neovim/neovim/commit/357f95a77eb799a6714117e50d9f06d72e9c0eaf

So, how do we proceed from here?

[clason]

Yes, that makes sense. 👍 Next step: someone opens an issue at the neovim repo (OP? brew install neovim --HEAD is easy enough to test fixes.)

[lervag]

Yes, it should be easy to test fixes. It should also be easy to test for the bug:

git clone https://github.com/lervag/vimtex
cd vimtex/test/issues/1595

# Here NVIM implies correct command line to start "current" version of neovim
NVIM -u test.vim test.tex

# Type in normal mode: www

I think, with this information at hand, @Zeta611 has all the information he needs to submit a good issue for neovim. If you do, please refer to this issue as well, then we may be able to help if possible.

[lervag]

Also, since I believe this is not really a vimtex bug, I'm taking the liberty of closing the issue. Feel free to continue the discussion though (I'll still follow it).

[andryandrew]

Hi, I am also affected by the bug (Arch linux, neovim-git). I can't find a related issue on the neovim tracker, I guess @Zeta611 did not submit it in the end. Do you have some more info since then? I could help by submitting the bug.

[lervag]

Hi @andryandrew! Thanks for volunteering! I don't have any more info, no, sorry, but I think the previously added info should suffice for a good issue report.

[adigitoleo]

Ya this one's annoying. @andryandrew Any progress? I'm a bit of a noob but if you tell me what to do I can try to help.

My neovim version

NVIM v0.4.4
Build type: Release
LuaJIT 2.0.5
Compilation: /usr/bin/cc -D_FORTIFY_SOURCE=2 -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -O2 -DNDEBUG -DMIN_LOG_LEVEL=3 -Wall -Wextra pedantic -Wno-unused-parameter -Wstrict-prototypes -std=gnu99 -Wshadow -Wconversion -Wmissing-prototypes -Wimplicit-fallthrough -Wvla -fstack-protector-strong -fno-common -fdiagnostics-color=always -DINCLUDE_GENERATED_DECLARATIONS -D_GNU_SOURCE -DNVIM_MSGPACK_HAS_FLOAT32 -DNVIM_UNIBI_HAS_VAR_FROM -I/build/neovim/src/build/config -I/build/neovim/src/neovim-0.4.4/src -I/usr/include -I/build/neovim/src/build/src/nvim/auto -I/build/neovim/src/build/include
Compiled by builduser

Features: +acl +iconv +tui

[andryandrew]

Hi! Sorry for not following with this, but I wanted to spend some time trying to reproduce the issue in neovim alone before submitting. After a lot of tinkering, I found which lines in vimtex triggered the bug and managed to "isolate" it (you'll see why the quotes).

My findings/hypotheses:

• Due to the refactor in the commit found by @lervag, neovim handles CursorMoved slightly differently: in particular, it is triggered on opening a new window (the quickfix window, in our case). In Vim, this is not the case.
• I don't know if this is intended behavior, but it seems like the event is triggered _after_ changing the window ID but _before_ changing the buffer.
• If the windows is changed again before a redraw, you get all sorts of unintended behaviors

Vimtex does this when opening the quickfix window and (in the default behavior) changing back the focus to the main window.

You'll notice something strange: the error line in the quickfix window has highlighted characters in the same positions as the matching braces _in the main window_. This very same bug (due to the default matchparen plugin) is present in vanilla neovim by just calling a function that opens the quickfix window and returns back to the main one.

In Vimtex, a call to SynID during the autocommand triggers the bug subject of this thread. I managed to reproduce this too. Both bugs go away if the vimtex CursorMoved autocommand has higher priority than the matchparen plugin.

Here the test.vim (to be used with the same test.tex above). Go to the closing brace in the first line and press f. Calling FixMP() before that will unload the matchparen autocommand and re-bind it _after_ the one defined in the script, which fixes the problem.

set nocompatible
filetype plugin on

nnoremap q :qall!<cr>

call setqflist([{'text': 'some chars of this will be highlighted'}])

function FixMP()
    NoMatchParen
    DoMatchParen
endfunction

function! Trigerror() abort
  if win_getid() == 1000 | return | endif
  let [lnum,cnum] = searchpairpos('{', '', '}', 'bn')
  call synID(lnum, cnum, 0)
endfunction

function Loadtrigger()
    augroup trigger
        autocmd!
        autocmd CursorMoved <buffer> call Trigerror()
    augroup END
endfunction

autocmd VimEnter * call Loadtrigger()

function! Test()
  copen
  call win_gotoid(1000)
endfunction

nnoremap f :call Test()<cr>

[andryandrew]

In conclusion, the bugs are definitely on neovim's side, but it seems that firing CursorMoved in more situations than vim is intended behavior (neovim/neovim#9838). There are many workarounds for the moment, like raising vimtex's autocommand priority or issuing a redraw before changing window in qf.vim:86.

Maybe the most elegant solution is avoiding autocommands when opening the quickfix window in qf.vim:84, with noautocmd. I can submit a PR if you want.

I might have found a closely related issue neovim/neovim#11878 . I'll keep you updated

[lervag]

Thank you for the detailed and well thought-out response and investigation, @andryandrew!