Vimtex: text inside {...} of \cref{...} is spell checked if not preceded by \begin{document}

[Konfekt]

Issue: Since a recent commit, if not preceded by \begin{document}, the text ... inside the braces of \cref{...} is spell checked.

Expected: No spell checking, independent of precedence of \begin{document} or not (just as for \ref{...}).

[Konfekt]

The commit is 62dc919 : Only load package syntax when necessary (fixes #1433)

[lervag]

Thanks. Could you provide a minimal example as well? It would be very helpful in quickly fixing this.

[Konfekt]

Something like

\documentclass{scrartcl}

\cref{speling}

\begin{document}
\end{document}

Because this does not compile, the error is only evident in files included in the main file, for example, a file intro.tex reading

\section{Intro}

By \cref{speling} ...

and

\documentclass{scrartcl}
\usepackage{cleveref}

\begin{document}
\include{intro.tex}
\end{document}

[Konfekt]

With

set nocompatible
let &rtp = '~/.vim/plugged/vimtex' . ',' . &rtp

syntax on
filetype plugin indent on

and

\documentclass{scrartcl}
\usepackage{cleveref}

\begin{document}
\cref{speling}
\end{document}

it should also be reproducible.

[lervag]

Hmm. It does not seem reproducible to me. But: In order for \cref to be properly highlighted, you need to have the \usepackage{cleveref} line in the preamble, or the document must be compiled so the .fls file is available (it contains a proper list of included package files).

Can you confirm that your issue is only present when you comment out the \usepackage{cleveref} line?

[Konfekt]

It is present with the \usepackage{cleveref} line. But I am sorry. I forgot to add set spell into the vimrc. It is

set nocompatible
let &rtp = '~/.vim/plugged/vimtex' . ',' . &rtp

syntax on
filetype plugin indent on
set spell

In any case, the higlighting of \cref{...} is different from that of \ref{...}.

[Konfekt]

Perhaps my distribution's vimrc in /etc/vimrc at https://build.opensuse.org/package/view_file/openSUSE:11.4/vim/suse.vimrc must be added to reproduce it? I had the same issue as in https://github.com/lervag/vimtex/issues/1433 before.

[lervag]

Hmm. It does not make sense to test this without having

let &rtp .= ',../../../after'

in the minimal vimrc file. Can you add it and test again?

[Konfekt]

Added, and the result is the same.

[Konfekt]

Not sure whethervim --clean -u ~/.viminrc loads /etc/vimrc.

[lervag]

You can see which scripts are loaded with :scriptnames.

I'm sorry that I can help, but I still can not reproduce this. :\

[Konfekt]

Removing out /etc/vimrc still highlights the spelling mistake.

Thanks for the hint, no scripts other than those coming with Vim and vimtex are loaded:

  1: ~/.config/.vim/viminrc
  2: /usr/share/vim/vim81/syntax/syntax.vim
  3: /usr/share/vim/vim81/syntax/synload.vim
  4: /usr/share/vim/vim81/syntax/syncolor.vim
  5: /usr/share/vim/vim81/filetype.vim
  6: /usr/share/vim/vim81/ftplugin.vim
  7: /usr/share/vim/vim81/indent.vim
  8: /usr/share/vim/vim81/plugin/getscriptPlugin.vim
  9: /usr/share/vim/vim81/plugin/gzip.vim
 10: /usr/share/vim/vim81/plugin/logiPat.vim
 11: /usr/share/vim/vim81/plugin/manpager.vim
 12: /usr/share/vim/vim81/plugin/matchparen.vim
 13: /usr/share/vim/vim81/plugin/netrwPlugin.vim
 14: /usr/share/vim/vim81/plugin/rrhelper.vim
 15: /usr/share/vim/vim81/plugin/spellfile.vim
 16: /usr/share/vim/vim81/plugin/tarPlugin.vim
 17: /usr/share/vim/vim81/plugin/tohtml.vim
 18: /usr/share/vim/vim81/plugin/vimballPlugin.vim
 19: /usr/share/vim/vim81/plugin/zipPlugin.vim
 20: /usr/share/vim/vim81/autoload/dist/ft.vim
 21: /usr/share/vim/vim81/syntax/tex.vim
 22: ~/.config/.vim/plugged/vimtex/ftplugin/tex.vim
 23: ~/.config/.vim/plugged/vimtex/autoload/vimtex.vim
 24: ~/.config/.vim/plugged/vimtex/autoload/vimtex/util.vim
 25: ~/.config/.vim/plugged/vimtex/autoload/vimtex/state.vim
 26: ~/.config/.vim/plugged/vimtex/autoload/vimtex/re.vim
 27: ~/.config/.vim/plugged/vimtex/autoload/vimtex/parser.vim
 28: ~/.config/.vim/plugged/vimtex/autoload/vimtex/parser/general.vim
 29: ~/.config/.vim/plugged/vimtex/autoload/vimtex/paths.vim
 30: ~/.config/.vim/plugged/vimtex/autoload/vimtex/view.vim
 31: ~/.config/.vim/plugged/vimtex/autoload/vimtex/view/general.vim
 32: ~/.config/.vim/plugged/vimtex/autoload/vimtex/view/common.vim
 33: ~/.config/.vim/plugged/vimtex/autoload/vimtex/compiler.vim
 34: ~/.config/.vim/plugged/vimtex/autoload/vimtex/compiler/latexmk.vim
 35: ~/.config/.vim/plugged/vimtex/autoload/vimtex/qf.vim
 36: ~/.config/.vim/plugged/vimtex/autoload/vimtex/qf/latexlog.vim
 37: ~/.config/.vim/plugged/vimtex/autoload/vimtex/toc.vim
 38: ~/.config/.vim/plugged/vimtex/autoload/vimtex/fold.vim
 39: ~/.config/.vim/plugged/vimtex/autoload/vimtex/fold/envs.vim
 40: ~/.config/.vim/plugged/vimtex/autoload/vimtex/fold/cmd_single_opt.vim
 41: ~/.config/.vim/plugged/vimtex/autoload/vimtex/fold/markers.vim
 42: ~/.config/.vim/plugged/vimtex/autoload/vimtex/fold/preamble.vim
 43: ~/.config/.vim/plugged/vimtex/autoload/vimtex/fold/sections.vim
 44: ~/.config/.vim/plugged/vimtex/autoload/vimtex/fold/cmd_addplot.vim
 45: ~/.config/.vim/plugged/vimtex/autoload/vimtex/fold/env_options.vim
 46: ~/.config/.vim/plugged/vimtex/autoload/vimtex/fold/cmd_multi.vim
 47: ~/.config/.vim/plugged/vimtex/autoload/vimtex/fold/cmd_single.vim
 48: ~/.config/.vim/plugged/vimtex/autoload/vimtex/cmd.vim
 49: ~/.config/.vim/plugged/vimtex/autoload/vimtex/complete.vim
 50: ~/.config/.vim/plugged/vimtex/autoload/vimtex/debug.vim
 51: ~/.config/.vim/plugged/vimtex/autoload/vimtex/delim.vim
 52: ~/.config/.vim/plugged/vimtex/autoload/vimtex/doc.vim
 53: ~/.config/.vim/plugged/vimtex/autoload/vimtex/echo.vim
 54: ~/.config/.vim/plugged/vimtex/autoload/vimtex/env.vim
 55: ~/.config/.vim/plugged/vimtex/autoload/vimtex/format.vim
 56: ~/.config/.vim/plugged/vimtex/autoload/vimtex/fzf.vim
 57: ~/.config/.vim/plugged/vimtex/autoload/vimtex/imaps.vim
 58: ~/.config/.vim/plugged/vimtex/autoload/vimtex/include.vim
 59: ~/.config/.vim/plugged/vimtex/autoload/vimtex/info.vim
 60: ~/.config/.vim/plugged/vimtex/autoload/vimtex/kpsewhich.vim
 61: ~/.config/.vim/plugged/vimtex/autoload/vimtex/log.vim
 62: ~/.config/.vim/plugged/vimtex/autoload/vimtex/matchparen.vim
 63: ~/.config/.vim/plugged/vimtex/autoload/vimtex/pos.vim
 64: ~/.config/.vim/plugged/vimtex/autoload/vimtex/misc.vim
 65: ~/.config/.vim/plugged/vimtex/autoload/vimtex/motion.vim
 66: ~/.config/.vim/plugged/vimtex/autoload/vimtex/process.vim
 67: ~/.config/.vim/plugged/vimtex/autoload/vimtex/profile.vim
 68: ~/.config/.vim/plugged/vimtex/autoload/vimtex/scratch.vim
 69: ~/.config/.vim/plugged/vimtex/autoload/vimtex/syntax.vim
 70: ~/.config/.vim/plugged/vimtex/autoload/vimtex/text_obj.vim
 71: /usr/share/vim/vim81/ftplugin/tex.vim
 72: ~/.config/.vim/plugged/vimtex/indent/tex.vim
 73: /usr/share/vim/vim81/indent/tex.vim

Vim version is

VIM - Vi IMproved 8.1 (2018 May 18)
Patch incluse: 1-892
Compilato da 'http://www.opensuse.org/'

[Konfekt]

1: ~/.config/.vim/viminrc reads

set nocompatible
let &rtp = '~/.vim/plugged/vimtex' . ',' . &rtp
let &rtp .= ',../../../after'

syntax on
filetype plugin indent on
set spell

[lervag]

For some reason, the after/syntax/tex.vim was not sourced. Can you try to use the same form, i.e.:

set nocompatible
let &rtp = '~/.vim/plugged/vimtex,' . &rtp
let &rtp .= ',~/.vim/plugged/vimtex/after'
filetype plugin indent on
syntax on

set spell

(I also changed the order of syntax on and filetype plugin indent on, but that should not really matter.)

[Konfekt]

Okay, that works around the issue in the minimal vimrc. In Vim, ~/.vim/plugged/vimtex/after is sourced and indeed no spell error in this minimal TeX file is marked.

That still leaves the initial problem open of a spell error being marked when \usepackage{cleveref} cannot be found; for example, in included files (into the main file).

[lervag]

... for example, in included files (into the main file).

And this is exactly what I want an example for. For me, the test files I've pushed works as expected both in the main.tex file and in the included.tex file.

[Konfekt]

Okay, I am sorry for the confusion. The command \RequirePackage{cleveref}instead of \usepackage{cleveref} was used, which lead to this issue.

[lervag]

Yes, that makes sense. I'll look at the PR immediately.

[lervag]

Thanks for the PR!

[Konfekt]

Somehow this fixes the issue in the minimal examples, but in a big project of mine it does not resolve it; I will have to distill another one.

[lervag]

In the big project, you can check if the b:vimtex_syntax variable includes a key for cleveref. It probably does not, which is why it fails...

[Konfekt]

Thank you, indeed it is empty despite a 500 line preamble. Perhaps you know of a common reason for this to happen?

[lervag]

No, this should not happen, I think. So it is probably a bug. But it is hard to say until I can reproduce it.

One possibility is for you to share your manuscript with me, although I clearly understand why you might not want to do that. If you did, e.g. by email, I would of course promise not to use it for anything but locating the issue.

The other alternative is that you again try to reduce your manuscript little by little to make a minimal working example which you may share here.

[Konfekt]

Mysteriously, it hinges on the existence of the *.fls file. I sent you an idiosyncratic example.

[Konfekt]

That is, b:vimtex_syntax is empty if the *.fls file exists, otherwise its value is correct.

[Konfekt]

I found the error. It is because I precompile a format file preamble.fmt that loads all packages, in particularcleveref. Therefore all packages loaded by the format file, in particularcleveref, do not appear in the fls file.

To correct this, one should use s:vimtex.parse_packages_from_fls() instead of s:vimtex.parse_packages() only if

• afls file exists, and
• it does not contain a line that matches \vINPUT \f+.fmt.

[Konfekt]

Here is the command to compile a preamble:

command! -buffer -bang -complete=file_in_path -nargs=1 TexCompilePreamble call s:TexCompilePreamble(<q-args>,<bang>0)

function! s:TexCompilePreamble(file, bang)
  wall
  if has('win32')
    let oldsx = &shellxescape
    set shellxescape=
    execute '!' . (a:bang ? 'start /b' : '') . s:preambleCmdWin32(a:file)
    echo "building preamble ..."
    let &shellxescape = oldsx
  elseif has('unix')
    execute '!' . s:preambleCmdUnix(a:file) . (a:bang ? '&' : '')
  endif
endfunction

function! s:preambleCmdWin32(file)
  let oldss = &shellslash
  set shellslash
  let mainfile = shellescape(expand(a:file))
  let &shellslash = oldss

  let cmd = 'pdftex -ini -interaction=nonstopmode -jobname=preamble "&pdflatex" mylatexformat.ltx ""' . mainfile . '""'
  return cmd
endfunction

function! s:preambleCmdUnix(file)
  let mainfile = shellescape(expand(a:file))
  let cmd = 'pdftex -ini -interaction=nonstopmode -jobname=preamble "&pdflatex" mylatexformat.ltx ' . mainfile
  return cmd
endfunction

[Konfekt]

To use the preamble, the tex code replaced by the preamble must be surrounded by the following lines:

%&preamble

  ...

\csname endofdump \endcsname

For example

%&preamble
  \documentclass[draft,DIV = calc,headings = big]{scrartcl}
  \usepackage[USenglish]{babel}
\csname endofdump \endcsname

[Konfekt]

This is from https://web.archive.org/web/20160712215709/http://www.howtotex.com:80/tips-tricks/faster-latex-part-iv-use-a-precompiled-preamble/

[Konfekt]

Note that, as https://web.archive.org/web/20160712215709/http://www.howtotex.com:80/tips-tricks/faster-latex-part-iv-use-a-precompiled-preamble/ states

It is recommended to write: \csname endofdump\endcsname instead of endofdump because if the format is not loaded, endofdump is an undefined cs, while \csname endofdump\endcsname expands to \relax.

[lervag]

Thanks, it seems you have located the problem. But I have to admit I don't quite follow your posts. It would be easier if you could just provide a relevant set of minimal LaTeX files. Preferably, if possible, ones that I could directly compile to generate the "faulty" fls file.

I would assume something like this:

% file: preamble.tex
% ------------------
\documentclass[draft,DIV = calc,headings = big]{scrartcl}
\usepackage[USenglish]{babel}
\usepackage{cleveref}

% file: main.tex
% --------------
% some preamble stuff here, I assume, to include the precompiled preamble
\begin{document}

Hello World!
\cref{speling}
\include{intro.tex}

\end{document}

% file: intro.tex
% ---------------
\section{Intro}

By \cref{speling} ...

[lervag]

Ok, I've managed to use the preamble trick, and I have sort of pinpointed the problem. For some reason, your fls file does include some packages, and when it does, vimtex assumes it includes _all_ packages. However, when I try to reproduce it, I can not get the fls file to include any packages. I.e., see the test files at test/issues/1471, where make will generate the preamble.fmt. When you now compile the minimal.tex, the fls file does not contain any links to the xspace package, even though it was added _after_ the endofdump line.

[Konfekt]

Vimrc file:

set nocompatible
let &rtp = '~/.vim/plugged/vimtex,' . &rtp
let &rtp .= ',~/.vim/plugged/vimtex/after'
filetype plugin indent on
syntax on

set spell

command! -bang -complete=file_in_path -nargs=1 TexCompilePreamble call s:TexCompilePreamble(<q-args>,<bang>0)

function! s:TexCompilePreamble(file, bang)
  wall
  if has('win32')
    let oldsx = &shellxescape
    set shellxescape=
    execute '!' . (a:bang ? 'start /b' : '') . s:preambleCmdWin32(a:file)
    echo "building preamble ..."
    let &shellxescape = oldsx
  elseif has('unix')
    execute '!' . s:preambleCmdUnix(a:file) . (a:bang ? '&' : '')
  endif
endfunction

function! s:preambleCmdWin32(file)
  let oldss = &shellslash
  set shellslash
  let mainfile = shellescape(expand(a:file))
  let &shellslash = oldss

  let cmd = 'pdftex -ini -interaction=nonstopmode -jobname=preamble "&pdflatex" mylatexformat.ltx ""' . mainfile . '""'
  return cmd
endfunction

function! s:preambleCmdUnix(file)
  let mainfile = shellescape(expand(a:file))
  let cmd = 'pdftex -ini -interaction=nonstopmode -jobname=preamble "&pdflatex" mylatexformat.ltx ' . mainfile
  return cmd
endfunction

TeX file:

% file: preamble.tex
% ------------------
%&preamble
\documentclass[draft,DIV = calc,headings = big]{scrartcl}
\usepackage[USenglish]{babel}
\usepackage{cleveref}
\csname endofdump \endcsname

% file: main.tex
% --------------
% some preamble stuff here, I assume, to include the precompiled preamble
\begin{document}

Hello World!
\cref{speling}
\include{intro.tex}

\end{document}

% file: intro.tex
% ---------------
\section{Intro}

By \cref{speling} ...

Then, open preamble.tex in Vim and

1. enter :TexCompilePreamble %<cr>
2. enter :VimtexCompileSS<cr>
3. enter :edit %<cr>

The format file preamble.fmt loads all packages between %&preamble and \csname endofdump \endcsname (or \endofdump), in particularcleveref.
Therefore all packages loaded by the format file, in particularcleveref, do not appear in the fls file.

Because preamble.fls contains INPUT preamble.fmt instead of INPUT INPUT /usr/local/texlive/2019/texmf-dist/tex/latex/cleveref/cleveref.sty, echo b:vimtex_syntax is empty;
but we expect it to contain cleveref.

To correct this, one should use s:vimtex.parse_packages_from_fls() instead of s:vimtex.parse_packages() only if

• afls file exists, and
• it does not contain a line that matches \vINPUT \f+\.fmt.

Most correctly,

• afls file exists, and
• it contains a line that matches \vINPUT (\f+)\.fmt,

then on top of all files found in the fls file, all packages between %&preamble and \csname endofdump \endcsname (or \endofdump) should be added. I think this is not worth it, because few people ever heard of preamble compilation; I am not even sure there is a tangible benefit --- perhapslatexmk` does something similar already by default?!

[Konfekt]

I can not get the fls file to include any packages. I.e., see the test files at test/issues/1471, where make will generate the preamble.fmt. When you now compile the minimal.tex, the fls file does not contain any links to the xspace package, even though it was added after the endofdump line.

Perhaps TeX detects that the xspace package is not used and therefore does not load it?!

[lervag]

Yes, perhaps. I'm not sure, but when adding a \xspace command it does work. Perhaps it was because I reloaded the preamble or because I changed to \endofdump command. In any case, it is reproduced, so I will find a solution perhaps later tonight.

[lervag]

Ok, I think I fixed it already. Could you test?

[Konfekt]

Works fine. Thank you!

[lervag]

Happy to hear it :)

[Konfekt]

Perhaps this regular expression is too permissive, as, from what I can see, most fls files contain the standard format file INPUT /usr/local/texlive/2019/texmf-var/web2c/pdftex/pdflatex.fmt. Maybe better check instead for fmt files in the home directory, that is,

\VINPUT ' . escape(expand($HOME) . (has('win32) ? '\' : '/'), '\')  . '\v\f+\.fmt'

[lervag]

Hmm. I'm also curious whether it is any point in checking... the code as is now looks like this:

function! s:vimtex.parse_packages() abort dict " {{{1
  let self.packages = {}

  " Only parse the preamble if fls file is not enough
  if self.parse_packages_from_fls() | return | endif

  " Parse preamble in the following

I think the initial idea was to optimize startup time, but I would hypothesize
that the impact is very minimal if we just replace with this

function! s:vimtex.parse_packages() abort dict " {{{1
  let self.packages = {}

  call self.parse_packages_from_fls()

  " Parse preamble in the following

The main effect is that the preamble is always parsed, and the fls file is parsed if present. There is no risk for conflicts or similar, as far as I can tell. And this change would solve the issue cleanly.

The only uncertainty is if it would introduce a noticable lag. I don't think it will, and I am tempted to just try this and be alert for possible reactions. If it should lead to undesired lags, then I could look into improvements after the fact.

[Konfekt]

Sounds good. I'd replace the current regex by the one proposed above and then remove the safeguards.

[lervag]

Good, I'll update. Note, with my proposal, there is no need to check for any .fmt files. :)

[clason]

@Konfekt Bit of a stupid question, but the test inludes a ~9MB .fmt file, increasing the size of the repo by nearly 1/3. Can this be reduced a bit?

[lervag]

I didn't notice that. This is my fault, not @Konfekt's. I could fix it, but it would mean a lot of people would need to "force pull". Would that be OK?

[clason]

Not a big deal, just wondering whether a smaller test example would do as well. Probably not worth breaking people's setup (not sure how vim-plug would handle a force pull...)

(I believe the "pro approach" would be a separate repository for the test data that the test script pulls in as a submodule, but, again, I'm not sure whether that's worth the effort.)

[lervag]

Ok. I've deleted the file, which should reduce the size a bit, but since it will live in history it can't be removed unless I break things for users who update often. I won't do that, since this is not a big deal. Thanks for notifying about the big file, though.

[clason]

git pull followed by git gc reduced the size of the repo on disk significantly, so that already made a big (and enough of a) difference!

[lervag]

Great, happy to hear it! :)

[clason]

The next biggest offender is your thesis, by the way (issue 1191) -- I guess that's one way of fulfilling the publishing requirement ;)

[lervag]

Haha! My way of pushing it to a larger audience! (My guess is that the number of people who have actually read it is ~5.) :)

[clason]

If the number is larger than (or equal to...) the number of reviewers, you're already ahead of the curve ;)