Vimtex: Syntax highlighting of AMS environments only correct after saving and reopening file.

[gillescastel]

Issue
Syntax highlighting of AMS environments only works after saving and reopening a file. When simply creating a new file, the syntax highlighting is incorrect – even after saving the file.

minimal.vim

```vim
set clipboard=unnamedplus
call plug#begin(stdpath('data') . '/plugged')
Plug 'lervag/vimtex'
let g:tex_flavor='latex'
let g:vimtex_view_method='zathura'
let g:vimtex_quickfix_mode=0
call plug#end()
map :echo map(synstack(line('.'), col('.')), 'synIDattr(v:val, "name")')

**input.tex**

  ```tex
\documentclass[a4paper]{article}
\usepackage{amsmath, amssymb}
\begin{document}
\begin{align*}

.\end{align*}
\end{document}
  ```
**Commands/Input**

- Execute `nvim -u minimal test.tex`, where `test.tex` is a non-existent file.
- Paste the contents of `input.tex` (`:r input.tex`)
- Go inside the `align*` environment and press F10. This prints the syntax layers (don't know the right name). It says `['texDocZone']`. (The F10 mapping is defined in `minimal.vim`)

However, directly editing `input.tex` (`nvim -u minimal.vim input.tex`) and doing the same gives `['texDocZone', 'texMathZoneAmsAS']`, which is the expected behaviour. 

<!-- Provide set of keys or command to reproduce the issue-->

**Observed Behaviour**
When creating a new file with ams environments, the syntax highlighting is not correct. It outputs `['texDocZone']`

**Expected Behaviour**
When creating a new file with ams environments, the syntax highlighting should be correct, i.e. `['texDocZone', 'texMathZoneAmsAS']`


**Output from VimtexInfo when creating a new file `test.tex`, after I pasted the contents of `input.tex`**

System info
OS: Ubuntu 18.04.4 LTS
Vim version: NVIM v0.5.0-dev
Has clientserver: true
Servername: /tmp/nvimkd4L8F/0

vimtex project: test
base: test.tex
root: /home/gilles/Desktop/vimtexalignissue
tex: /home/gilles/Desktop/vimtexalignissue/test.tex
out:
log:
aux:
fls:
main parser: current file
compiler: latexmk
backend: nvim
output: /tmp/nvimkd4L8F/1
configuration:
continuous: 1
callback: 1
latexmk options:
-verbose
-file-line-error
-synctex=1
-interaction=nonstopmode
latexmk engine: -pdf
viewer: Zathura
xwin id: 0
qf: LaTeX logfile
config:
packages:
default: 1
default: 1
document class:

**Output from VimtexInfo when editing the existing file `input.tex`**

System info
OS: Ubuntu 18.04.4 LTS
Vim version: NVIM v0.5.0-dev
Has clientserver: true
Servername: /tmp/nvimVfT4Jg/0

vimtex project: input
base: input.tex
root: /home/gilles/Desktop/vimtexalignissue
tex: /home/gilles/Desktop/vimtexalignissue/input.tex
out:
log:
aux:
fls:
main parser: current file verified
compiler: latexmk
backend: nvim
output: /tmp/nvimVfT4Jg/1
configuration:
continuous: 1
callback: 1
latexmk options:
-verbose
-file-line-error
-synctex=1
-interaction=nonstopmode
latexmk engine: -pdf
viewer: Zathura
xwin id: 0
qf: LaTeX logfile
config:
packages:
default: 1
default: 1
document class: article
packages:
amsmath
amssymb
```

I use the syntax highlighting groups to detect whether or not to expand UltiSnips snippets.

[gillescastel]

After I compile, the syntax highlighting is correct. Seems like Vimtex is not updating the list of used packages when you save a file, but only when compile / open?

[lervag]

Ah, yes. This is currently a known issue and more of a limitation really. See e.g. #1546.

The problem is that it is difficult to parse the list of packages. There are two methods: 1) simple (parse usepackages in preamble) and advanced (parse fls file). The advanced only works when the document is compiled, as the .fls file is generated during compilation.

I have added support for reparsing the package list and loading syntax rules after compilation through the callbacks, which slightly alleviate this. But I'm sorry to say I don't really know a good solution to this.

We could add a rule to reload the package list on save, but I'm afraid it would take too much time. But perhaps I'm just paranoid; I've recently improved some of the slow parsing problems, so perhaps it is actually fine to add some autocommands to buffer writing...

[lervag]

I've been thinking: A possible solution could be to add an option to specify syntax packages to always load. E.g.:

I've been thinking: A possible solution could be to add an option to specify syntax packages to always load. E.g.:

let g:vimtex_syntax_autoload_packages = ['amsmath', 'tikz']

This would specify to always load the amsmath and tikz syntax rules, and to optionally load other package syntax rules when the packages are detected.

Opinions? (Possibly interested people: @ddrake @kiryph @clason)

[clason]

I saw that issue, and was thinking in a somewhat similar direction: There are some packages that are so fundamental to LaTeX that hardcoding their syntax would be worth it (pretty much the AMS classes, and maybe graphicx; TikZ, on the other hand, is a bit heavy-weight for that -- and I know people who strongly prefer asymptote). Of course, people from humanities would disagree...

So I guess making it an option is indeed the only sensible course of action? (One could discuss about possible -- non-empty -- default values, though. I think amsmath would be lightweight and ubiquitous enough to qualify, as would graphicx.)

[lervag]

@clason You're fast! :)

I saw that issue, and was thinking in a somewhat similar direction: There are some packages that are so fundamental to LaTeX that hardcoding their syntax would be worth it (pretty much the AMS classes, and maybe graphicx; TikZ, on the other hand, is a bit heavy-weight for that -- and I know people who strongly prefer asymptote). Of course, people from humanities would disagree...

So I guess making it an option is indeed the only sensible course of action? (One could discuss about possible -- non-empty -- default values, though. I think amsmath would be lightweight and ubiquitous enough to qualify, as would graphicx.)

There is actually no added syntax rules for graphicx, so nothing to do there. I agree wrt. TikZ and asymptote, no need to make this automatically loaded. I've not met people who prefer asymptote, but then again, I don't really personally know many LaTeX power users.

I think the default setting should try to balance between being conservative and pragmatic. I've already pushed an update on this where I've added amsmath to the default list. Perhaps we should add more. Here's the list of currently available syntax packages; some of them are only "links" to others (e.g. mathtools load amsmath).

• amsmath
• array
• asymptote
• beamer
• biblatex
• breqn
• cases
• cleveref
• csquotes
• dot2texi
• glossaries_extra
• glossaries
• gnuplottex
• hyperref
• listings
• luacode
• markdown
• mathtools
• minted
• moreverb
• natbib
• pdfpages
• pgfplots
• pythontex
• subfile
• tabularx
• tikz
• url
• varioref
• wiki

[clason]

@lervag And you're even faster, implementing it :)

Pragmatic is a good mindset here, I agree. Regarding the specific packages, the only ones I _always_ use are mathtools and hyperref (and either cleveref _or_ beamer). I think it might be good to keep the goal in mind here (correct syntax highlighting, especially of environments), for which I only see mathtools (and beamer, but that is not a good default?) being relevant. (On the other hand, I hadn't even noticed the original issue before... So maybe wait until someone complains about missing or wrong syntax highlighting, and then consider adding that package?)

[lervag]

@lervag And you're even faster, implementing it :)

Hehe. I like the "simple" things that are easy to fix/improve. Should probably try and get more time to solve the harder problems as well...

Pragmatic is a good mindset here, I agree. Regarding the specific packages, the only ones I always use are mathtools and hyperref (and either cleveref or beamer). I think it might be good to keep the goal in mind here (correct syntax highlighting, especially of environments), for which I only see mathtools (and beamer, but that is not a good default?) being relevant. (On the other hand, I hadn't even noticed the original issue before... So maybe wait until someone complains about missing or wrong syntax highlighting, and then consider adding that package?)

Ok, I think we'll leave things as they are now, then. mathtools simply load amsmath, and so I think amsmath makes for a useful default. And I'll consider to add more if there are issues. Currently, I've noticed perhaps two issues related to this where I think both reacted to the lack of AMS support.

@gillescastel Does this new option adequately solve your original issue?

[clason]

Ok, I think we'll leave things as they are now, then. mathtools simply load amsmath,

Are you sure that's correct? If I recall, mathtools adds some environments as well (such as multlined -- but those go inside math mode, so shouldn't influence syntax highlighting). But yes, let's wait for explicit demands.

[lervag]

Are you sure that's correct?

Yes. Here's the content of autoload/vimtex/syntax/p/mathtools.vim:

" vimtex - LaTeX plugin for Vim
"
" Maintainer: Karl Yngve Lervåg
" Email:      [email protected]
"

scriptencoding utf-8

function! vimtex#syntax#p#mathtools#load() abort " {{{1
  if has_key(b:vimtex_syntax, 'mathtools') | return | endif
  let b:vimtex_syntax.mathtools = 1

  " Load amsmath
  call vimtex#syntax#p#amsmath#load()
endfunction

" }}}1

Note, this does not mean that the current implementation is "perfect". There might be macros or environments from mathtools that should/could be added/improved. But as far as I can recall, noone's asked me to do it.

[clason]

Oh, sorry, I misunderstood -- somehow I thought you were talking about mathtools.sty itself, not the vimtex syntax support...

Yes, leave it minimal until someone complains requests a specific inclusion.

[lervag]

Oh, sorry, I misunderstood -- somehow I thought you were talking about mathtools.sty itself, not the vimtex syntax support...

Ah, I see.

I'll take the liberty of closing this issue. @gillescastel Feel free to reopen if you feel something is still missing.