Latex-workshop: Adding language support for LaTeX Expl3 and DocTeX

[Liam0205]

Issue Type: Feature Request

I'm a TeX/LaTeX macro developer.

When developing with LaTeX Expl3 and the DocTeX util, I find that LaTeX workshop for VSCode could not display code highlighting well. Please consider adding the support for the combination of them.

Extension version: 5.3.0
VS Code version: Code 1.22.2 (3aeede733d9a3098f7b4bdc1f66b63b0f48c1ef9, 2018-04-12T16:32:53.389Z)
OS version: Darwin x64 16.7.0

System Info

|Item|Value|
|---|---|
|CPUs|Intel(R) Core(TM) i5-4278U CPU @ 2.60GHz (4 x 2600)|
|Load (avg)|2, 2, 2|
|Memory (System)|8.00GB (0.03GB free)|
|Process Argv|/Applications/Visual Studio Code.app/Contents/MacOS/Electron|
|Screen Reader|no|
|VM|0%|

[jlelong]

Can you be more precise on what you expect or even give us an example?

[Liam0205]

Sorry for anything that is not precise enough.

In LaTeX, % is used for comment. It works like # in Python. However, in DocTeX mode, the leading %s will be striped for literate programming. That is to say, in DocTeX mode, comments behind the leading %s forms the documentation of those code are not being commented. This is why VSCode highlights contents behind leading %s in DocTeX mode.

The commands in LaTeX are begun with the escape char \ and followed by a uninterrupted sequence of English letters (called control word) or followed by a single sign (called control sign). In LaTeX(2e) mode, LaTeX expl3 commands like \@@_alternate_cs:n are invalid in most time, hence, in most LaTeX highlighting extensions, including DocTeX mode in VSCode, this kind of commands will not be highlighted correctly. In short, LaTeX expl3 allows @, _, and : in valid commands, while LaTeX(2e) does not.

When turning VSCode working in LaTeX Expl3 mode, these LaTeX Expl3 commands could be highlighted correctly, however, if one is developing LaTeX Expl3 with DocTeX utility, VSCode will simply ignore to highlight the DocTeX documentation parts in the file, and treat them as common comments.

Please note that LaTeX(2e) and LaTeX Expl3 are two versions of syntax of LaTeX (like Python2 and Python3), while DocTeX is a utility that could be used together with both LaTeX(2e) and LaTeX Expl3. Hence, I'm here asking for adding language highlighting support for the combination of LaTeX Expl3 and DocTeX.

Code snippet for testing.

% \begin{macro}{\@@_check_family:n}
% 删除重复的定义，清除替代字体的先前设置。
%   \begin{macrocode}
\cs_new_protected_nopar:Npn \@@_check_family:n #1
  {
    \prop_gpop:NnNT \g_@@_family_font_name_prop {#1} \l_@@_tmp_tl
      {
        \cs_undefine:c { \@@_family_csname:n {#1} }
        \cs_undefine:c { \@@_alternate_cs:n {#1} }
        \prop_gpop:NnNT \g_@@_family_name_prop {#1} \l_@@_base_family_tl
          {
            \use:c { \@@_alternate_cs:n { clear / #1 } }
            \cs_undefine:c { \@@_alternate_cs:n { clear / #1 } }
            \cs_undefine:c { \@@_alternate_cs:n { reset / #1 } }
            \prop_gremove:Nn \g_@@_reset_alternate_prop {#1}
          }
        \msg_warning:nnxx { ctex } { redefine-family } {#1} { \l_@@_tmp_tl }
      }
  }
\tl_new:N \l_@@_tmp_tl
\msg_new:nnn { ctex } { redefine-family }
  { Redefining~CJKfamily~`\@@_msg_family_map:n {#1}'~(#2). }
%    \end{macrocode}
% \end{macro}

[jlelong]

while DocTeX is a utility that could be used together with both LaTeX(2e) and LaTeX Expl3

This is in principle true but this is not the way it is handled by LaTeX Workshop. The syntax used depends on the filename extension. Currently, only .dtx files trigger the DocTeX syntax, which in turn includes the LaTeX one.

The simplest way to achieve what you are asking for is to merge the LaTeX(2e) and LaTeX Expl3 syntaxes. In my opinion, this is related to #461. I think allowing @, _ and : in macro names in the LaTeX syntax actually make sense, all the more so as it can happen to have a @ in a macro name thanks to makeatletter. Then, DocTeX could be used with both of them.

Any comment on the idea of merging LaTeX(2e) and LaTeX Expl3 syntaxes?

[Liam0205]

Well, I know the author of issue 461, yet I didn't know that he had posted the issue.

brief introduction to catcode in TeX and @, _, and :

Basically speaking, synbols, such as @, _ and :, are not allowed in control sequence, in user mode, except for control signs, say \@, \_ and so on. This is because TeX treats input not only by character codes (for example, ASCII), but also by category codes. The category code of a specific character code is preset by TeX's format, and could be revised during the process of TeX. That is why \makeatletter makes @ valid in the name of control sequences in user mode, this macro changes the catcode of @ from "sign" to "character".

I emphasised user mode above, for the reason that, in .sty-files and .cls-files the catcode of @ will be set to character automatically.

For LaTeX3's syntax, _ and : are allowed in the name of control words. However, LaTeX3 is still under developing, and hence a experiment layer is developed to work with current LaTeX2e, which is exact the Expl3. Just like what the pair, \makeatletter and \makeatother, does, the pair, \ExplSyntaxOn and \ExplSyntaxOff, changes the catcode of _ and :.

When dealling with .dtx files, LaTeX3 introduce @ as a shortcut of current package name. That is to say, \@@_foobar: will be replaced by the DocStrip utility to \ctex_foobar:. Hence, although @ is not allowed in the name of macros in LaTeX3, by default, DocTeX mode highlighting should also treat it as
a valid character in LaTeX3 mode.

Currently, merging LaTeX2e and LaTeX Expl3 is not a good idea, for the following reasons, at least.

• Most general users know nothing about LaTeX3 and its syntax, merging syntax highlighting without noticing breaks basic rules of software developing.
• This kind of merging could be puzzled. For example in LaTeX2e's math mode, \epsilon_i means the variable \epsilon with underscript i, however in LaTeX3's syntax, \epsilon_i is a valid control sequence.

Dynamically changing syntax highlighting scheme, based on the macro pairs (\makeatletter, \makeatother and \ExplSyntaxOn, \ExplSyntaxOff), is still not a good idea too. There are too many edge case to handle.

---

According to your post, LaTeX workshop firstly detects the file extension, say .dtx, and then triggers DocTeX mode, which in turn highlights codes based on LaTeX2e. I wonder if we could specify which kind of LaTeX syntax will be triggered by DocTeX mode? That seems to be the only solution right now.

[Liam0205]

Any suggestions or clues for this thread? @Stone-Zeng

[stone-zeng]

At present, the extension-language binding is the following:

| ID | Name | File Extensions |
|:-:|:-:|:-:|
| tex | TeX | .sty .cls .bbx .cbx |
| doctex | DocTeX | .dtx |
| latex | LaTeX | .tex |
| bibtex | BibTeX | .bib |
| latex-memoir | LaTeX Memoir | |
| latex-expl3 | LaTeX Expl3 | |

latex-expl3 doesn't bind to a specific file extension, so similarly, we may have a language like doctex-expl3. Anyway, which extension corresponds to which language can be set by the user.

[jlelong]

Considering the level of expertise required to use doctex, I think accepting both LaTeX(2e) and LaTeX Expl3 syntaxes within doctex would make sense and be an acceptable solution.

[stone-zeng]

By the way, I think the snippets like \begin ... \end and \usepackage should be enabled in both expl3 and doctex mode.

[jlelong]

OK for the snippets but the \begin ... \end would insert

\begin{env}

\end{env}

instead of

%  \begin{env}
%
%  \end{env}

I think that adding the leading % would require to either redefine all the snippets or heavily tweak the Completer implementation. I do not have time to go into this right now.

Concerning syntax highlighting, here is my

• [x] Related to #461, I plan to merge latex-memoir into latex. The main difference between the two is that in latex-memoir, sectioning commands may take two optional arguments instead of one.
• [X] Related to 611bf18 and #535, update the scopes used by latex-expl3 to match those used by latex.

[stone-zeng]

I think the leading % should not be add, since environment can exist both in "code" and "document". It depends on the user.

[jlelong]

Ok, I have made a PR to enable snippets in doctex mode.

[jlelong]

I think this issue can be closed as latex3 syntax is now enabled in doctex mode, see #582.
I will continue my work on unifying/simplifying syntax files. I suggest to continue our discussion in #461.