Vimtex: 'build_dir' option for `g:vimtex_compiler_arara`

[xarthurx]

Issue
Following this issue:https://github.com/lervag/vimtex/issues/907

We can set the default compiler method:

let g:vimtex_compiler_method = 'arara'

And from help file, there's also another option:

    let g:vimtex_compiler_arara = {
        \ 'options' : ['-v'],
        \}

However, there seem to be no more options available.

As we need to give the command to arara by ourself, this is understandable.
However, the build_dir seems to be a good default option to be added.

Currently, if I set

let g:vimtex_compiler_latexmk = {
    \ 'build_dir' : 'build',
    \}

arara seems also recognize that folder setting and this is a bit confusing....
Shall we add a build_dir to the g:vimtex_compiler_arara itself?

minimal.vim

" using arara as automation script engine
let g:vimtex_compiler_method = 'arara'

" default using pdflatex
let g:vimtex_compiler_latexmk = {
    \ 'build_dir' : 'build',
    \}

let g:vimtex_compiler_arara = {
      \ 'options' : ['-v'],
      \}

% arara: pdflatex
\documentclass{article}

\begin{document}
Hello world.

\end{document}

[xarthurx]

Additional comments, I made a mistake when latexmk was not stopped while starting arara.

The settings for build_dir in latexmk option doesn't apply for the g:vimtex_compiler_arara.

[lervag]

And from help file, there's also another option:

    let g:vimtex_compiler_arara = {
        \ 'options' : ['-v'],
        \}

However, there seem to be no more options available.

Correct.

As we need to give the command to arara by ourself, this is understandable.
However, the build_dir seems to be a good default option to be added.

As far as I understand from the arara documentation, it is not supported. The way build_dir works for latexmk and latexrun is that the option is passed on to the build tool. Vimtex does not do the actual work. For arara, it seems this is not possible, and so I don't know how to add this option. Sorry.

[xarthurx]

Thanks for the quick response.
That's a bit sad... I'll open an issue on the arara repo.

[lervag]

Yes, please do. If you get a response and the feature is added, let me know.

[LambdaP]

Hi. Sorry if there is a simple workaround somewhere else, I’ve been looking for a solution for some time now, to no avail.

The problem of the lack of build_dir instruction for g:vimtex_compiler_arara is that the rest of Vimtex does not know what the build directory is. In my current project, the build directory is /path/to/root/out/ and I’m using the arara compiler, which knows what to do from the arara directives. This creates problems for :VimtexView:

vimtex: Viewer cannot read PDF file!
        /path/to/root/main.pdf

well, duh.

From what I gather from reading the documentation of arara, giving it a general build_dir argument is not in the spirit of the tool at all and I don’t expect the tool to ever have such a feature. My arara directives in main.tex look like this:

% arara: lualatex: {
% arara: --> options:     [ '--output-directory=out', ... ],
% arara: --> ...
% arara: --> }
% arara: biber: {
% arara: --> options: [ '--input-directory=out', '--output-directory=out' ],
% arara: --> ...
% arara: --> }
% arara: lualatex: {
% arara: --> options:     [ '--output-directory=out', ... ],
% arara: --> ...
% arara: --> }

…and if there’s a way around doing it like that, I have not found it (the two lualatex directives don’t have the same arguments, otherwise I could factor them out in a preamble and gain a few lines).

The build directory is obviously a project-dependent matter, and I guess the correct way for Vimtex to learn it is the way it’s done with latexmk: go get the information in the configuration file. For the above reasons, there isn’t (according to my understanding, and I think @lervag's as well) a clean way to do the same thing with arara, and I think there may never be.

Given that, could it be possible to to tell Vimtex (either/or):

• what the pdf file is with something like let b:vimtex_pdf_file = './out/main.pdf'
• a list of fallback paths to try before quitting with something like let g:vimtex_output_fallback_dirs = [ './out', './build' ]

I don’t have any other ideas right now but this seems like a worthwhile thing to think about.

As this is my first (methinks) time commenting here, here comes the obligatory (but utterly sincere) part where I thank everyone, and @lervag in particular, for building and maintaining this tool. It truly is a delight to use.

[lervag]

I don’t have any other ideas right now but this seems like a worthwhile thing to think about.

I agree. Possibilities:

1. Add a build_dir option to g:vimtex_compiler_arara. This would only be used for the viewer to find the pdf file, though, and might be confusing since it does not really tell arara anything.
2. Vimtex could parse the arara configuration lines at the top of the main file for a string like output-directory=... and use that if available.
3. b:vimtex_pdf_file as mentioned
4. g:vimtex_output_fallback_dirs as mentioned

Of the above options, I personally think only number 2. is sensible. The first one is confusing because it seems similar to e.g. g:vimtex_compiler_latexmk.build_dir. The third is possible, but would be a "global" solution that would work regardless of arara, and it seems slightly inconvenient. The 4th also seems slightly inconvenient, and it could lead to errors where the wrong document was opened.

What do you think about idea 2., parsing the configuration lines? To implement it, I would need one or more examples of various arara based LaTeX files. I want to make a couple of tests.

As this is my first (methinks) time commenting here, here comes the obligatory (but utterly sincere) part where I thank everyone, and @lervag in particular, for building and maintaining this tool. It truly is a delight to use.

Thanks!

[LambdaP]

Hey, thanks for the quick reply.

I’ve thought about that before, and I think an approach like 2. will feel very hacky, will never be entirely robust, and might require more work over the long run. Let me explain.

arara is more like make than like latexmk or variants – it’s a tool to which you specify the steps to build a project, and then it runs through them when you call the tool. It comes with a predefined set of directives that more or less cover everything a latex user could want to do, and so there is a directive for basically all existing compilers (latex, pdflatex, xelatex, lualatex, luahblatex, context, etex…), external tools necessary to produce the output (bibtex, biber, makeindex), but also various tools such as latexindent and whatnot. Directive instructions take the form of

% arara: directive_name: { key1: val1, … keyN: valN }

…which I guess is YAML? There’s also a syntax for breaking line, which may or may not be YAML, I don’t know. Here’s the unabridged version of the arara directives I posted above.

% arara: lualatex: {
% arara: --> options:     [ '--output-directory=out',
% arara: -->                '--file-line-error' ],
% arara: --> interaction: nonstopmode,
% arara: --> synctex:     yes,
% arara: --> draft:       yes,
% arara: --> shell:       yes,
% arara: --> }
% arara: biber: {
% arara: --> options: [ '--input-directory=out',
% arara: -->            '--output-directory=out' ],
% arara: --> } if found (toFile('out/main.log'),
% arara: -->             'Please \\(re\\)run Biber on the file')
% arara: -->   || changed (toFile('bibliography.bib'))
% arara: lualatex: {
% arara: --> options:     [ '--output-directory=out',
% arara: -->                '--file-line-error' ],
% arara: --> interaction: nonstopmode,
% arara: --> synctex:     yes,
% arara: --> shell:       yes,
% arara: --> }

(NB: I would not take this as a great example of good arara code; for instance it looks like it’s re-running biber at every invokation even though I’d like it to only do so when needed.)

So I guess you could just parse the entire thing and look for '--output-directory=foo' and that would work in that case. I have not checked exhaustively but all TeX compilers I looked into seem to use the same syntax, thanks heaven. Obviously this does nothing to protect you against somebody deciding to write scalalatex and use '--bin_dir=bar' instead, but I guess that’s maybe not a realistic concern to be sweating over.

But the plot thickens. To continue with my own example, the only difference between the two lualatex directives is that the first contains

% arara: --> draft:       yes,

so that the first run does not produce an output. To be completely honest, that’s a very mediocre optimization (I guess I could remove synctex the first time around too?). If I didn’t care about that, I could just define my own directive (I don’t know the syntax as I’ve never done it, but it’s pretty clearly explained in the documentation for arara) my_lualatex, which calls the lualatex directive with { options: [ '--output-directory=out', '--file-line-error' ], interaction: nonstopmode, synctex: yes, shell: yes, } and turn the arara directives in main.tex into:

% arara: my_lualatex
% arara: my_biber
% arara: my_lualatex

Obviously, if I so desire I can turn the latter into its own directive my_build, and potentially reduce the preamble of main.tex to

% arara: my_build

and from there, since arara accepts the --preamble TEXT option (see the arara doc pp. 33-34), I can just let g:vimtex_compiler_arara = { 'options' : ['-v', '--preamble my_build'] } to do the above without having a single line of arara directive in my .tex file.

This, of course, relies on the arara config file, and the way arara determines which file(s?) to use is described in Section 4.1 of the documentation. Basically there are two directories (one global and one local) and four possible file names that are checked with some priority defined. I guess latexmk does something similar, so that sort of effort might be unavoidable.

This is all to say that I don’t really think option 2. would be particularly pleasant to implement, would need maintenance for when people add new compilers or options to update the parser, and would likely keep breaking in weird corner cases and have bugs.

I don’t think option 4. is a good idea for the reasons you mention, plus the fact that it could possibly introduce security issues. Option 1. is clearly a bit ridiculous too for the reason you say, but I think maybe we’re bumping into something slightly deeper here: why should the viewer rely on options passed to the builder? I understand that when the builder can provide useful information such as the build directory it’s nice for the viewer to be able to use it, but I feel I should be able to build my project however I like (say, with :make) and still be able to use Vimtex’s viewer to benefit from e.g., backwards/forwards search. This begins to look a lot like option 3, and so I think it’s worth really thinking about what would be so inconvenient about it.

I understand that if we take this idea far enough it might represent a serious refactoring of some of what Vimtex does – since basically it would mean for the viewer to do its own thing separately from the builder and Vimtex try to fill out the missing information from the former when needed. It does, however, feel a lot more principled and less hacky than option 2.

Do bear in mind that I know basically nothing about Vimtex’s internals, so if I said something egregious above it’s not because of malice, but only because of ignorance. I don’t mean at all to waltz in and tell you how to run your project.

[LambdaP]

Something else that comes to mind that’s probably a lot less work is to deprecate some of the options to g:vimtex_compiler_latexmk and the like in favour of a dictionary g:vimtex_compiler_options. The keys that come to mind are build_dir of course, but callback and possibly hooks as well. I’m a bit confused about hooks: I don’t use it personally, and the documentation of g:vimtex_compiler_latexmk says

If running in continuous mode, each hook will be called for each line of output from latexmk, with that line as argument.

…but that sounds like a behavior that might be desired for single-shot compilation tasks as well? Similarly I’m sure one might like to use callbacks after compiling with other builders than latexmk, although maybe callbacks like these are being phased out in favour of the VimtexEventCompile* events?

Anyway we’d have something like

let g:vimtex_compiler_options = {
    \ 'build_dir' : '',
"   \ 'callback' : 1,
"   \ 'hooks' : [], 
    \}

…and this might then be written by Vimtex depending on g:vimtex_compiler_method if the user so desires, by reading latexmkrc for instance. Then the viewer can rely on g:vimtex_compiler_options to know where to find the pdf file as it’s doing already with g:vimtex_compiler_latexmk, and we can add a remark in the documentation that the build_dir option does nothing for arara but is nonetheless useful for the viewer, which is a lot less egregious than creating a whole new build_dir key for g:vimtex_compiler_arara that arara itself does not use at all.

Finally, but if I remember from somewhere else you don’t seem too keen on that, you could just add a manual option to g:vimtex_compiler_method and a corresponding g:vimtex_compiler_manual dictionary with about the same keys as the g:vimtex_compiler_latexmk one and leave the divas work out their own problems. This option offers the benefit of being able to use the VimtexEventCompile* events while keeping a finer control over the build chain, something people might want (that’s not a thought experiment, I have somewhere a makefile that compiles a front and back cover for a document from a weird, buggy, hacky template I really don’t want to integrate in the main project, and then import it into the main document with pdfpages).

[lervag]

Thanks, @LambdaP, you make a good case why my suggestion 2 is not good. Further:

but I think maybe we’re bumping into something slightly deeper here: why should the viewer rely on options passed to the builder? I understand that when the builder can provide useful information such as the build directory it’s nice for the viewer to be able to use it, but I feel I should be able to build my project however I like (say, with :make) and still be able to use Vimtex’s viewer to benefit from e.g., backwards/forwards search. This begins to look a lot like option 3, and so I think it’s worth really thinking about what would be so inconvenient about it.

This is also a valid and good point. An important note: we are not just talking about the viewer, but all compiled files (e.g. .log, .fls, .blg, ...). Many such files are for various reasons parsed by vimtex for particular purposes (quickfix, completion, detecting which packages are loaded, etc).

Do bear in mind that I know basically nothing about Vimtex’s internals, so if I said something egregious above it’s not because of malice, but only because of ignorance. I don’t mean at all to waltz in and tell you how to run your project.

Don't worry, I think you are being both polite and constructive! :)

Something else that comes to mind ...

Thanks, I like when people share ideas like this! But: We are now mixing topics, and as I don't have so much time for vimtex maintenance/development, it is quite important to keep each thread "simple" and "focused". The present threads topic is to support a custom build dir for arara, while your new comment is much more general. As such, I would be happy if you could raise it as a new issue (or several issues).

---

Let's take a step back. The compiler interface currently has a role of specifying the build dir. This partly makes sense, since it is the compiler that specifies the build dir. This works very much as expected for latexmk, in which vimtex can specify default which can be overridden by a local latexmkrc. It should also work for latexrun, although there is no way to override the vimtex option locally. And, as is clear, there is no such option for arara.

One is typically working with several different kinds of projects which may differ in things like the build dir specification. So, adding an option like g:vimtex_compiler_arara.build_dir (or g:vimtex_compiler_options.build_dir for that matter) would only work in projects that aligns to this config. Essentially, I think what is lacking is a method for specifying project local configuration. Something like: /path/to/mainfile/.vimtex.cnf. This should have a small set of project specific configuration options. Or perhaps it could be vimscript, which would allow full flexibility to customize vimtex for a particular project?

[LambdaP]

An important note: we are not just talking about the viewer, but all compiled files (e.g. .log, .fls, .blg, ...). Many such files are for various reasons parsed by vimtex for particular purposes (quickfix, completion, detecting which packages are loaded, etc).

Ah right, I did suspect that something like this might be taking place.

Thanks, I like when people share ideas like this! But: We are now mixing topics, and as I don't have so much time for vimtex maintenance/development, it is quite important to keep each thread "simple" and "focused". The present threads topic is to support a custom build dir for arara, while your new comment is much more general. As such, I would be happy if you could raise it as a new issue (or several issues).

You’re right. I got carried away a little. I’ll try to write something up.

Let's take a step back. The compiler interface currently has a role of specifying the build dir. This partly makes sense, since it is the compiler that specifies the build dir. This works very much as expected for latexmk, in which vimtex can specify default which can be overridden by a local latexmkrc. It should also work for latexrun, although there is no way to override the vimtex option locally. And, as is clear, there is no such option for arara.

One is typically working with several different kinds of projects which may differ in things like the build dir specification. So, adding an option like g:vimtex_compiler_arara.build_dir (or g:vimtex_compiler_options.build_dir for that matter) would only work in projects that aligns to this config.

I think this is a very generic problem, and a difficult one with that. I think that at least exposing something like g:vimtex_compiler_options.build_dir would mean that the user could be able to hack their way around, with a modeline or using https://github.com/embear/vim-localvimrc or what have you. That’s not ideal, but a lot better than having to choose between a build directory or a build method.

Essentially, I think what is lacking is a method for specifying project local configuration. Something like: /path/to/mainfile/.vimtex.cnf. This should have a small set of project specific configuration options. Or perhaps it could be vimscript, which would allow full flexibility to customize vimtex for a particular project?

That does not sound unreasonable, but I can see people being annoyed at having to drop yet another configuration file in their directory unless you get Vimtex to generate it for them in a semi-automatic manner. I’m also not sure the issue of the build directory by itself is worth the effort.

On another note, I checked with the maintainers of arara who don’t seem entirely hostile to having a config file specify a build directory someday. The conversation is happening over there: https://gitlab.com/islandoftex/arara/-/issues/46.

[lervag]

You’re right. I got carried away a little. I’ll try to write something up.

Great, thanks.

I think this is a very generic problem, and a difficult one with that. I think that at least _exposing_ something like g:vimtex_compiler_options.build_dir would mean that the user could be able to hack their way around, with a modeline or using https://github.com/embear/vim-localvimrc or what have you. That’s not ideal, but a lot better than having to choose between a build directory or a build method.

Hmm, yes, perhaps you're right.

Essentially, I think what is lacking is a method for specifying project local configuration. Something like: /path/to/mainfile/.vimtex.cnf. This should have a small set of project specific configuration options. Or perhaps it could be vimscript, which would allow full flexibility to customize vimtex for a particular project?

That does not sound unreasonable, but I can see people being annoyed at having to drop yet another configuration file in their directory unless you get Vimtex to generate it for them in a semi-automatic manner. I’m also not sure the issue of the build directory by itself is worth the effort.

Yes, it does require some effort. Note, though, I did not mean that you _need_ local configuration, only to allow adding it. Thus people who would be annoyed would not need to use it. Still, it might be overkill for the problem at hand!

On another note, I checked with the maintainers of arara who don’t seem entirely hostile to having a config file specify a build directory someday. The conversation is happening over there: https://gitlab.com/islandoftex/arara/-/issues/46.

Great, thanks!

---

For context, here's how vimtex currently finds the generated files:

https://github.com/lervag/vimtex/blob/513e8c916485e5f1969d8444ddef60cef945caf2/autoload/vimtex/state.vim#L761-L771

---

So, the question remains:

1. Should we add g:vimtex_compiler_arara.build_dir or not. It might also depend on the outcome of the discussion of g:vimtex_compiler_options.

2. Alternatively, should we add a mechanism to locally specify an out directory? Perhaps it oculd be an idea to use an environment variable? E.g. something like this:

   " Next check for file based on env var
   if !empty($VIMTEX_BUILD_DIR)
     let cand = self.root . '/' . $VIMTEX_BUILD_DIR
           \ . '/' . self.name . '.' . a:ext
     if a:0 > 0 || filereadable(cand)
       return fnamemodify(cand, ':p')
     endif
   endif
   

   Here we could also allow a buffer variable, e.g. b:vimtex_compiler_build_dir. Essentially, I think something small like this should "just work" and might actually solve your issue.

Personally, I still don't really like option 1, because it sort of implies that vimtex also passes the option to the compiler itself, which for arara will not be true.

[LambdaP]

2. Alternatively, should we add a mechanism to locally specify an out directory? Perhaps it oculd be an idea to use an environment variable? E.g. something like this:
   ```viml
   " Next check for file based on env var
   if !empty($VIMTEX_BUILD_DIR)
     let cand = self.root . '/' . $VIMTEX_BUILD_DIR
           \ . '/' . self.name . '.' . a:ext
     if a:0 > 0 || filereadable(cand)
       return fnamemodify(cand, ':p')
     endif
   endif
   ```

If you go see the conversation over at the arara gitlab, you’ll see that they suggest environment variables as a way to pass arguments to arara. I think this is probably the best option. It’s lightweight, tool-agnostic, and likely to be somewhat future-proof. I agree with you that g:vimtex_compiler_arara.build_dir isn’t really meaningful, and that means it’d probably be a bad move to add it.

   Here we could also allow a buffer variable, e.g. `b:vimtex_compiler_build_dir`. Essentially, I think something small like this should "just work" and might actually solve your issue.

Thinking back, I’m not sure that a _buffer_ variable is best, as you’ll typically be compiling from whatever buffer in your project. But I don’t know viml or vim’s internal machinery that well, and this may just be a non-problem.

How should we proceed from here?

[lervag]

If you go see the conversation over at the arara gitlab, you’ll see that they suggest environment variables as a way to pass arguments to arara. I think this is probably the best option. It’s lightweight, tool-agnostic, and likely to be somewhat future-proof. I agree with you that g:vimtex_compiler_arara.build_dir isn’t really meaningful, and that means it’d probably be a bad move to add it.

Great. It is a simple thing to implement, but we should think careful about the name of the environment variable. It should not be arara specific. It can be vimtex specific (not really a problem if people have duplicate env vars), but if there exists a "standard" or similar, then I don't mind adhering to it.

Thinking back, I’m not sure that a _buffer_ variable is best, as you’ll typically be compiling from whatever buffer in your project. But I don’t know viml or vim’s internal machinery that well, and this may just be a non-problem.

I agree. I think allowing an environment variable would suffice in any case.

How should we proceed from here?

Let's say I implement an option to specify the build dir with an environment variable, would that suffice to close this issue? Or have we lost some important part of the thread in our discussion?

As far as I can understand, it should now suffice that agree on the name of the environment variable. Then I'll implement it and update the docs.

There is no rush, IMHO, so it can be smart to see what the arara devs are thinking and look for possibilities for "overlapping" configuration.

[LambdaP]

Cool. What comes to mind is 1) $VIMTEX_OUTPUT_DIRECTORY, because TeX engines seem to generally use the option --output-directory, 2) $VIMTEX_BUILD_DIR, because that’s what latexmk uses, (and 3) $VIMTEX_BUILD_DIRECTORY because I like it when we write words in full :-) )

I like 1) the best, because I think TeX engines are a sounder source for nomenclature than latexmk, but 2) is more in line with what Vimtex already calls things.

I agree that it’s a good idea to try to see if some names are standard. Cargo, of Rust fame, uses for example $CARGO_TARGET_DIR. I still think that following the TeX nomenclature is better.

[lervag]

Ok, I've added support for $VIMTEX_OUTPUT_DIRECTORY. I think that is a good name for it. I've added warnings for e.g. if one should define g:vimtex_compiler_latexmk.build_dir and also defines the environment variable to a different path. I think this should work well, but please test. I'll add the docs after some testing period (let's call this a beta version for now!).

[lervag]

And there, I've added tests and docs. Please let me know what you think.

[LambdaP]

Thanks so much!

I tried and the viewer seems to be working, as far as I can tell. When compiling, however, I ran into a bug that I think is unrelated.

So I’m running :VimtexCompile which launches arara in background. If, when compilation finishes, the active buffer happens to be a tex buffer, all is well. However, when it is not the case, nvim yells at me:

Erreur détectée en traitant function <SNR>90_callback_nvim_exit :
ligne    1 :
E121: Undefined variable: b:vimtex
Appuyez sur ENTRÉE ou tapez une commande pour continuer
Erreur détectée en traitant function <SNR>90_callback_nvim_exit :
ligne    1 :
E15: Expression invalide : self.target !=# b:vimtex.tex ? self.target : ''

Sure enough:

https://github.com/lervag/vimtex/blob/e2e9dfee5d4698bbbd55da039f0e0e1a8a787d5a/autoload/vimtex/compiler/arara.vim#L178-L183

it’s trying to access b:vimtex, which doesn’t exist.

What do you suggest we do? Take this somewhere else? I’m afraid I don’t have time to work the thing down to a MWE but the cause of the problem seems rather straightforward.

[lervag]

Thanks so much!

My pleasure, and thanks for contributing with a very good discussion and good points!

I tried and the viewer seems to be working, as far as I can tell. When compiling, however, I ran into a bug that I think is unrelated.

I think it is fixed now.

[clason]

Sorry to butt in here, but I think I noticed the same issue (vimtex error when callback triggers if I'm in a non-tex buffer) with latexmk as well. Is the fix arara specific, or should I see if this fixed things on my end as well?

[clason]

(or I could just look at the fix to see that it's not araraspecific at all 😆 )

[lervag]

Sorry to butt in here, but I think I noticed the same issue (vimtex error when callback triggers if I'm in a non-tex buffer) with latexmk as well. Is the fix arara specific, or should I see if this fixed things on my end as well?

Actually, the fix _was_ arara specific. I'll check if I can see a similar issue with latexmk in the code now.

[clason]

Ah, if I check, I should also check the file name...

In this case, I'll just keep an eye out and try to make a reproducing example out of it when it happens again. No need to stay up late!

[lervag]

I've applied similar fixes now. Feel free to open a new issue if it did not fix what you've noticed.

[lervag]

I think this issue is resolved; please reopen and help me see what is lacking if I'm wrong.