Latex-workshop: subfiles: cwd for latexmk

[vogler]

For a project using subfiles and building with a .latexmkrc ($out_dir = '.build';):
~
.build/
.latexmkrc
main.tex
foo/bar.tex
foo/...
baz/...
~
where foo/bar.tex would start with
~
\documentclass[../main.tex]{subfiles}
\begin{document}
...
~
I can build both latexmk main and latexmk foo/bar yielding .build/main.pdf and .build/bar.pdf, respectively.

Building in vscode with the default latexmk (latexmkrc) recipe and selecting foo/bar.tex as a root file fails because it runs latexmk in foo/ where there is no .latexmkrc.
I first solved it by cd foo; ln -s ../.latexmkrc which then results in foo/.build/bar.pdf (built in vscode).
However with "latex-workshop.latex.outDir": "%DIR%/.build" it tries to open .build/foo/bar.pdf which of course does not exist. Again symlinking fixes it, but that's just a hack.

I then tried to define a custom recipe/tool and was missing an option to set the directory, which led me to this:
~
{
"name": "sub_latexmk",
"command": "bash",
"args": [
"-c",
"cd ..; latexmk %DOC% && cd .build && mkdir -p $(basename %DIR%) && cd $(basename %DIR%) && ln -sf ../%DOCFILE%.pdf && ln -sf ../%DOCFILE%.synctex.gz"
],
"env": {}
},
~
The cd .. is still a hack, it should go to the project root (I didn't find a variable for it) instead so that it works for arbitrarily deep paths.
This fixes building and previewing for the above subfiles setup.

Regarding a general solution:

• Introduce a cwd/path field for "latex-workshop.latex.tools" such that the bash hack is not needed.
• Isn't cwd = project_root a better default than %DIR% for spawning tools? Looks like the argument variables are absolute paths anyway.
• The preview for subfiles should look for .build/bar.pdf instead of .build/foo/bar.pdf (maybe walk up the path until in root of outDir?).

[jlelong]

This is also probably related to #1860. I will look into the whole subfiles mechanism as soon as possible.

[jlelong]

From the subfiles package documentation https://github.com/gsalzer/subfiles/blob/master/subfiles.pdf: each of the three files can be typeset individually in its respective directory, where LATEX is able to locate all included text files and images.

In particular, if you look at the examples of https://github.com/gsalzer/subfiles/tree/master/tests/hierarchy, you will see that every subfile is supposed to be compiled from its own directory. So setting cwd as the project is not compatible with the package documentation and it breaks the compilation.

• I do not know what to do for the issue on the location of .latexmkrc. I guess this is on the user.
• The remaining question, this extension has to handle is the -output-directory option. Currently, the output-directory is created at the top level of the project and we reproduce the full project hierarchy inside (in the same way as CMake does for instance)

[vogler]

If I cp .latexmkrc foo/, set $out_dir = '../.build';, and cd foo; latexmk bar I run into the problem that minted expects to find .build/bar.pyg instead of ../.build/bar.pyg because I have \usepackage[outputdir=.build,kpsewhich]{minted} such that I can build latexmk main in the root directory.
If I cd foo; ln -s ../.build it builds, but still the output will be .build/bar.pdf and the extension expects to find .build/foo/bar.pdf for the preview.

[jlelong]

My feeling is that to make everything interact properly (especially with packages outputting extra auxiliary files), we should generate foo/.build/bar.pdf instead of .build/foo/bar.pdf. Where is the line
\usepackage[outputdir=.build,kpsewhich]{minted}? Inside main.tex?

[vogler]

we should generate foo/.build/bar.pdf instead of .build/foo/bar.pdf

Okay, that works if I link ../.latexmkrc. The only thing that needs to be adjusted then is where it looks for the pdf in case of subfiles.

Where is the line \usepackage[outputdir=.build,kpsewhich]{minted}? Inside main.tex?

Yes, indirectly via \input{latex/packages}.

[jlelong]

I would rather have build at the top level with the project hierarchy duplicated inside it but I think this is not compatible with

• setting paths in the main file
• using the same .latexmkrc (by making a link)

Let me know if you think of an other approach

[vogler]

I also prefer .build at the top level.
Regarding 'project hierarchy duplicated inside': subfiles will put the pdf at the top level of outputdir in any case.
As far as I understand it, there are two scenarios that should be supported:

• Nested $root/foo/.latexmkrc:
  
  
  
  • build works with default latexmkrc recipe, produces $root/foo/.build/bar.pdf (same with cd $root/foo; latexmk bar)
  
  
  • extension expects $root/.build/foo/bar.pdf, despite "latex-workshop.latex.outDir": "%DIR%/.build" where %DIR% = root/foo -- this seems like a bug, respecting %DIR%, it should be $root/foo/.build/bar.pdf or $root/foo/.build/foo/bar.pdf
  
  
• One top-level $root/.latexmkrc:
  
  
  
  • build does not work since cwd = $root/foo, cd $root; latexmk foo/bar produces $root/.build/bar.pdf
  
  
  • extension expects $root/.build/foo/bar.pdf
  
  

Maybe a solution ($root being the workspace root):

• build:
  ~
  cwd = %DIR%
  while ! exists "$cwd/.latexmkrc" || $cwd == $root
  cwd = "$cwd/.."
  spawn command in $cwd
  ~
• preview:
  ~
  pdf = %DOCFILE%.pdf
  rel_dir = relative_to(%DIR%, $root)
  cwd = "%OUTDIR%/$rel_dir"
  while ! exists "$cwd/$pdf" || $cwd == %OUTDIR%
  cwd = "$cwd/.."
  open "$cwd/$pdf"
  ~
  Still, outDir would have to be set to the right directory which is different for the two use cases.
  Maybe also check for %DIR%/%OUTDIR% and walk up the path until found.

---

In case it might help someone until this is resolved, my workaround for the top-level .latexmkrc with \git alias root '!pwd':
~
{
"name": "sub_latexmk",
"command": "bash",
"args": [
"-c",
"r=$(git root); cd $r; latexmk %DOC% && cd .build && d=%DIR%; dr=${d#$r/}; mkdir -p $dr && cd $dr && df=$r/.build/%DOCFILE%; ln -sf $df.pdf && ln -sf $df.synctex.gz"
],
"env": {}
},
~

[jlelong]

Let us summarise the different cases/situations. I suggest to focus only on compilation issues for now as preview can easily be adapted to look for the .pdf files at the right place. Consider the following example based on https://github.com/gsalzer/subfiles/tree/master/tests/hierarchy and yours.

.latexmkrc
main.tex
  A/
     subfile1.tex
     B/
       subfile2.tex
       infile.tex

with

% main.tex
\documentclass{article}
\usepackage[outputdir=.build,kpsewhich]{minted}
\usepackage{subfiles}
\begin{document}
\subfile{A/subfile1}
\end{document}

% subfile1
\documentclass[../root.tex]{subfiles}
\begin{document}
\subfile{B/subfile2}
\end{document}

% subfile2
\documentclass[../../root.tex]{subfiles}
\begin{document}
input{infile}
\end{document}

# .latexmkrc
$pdf = 1;
$pdflatex = 'pdflatex --shell-escape %O %S';
$out_dir = '.build';

Compiling main.tex

Call latexmk from the top level directory

Compiling subfile1.tex

1. From the top level directory, it fails: LaTeX Error: Filesubfile2.tex' not found.`

2. cd A; latexmk -r ../.latexmkrc subfile1. This works but produces all outputs (.pyg, .aux, .pdf) in A/.build. Note you can also use latexmk -cd A/subfile1, which avoids the need of symlinking .latexmrkc into A/.

3. Create A/.latexmkrc containing

   # .latexmkrc
   $pdf = 1;
   $pdflatex = 'pdflatex --shell-escape %O %S';
   $out_dir = '../.build';
   

   Then, cd A; latexmk subfile1 fails with .build/subfile1.pyg: No such file or directory because of the line \usepackage[outputdir=.build,kpsewhich]{minted} in main.tex. It can only be fixed by manually tweaking that line, which is not acceptable.

Conclusion

Solution 2 is the only approach to successfully compile the subfiles. It outputs everything into A/.build but unfortunately we cannot do anything about it.

Currently, we run the recipe commands with cwd set as the dirname of the file to compile, it would probably be smarter for latexmk recipes only to set cwd to the top level directory and use the -cd flag.

If nobody complains, I will make a PR with this new approach.

[jlelong]

As reported in #1932, the -cd option can be harmful because of a bug in latexmk/makeindex. As of now, the simplest solution is to remove the -cd from the default configuration and those who really need it should create a .latexmkrc file containing $do_cd = 1;

[jmussman]

@jlelong asked me to comment on this because I missed the change in the FAQ and raised an issue as #1967. Sorry if this is a bit long, but I want to cover a lot:

First I want to say that since all I have to do is add -cd back to the latexmk options I do not have any problem with the change. But, in general principle I do not like having a tool wrapping other tools make changes on how they normally work unless there isn't another option. Wait, but -cd was added and then taken away! Well, I'll get to that in two paragraphs.

Number one, I use .latex-temp instead of .build, but that is just me, and I'll use it here.and that

Number two, if latex-workshop changes the behavior, then I expect that behavior as the default in other tools in other IDEs, and at the command line when I use them. Now this isn't exactly true for what was changed; latex-workshop added -cd so subfiles in a subdirectory were compiled in that directory. At the command line I experience the same thing when I run latexmk in the directory, but then I don't need -cd. This by the way is why I don't have a problem with the change per se, because I have to add it myself when I want to manually compile a subfile in a subfolder. But, I believe that by default latex-workshop should launch the compile in the directory where the file being compiled is, simply because that is where the file is. It should be up to the LaTex "programmer" to make it work outside of the extension, so that is why I like -cd. I'll get back to the original problem in two more paragraphs. But first, a little background.

The original conception of subfiles was to break apart the LaTex document into multiple files, which is a good idea. It's Bob Martin's Single Responsibility principle from programming. Having a big book with multiple chapters all in one file, yuck. So break it into chapters. But wait, the chapters have sections. So maybe I want to group the sections in subfolders? Yea Bob! Enter the problem: when you are compiling the whole document at the top level, you need to have files in one path, but when you are compiling just one piece in a subfolder, the path changes to get to the master which still has to be referenced, and more importantly, TO WHAT THE MASTER INCLUDES. That is what the original problem up above is.

Big problem though: if you compile everything into .latex-temp at the master level, you'll get conflicts if you are using the same subfile names in the folders, like "section-0", "section-1", etc. So @vogler discussed that up above. It could be put into a subfolder in the master .latex-temp folder, but that requires some juggling of the output directory. And, it seems that since I am just building the section in order to proof it I really would prefer .latex-temp at the subfolder level to keep everything together. I'm not following the issue that @vogler mentioned, because I don't have any problem getting latex-workshop to find the PDF in .latex-temp in the subfolder in any version that I have used.

So back to the original problem posted way up above (as I read it): how do you build at both the master and subfolder level if the paths have to be different for the build? This is what we are really trying to fix! Well this is something I had to solve long before VSCode. The trick is that you can define a variable at the master level that will not get override in the subfiles when you are compiling the master:

\providecommand{\main}{.}

This solution extends another one documented more recently (four years ago) at https://tex.stackexchange.com/questions/289450/path-of-figures-in-different-directories-with-subfile-latex, so you may want to look at that.

Then, I reference main in my includes in master, like this:

\graphicspath{{\main/_Images/}}
input{\main/workbook-titles}
input{\main/_Preambles/workbook-def}

So, what about the subfiles in the subfolders? Well, I change main in the subfiles to reference the master folder, in this case "..". Note how the master is still referenced with a .. here, because that is absolutely where the master file is going to be all the time and using \main won't work there:

\providecommand{\main}{..}
\documentclass[../workbook]{subfiles}
\graphicspath{ {\main/_Images/} }

The trick is that the OTHER STUFF, like the graphics path, and the files that the master includes, actually has to be different when you are compiling the subfile individually instead of the whole project together. But \providecommand is ignored in the subfile when the master is compiled, so the path stays ".". \main is ONLY set in the subfile when it is compiled individually, and then it causes the compiler to bump up to the master directory to get images and the other files included by master in the example above, like "input{\main/workbook-titles}".

A real-life example of this can all be found in the project https://github.com/nextstepitt/standard-course-template, which by the way I use latex-workshop for editing, but build scripts for actually putting the project together. And since it happens across MacOS, Windows, and Linux, I use different LaTeX compilers.

Thank you for your patience, and I am going to say I really appreciate all the effort put in by everybody to continue to make latex-workshop the awesome tool I need!