Latex-workshop: Proposal - make synctex command configurable

[harmishhk]

Will it be possible to make synctex command configurable from the settings file?

I am trying to run all LaTeX commands within docker containers. I am able to run both toolchain and chktex commands, for example, with these settings:
"latex-workshop.linter_command_active_file": [ "sh", "-c", "docker run --rm -i -v $(pwd):$(pwd) -w $(pwd) harmish/latex chktex -wall -n22 -n30 -e16 -q -I0 -f%f:%l:%c:%d:%k:%n:%m\n" ], "latex-workshop.linter_command_root_file": [ "sh", "-c", "docker run --rm -i -v $(pwd):$(pwd) -w $(pwd) harmish/latex chktex -wall -n22 -n30 -e16 -q -f%f:%l:%c:%d:%k:%n:%m %DOC%.tex\n" ], "latex-workshop.toolchain": [ "docker run --rm -t -v $(pwd):$(pwd) -w $(pwd) harmish/latex latexmk -synctex=1 -interaction=nonstopmode -file-line-error -pdf %DOC%.tex"

Same way, it would be great if I can invoke the synctex command also from within a container.

I quickly checked by replacing synctex with docker run --rm -i -v $(pwd):$(pwd) -w $(pwd) harmish/latex synctex in lines 66 and 78, which works fine.

And thank you, the authors, for creating this nice and powerful extension.

[jabooth]

Hey @harmishhk,

wow that's a wild use case!

It's funny actually, as I was thinking of deprecating these options to fully customise the commands that are ran. The trade off here is that some parts of these commands have to be retained for the extension to work, e.g. for chktex:

-f%f:%l:%c:%d:%k:%n:%m

Is the exact format our parser expects. Now you have customised the command in your local config which is great, but what if in 6 months time we realise we can get more info out of chktex if we use a different set of flags? Now you have hardcoded the 'old' config setting and of course things are going to break.

Saying that, I hadn't considered that some users would be using these settings in such interesting ways, so it's a trade off between power and reliability.

At least for now I have no issue with making the synctex command customisable - as long as @James-Yu has no objections I would happily accept a PR if you fancy having a go @harmishhk?

Should be fairly straightforward to mirror how we name/access the config settings for chktex for instance (i.e. do a list of strings rather than a single string with whitespace sep args).

name the setting latex-workshop.synctex_command.

If you aren't up to that no worries we can leave this open and I will try and get around to it some time.

[James-Yu]

Easy fix!

[James-Yu]

Looks good. Will PR.

[harmishhk]

Wow, that was quick, thanks.

Now you have hardcoded the 'old' config setting and of course things are going to break.

Yes, I see these configuration options could make problems in the future. In fact, when using chktex with docker, as I'm doing with my configurations, I only see one warning for chktex. Once I correct that warning the next one appears in the 'PROBLEMS` section of vscode. But I need to check carefully what I'm doing wrong, or how these configuration strings are passed on to the docker container. But, this is for later, may be with filing another issue or PR :)

it's a trade off between power and reliability

What about removing the detailed configuration options but just keeping an option to specify chktex executable? Similarly to what @James-Yu have done with synctex, something like latex-workshop.chktex_command.

[jabooth]

Wow, that was quick, thanks.

Yup, blink and you will miss @James-Yu 😉 the man is non stop!

when using chktex with docker, as I'm doing with my configurations, I only see one warning for chktex

That will be related to bash escaping the newline character at end of the -f flag. It's important - it actually separates out each problem. If it doesn't exist all the problems come on one line and our parser will only catch the first.

I had thought about using a different sentinel character, but then you have all the worries of it maybe occurring else in the line...

Let me know if you figure out how to reliably solve this problem!

On another question @harmishhk - why are you going with dockerized latex approach? Is the main appeal that you don't have to wrestle with installing texlive on your machine?

I'm asking as I'm working on an alternative approach for 'bundling' a minimal latex toolchain with the extension, so we can provide a consistent latex experience and users won't need to have anything else installed...just wondering if that would appeal to you or if you have a different goal in mind!

[James-Yu]

tbh I have some idea on revamping the command-related configurations. However that may break the backward compatibility. I am always hesitated when doing so.

imo commands are composed of three parts: command+optional parameters+required parameters. The first two components can be user configurable while the third one is hard-coded, e.g., -f%f:%l:%c:%d:%k:%n:%m. But that is definitely incompatible with existing settings. In fact, there are two different patterns to define commands now, i.e., toolchain where command arguments are listed in one string and chktex where arguments are in an array. Either one has its own advantage.

By the way, I often seen others using docker doing things. @harmishhk I am also quite curious on your reason of using that. Purely interest 😆

[jabooth]

tbh I have some idea on revamping the command-related configurations.

imo commands are composed of three parts: command+optional parameters+required parameters

Yeah this is very similar to what I was thinking too! I also think a PR to make this all consistent would be good for the long-term health of the extension, but it is a breaking change which is annoying.

Saying that, the kind of issue I mentioned above (where users are hard coding required parameters every time they want to customise) is getting worse every day as more of our 21,000 (!) users have the need to customize something. That means the longer we leave this the more users will be inconvenienced by the changes. So maybe it's something we should do sooner rather than later?

In fact, there are two different patterns to define commands now, i.e., toolchain where command arguments are listed in one string and chktex where arguments are in an array. Either one has its own advantage.

Perhaps we should make an issue to discuss the overall structure of the settings for latex workshop and decide:

1. how to standardise things (for instance, decide which way we will do subprocess calls - strings or arrays of strings)
2. how to handle the fact that this would be a breaking change

If we feel that we come to good solutions to 1 and 2, and if we feel the upside is worth the disruption maybe we could maybe consider making the breaking change? Hopefully it would be a one off, and would mean we would have a strong convention for going forwards without more breaking changes?

[James-Yu]

Great! Will work on that.

What I suppose is that the API should support reading configs even not registered in package.json. If so, we can pop red alerts when the global/workspace config has obsolete config keys and encourage users to adopt the new pattern. I'll have a short AFK and return with some more things!

[jabooth]

OK mate sounds good, look forward to seeing what you come up with!

[harmishhk]

Regarding containers:

Yes, the main appeal of docker for me is that I don't have to wrestle with installation of texlive again and again, and it keeps my system clean :). It is quite fast to just download the latex image from docker hub. Additionally, it works "seamlessly" on Linux or Windows (or for that matter whatever OS docker supports) and the overhead is negligible.

It's also probably my obsession nowadays that I'm using-docker-for-whatever-I-do, because of problems I had with working with different versions of some software and different OSs. For example, I work in robotics and use library called ROS , different version of ROS required different Ubuntu versions and I need to test my algorithms on different robots that are working different versions as well. For LaTeX, I use docker also because if it compiles once, it will always compile.

latexjs looks a credible undertaking. That would eliminate me from installing LaTeX myself. Only reservation I would have is when I suddenly need some new packages but I'm working offline. That can be solved by providing an option in when using latexjs to download full version of texlive before going offline. In fact, I am just installing texlive-full inside the docker image I'm using.

Regarding proposal on structure of settings:

Sounds good. I like when extensions provide more configuration options, makes it easy to try out things like using docker containers.

[jabooth]

@harmishhk thanks for the feedback, useful. Yeah definitely have similar (positive) feelings towards docker. it's such a minimal overhead and the rock solid guarantee that it will just work is awesome.

It feels to me like we may see a move to containers being used liberally in OS's in place of apps now, basically taking the sandboxing of apps to the next level...

anyway, that's a ways off, and the nice thing about JS is it's already a fairly consistent runtime that's available everywhere (very handily inside any electron-based editor like vscode). If we can get the performance within 2-3x of native it should be a nice solution - currently 8x slower than native for me, but I've done little to optimise.

Only reservation I would have is when I suddenly need some new packages but I'm working offline.

Yeah that's fair, as you said the on-demand aspect of it is optional and we could let users download the whole thing if they aren't bothered about disk space/network.

[harmishhk]

Let me know if you figure out how to reliably solve this problem!

@jabooth it was very simple, just had to put double prime around -f%f:%l:%c:%d:%k:%n:%m\n, that is \"-f%f:%l:%c:%d:%k:%n:%m\n\"

[jabooth]

@harmishhk! Nice, thanks! Glad you've got it working.

[jabooth]

@harmishhk just to let you know in #89 we decided to bite the bullet and move to hopefully a more sensible set of configuration options for most, but it will mean you need to do a little more to adapt for your Docker workflow.

Now we require that you specify an executable to run along with the args that should be passed to it for all commands seperately. We also reserve the right to inject additional required args (e.g. the format string for chktex) internally, so we can change these things without breaking things for people with custom configs all the time.

Shouldn't be too hard to adapt, but let us know if you have issues.

[harmishhk]

@jabooth thank you for the information. The new way of defining configurations seems better. However, my docker related configurations had problems with this solution. It is combination of how spawn of child_process works and how configuration strings are passed on to docker. But, I have found a better way of defining LaTeX commands with docker. Rather then playing with configurations, I just added a small executable script latexmk in my PATH, for Linux it would be like:

#!/bin/bash
docker run --rm -i -v $(pwd):$(pwd) -w $(pwd) harmish/latex latexmk "$@"

With this I do not need to change configuration in LaTeX-Workshop at all, and this should work regardless of using spawn or exec and whatever tool I'm using to call latexmk. Similar script for chktex also works fine :).

[mathematicalmichael]

@jabooth thank you for the information. The new way of defining configurations seems better. However, my docker related configurations had problems with this solution. It is combination of how spawn of child_process works and how configuration strings are passed on to docker. But, I have found a better way of defining LaTeX commands with docker. Rather then playing with configurations, I just added a small executable script latexmk in my PATH, for Linux it would be like:

#!/bin/bash
docker run --rm -i -v $(pwd):$(pwd) -w $(pwd) harmish/latex latexmk "$@"

With this I do not need to change configuration in LaTeX-Workshop at all, and this should work regardless of using spawn or exec and whatever tool I'm using to call latexmk. Similar script for chktex also works fine :).

thanks so much for this. I'm trying to do exactly what you set out to do (soooo much easier than managing miktex). I was mounting my volume in to /tmp, and not using -i. I'm trying to use Atom for my editor, and this change allowed it to run, and open the right file. Didn't have to mess with settings at all, simply create the right executables in my PATH.

My use case: thesis in applied mathematics. I want _every_ last aspect of it completely dockerized for total compatibility, as I'm demonstrating a case study in software reproducibility (novel methods are nothing without good implementations).