Readthedocs.org: Xelatex for PDF generation

[NoralK]

I was wondering if xelatex is supported for PDF generation? If so is the font Symobla.ttf installed as well? If not do you know when it will be supported.

Thank you

[gregmuellegger]

That's a question for @agjohnson or @ericholscher. Let's see what they say :)

[agjohnson]

Ubuntu offers the texlive-xetex package, which is installed. Past that, I don't know anything about xelatex, so I don't know if your case would be supported. You can certainly try though.

Symbola.ttf is installed via ttf-ancient-fonts-symbola, which is not installed, and isn't on or backported to the ubuntu 14.04 release. We would install this if there was a supported package, but won't install and support third party packages like this manually.

[NoralK]

Locally, I had to modify the Make.bat to give an option for xelatex and then script changes to the makefile using SED. RFTD is calling the pdflatex option and a new option would need to be created for xelatex.

For this to work I am guessing an advanced option preference would need to be added to select pdflatex (recommended) and xelatex. Based on that option there would be scripting to correct the makefile then build using xelatex.

BTW - It would also be nice to add in some "preprocessing" capabilities. I currently use SED to make changes to the "whateverthefilenameis.tex". Some changes I do before calling the makefile is sed -i s/\\RequirePackage{color}/\\RequirePackage{xcolor}/ sphinx.sty and sed -i s/]{hyperref}/,unicode]{hyperref}/ sphinx.sty - these are some changes that may be needed, not all the time required, when using xelatex.

Thank you for the help.

[NoralK]

BTW - here is my xelatex section of make.bat:

if "%1" == "xelatex" (
    %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex
    cd %BUILDDIR%/latex
    echo "Running XeLaTeX files through xelatex..."
    sed -i s/pdflatex/xelatex/ Makefile
    sed -i s/\\RequirePackage{color}/\\RequirePackage{xcolor}/ sphinx.sty
    sed -i s/]{hyperref}/,unicode]{hyperref}/ sphinx.sty

    rem WINDOWS batch files require an escape for & using ^!
    sed -i s#\\{\\textbackslash{}faWrench\s\\}\s\\^&\\\#x1F527#{\\faWrench}#g whateverthefilenameis.tex

    del sed*
    make all-pdf
    cd %~dp0
    echo.
    echo ****
    echo %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex
    echo ****
    echo "xelatex finished; the PDF files are in $(BUILDDIR)/latex."
    goto end
    )

[agjohnson]

Running xelatex would require special processing in our build environments. Our build environments disregard makefiles completely, you would probably want to implement your special processing as a sphinx extension. The changes you are describing are slightly outside the realm of RTD though.

Marking this community support if anyone wants to take this on, it's off our roadmap for now.

[skirpichev]

Running xelatex would require special processing in our build environments.

I think that no severe changes would be needed. Just a replacement pdflatex -> xelatex - will work for most cases.

@agjohnson, what about extending yml config with options to allow this?

I see following variants:

• boolean option (e.g. use_xelatex)
• string option, i.e. "command name" for pdflatex-replacement

[asr]

Blocking https://github.com/agda/agda/issues/2153.

[skirpichev]

@agjohnson, there is a patch that sits in queue for month. Apparently, this is an important issue for several projects and require minimal changes. How else I can help to get it fixed?

BTW, another possible option (other than those mentioned above) is to query conf.py for 'latex_engine' value. Let me know if you prefer this.

[ricpelo]

It seems that the latex_engine option doesn't work because readthedocs uses sphinx-build -b latex and then pdflatex over the .tex files that sphinx-build generates.

[skirpichev]

It seems that the latex_engine option

Yes, because readthedocs doesn't use it now, which I suggest.

and then pdflatex

To select correct binary, we can query 'latex_engine' option.

[agjohnson]

@skirpichev it seems the purposed fix would be to simply just replace the binary executed with the binary name from the readthedocs-build config module? If there is feature parity there, this sounds fine, however I'd want to note that we only officially support pdflatex in the docs. A PR should update our docs to describe this change.

[skirpichev]

@skirpichev it seems the purposed fix would be to simply just replace the binary executed with the binary name from

Yes. But after "from" - a number of options.

1. We can configure binary name in readthedocs-build
   a. string option
   b. boolean option (like "use_xelatex")
2. We can configure binary name from the projects conf.py option "latex_engine".

I'd want to note that we only officially support pdflatex in the docs.

But can we be sure that xelatex doesn't disappear from the environment in future?

[cyberorg]

https://github.com/rtfd/readthedocs.org/issues/2620

Sphinx allows the use of following:

make PDFLATEX=xelatex -e SPHINXOPTS="-D language='gu'" latexpdf

All we need on readthedocs is the way to set PDFLATEX=xelatex

[humitos]

To add more info to this issue, I did this stupid change in my local instance and it built the PDF from #2620 (https://readthedocs.org/projects/mss-doc-gu)

diff --git a/readthedocs/doc_builder/backends/sphinx.py b/readthedocs/doc_builder/backends/sphinx.py
index c6bd529e..45ad7e32 100644
--- a/readthedocs/doc_builder/backends/sphinx.py
+++ b/readthedocs/doc_builder/backends/sphinx.py
@@ -292,7 +292,7 @@ class PdfBuilder(BaseSphinx):

         # Run LaTeX -> PDF conversions
         pdflatex_cmds = [
-            ['pdflatex',
+            ['xelatex',  # 'pdflatex',
                 '-interaction=nonstopmode',
                 tex_file]
             for tex_file in tex_files]

Besides, I uncommented the lines from conf.py regarding latex_elements. So, giving an _experimental_ way of changing the pdflatex command to xelatex to the user could be a way to build, at least, a couple of builds.

The result, that wasn't linked in the _Downloads_ section of the project for some reason, but I opened it from user_builds/h-gu/artifacts/xelatex/sphinx_pdf/h-gu.pdf

[skirpichev]

@humitos, please see https://github.com/rtfd/readthedocs-build/pull/18

The result, that wasn't linked in the Downloads section of the project for some reason

I don't know you local settings, but perhaps that could be https://github.com/rtfd/readthedocs.org/issues/2582 (which also has patch, awaiting for review).

[agjohnson]

@humitos good point. Seems we'll need more logic on the RTD side than just swapping out the binary used. The download path is probably because of some difference in the output, and may take a patch to fix the sphinx builder in RTD.

[skirpichev]

@agjohnson, what about conf.py option? Should we add something to yml config or just utilize 'latex_engine'?

Correct me, but rtd now don't parse & query for any conf.py settings, right?

[chilibreaker]

This is my way to support chinese build pdf locally:

1.vim doc_builder/backends/sphinx.py
pdflatex_cmds = [
['xelatex',
'-interaction=nonstopmode',
tex_file]
for tex_file in tex_files]

2.update xetex to support xelatex

apt-get update texlive-xetex

3.upload "simhei.ttf,simsun.ttf"(can be found in C:\Windows\Fonts on windows) to system path /usr/share/fonts/truetype/

4.flush the fc-cache

fc-cache -fv

5.add "latex_elements" to the conf.py in your docs repo :

    latex_elements = {
    # The paper size ('letterpaper' or 'a4paper').
    'papersize': 'a4paper',

    # The font size ('10pt', '11pt' or '12pt').
    'pointsize': '12pt',

    'label': '\\usepackage[english]{babel}',

    # Additional stuff for the LaTeX preamble.
    'preamble': '''
    \usepackage{xeCJK}
    \usepackage{indentfirst}
    \setlength{\parindent}{2em}
    \setCJKmainfont[BoldFont=SimHei, ItalicFont=SimSun]{SimSun}
    \setCJKmonofont[Scale=0.9]{SimSun}
    \setCJKfamilyfont{song}[BoldFont=SimSun]{SimSun}
    \setCJKfamilyfont{sf}[BoldFont=SimSun]{SimSun}
    \XeTeXlinebreaklocale "zh"
    \XeTeXlinebreakskip = 0pt plus 1pt
    '''
    }

Then you can build chinese pdf and this works for me.

[agjohnson]

So i believe the only missing piece is maybe the Symbola font family? Does rtfd/readthedocs-docker-images#61 address this correctly? We already install texlive-xetex, so i believe our latest image should support this. If our 2.0 image doesn't, our docs describe how to set build.image in our readthedocs.yml.

I'm not sure which parts of the latex_elements blob above are stylistic and which are required, but this would be a great blob to have in our docs. We have some similar content under /guides for now, but we really new a strong section on translations and language support as well.

[skirpichev]

So i believe the only missing piece is maybe the Symbola font family?

@agjohnson, so rtfd could be configured to run xelatex instead of hardwired pdflatex?

[stevepiercy]

FWIW, my local build of the Pyramid docs in PDF works, but it fails on RTD. There might be some useful configuration items in my PR that got it to work locally. I would really like a configurable option to run xelatex.

[stsewd]

@stevepiercy maybe this would help https://github.com/rtfd/readthedocs.org/issues/4021?

[stevepiercy]

@stsewd heh, yeah, I subscribed to that issue last night. 😉

I am looking for a way to resolve my failed PDF build:

pdflatex -interaction=nonstopmode /home/docs/checkouts/readthedocs.org/user_builds/pyramid/checkouts/master/docs/_build/latex/pyramid.tex
# <snip>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Fatal fontspec error: "cannot-use-pdftex"
! 
! The fontspec package requires either XeTeX or LuaTeX.
! 
! You must change your typesetting engine to, e.g., "xelatex" or
! "lualatex"instead of plain "latex" or "pdflatex".
! 
! See the fontspec documentation for further information.
! 
! For immediate help type H <return>.
!...............................................  

/me raises hand to be a guinea pig! 🖐

[stsewd]

I'll add this as something to consider for the next version of the yaml file https://github.com/rtfd/readthedocs.org/pull/4084

[agjohnson]

A RTD option is not needed to use xelatex, this is a Sphinx option. See http://www.sphinx-doc.org/en/master/usage/configuration.html#confval-latex_engine

As long as the engine is installed, it should just build with the specified engine. If there is something else that we need for fonts or other latex magic, we'll likely need to make a change to our build image.

Edit: nevermind, skimmed this harder. I'm -1 on making an RTD option that just mimics a Sphinx configuration option. Perhaps we can alter our commands to be more generic instead.

[skirpichev]

As long as the engine is installed, it should just build with the specified engine

@agjohnson, how it possible given rtfd coming with hardcoded pdflatex command?

[stsewd]

Maybe what @agjohnson mean is that rtd respecting that configuration is a new feature? (in favor of adding a new option or #4021)

[stevepiercy]

Could I get some clarity on what is the proposed direction to resolve this issue? I'm eager to help resolve this issue prior to Pyramid 1.10's looming release. Just tell me what would help, and I'll do my best to do it. Otherwise I'll have to revert changes and drop support for characters in tree diagrams that come only with Unicode support, and that would make me sad. Thank you!

[humitos]

@stevepiercy I read the whole thread again (including PRs and issues linked) and it's not clear what is the direction to take.

I'd say that the best approach is to respect the latex_engine option from Sphinx's conf.py file (http://www.sphinx-doc.org/en/master/usage/configuration.html#confval-latex_engine) making pdflatex the default if it's not defined.

Although, this makes a little of noise to me because:

1. @agjohnson has said that we can "guess" which binary to use in https://github.com/rtfd/readthedocs.org/issues/4021
2. we are not parsing/importing/using conf.py at all in our source code --so, we should implement this and I'm not sure that we want to do that :/

I'd be :+1: on relying on the user decision (as long as we have installed all the binaries in our Docker image).

[stevepiercy]

In #4021 the proposed guessing algorithm is based on the specified language of the documentation, which would skip English docs. I much prefer a user setting.

I can't be the only English language docs maintainer that wants this. I imagine that it would also be useful mathematics and scientific symbols (nevermind that math and science appear to be foreign languages in America these days 😛 ).

[RichardLitt]

I think a user setting would make more sense, too. However, I don't know how much more effort it would take to implement this.

[skirpichev]

In #4021 the proposed guessing algorithm

Just not guessing, please. Unless this algorithm will specify defaults for some option(s), that will be also configurable by user (via rtfd.yml or conf.py).

I think a user setting would make more sense, too.

I would suggest approach in https://github.com/rtfd/readthedocs-build/pull/18 (but, perhaps, with better naming for option). With another simple few-liner PR for rtfd.org - it will solve really annoying problem, which kills PDF builds for many projects. Practicality beats purity.

Parsing conf.py file would be a right approach, if you could suggest right now several other use-cases for this functional. Any other option you would like to take from sphinx config?

[stevepiercy]

ping @stsewd @agjohnson. I'm available to serve as a test subject for resolving this issue. I appreciate that the migration of RTD to Azure was successful, and that us Merkins got a 3-day weekend coming up. Please let me know what, if anything, I can do to help.

[stsewd]

I think the main decision to take here is from where to get the type of pdf engine, I'm -1 on guessing based on the language, as users like @stevepiercy want to use it in english docs. I'd say we can have a feature flag to change the engine to xelatex in the meantime, and see if this works without any major changes. If so, I think we can move to parse the conf.py file or whatever makes more sense. Thanks @stevepiercy btw.

[humitos]

I agree with @stsewd on these:

• -1 on guessing: this is always a headache :)
• I think a quick implementation for this feature is a Feature flag for the project that want this
  
  
  
  • the problem with this is that a Feature flag is a boolean decision, and there are more than 2 options for latex_engine (4 actually): http://www.sphinx-doc.org/en/master/usage/configuration.html#confval-latex_engine
  
  
• I suppose we should start thinking on writing the code that read in a sanity way configs/options/settings from standard Sphinx's conf.py and extends our .yaml configuration file. This will be useful for this particular case, but also in the future to read more Sphinx's configs without too much effort.

[stsewd]

Just adding that 3 of those engines are included in the texlive-full package https://packages.debian.org/sid/texlive-full (Not, sure about pLatext, it is the same as Plain Latex?)

[stsewd]

To parse the conf.py file, the file needs to be executed, I'm not sure if users want to execute the file twice (or maybe it doesn't matter).

Sphinx docs

The configuration file is executed as Python code at build time (using execfile(), and with the current directory set to its containing directory), and therefore can execute arbitrarily complex code. Sphinx then reads simple names from the file’s namespace as its configuration.

We can use https://github.com/sphinx-doc/sphinx/blob/bc02abcb77143d12c08265c4c10ba9c6cd003832/sphinx/config.py#L183 to read the file from the rtd application.

[stevepiercy]

(Not, sure about pLatext, it is the same as Plain Latex?)

I think they are different, based on my parsing of this post. platex allows for vertical typesetting for languages including Japanese.
https://tex.stackexchange.com/questions/397182/what-is-the-difference-between-platex-and-pdflatex

Here's the repo for platex.
https://github.com/texjporg/platex

It would be good to solicit information from someone who can read and write Japanese to advise, perhaps from the Sphinx project or platex?

[stevepiercy]

With the looming release of Pyramid 1.10a, we must revert to using only ASCII characters in its PDFs. I will be going to Tokyo for PloneConf 2018, and perhaps some of developers there can assist with this issue. Please let me know how I can help.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[skirpichev]

Wow, an original method of fixing issues.

[asr]

Indeed!

[stsewd]

This is still valid, but we haven't had the time to check in to this yet.

[skirpichev]

we haven't had the time to check in

(You had a working PR (or PRs) year ago (or more?), but prefer the Code of Conduct, instead of code...)

[humitos]

We can use sphinx-doc/sphinx:sphinx/config.py@bc02abc#L183 to read the file from the rtd application.

This will be a problem.

If the user is executing code that writes a file, calculate something or whatever this will be executed twice and, besides it's a waste of resources, it could bring other issues.

[agjohnson]

Feature flag is a good direction temporarily. How we arrive at a binary to use will probably end up being a config file change, as it seems we can't automate this before we execute the conf.py. We can't rely on the makefile, but there might be a third option such as using a custom RTD builder that is a wrapper for pdflatex/xelatex depending on the conf. Keeping this out of our config would be helpful.

Either way PRs welcome here. Our code of conduct is not going to be removed, please stop implying we should remove it.

[skirpichev]

please stop implying we should remove it.

Sorry, unsubscribed.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stevepiercy]

Please remove the status: stale label.

There is some relevant discussion taking place in #4454 about how to avoid a makefile.

[skirpichev]

@ericholscher, you closed this issue.

Unless I misunderstood something, new feature (from #5437) is not working on the https://readthedocs.org/, it may be enabled only for custom installations.

If so, I don't think the original issue was solved. Or this is closed as "will not fix" and the https://readthedocs.org/ will never support xelatex?

[stevepiercy]

I followed this issue in hopes that it would resolve https://github.com/Pylons/pyramid/issues/3290. Although #5437 was merged, this issue was not resolved, unless I have missed some configuration or other user-configurable option. Should I open a new issue or is there another that I should follow?

[stsewd]

Please see https://github.com/rtfd/readthedocs.org/pull/5453

[skirpichev]

@stsewd, it seems - it's impossible to use xelatex on the readthedocs.org, I don't think your reference to #5453 could help. For example, I set latex_engine="xelatex" in conf.py, but rtfd still uses pdflatex: see failed build.

[humitos]

The issue was closed automatically because the PR was merged.

If you want to test/try this new feature in your project, please open a new issue asking to enable USE_PDF_LATEXMK (https://docs.readthedocs.io/en/latest/guides/feature-flags.html)

Thanks!

[skirpichev]

If you want to test/try this new feature in your project, please open a new issue asking to enable

Ok, thank you for the explanation, see #5494.