Jupyter-book: Tex Macros in _config.yml not being applied [html]

[mmcky]

Describe the bug

It doesn't seem that the TeX Macros when specified in _config.yml are taking affect for mathjax configuration.

To Reproduce

Build jb build docs and open docs/_build/html/advanced/sphinx.html

Expected behavior

The red \bmat should be replaced with definitions defined by the TeX macro.

Environment

• CI environment reproduces issues

Additional context

Looking to see if mathjax can support \DeclareMathOperator

[bryanwweber]

I was just going to open a new issue for this...

I think there are two causes. First, the sphinx:config values are not read from the _config.yml file by default, only the extensions are read in yaml.py:
https://github.com/executablebooks/jupyter-book/blob/7da6b865256078628affd7ef061b5e88c6f4502b/jupyter_book/yaml.py#L87

This issue can be overcome (to continue debugging) by passing the path to the config file in the --config switch because then it gets put into the confoverrides dictionary in sphinx.py: https://github.com/executablebooks/jupyter-book/blob/7da6b865256078628affd7ef061b5e88c6f4502b/jupyter_book/sphinx.py#L100-L102

However, the bigger problem is in Sphinx itself. The sphinx_config is correctly passed into the creation of the Config() object: https://github.com/sphinx-doc/sphinx/blob/72ad5f2a98c60d2b1cd856b6b9057ac5b5c85edd/sphinx/application.py#L216-L217
However, since confoverrides (the Sphinx variable, not the JB variable) are used, these values aren't initialized into the configuration until after the extensions are loaded: https://github.com/sphinx-doc/sphinx/blob/72ad5f2a98c60d2b1cd856b6b9057ac5b5c85edd/sphinx/application.py#L268
is where the values are moved from overrides into the main configuration, and
https://github.com/sphinx-doc/sphinx/blob/72ad5f2a98c60d2b1cd856b6b9057ac5b5c85edd/sphinx/application.py#L244-L245
is where extensions are set up. Actually, the problem is more than that, because only default Sphinx config values can be set in the overrides, see: https://github.com/sphinx-doc/sphinx/blob/72ad5f2a98c60d2b1cd856b6b9057ac5b5c85edd/sphinx/config.py#L261-L264

So the mathjax extension never sees the custom values of the mathjax_config variable. I'm not sure what the best way to resolve this is. It seems as though the only way to read extension configuration is directly from a conf.py file, which must be an actual file on the disk, it can't be a buffer of some sort. However, the method of specifying \newcommand... in a {math} block works, so it might be best to just update the documentation.

[chrisjsewell]

Heya, so firstly I would ask what version of myst-nb are you using?

I've just realised today that the current jupyter-book does not correctly pin myst-nb to version 0.8, which it should since I've just released v0.9 that has some breaking changes that I was planning to add to jupyter-book today.

In particular, it uses the new version of myst-parser which does some overriding of mathjax, see https://myst-parser.readthedocs.io/en/latest/using/syntax.html#mathjax-and-math-parsing

It may be that I have inhibited these tex macros there, so will look into it now...

[chrisjsewell]

But yeh I reckon I can get this solved upstream in myst-parser

[bryanwweber]

My version of myst-nb is 0.9.1 and myst-parser is 0.12.2. jupyber-book is from the tip of the master branch in this repo.

[chrisjsewell]

Ok hopefully this doesn't break anything for you, but I have just messed around with the very recent commit history, to pin myst-nb~=0.8.0, and released a jupyter-book v0.7.5. (see https://github.com/executablebooks/jupyter-book/commits/master)

Can you update your master branch, create a fresh environment, and try again. Then we shall see if it is an issue introduced in the latest myst-parser

[chrisjsewell]

Either way, I'm going to look at a test/fix for this in myst-parser now

[chrisjsewell]

Ok, so remembering how I implemented this now lol:

In the new myst-parser, with no changes to jupyter-book, you will need to add:

sphinx:
  config:
    myst_override_mathjax: false

I will make sure to document this in the myst-nb upgrade.

BUT, I think there could be a better solution than this, that I can look to implement in myst-parser.

There should be a separate myst_tex_macros configuration, which aims to work consistently across all builders, i.e. rather than having to set the Mathjax config (which is only relevant for HTML with mathjax) and adding a {math} block to enable rendering in TeX/PDF builds and/or adding LaTeX customisation.
This config would add these automagically.

I'm suprised there is nothing like this already available in Sphinx, but I don't see anything?

[chrisjsewell]

FYI, I've opened https://github.com/executablebooks/MyST-Parser/issues/230

[bryanwweber]

@chrisjsewell Thanks, I installed Jupyter Book from pip in a new virtual environment and 0.7.5 works with the definition in the config file 🎉

[bryanwweber]

However, building a single page using jb page (with 0.7.5) or jb build (with the master branch) does not apply the configuration.

[chrisjsewell]

Hey @bryanwweber you might want to check out #911 where I'm finishing up the upgrade.
The configuration looks to be working well 👍 https://deploy-preview-911--jupyter-book.netlify.app/advanced/sphinx.html#defining-tex-macros

[bryanwweber]

Thanks @chrisjsewell everything works with #911. The only small thing is that I have to provide the path to the config file when building a single page, because the config file is not adjacent to the file I'm building, but that's a separate issue/feature request. There are also two deprecation warnings from Sphinx 3.2.1 which are issued:

jb build --config orbital-mechanics/_config.yml orbital-mechanics/chapter-1/3-kinematics.md
Running Sphinx v3.2.1
/Users/bryan/.pyenv/versions/orbital/lib/python3.8/site-packages/jupyter_sphinx/__init__.py:281: RemovedInSphinx40Warning: app.add_lexer() API changed; Please give lexer class instead instance
  app.add_lexer("ipythontb", IPythonTracebackLexer())
/Users/bryan/.pyenv/versions/orbital/lib/python3.8/site-packages/jupyter_sphinx/__init__.py:282: RemovedInSphinx40Warning: app.add_lexer() API changed; Please give lexer class instead instance
  app.add_lexer("ipython", IPython3Lexer())

[chrisjsewell]

Yep thanks, thats a known (and annoying) issue that will be fixed hopefully in the next few days when jupyter-sphinx 0.3 is released

[chrisjsewell]

The only small thing is that I have to provide the path to the config file when building a single page, because the config file is not adjacent to the file I'm building

Hmm, I thought this should not be an issue, since it should look for the config in parent folders: https://github.com/executablebooks/jupyter-book/blob/06b8134af1266655717df474438bed2569b14efe/jupyter_book/commands/__init__.py#L306

Is this not happening?

[bryanwweber]

It seems not to be. The macros that I define are shown as undefined and when I specify the path to the config file, I get an extra warning about the missing logo, which implies to me that it's not trying to find the logo when I don't include the config file. I'm happy to provide more info/complete output in the morning 😊

[chrisjsewell]

I guess a good initial good improvement would be to print out some information about the paths that will be used, I'll add that now

[chrisjsewell]

Just committed a fix in #911, let me know how that works for ya?

I guess relative paths in the _config.yml will be broken, i.e. the logo, because its working from a different source path. You could think about updating these based on the relative config -> source location.
Probably leave that as a fix for another day though

[bryanwweber]

Thanks @chrisjsewell the changes in #911 fixed the config file issue 🎉