Jupyter-book: [pdflatex] nested folder issue for builds with subdir structure

[mmcky]

Current issues being report by sphinx when building using --builder pdflatex

/home/circleci/project/docs/start/build.md:13: WARNING: None:any reference target not found: overview
/home/circleci/project/docs/start/publish.md:10: WARNING: None:any reference target not found: build
/home/circleci/project/docs/use/notebooks.ipynb:10007: WARNING: image file not readable: _build/jupyter_execute/notebooks_2_0.png
/home/circleci/project/docs/use/notebooks.ipynb:10007: WARNING: image file not readable: _build/jupyter_execute/notebooks_4_0.png
/home/circleci/project/docs/use/notebooks.ipynb:60002: WARNING: image file not readable: _build/jupyter_execute/notebooks_9_0.png
/home/circleci/project/docs/use/glue.md:41: WARNING: image file not readable: _build/jupyter_execute/glue_10_2.png
/home/circleci/project/docs/use/glue.md:41: WARNING: image file not readable: _build/jupyter_execute/glue_10_3.png
/home/circleci/project/docs/use/glue.md:161: WARNING: image file not readable: _build/jupyter_execute/glue_10_0.png
/home/circleci/project/docs/use/glue.md:192: WARNING: image file not readable: _build/jupyter_execute/glue_10_0.png
/home/circleci/project/docs/use/glue.md:194: WARNING: image file not readable: _build/jupyter_execute/glue_10_1.png
/home/circleci/project/docs/use/glue.md:260: WARNING: image file not readable: _build/jupyter_execute/glue_10_0.png
/home/circleci/project/docs/use/glue.md:330: WARNING: image file not readable: _build/jupyter_execute/glue_10_0.png
/home/circleci/project/docs/use/glue.md:330: WARNING: image file not readable: _build/jupyter_execute/glue_10_1.png
/home/circleci/project/docs/use/layout.md:16: WARNING: image file not readable: _build/jupyter_execute/layout_1_0.png
/home/circleci/project/docs/use/layout.md:80: WARNING: image file not readable: _build/jupyter_execute/layout_3_0.png
/home/circleci/project/docs/use/layout.md:80: WARNING: image file not readable: _build/jupyter_execute/layout_8_0.png
/home/circleci/project/docs/use/launchbuttons.ipynb:20006: WARNING: image file not readable: _build/jupyter_execute/launchbuttons_2_1.png
/home/circleci/project/docs/use/hiding.md:68: WARNING: image file not readable: _build/jupyter_execute/hiding_1_0.png
/home/circleci/project/docs/use/hiding.md:68: WARNING: image file not readable: _build/jupyter_execute/hiding_4_0.png
/home/circleci/project/docs/use/hiding.md:68: WARNING: image file not readable: _build/jupyter_execute/hiding_6_0.png
/home/circleci/project/docs/use/hiding.md:68: WARNING: image file not readable: _build/jupyter_execute/hiding_10_0.png
/home/circleci/project/docs/use/markdown.md:: WARNING: a suitable image for latex builder not found: ['image/gif'] (https://media.giphy.com/media/yoJC2A59OCZHs1LXvW/giphy.gif)

[choldgraf]

Huh, that's weird...do those files still exist?

[AakashGfude]

Looks like the images that should be created after execution by jupyter-cache not found in the jupyter-execute folder?

[mmcky]

@AakashGfude this issue pertains to this repo

https://github.com/ExecutableBookProject/cli/tree/master/docs

[AakashGfude]

The above error arises because of the non-existence of those paths. They are reading paths like _build/jupyter_execute/glue_10_2.png whereas the image is actually in _build/jupyter_execute/use/glue_10_2.png.

The problem here lies in the fact that the latex build creates one single ast comprising of the whole source folder documents, which create one single tex.
So in this case, it goes to the transform function only once with a huge ast, with env.docname variable pointing to the first file it encounters, which in this case is intro.md(which is in the root path of the source directory).
Whereas in html build where the transform function is visited individually for every document ast, the env.docname points to the correct documents that are being processed one at a time.

output_dir = sphinx_abs_dir(self.env) in line 2 of the apply function below uses the path of env.docname to construct this variable. Which in case of latex is _build\jupyter_execute for every node of this single huge ast , irrespective of the fact that the image node which is being processed might be the part of a file inside a subfolder like use.

[AakashGfude]

Not sure how to tackle this issue. Have to create output_dir variable inside the cell_output_to_nodes function individually for each node in case of latex?

[choldgraf]

I believe this is now fixed by ExecutableBookProject/cli#73

edit: ah nope, I don't think it is. @mmcky or @AakashGfude correct me if I'm wrong

[mmcky]

@choldgraf I think this issue is outstanding. The issue is when we have sub-directory structure with documents then when the inline get's done the full path for static assets get's written ahead of time so the full path is incomplete.

@AakashGfude are you able to work on this?

[choldgraf]

I think this one should supercede #501 because it means that any book with sub-directories is un-buildable as PDF, no?

[mmcky]

I agree - this is higher priority. It also breaks singlehtml builds.

[AakashGfude]

@mmcky @choldgraf have made a PR here https://github.com/jupyter/jupyter-sphinx/pull/114 for this.
About to push a test case in that repo to test this feature which should finish that up.

[mmcky]

thanks for the update @AakashGfude

[AakashGfude]

Also, @choldgraf when do you think will be the next release cycle of https://github.com/jupyter/jupyter-sphinx ? If not soon then we can meanwhile probably work with this PR https://github.com/ExecutableBookProject/MyST-NB/pull/165 ?

cc: @mmcky

EDIT: Have pushed a test case in this PR https://github.com/jupyter/jupyter-sphinx/pull/114

[choldgraf]

not sure - we can probably offer to help @akhmerov make a release

[akhmerov]

I can mint a release essentially any time.

It probably makes sense to try and merge @choldgraf's reorganization first though.

[AakashGfude]

The PR https://github.com/jupyter/jupyter-sphinx/pull/114 has now been merged. Thanks @akhmerov .
Will close this issue, once we have an update.

[choldgraf]

hey @AakashGfude just a note that jupyter-sphinx now has a new version out