Jupyter-book: Jupyter Book legacy upgrade guide

[choldgraf]

This issue is to start writing content that we can use in an upgrade guide. This will eventually make its way into a changelog or a blog post somewhere.

MOVED to the jupyter book wiki

Feel free to continue commenting and discussing here, or propose an edit to the wiki!

[choldgraf]

@martinagvilas it would be great to have your input here...what do you think I am missing? This should be a comprehensive guide since that's what the documentation is for, but what are the main pain points for migrating books?

[martinagvilas]

I think this is great! The only comments I have are:

• Most people I've met actually used the old jupyter-book through the docker image. So I would:

1. Add to the following sentence something like:

The old version of jupyter book required you to pip install jupyter book, then run a command to install Ruby and Jekyll locally. Building books required first building each page's HTML with Python, and then building the book with Jekyll (or using a docker image to run these commands).

... to make sure all users understand what it is being described.

1. This is very opinionated, but I would add to the build process a description of the local serve step. When running the docker image both the build and the local serve were executed, so it might be useful to make this step explicit in case users unfamiliar with html are expecting the same behavior just by running the build.

Maybe adding something like:

The new version of Jupyter Book uses Sphinx the whole way through. This means that installing Jupyter Book is __just__ running pip install jupyter-book, and building a book is __just__ running jupyter-book build mybook/. You can then preview your book locally by running something like {option} That's it!

I know the intention of this change log is not to give a step by step tutorial of the new jupyter-book, but this is a step that in my opinion it is not super clear yet in the documentation (see my comment in ExecutableBookProject/cli#89), and is a basic part of the jupyter-book workflow process.

If the documentation gets improved in this aspect, then maybe the text could say:

You can then preview and serve your book online following these steps

But then again, this last point is very opinionated 😆

[choldgraf]

Nice - I agree w/ both points. I'll try to find a way to work them in there

[kyleniemeyer]

Hi there, thanks for your work on this! I've been using the (now legacy) Jupyter Book for a few projects—currently building https://kyleniemeyer.github.io/computational-thermo/intro.html as I teach a class.

Today I've been trying to migrate this project over to the new version of Jupyter Book. This has mostly gone smoothly, with a few hiccups.

First, the big changes I needed to make were:

• remove _includes, _layouts, assets, scripts, _sass
• remove Gemfile
• remove old _build and _site
• moved _data/toc.yml to _toc.yml
• make some changes to _config.yml

Just after doing this, I realized that a number of the keys used in _toc.yml will actually lead to Sphinx errors, and needed to change things slightly to just use file:, title:, expand_sections: (but without true or false—just having the key seems to flag it as true), and sections:.

The only real issues I'm still running into are:

1. I get a Sphinx error if the first non-blank line in _toc.yml is not a file. (For example, it doesn't like header.)

2. It seems that the first file in the table of contents cannot have sections. If it does, the pages listed simply do not display, and when building I get an error from Sphinx: WARNING: document isn't included in any toctree for each of the files listed in sections:.

3. Although I can include static images in pure Markdown files, and those are carried over to the build, the same does not seem to work when including static images in Jupyter notebooks in Markdown cells. It looks like the build process does not copy those over for some reason; this worked on the legacy Jupyter Book.

4. This is minor, but bothering me... for some reason the copyright date is not being read from _config.yml. The first five lines are

title: Computational Thermodynamics
author: Kyle Niemeyer
email: [email protected]
copyright: '2020'
logo: images/logo/logo.png

which seems to match the examples, yet I just get "© Copyright ." on the build pages. 🤔

1. I haven't dug into this deeply yet, but trying to build a combined PDF via LaTeX seems to fail due to the inclusion of SVG files on some Markdown pages. Not totally sure how to get around that yet, though this is a lower priority.

Most of these issues are just figuring out what doesn't work on the new Jupyter Book but worked on the legacy version, though in some cases the behavior might not be desired.

[kyleniemeyer]

Ah! Minor update, after having a thought immediately after posting that last comment.

Regarding the issue of static images not displaying properly in Markdown cells of Jupyter notebooks: the issue was that I was using HTML to include the figure:

<figure>
  <center>
  <img src="../../images/image.png" alt="image" style="width: 400px;"/>
  <figcaption>Figure: caption</figcaption>
  </center>
</figure>

If I instead simply use the native Markdown image command ![](../../images/image.png), the image shows up in the build HTML page properly.

[choldgraf]

ah this is all super helpful - I'll incorporate into the guide (and also fix a few bugs that you have uncovered!)

[kyleniemeyer]

@choldgraf great! One more thing to note: it looks like the navigation object in the upper right ("on this page:") does not currently work.

However, I can still manually make intra-page links in Markdown cells, though one different behavior is that sections (e.g., # Introduction) have to be referenced using all lowercase: (Link)[#introduction]

[choldgraf]

quick responses below

One more thing to note: it looks like the navigation object in the upper right ("on this page:") does not currently work

Could you chime in on this issue re: the "on this page" bar? https://github.com/executablebooks/sphinx-book-theme/issues/80

However, I can still manually make intra-page links in Markdown cells

A more reliable way to do this now is to manually assign a section label, and then reference that label later on. You can do so like this:

(my-label)=
# My section

and later on:

here's a reference to [the section](my-label)

[kyleniemeyer]

@choldgraf quick update: I resolved the copyright issue—that was an error on my part.

One new issue I've encountered: I'm creating a separate bibliography page to hold all references; however, I have to manually touch/save that page to get the reference to appear and be linked from the page with the {cite} command.

This may be related to using execute_notebooks: cache in my _config.yml file—but with a good number of notebooks that take some time to rebuild, I'd prefer not to have to rebuild everything each time I build the book.

[kyleniemeyer]

(Sorry to pile on... just wanted to document things here)

I can't seem to get the margin or full-width tags to work with my Jupyter notebooks—the equivalents in legacy Jupyter Book worked.

[kyleniemeyer]

Oh! The hide_input tag works, in the same Jupyter notebook that margin does not seem to work. Not sure if that helps.

[choldgraf]

@kyleniemeyer this is very helpful, thanks! A few responses:

I resolved the copyright issue—that was an error on my part.

I don't think so - I patched a bug in jupyter-book that I found because of your comment 🤣

I'm creating a separate bibliography page to hold all references;

could you send a link to this? I'd love to see how it looks :-) I suspect we may still need to iron out kinks and documentation around citations.

The citations shouldn't be related to caching the book content - cache only applies to the code and code outputs. You can "clean up" your book content without deleting the execution cache by running jupyter-book clean mybook/ on your book (ife that command doesn't exist, try out the master branch!)

I can't seem to get the margin or full-width tags to work with my Jupyter notebooks—the equivalents in legacy Jupyter Book worked.

Can you provide a link to your notebook or doc? That reminds me that we should add a section about tags to the migration guide. All of the old tags "should" work, so if not it is a bug :-)

I just added a "Notebook cell tags" section to the migration guide...wanna check it out and see if that clarifies things?

[kyleniemeyer]

@choldgraf

could you send a link to this? I'd love to see how it looks :-) I suspect we may still need to iron out kinks and documentation around citations.

unfortunately I don't have the new book online yet (GitHub Pages currently hosts the content build using legacy Jupyter Book), but the source is available here: https://github.com/kyleniemeyer/computational-thermo/tree/newbackend

(If you try to build, note that there's one conda-only dependency, but the environment is specified in environment.yml.)

To get the latest version of jupyter-book, do I need to have pip install directly from GitHub? Right now I have it installed with pip install -U jupyter-book --pre.

Can you provide a link to your notebook or doc? That reminds me that we should add a section about tags to the migration guide. All of the old tags "should" work, so if not it is a bug :-)

I just added a "Notebook cell tags" section to the migration guide...wanna check it out and see if that clarifies things?

Regarding the tags, I have not tried these for Markdown files—only Jupyter notebooks, where the full_width and popout tags worked before. I've now added full-width and margin tags to these cells, but that doesn't seem to be working. The hide_input tag does work as expected though.

The citations shouldn't be related to caching the book content - cache only applies to the code and code outputs. You can "clean up" your book content without deleting the execution cache by running jupyter-book clean mybook/ on your book (ife that command doesn't exist, try out the master branch!)

Hmm... here is the workflow I'm trying:

1. Add item to .bib file, then add citation to a Jupyter notebook or Markdown file. For example, I'm adding a reference that uses the key scipylecture. Save the file.
2. Do jupyter-book build .
3. From Sphinx, I get a warning, associated with the file where I added the citation: WARNING: citation not found ...

At this point, if I view the page where I added a citation, I see just [scipylecture] without a link.

1. I tried doing another jupyter-book build ., but nothing changed.
2. At this point I did jupyter-book clean ., followed by a fresh jupyter-book build .. The output from this step included WARNING: citation not found messages for all of my citations, and now my bibliography is empty and none of the citations are working links.
3. Resaving/touching the Markdown file that contains the bibliography command, and then rebuilding, does not make the citations appear.

At this point I couldn't seem to get the references to show up at all, no matter what I did... it was only after some combination of opening and saving the bibliography page, as well as page(s) that include citations, and rebuilding multiple times, did the citations show up.

So, either I'm missing something with how to properly build the bibliography, or something is a bit buggy here...

[choldgraf]

hmmm - it looks like Sphinx isn't finding your bibtex file for some reason...

do you see an output line that looks something like:

parsing bibtex file <path-to-your-bibtex-file>... parsed 5 entries

?

[kyleniemeyer]

Yes, this is the output from my last build command, done after a clean, after which things seemed to work:

Running Sphinx v2.4.4
making output directory... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 27 source files that are out of date
updating environment: [new config] 27 added, 27 changed, 0 removed
Did not execute content/bibliography. Set jupyter_execute_notebooks to `force` to execute                                                                     
checking for /Users/niemeyek/projects/computational-thermo/book/references.bib in bibtex cache... not found
parsing bibtex file /Users/niemeyek/projects/computational-thermo/book/references.bib... parsed 6 entries
Executing: content/exergy/exergyt/exergy/exergy                                                                                                               
Executing: content/exergy/second-law-efficiency-law-efficiency                                                                                                
Executing: content/first-law/air-tankst-law/air-tank                                                                                                          
Executing: content/first-law/hydrogen-storage-tanksn-storage-tanks                                                                                            
Executing: content/first-law/portable-cooling-system-cooling-system                                                                                           
Executing: content/first-law/steam-water-equilibriumter-equilibrium                                                                                           
Executing: content/intro-solving-problemsolving-problems                                                                                                      
Did not execute content/intro-using-coolprop. Set jupyter_execute_notebooks to `force` to execute                                                             
Executing: content/mixtures/mixturesxtures/mixtures                                                                                                           
Executing: content/mixtures/multicomponent-phase-equilibriumase-equilibrium                                                                                   
Executing: content/properties-pure/properties-pureproperties-pure                                                                                             
Executing: content/properties-pure/reduced-helmholtzduced-helmholtz                                                                                           
Executing: content/second-law/heating-waterw/heating-water                                                                                                    
Executing: content/second-law/second-law-law/second-law                                                                                                       
Executing: content/second-law/solar-powered-pumpar-powered-pump                                                                                               
Executing: content/second-law/turbine-efficiencybine-efficiency                                                                                               
Executing: content/second-law/vortex-tubelaw/vortex-tube                                                                                                      
Did not execute content/setting-up-environment. Set jupyter_execute_notebooks to `force` to execute                                                           
Executing: content/stat-thermo/stat-thermormo/stat-thermo                                                                                                     
Executing: content/using-scipyent/using-scipy                                                                                                                 

looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] content/using-scipy                                                                                                                  
/Users/niemeyek/projects/computational-thermo/book/content/exergy/exergy.ipynb:10004: WARNING: citation not found: rant:1956
/Users/niemeyek/projects/computational-thermo/book/content/intro.md:7: WARNING: citation not found: KleinNellis2017
/Users/niemeyek/projects/computational-thermo/book/content/mixtures/mixtures.ipynb:120008: WARNING: citation not found: obert1960
/Users/niemeyek/projects/computational-thermo/book/content/mixtures/mixtures.ipynb:150004: WARNING: citation not found: Soave1972
/Users/niemeyek/projects/computational-thermo/book/content/properties-pure/reduced-helmholtz.ipynb:20002: WARNING: citation not found: Wagner2002
/Users/niemeyek/projects/computational-thermo/book/content/using-scipy.ipynb:20002: WARNING: citation not found: scipylecture
generating indices...  genindexdone
writing additional pages...  searchdone
copying images... [100%] _build/jupyter_execute/content/using-scipy_44_0.png                                                                                  
copying static files... ... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded, 6 warnings.

The HTML pages are in _build/html.

[kyleniemeyer]

It's strange—I do see the parsing bibtex file /Users/niemeyek/projects/computational-thermo/book/references.bib... parsed 6 entries line, but then later on also the WARNING: citation not found commands for each citation.

[choldgraf]

@kyleniemeyer could you open up a new issue about this in this repository? Sounds like a bug we should run-down and I don't want it to get lost

[choldgraf]

hey @sidneymbell - it would be great if you wanted to try out the new jupyter-book and give your feedback :-) just pinging you here since we discussed the new build system before and now it is ready for testing out!

[sidneymbell]

Hey @choldgraf! This is super awesome. I'd love to take it on a test drive later on, but at the moment I'm quite focused on the COVID data.

[gharp]

Something to be aware of in migration of citations is that the bibtex types allowed by sphinxcontrib-bibtex appear to be far more restrictive than what was allowed and supported by Jekyll-Scholar — which even provided an option to map legacy reference types to new types, I think.

I was using a custom types, like @software, @blog, etc., and I think sphinxcontrib-bibtex may only allow the basic types from the original bibtex specs, as far as I can tell. In parsing the bibliography, I kept getting errors until I figured out that I had specified a few custom types of references. reverting back to only the legacy list of types meant changing most of the custom ones to @misc, and then it all worked fine.

I'm assuming a feature request for in-line citation styles and APA, MLA, etc formats would go to sphinxcontrib-bibtex or pybtex ... but I'm not sure.

Wish I read this issue thread more fully this morning before embarking on the migration :) Not too bad though; the TOC was the trickiest thus far.

Thank you! Great work!

[choldgraf]

Ah that's definitely helpful to know. In the future hopefully we can upstream some of these things to sphinxcontrib-bibtex. Do you wanna open an issue to track this one?

[choldgraf]

closing because I think this is now done!