Jupytext: Clarify improvements / differences from nbconvert

[choldgraf]

This looks like a really interesting project! One question I've been trying to figure out is "how is this different from nbconvert?" It seems like they have partially overlapping functionality, so I'm curious how you see these two projects differentiating one another.

[mwouts]

Hello @choldgraf , thanks for asking!

As you state, jupytext and nbconvert can both export Jupyter notebooks to markdown and scripts. The added value of jupytext versus nbconvert is the support for the reverse conversion: with jupytext, you can convert a notebook to a script, edit the script, and then _update_ the original jupyter notebook.

There are two ways you can use jupytext:

• on the command line. Use the --update flag to preserve output cells for unchanged input cells.
• as a plugin for Jupyter Notebook or Jupyter Lab. See how we _pair_ a text representation to a Jupyter notebook in the demo, edit the text representation, and get the updated notebook in Jupyter

A few more comments:

• Unlike nbconvert, jupytext will preserve notebook and cell metadata in some of the text representations (R Markdown, light and percent scripts).
• Round trip conversion is robust, and was tested on real notebooks at #99
• The plugin (actually, a contents manager) for Jupyter also allows to load any Python script or Markdown document as a notebook. A potential application is the matplotlib binder which allows to browse the collection of sphinx-gallery scripts and open them as notebooks, cf https://github.com/matplotlib/matplotlib/pull/12116.

[choldgraf]

@mwouts very cool - thanks for the explanation :-)

A couple quick questions:

• How lightweight is this compared with nbconvert? At sphinx-gallery we automatically generate notebooks from the .py files that are there. We're currently doing this with custom code because we want to avoid too many heavy dependencies (and nbconvert can be heavy).
• To that point, have you considered separating out the "converting text formats from one to the other" code from the "running a paired server" code? I think one of the challenges of maintaining/growing nbconvert was that it tries to do a lot which makes it hard for others to contribute and understand the codebase.
• I looked through the Sphinx Gallery code and the formatting isn't quite right (Sphinx Gallery python files need to begin with an rST-formatted docstring). If you point me in the right direction perhaps I can give a patch!

[mwouts]

Thanks @choldgraf for the feedback! Let me try to answer your questions...

How lightweight is this compared with nbconvert?

I am not sure! Jupytext has very few dependencies. And its code is about 3k lines. Do you quantify that as light?

To that point, have you considered separating out the "converting text formats from one to the other" code from the "running a paired server" code?

Well, Jupytext only focuses on reading/writing notebooks. The contents manager (i.e. the plugin for Jupyter) is just one file (389 lines). There is no server within Jupytext.

I looked through the Sphinx Gallery code and the formatting isn't quite right (Sphinx Gallery python files need to begin with an rST-formatted docstring).

I see! You're right. The first markdown cell should be exported as a docstring. To achieve this, we would just need to add a cell metadata 'cell_marker' equal to """ for the first markdown cell of the notebook, in that function. But I am afraid the YAML header is not very Sphinx Gallery compliant... Do you see a way to have some content in the sphinx file that is ignored by Sphinx Gallery?

[mwouts]

@choldgraf , on the branch 0.8.4 the first markdown cell is now exported as a docstring. Is this better? Please also let me know if you see how to mark the YAML header so that Sphinx will ignore it.

[mwouts]

A few improvements, including the doctring for the first cell, are available in a pre-release version:

pip install jupytext==v0.8.4rc0

The YAML header will not be created if an option is set on Jupytext's content manager, cf. the documentation.

@choldgraf , I will close this issue as I think we've answered your questions here. Please let me know if you have other questions!

[choldgraf]

hey - just saw that I hadn't responded before, my bad! re: the YAML header, I don't think there's any way to get sphinx to ignore it unfortunately, so it'd have to be done with some kind of post-processing if it isn't possible to control with jupytext :-/ I'll take a stab at using this when I have a moment and maybe can make some PRs!

[mwouts]

Sure, I will be glad to receive your pull request, they are welcome!

In the 0.8.4 release candidate Jupytext does have the option to not generate the YAML header (this is documented on the README). Still I am sure you will find room for more improvements... Let me know!

[choldgraf]

It seems like you're one step ahead of me here 😊