Jupytext: How to execute a notebook created with Jupytext?

[mwouts]

When a notebook is created from a .py or .md file using Jupytext, it has no outputs.

We should document how to recreate those using either

• jupyter nbconvert --execute
• papermill

Example use cases are

• #226, contributed by @hoangthienan95 : inject parameters in a script using papermill, and run... the script !!
• #230, contributed by @psychemedia : convert .md files to Jupyter notebook using Jupytext, execute them, and render the collection of notebooks as a static site using Jupyter Book.

[mwouts]

Experiments on this subjects are available at https://github.com/mwouts/papermill_jupytext. And the discussion with the papermill team is at https://github.com/nteract/papermill/issues/365.

The less intuitive part is probably the kernel information. By default, Jupytext does not set any kernel when the file is converted, the user has to set it explicitly using the --update-metadata flag. This is not very user friendly. Do we want to set a kernel by default (e.g. the current python env for Python scripts, otherwise the first kernel matching the script language)?

Note that there are other examples of jupyter tools that requires a kernel (jupyter-book also does, cf. #230).

[psychemedia]

I think https://github.com/mwouts/jupytext/issues/119 is also relevant - a sphinx workflow.

One thing I wonder is: could I edit jupytext dualled notebooks, save rst/md or whatever sphinx feeds from, and let Github, for example, automatically generate a Sphinx site from md/rst duals saved from a Jupytext enabled notebook authoring environment?

[psychemedia]

In terms of generating publishing routes, I wonder if it would be possible to generate a form of the chapters/slides/exercises md/py used in the spacy course (see also the course-starter-python and course-starter-r).

This would then provide a route for generating the md/py feedstock required to publish an interactive course from materials authored in a Jupyter notebook environment?

[mwouts]

Nice links, thanks @psychemedia . Sure there's something to be done in this direction. Maybe we could document what you've done with Jupyter Book: turn a collection of .md files into notebooks and then use Jupyter Book to create a static site? Or maybe we could event see if Jupyter Book could open the .md file as a notebook directly?

I am also interested in the other frameworks like Sphinx or the spacy course, but maybe there editing these documents as notebooks will not help much, since the final rendering would be done using a different framework. What do you think?

[psychemedia]

Re: Jupyter Books things: yes, documenting that is on my to do list; just need to try out a couple of other workflow attempts to see if I can smooth things a bit first.

[mwouts]

@psychemedia, I have added a new --execute command to Jupytext CLI.
It works as follows:

jupytext --to ipynb --execute *.md               # create and execute all md files (in the current python env)
jupytext --set-formats md,ipynb --execute *.md   # idem, plus pairs the resulting ipynb notebook to the original md files.

@Chichilele, I know you did not recommend to provide the --execute option at #254, but I felt that piping into jupyter nbconvert with

jupytext --to ipynb --pipe-fmt ipynb --pipe 'jupyter nbconvert --stdin --stdout --to notebook --execute' script.py

(which also works now) was a bit uncomfortable!. And please be reassured, the implementation is super simple and obviously uses nbconvert, so it is able to execute in any language for which a kernel is available.