Jupytext: Hide inputs, include outputs in Markdown format

[mwouts]

The Markdown format for notebooks is a great fit for READMEs on GitHub. For this usage, I would like to be able to include (selected) outputs in the Markdown file. And from time to time I would like to hide a few input cells.

Possible implementation...

• Outputs are delimited with HTML comments. Outputs are ignored when the notebook is loaded, and updated when the notebook is saved.
• Images are created in a configurable directory (say, figure by default?), and the base name for the image is the cell name, if any.
• Code cells are hidden in a HTML comment when marked as not visible by the hide input extension.

[choldgraf]

Just a note that I bet some folks in

https://discourse.jupyter.org/t/generating-reports-for-jupyter-notebooks/279/15

would find this interesting and useful. I was also thinking about improving the story for "one notebook file -> one md file" creation. It seems like one tricky thing about turning certain code blocks / outputs / etc ON or OFF is that this makes two-way sync more difficult.

Do you think there'd be value in building in some functions in jupytext that break two-way synchronization? Otherwise as you mention, you could try keeping track of all the "hidden" pieces by using comments.

The other question is how things like interactive outputs would work in markdown. I guess those would be embedded as HTML snippets?

[mwouts]

Hi, I will soon start thinking of how to best implement this. Currently some ideas I have are listed below, please feel free to discuss those!

Global behavior

• including the outputs in the notebook has to be an explicit choice from the user. We don't want to surprise anyone with this.
• the user should be allowed to retain only certain types of outputs, either per cell or globally for the notebook. For instance, maybe he/she only wants to store the text output. This is something that is already available in nbconvert with the option `NbConvertBase.display_data_priority (only the first output type that matches the list is retained). Should we use the same option name, i.e. display_data_priority? Should we allow an user to display the multiple representation for a given output (e.g. text and HTML)?
• Reloading the .md document in Jupyter, and saving again (i.e. a round trip in Jupytext) should preserve the outputs that were stored in the .md document. Similarly, outputs that are not stored in the .md documents should be preserved in the .ipynb file.

Saving outputs

• textual output may be included in the text document itself - provided that it is not too long?
• images, html and javascript content should go to files in a sub folder (provided that they still work fine when we compile the resulting .md file to, say, HTML). If the current document is named _sample notebook.md_, then following the example of R Markdown we could store the images, etc in a sub-folder _sample notebook_files_.
• naming the files is very important. For cells that have a metadata name (has to be unique in the notebook) or title, we could use that name as a prefix. Then add -1.png for the first image, -2.png for the second one, etc, as R Markdown does.

Hiding code
I'd like to offer an option to hide one or all code cells. The code cell would obviously remain in the .md document, but in a part of the document that is commented. We should use a standard cell metadata for triggering this, e.g. the tag hide_input as in Jupyter Book.

[mattharrison]

Would love the ability to show output in markdown export (and run doctest on it). I'm authoring a book and this would make be wonderful! :)

[mwouts]

Hello everyone! I have done some research on this subjects and looked into how it would be possible to include (selected or all) outputs in the Markdown file.

I'd follow the default nbconvert's display_data_priority (probably ['html', 'application/pdf', 'svg', 'latex', 'png', 'jpg', 'jpeg', 'text']), and include just one representation of the output, and e.g. drop the text output on plots which looks like <Figure size 432x288 with 1 Axes>, unless the output is required for a visual identity of the notebook (e.g. the JSON data attached to plotly graphs may have to be preserved).

For each output type I plan to offer both _inline_ and _include_. Include would be the default for images (as this works well in VSCode, PyCharm and GitHub), and inline would be the default for text. For HTML I would prefer to go for _include_ by default, but that won't work on GitHub, so please advice! For Javascript and JSON files I'll choose _include_ by default, as neither include nor inline will work on GitHub (I think, but still have to check, that both forms work in VScode when you turn security settings off).

Now let me describe what I have in mind for each output type

Text

Outputs like

{
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }

could be represented as

```output_execute_result
2
```

similarly to jupyter nbconvert which sets class="output_text output_subarea output_execute_result" on these outputs.

{
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "using print\n",
      "using sys.stdout.write"
     ]
    }

could be represented as

```output_stdout
using print
using sys.stdout.write
```

unless we want to preserve the 'name' and 'output_type' more explicitly. Note that jupyter nbconvert sets class="output_subarea output_stream output_stdout output_text" on these outputs.

Similarly, for outputs on stderr we could use pseudo code blocks starting with ```output_stderr. Again, nbconvert sets class="output_subarea output_stream output_stderr output_text"

I also plan to allow including the output in a .html file, that would be useful especially for long outputs like logs.

Images

By default, images would use the standard Markdown inclusion like

![](notebook_name_outputs/unnamed_code_cell_5_1.png)

Jupytext would know that this is an image output just by matching the image name: if it is in the output folder (by default {notebook_name}_outputs) and if the image name matches {code_cell_name}_{output_count}.{mime_ext}.

Inline images exist in Markdown, and work in VSCode, but are not displayed by GitHub.

![](data:image/png;base64,iVBORw0KGgoAAAA...)

HTML

By default we could include the HTML inline, like here:

	0
0	4

Maybe I should filter out the style section which does not display nicely on GitHub.

I will also offer the option to _include_ the HTML file using <object>:

<object type="text/html" data="notebook_name_outputs/unnamed_code_cell_3_1.html"></object>

or <iframe>:

<iframe src="notebook_name_outputs/unnamed_code_cell_3_1.html" seamless frameborder="0"></iframe>

These kind of work in VSCode (they add an unesthetic extra box), but not on GitHub.

Javascript

By default scripts would be _included_ using <script>:

<script src="notebook_outputs/unnamed_code_cell_12_4.js"></script>

This may work in VSCode, but not on GitHub.

Metadata

I've not discussed metadata here, but we'll need to find a way to store the metadata outputs. Probably using JSON in HTML comments.

Summary

• images are super easy to include, and would work well
• text and html would be supported by GitHub only if we inject them inline, but then we get unreadable diffs again.
• an alternative is to _include_ the text and html outputs (ok with VSCode and pandoc). Which of <object> and <iframe> is the most standard?

[choldgraf]

Is there a way that, instead of creating our own folders etc, we could piggy-back on the ipynb format for including data/images/etc in the text-file representation?

E.g. if we could just treat the ipynb file as a dictionary of unique cell IDs -> outputs. Then we could reference the cell ID in the notebook and its output from within the text file? I'm not sure how that'd work with something like vscode/atom/etc, but if it were possible I think it'd be a nice way to add outputs to the text files without changing the files too much

[mwouts]

Sure, the .ipynb format is a great container. And we already do something like what I think you suggest (if I understood correctly): when the user pairs a notebook to a .md file, we use the .ipynb file for storing the outputs (and filtered metadata), and the .md file for storing inputs. Is that what you were thinking of?

This is available in Jupyter (pair a notebook to a text format, close it and reopen the text document), and also on the command line (jupytext --update). The limitation is that the outputs are associated to the input text itself, so in case the inputs are severly modified (other than code reformating with e.g. black), the corresponding ouputs are removed.

[mwouts]

Hi there! It has been some time since we last discussed this... and I am still interested!

The problem that I have with this issue is that I would like to include HTML outputs as external files for the sake of version control. Say e.g., if my notebook sample_notebook.md outputs a dataframe in a cell named main_result, the .md file should contain not the actual table, but instead _include_ the file sample_notebook_outputs/main_result_1.html where the table would be stored.

Including external HTML in Markdown files (with either iframe or object) seems to work in some editors like VS Code. However I am looking for a solution that also works on GitHub.

Does anyone know if that is possible? I am aware of https://stackoverflow.com/questions/14951321/how-to-display-html-content-in-github-readme-md, which explains that iframe and probably also object are sanitized out (so not displayed on GitHub), and also of https://github.com/github/markup/issues/346 and https://github.com/github/markup/issues/1159, so the answer seems to be negative, but maybe I am missing something? @inc0, Do you know if there is any way to embed, in a GitHub Markdown file, say, a table from an HTML file in the same repository?

[choldgraf]

I don't believe that this is possible in any kind of generic way - it's definitely not part of the commonmark spec. You can do this in some flavors of markdown - I think SSG projects like Jekyll and Hugo, or MyST support it, but not in a "works in any interface" kinda way. I'd recommend on trying this not with HTML, but instead with the JSON mimetypes that are generally output from running cells. Otherwise you'll have two different kinds of jupyter outputs - ones that work in ipynb, and ones that are HTML-only and meant for use with jupytext. That said, I think this could become complex quickly (my 2 cents is still that if people want outputs, the easiest path forward is to just use an ipynb file...)

[mwouts]

Hi @choldgraf

I don't believe that this is possible in any kind of generic way - it's definitely not part of the commonmark spec. You can do this in some flavors of markdown - I think SSG projects like Jekyll and Hugo, or MyST support it, but not in a "works in any interface" kinda way

Oh that's right! I completely agree, if we ever implement a prototype for a notebook with outputs, we should start with one of these three frameworks. Can you remind me of the links between MyST and the other two? I mean, is Jupyter Book more Jekyll-based or more Hugo-based? What happens when the MyST inputs are compiled into a website? Will that work if I manually insert an complex output (e.g. the data of a plotly plot) into the MyST file?

That makes me think that last year I did some research about what kind of outputs I could include in Hugo, and the result was pretty nice. I could get footnotes, HTML tables, interactive tables, plotly graphs and even linked Jupyter widgets working (but not math formulae). See https://github.com/mwouts/first_steps_with_hugo for the code and https://my-first-steps-with-hugo.netlify.com/ for the corresponding website.

I'd recommend on trying this not with HTML, but instead with the JSON mimetypes that are generally output from running cells.

In the Hugo test, I extracted the JSON data from the notebook output for both the widget and the plotly graph. Both used two different _Hugo shortcodes_, but maybe indeed we could think of a _shortcode_ that could display any output from a notebook (or, the equivalent for Jekyll, but maybe you have that already in Jupyter Book?).

[choldgraf]

Can you remind me of the links between MyST and the other two?

There aren't any links between them, but they do some similar things. MyST is a different kind of tool than Hugo/Jekyll in that it parses markdown into a "document representation" as opposed to just converting to HTML. For example, it can use Sphinx to handle cross-references, citations, etc. That's what Jupyter Book does (but it doesn't use Hugo/Jekyll for any of this)