Jupytext: Hide metadata header in markdown

[sammorley-short]

Is it possible, or would it be possible to hide the metedata header added when reading and writing to markdown?

For example, this is the header on one markdown document:

---
jupyter:
  jupytext:
    text_representation:
      extension: .md
      format_name: markdown
      format_version: '1.2'
      jupytext_version: 1.4.2
  kernelspec:
    display_name: Python 3
    language: python
    name: python3
---

Instead, it would be great if this could be put in an inline comment so that it isn't visible when the document is rendered outside of jupyter.

(P.s. thank you very much for your work on this, it is a really excellent plugin you've built here 😄)

[mwouts]

Hello, thank you @sammorley-short !

Yes that's a great idea - we were discussing something like that yesterday at #517. We were thinking of adding a new option, possibly comment_notebook_metadata, to embed the YAML metadata in an HTML comment, so that I will not be rendered on other viewers. Do you think it will fit your own use case?

Note that, with the current version of Jupytext, you can make the following choice: include, or not, the notebook metadata. If you want to give that a try, uncheck 'Include metadata' in either the Jupytext Menu (Jupyter Notebook) or the Jupyter commands (Jupyter Labs), or execute

jupytext --update-metadata '{"jupytext": {"notebook_metadata_filter":"-all"}}' notebook.ipynb

[sammorley-short]

My use-case here is that we are actually deprecating the use of jupyter notebooks altogether by converting any we have to markdown. We do this by initially pairing any notebooks that currently exists, then unpairing them and finally deleting the old notebooks. That way, the markdown versions can be version controlled as well as rendered in GUIs such as GitLab, but anyone who wants to run them as notebooks can simply use the jupytext extension to edit and run them as notebooks.

Given this workflow, would removing the notebook metadata work? I assumed that this was needed to distinguish between a "jupytext" markdown file and a standard one. If this is not the case, then presumably your solution should work perfectly 🙂.

With regards to the way of hiding the metadata, I think the HTML solution would work fine. But out of interest, why would one not simply encapsulate it in a markdown comment?

[mwouts]

We do this by initially pairing any notebooks that currently exists, then unpairing them and finally deleting the old notebooks.

Impressive! Yes that is an option now, especially if you don't need the outputs in GitHub/GitLab.
By the way, if you want to do that on the command line, you can give a try to

jupytext --to md *.ipynb

(and if you want to rewrite the full history, have a look at https://jupytext.readthedocs.io/en/latest/faq.html#can-i-re-write-my-git-history-to-use-text-files-instead-of-notebooks )

The header is not required, in the sense that .md files with no header are still rendered as notebooks in Jupyter. But I think it's better to keep it when possible, as it contains 1) the kernel information and 2) the jupytext version information, which may be useful to ensure forward compatibility (and 3), indeed some programs like e.g. Jupyter Book use that to distinguish between plain Markdown and Jupytext Markdown files)

With regards to the way of hiding the metadata, I think the HTML solution would work fine. But out of interest, why would one not simply encapsulate it in a markdown comment?

By HTML comment I mean something like the below. Maybe we are using different names for the same thing? Otherwise I'd be curious to know how you do markdown comments!

<!--

---
jupyter:
  jupytext:
    text_representation:
      extension: .md
      format_name: markdown
      format_version: '1.2'
      jupytext_version: 1.4.2
  kernelspec:
    display_name: Python 3
    language: python
    name: python3
---

-->

[sammorley-short]

Ah, apologies. I am not an HTML programmer I had assumed that "HTML comment" meant something encapsulated by HTML tags. But the above comment format would be ideal! Nice!

[inc0]

@sammorley-short https://discourse.jupyter.org/t/jupyter-and-github-alternative-file-formant/4972/31 in this thread, we're discussing ability to plug format like yours to backend of jupyter and make jupyter itself save and load notebooks from text-readable files. Would that work for your use case?

I've made very similar prototype to yours, but I'm wondering if MyST format isn't better in this case as it has yaml blocks in the syntax rather than hacks with html comments, but I understand that this can be rendered with existing tools.

[mwouts]

@choldgraf , @chrisjsewell , I plan to address this issue soon. Starting from Jupytext 1.6.0, the metadata header in Jupytext Markdown notebooks will look like this:

<!--
jupyter:
  jupytext:
    text_representation:
      extension: .md
      format_name: markdown
      format_version: '1.2'
      jupytext_version: 1.4.2
  kernelspec:
    display_name: Python 3
    language: python
    name: python3
-->

and thus the metadata will be hidden on GitHub and also when the .md file is rendered as HTML.

Notebooks that were created with Jupytext < 1.6.0, when opened with Jupytext >= 1.6.0, would get an option hide_notebook_metadata: False to preserve the visible YAML header and interoperability with the previous versions of Jupytext.

Now my question for you, @choldgraf and @chrisjsewell: would you like me to do the same for the MyST-Markdown format?

[chrisjsewell]

eurgh, my intuitive sense is this may break myst 😕

Is there reasons you are doing this other than the discussion in this issue?
YAML front matter in Markdown is pretty well established and this seems to go against the grain of most Markdown parsers/renderers, e.g. Github already has a pretty nice representation of them (i.e. via GitHub flavoured MD) and VS Code previews (which uses markdown-it) already hides them?

[choldgraf]

yeah my 2 cents is actually that commenting out the top-level metadata in a markdown file is not a great idea. It's a standard practice across many different communities and technologies. I think a better approach would be to have the default as "shown" and let jupytext users flag that they want it hidden, rather than the reverse.

[mwouts]

I think a better approach would be to have the default as "shown" and let jupytext users flag that they want it hidden, rather than the reverse.

Thanks @choldgraf , that is probably much wiser than what I described in my previous comment :smile: I'll do this way, then.

Github already has a pretty nice representation of them and VS Code previews (which uses markdown-it) already hides them

I was aware of the table-like representation of GitHub for the YAML header, but I had missed the VS Code preview... very nice indeed. Thanks @chrisjsewell

[sammorley-short]

@inc0 We're actually hosting our code on GitLab, so although it sounds great, this wouldn't be applicable in my case. Either way, if you can configure jupytext to keep the metadata in an HTML comment, then this would solve the issue in a renderer-independent way. This seems like a good alternative solution for renderers that don't do this (yet).

[chrisjsewell]

But gitlab does recognise front matter https://docs.gitlab.com/ee/user/markdown.html#front-matter?

[sammorley-short]

@chrisjsewell: It recognises it, but it is still displayed as-is. For example, see here: https://gitlab.com/gitlab-org/gitlab/blob/master/doc/README.md. Being able to wrap the data in an HTML comment rather than the other delimiters would hide it from such renderings.

[mwouts]

Hi @sammorley-short , Jupytext 1.6.0 will have an option hide_notebook_metadata to hide the YAML header (for the Markdown format only).

It will take me a few more weeks to release that version, so if you wish to give a try to the dev version, with

pip install git+https://github.com/mwouts/jupytext.git

[roachsinai]

Hi @mwouts thanks for your work. But I failed to run jupytext with hide_notebook_metadata.

jupytext --to=md --hide_notebook_metadata=true  -o- test.ipynb
usage: jupytext [-h] [--from INPUT_FORMAT] [--pre-commit] [--to TO]
                [--format-options FORMAT_OPTIONS] [--set-formats SET_FORMATS]
                [--set-kernel SET_KERNEL] [--update-metadata UPDATE_METADATA]
                [-o OUTPUT] [--update]
                [--version | --paired-paths | --sync | --warn-only | --test | --test-strict]
                [--stop] [--pipe PIPE] [--check CHECK] [--pipe-fmt PIPE_FMT]
                [--execute] [--quiet]
                [notebooks [notebooks ...]]
jupytext: error: unrecognized arguments: --hide_notebook_metadata=true

My version is jupytext 1.6.0.dev0

[mwouts]

Oh, I see! It is always a challenge to document how to use these options... Since hide_notebook_metadata is a format option, you need to use --opt, i.e.:

jupytext --to=md --opt hide_notebook_metadata=true -o- test.ipynb

If you want this for all the notebooks in a given repository, you can also create a .jupytext.toml file with this content:

hide_notebook_metadata=true

[roachsinai]

@mwouts Thanks!