Jupytext: Standardizing how to add cell metadata between code and markdown cells

[choldgraf]

Something I've noticed adds some (I think unnecessary) cognitive burden re: cell tags.

Currently there are two different syntaxes for cell metadata.

• If the cell is a code cell, you provide metadata like so (ignore the backslashes):
  \python tags=["tag1", "tag2"]
  code
  ```
  ```

• If the cell is a "region" (aka markdown cell), you provide it like this:

  ```
  
  sometext
  

I find myself often using the first style in markdown cells, e.g. doing:

<!-- #region tags=["tag1", "tag2"]} -->
sometext
<!-- #endregion -->

This then gets read in as a "title" metadata instead of cell tags.

I feel like this is an unnecessary difference in syntax, at least from a user's perspective. Is there a strong technical reason that it needs to be different between the two?

(a related point, it'd be nice to relax some of the strong JSON requirements for that metadata. for those who are writing it directly into markdown files, it's annoying to have to remember to, e.g., use double-quotes only instead of single quotes etc.)

[mwouts]

Hello @choldgraf , I like the idea, sure we could use the key=value (value being a JSON object) format everywhere in the Markdown file. Indeed it is easier to write. Maybe I'll also preserve the 'title' part, for compatibility with VS Code in which region can have names.

I feel like this is an unnecessary difference in syntax, at least from a user's perspective. Is there a strong technical reason that it needs to be different between the two?

No, I don't think so.

it'd be nice to relax some of the strong JSON requirements for that metadata

I'd be happy to! Do you have an idea how to do that? Also, you see no issue if Jupytext normalizes the JSON if the document is opened and saved?

[mwouts]

By the way, Chris, do you think we should also review how metadata are stored in the other formats? By simplicity I have used simple JSON there:

https://github.com/mwouts/jupytext/blob/bf6049b9107fef171647345f84364cdb4c5aaf5e/tests/notebooks/mirror/ipynb_to_script/Notebook%20with%20metadata%20and%20long%20cells.py#L42-L49

https://github.com/mwouts/jupytext/blob/bf6049b9107fef171647345f84364cdb4c5aaf5e/tests/notebooks/mirror/ipynb_to_percent/Notebook%20with%20metadata%20and%20long%20cells.py#L45-L52

[mwouts]

By the way, if we modify this part then maybe we could also offer a more concise syntax for markdown cells. Chris, do you think it would be worth allowing #md/#endmd (and #markdown/#endmarkdown) to explicitly mark markdown cells? e.g.:

<!-- #md tags=["tag1", "tag2"]-->
sometext
<!-- #endmd-->

[choldgraf]

A couple quick responses!

[re: relaxing the JSON constraints] Do you have an idea how to do that? Also, you see no issue if Jupytext normalizes the JSON if the document is opened and saved?

Yeah IMO that's fine - I'm just thinking that most people, when they write JSON, aren't really thinking in terms of "should I use single or double quotes" etc so it just seems like unnecessary cognitive overhead. Instead it seems we should allow the same behavior that Python would allow in construction a dictionary? (e.g. you can put single quotes within a double-quoted string, and vice versa). What do you think?

re: "how metadata are stored in other formats". I think the most important thing is just consistency. For me the confusing thing was that one format used YAML-style metadata, while another used JSON-style metadata. I feel that it's better to just pick the official spec, and then apply it throughout Jupytext formats. That said, I think that markdown is probably the "more important" of the formats right now, because IMO it's much more likely someone will write a markdown document from scratch, while I imagine folks might create the others using jupytext from a notebook. I might be wrong there though...

re: concise syntax for markdown cells. I agree that "region" and "endregion" seem a bit verbose. What if the convention was something like <!-- #<cell_type> -->. So the assumption is that whatever came after the # would be the cell_type field in the metadata. Then you could explicitly do something like <!-- #raw --> and <!-- #endraw -->. Maybe md could also be supported as a special case of #markdown.

Another option would be to support something closer to Pandoc-flavored markdown and use ::: {metadata=options} for markdown divs. Though that might also render improperly in vscode etc.

[mwouts]

it'd be nice to relax some of the strong JSON requirements for that metadata

Well, maybe I could give a try to ast.literal_eval(node_or_string):

Safely evaluate an expression node or a Unicode or Latin-1 encoded string containing a Python literal or container display. The string or node provided may only consist of the following Python literal structures: strings, numbers, tuples, lists, dicts, booleans, and None.

This can be used for safely evaluating strings containing Python values from untrusted sources without the need to parse the values oneself. It is not capable of evaluating arbitrarily complex expressions, for example involving operators or indexing.

[mwouts]

Hello @choldgraf, I've pushed a few commits to the markdown_format_update branch.

With the version in that branch, you can type the metadata as you wish (hopefully!). Give a try to
<!-- #md key='value'--> or <!-- #markdown param=5-->. And let me know if you can find a pattern that breaks the parser!!

With that version all the formats can read metadata in either the key=value syntax, or JSON.

I have not yet changed how Jupytext writes the metadata. I have to think a bit more about this... I do have a preference for the key=value format. But I don't want Jupytext users to see big diffs in their files just because they switch to a new version of Jupytext... Maybe I could use a Jupytext-specific metadata to trigger either the JSON or the key=value metadata encoding?

[choldgraf]

I quite like it! It works fine for me, I tried a few combinations of varying objects etc and wasn't able to "break" things :-)

That said - one strange behavior I noticed was that if you write python code directly into the values, it'll get executed. e.g., if I write

<!-- #md meta=2+2} -->
test 1
<!-- #endmd -->

Then the output will be "4". That seems vaguely insecure to me, no?

[mwouts]

Thanks @choldgraf . I'm glad you liked it! I do expect the new parser to be more robust than the previous one - hopefully it will even catch parsing errors and preserve your imparsible inputs in a round trip...

Oh, I did not expect that 2+2=4 :smile: ! But it's easy to explain, and we can reproduce it with:

import ast
ast.literal_eval('2+2')

That seems to be consistent with the documentation for literal_eval, which I quoted above (_safe evaluation_, etc). Now I see there is also a warning:

It is possible to crash the Python interpreter with a sufficiently large/complex string due to stack depth limitations in Python’s AST compiler.

I don't think that is a problem, but let me know if you think otherwise.

And regarding the name of the Jupytext option to select the JSON format for metadata, what do you think of cell_metadata_json: True in the jupytext section of the YAML header? It would be set automatically by Jupytext when opening a document that was created with versions < 1.3, to avoid introducing diffs, and then the user can choose to delete that option if he/she prefers key/values.

[choldgraf]

IMO the potential to crash python isn't that big of deal - I really doubt people will use metadata that is this complex (but you never know haha)

For the JSON metadata question - is the idea that if the cell_metadata_json is set to True, then it'll expect "old style" JSON in the cell metadata, and if false, jupytext will expect key=value metadata? That approach seems reasonable to me (and again, probably fine to still write metadata in JSON format during conversion)

[mwouts]

Sure!

Well, not exactly: the cell_metadata_json option is just the format in which the metadata will be _written_ by Jupytext. Whatever the value of the option it will be able to read metadata in either form, key/value or JSON. Only when it writes the notebook back to text, Jupytext will apply that option. Again, the aim here is to minimize the impact of format changes on the existing text notebooks.

[mwouts]

key="value" looks way nicer than the previous {"key": "value"} representation! I just regenerated the collection of files that mirror the test notebooks at https://github.com/mwouts/jupytext/commit/18e6e7b483e30fad7009f2fdf4ae8619c1a2a8ea.

On that commit the change is huge and affects every notebook with metadata. That is exactly what I wanted to avoid with the automated cell_metadata_json option! On that example, it did happen because I regenerated the text representation of the notebooks from the .ipynb file. But, if one works from the text file, the metadata formatting should be preserved, cf. this example:

https://github.com/mwouts/jupytext/blob/b32f54f5b596557ed09df250fc8b9c7225fc8bab/tests/test_read_simple_markdown.py#L227-L234

When I finish a few other subjects I will release another RC so that we can make sure that this really works in practice!

[choldgraf]

Wohoo! I agree I think it's much cleaner 😊