Jupytext: Need Developer Docs for adding a new language/kernel

[dsblank]

Great tool! Would be great for me and future generations to document the steps needed to add a new language.

Consider Scheme. There is a link to a live kernel here:

Details:

• file extension (that we use): .ss
• comment from here to end of line: ; (often times we use two or more)
• kernel: https://github.com/Calysto/calysto_scheme

Anything else one needs?

[mwouts]

Thanks @dsblank , that's a very nice suggestion.

Well, I'm not sure the current organisation of the code is prepared for being documented... It has improved a lot with v0.6.0, but adding a new format still requires modifying the code at lots of places (latest example was Julia #56).

So I think that we should first refactor the code a bit further. There are already three more formats in the pipe, and the refactored code should allow an easier inclusion of those:

• scheme, as you suggest (language infered from the extension, like Julia, Python and R scripts)
• Hydrogen / VScode / Spyder cells (language infered from the extension, but format is language agnostic) #59
• Org-mode (language agnostic, like Markdown and R Markdown) #62

[dsblank]

Org mode looks very interesting, and perhaps a nice way to combine jupyter notebooks and jupytext ideas in a scalable fashion.

Some other links regarding org mode and jupyter:

• http://kitchingroup.cheme.cmu.edu/blog/2017/01/21/Exporting-org-mode-to-Jupyter-notebooks/
• https://news.ycombinator.com/item?id=16842786 (mentions org-babel)
• https://github.com/gregsexton/ob-ipython (org-babel)
• Emacs IPython Notebook (EIN) also has some emacs code on topic: https://github.com/tkf/emacs-ipython-notebook/blob/d62dc59869d436385efa940ba20a737bb8b9e5fd/lisp/ein-org.el

[mwouts]

Hello @dsblank , I have been working on refactoring, and was able to add a new format (the double percent scripts #59). It now requires

• Adding a new class derived from BaseCellReader for reading cells from text
• A class that derives from BaseCellExporter for exporting cells to text
• appending the definition of the format to formats.py

The corresponding commit (that also includes a bit of noise, sorry) is https://github.com/mwouts/jupytext/commit/3f3a12b2a03461acd74798a736f13a06e5480831

Do you think that structure is sustainable? Would you recommend another organization? I feel recommendations on this could be very useful! Thanks.

[dsblank]

I don't really have a feel for what this code is doing. Why does jupytext even need to be parsing the cell data? It seems that code cell data should be either sent directly to the kernel, or saved as-is in a section marked as "code cell".

[mwouts]

@dsblank , I have just implemented the _double percent_ format for Python/Julia/R (#59). That format is interesting as it is almost language agnostic. Would you like an adaptation of that format for scheme (that's easy, I just need to replace the comment char # with the ;;...) ?

Two quick questions:

• Could you contribute a sample scheme notebook to the tests/notebooks directory ? (or, may I copy the Mandelbrot one?)
• Should jupytext comment the Jupyter magic commands in the resulting scheme script?

[dsblank]

The line comment per language seems like it would be something that the kernels should be able to identify. (I recommended this to jupyter core 4 years ago, but it wasn't adopted at that point).

The magics in Scheme are language agnostic: they come from Metakernel. (We could have metakernel identify the line comment character(s)).

https://github.com/Calysto/metakernel/blob/master/metakernel/magics/README.md

The current rules for Metakernel magics:

1. they start with %, %%, or %%%.
2. they have to appear at the beginning of the cell
3. they can have flags and arguments, like the command-line

The Metakernel magics also provide the shell interaction.

The Mandelbot one is one of the most complicated, as it uses the full power of Metakernel. Feel free to use any of the notebooks from our repos. Perhaps a better representative notebook would be:

https://github.com/Calysto/calysto_scheme/blob/master/notebooks/Reference%20Guide%20for%20Calysto%20Scheme.ipynb

FYI, perhaps the most popular Metakernel languages are octave and matlab:

• https://github.com/Calysto/octave_kernel/
• https://github.com/Calysto/matlab_kernel/

[mwouts]

Hello @dsblank , the branch v0.7.2 has support for scheme! The resulting scripts are here (percent format, which you can open with Hydrogen) and here (light format, which has fewer explicit cell markers).

Would you like to test this and provide feedback? Currently I have implemented the support for just one comment character per script extension - for scheme I have choosen ;;. Do you think that is good enough? Would you prefer just ; ?

By the way - extending the light and percent formats for more file extensions is easier than ever before:

• add one line to _SCRIPT_EXTENSIONS in languages.py
• contribute a sample notebook in tests\notebooks\ipynb_[language]
• add two tests in test_mirror.py: one for the light format, one for the percent format...

... Which file extension do you want to do next ? 😄

[dsblank]

Very cool! I will test this out!

[mwouts]

@dsblank , I will soon review the treatment of magics in Jupytext (and possibly make it simpler, cf. #94). May I ask you two questions?

1. Do you think I can consider any command that starts with one or two percent sign as a 'magic'? Are there legitimate Python commands that start with one percent sign?
2. Above you mention that metakernel magics may start with up to three percent signs. I have never seen one of those, may I ask you to confirm?
   Thanks,

[dsblank]

1. I believe that is correct that there are no regular python code that can start with a percent.
2. Other languages can start with percents, but we decided for metakernel, we would put those at very top of cell. Metakernel can have 1, 2, or three percents (3 means sticky magic, and continues to apply)
3. Note that ipython's magics can appear anywhere, even on right of an equal sign.
4. We are considering allowing some metakernel magics in middle of code, but still at left of line (like %include)

[mwouts]

A brief documentation on how to extend the light and percent format is available in the readme in version 0.8.0. Scheme and C++ are supported already!

[mwouts]

Well, I just noticed with #97 that the 'mirror' test was not covering properly the round trip conversion. This has been fixed in v0.8.1 which is thus the first version with proper support for C++ and Scheme.