jupytext ignores lines starting with %

[tordbb]

I am using jupytext --to ipynb *.Rmd or jupytext --to Rmd *.ipynb to keep the pairs of Rmd and ipynb files in sync. This works nicely if I add code in either document and use the jupytext calls to copy these to the other file type.

However, it does not work if that line of code starts with either % or # %, as in %autosave 0 or # %autosave 0. Such lines of code are simply not transferred between the formats. As a result, since we only edit the ipynb file and make code review of the .Rmd file, such code edits go unnoticed.

Could you please make jupytext stop ignoring such code changes?

[mwouts]

Hello @tordbb , thanks for asking.

It is expected that Jupytext comments out magic commands (by default, because they don't make sense in a .Rmd document), but it should not remove any line when changing from one format to the other. If you have a reproducible example I would like very much to have a look.

Now, you can change that behavior with the comment_magics option (jupytext --opt comment_magics=False).

You can also change that for a single line with e.g.

%autosave 0 #noescape

[tordbb]

Hi @mwouts, thanks for your helpful response!

We are saving our notebooks as.Rmd and .ipynb since it is easier to version control changes in an Rmd file than in a messy ipynb file with all its outputs and metadata. Still, we are committing to git both Rmd files for easy version control, and also ipynb so that we can see the notebook outputs without having to rerun the notebook. And we are using jupytext to transfer changes we make in the ipynb files over to the rmd files which we version control.

This method is based on the following recipy:

In this sense, it is damaging when magic commands are being deactivated or ignored during this translation.

We have created a task to implement the changes you suggested. Feel free to let me know if you would suggest a better system for achieving both easy version controlling and sharing notebooks with outputs intact.

[mwouts]

Hi @tordbb , you're welcome.

Note that I think there's one more thing we should do, before you change anything to your setup...

If I understand well, you experience that change in magic commands in Jupyter notebook are not reported in the Rmd representation. This is not expected - Jupytext comes with a battery of test to ensure round-trip robustness, so changes should be reported in the Rmd file (but possibly not verbatim).

Could you please check what you see in the Rmd document when you have

%autosave 0

in the notebook (expected: # %autosave 0), and then, when the notebook contains

# %autosave 0

(expected: # # %autosave 0, i.e. commented twice) ? What does it gives for you? Is the issue with another magic command?

[mwouts]

Also, is there any specific reason why you are using R Markdown (i.e., do you use R?). Otherwise, I would suggest you give a try to the Markdown format (and that one does not comment out magic lines, because they are not meant to be executed in that format).

[anineharto]

Hi @mwouts

Thanks for helping us out. I just verified that when we have the following in our notebook
# %autosave 0
it is saved as
# # %autosave 0 in the Rmd file.

The reason to why we are using Rmd is due to this article, but if I understand you correctly storing ipynb to md is just as effective?

[mwouts]

Hi @anineharto , you're welcome!

storing ipynb to md is just as effective

Yes, it is! And it may even look better on GitHub :smile: So really, unless if you use either of RStudio or knitr, you will be better off with the .md than with the .Rmd format.

[anineharto]

@mwouts I tried storing the files as md but now I'm experiencing complaints regarding: MD031/blanks-around-fences: Fenced code blocks should be surrounded by blank lines. It seems like it's not automatically adding space between code cells and markdown cells. Do you have experience with this, or a way to solve this?

[mwouts]

Actually there are three flavors of the .md format for Jupyter notebooks:

• base md (Jupytext's one)
• pandoc's representation for Jupyter notebooks as Markdown files
• myst Markdown notebooks

Cf. also https://jupytext.readthedocs.io/en/latest/formats.html

The error you are getting seems to suggest that you are using myst:md. If you want to use Jupytext's plain Markdown format, then you should use just md as the format name.

[anineharto]

Seems odd. I use jupytext --to md notebook.ipynb when transforming to md file.

[mwouts]

Indeed. Well, could you share one notebook with that issue? Or send it to me over email?

[anineharto]

I could, but you won't be able to run it since it's dependent on confidential data.

[mwouts]

Well, I don't think I need to run it to reproduce the issue :smile:

Feel free to sent it to me over email (see my github profile).

If you prefer, note that it is possible that just having a look at the notebook metadata can tell us what goes wrong here. Would you like to open your notebook in Jupyter Notebook, click 'Edit -> Edit Notebook Metadata', and paste the result here (or at least the parts that are not confidential?). Also, please check that you don't have myst anywhere in your config files.

[anineharto]

I found the source of the error, so I managed to fix it :) Thanks for your help!

[mwouts]

Great news! Let me know if you need anything else, or if we can close this issue.