Jupytext: Automatic conversion to .py upon git commit?

[lebigot]

For those of us who prefer not to change the global Jupyter config or notebook metadata (and pair a notebook with a .py or something else), it would be nice to have a documented git pre-commit (?) hook that automatically converts some .ipynb notebooks into versioned .py files.

I tried once but failed, as I am not fully sure yet of how pre-commits work (the .py was generated, but effectively after the commit).

If I were to investigate this again I would either:

• Ask someone who knows how these hooks work, or
• Understand things clearly _before attempting something plausible_ but that can fail.

Some references (that I don't master yet):

• Official doc: https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks
• Tips: https://codeinthehole.com/tips/tips-for-using-a-git-pre-commit-hook/
• Blog 1: https://delicious-insights.com/en/posts/git-hooks/
• Blog 2: https://medium.com/the-andela-way/git-hooks-beautifully-automate-tasks-stages-bfb29f42fea1
• Examples: https://github.com/pre-commit/pre-commit-hooks

[mwouts]

Thanks @lebigot , that's a very nice idea. I think a pre-commit hook that generates the .py file should be feasible. Another hook that updates the notebook on git pull would be nice as well.

I am interested by the subject, and will try a bit, but I at this time I do not have a lot of experience with git. If someone can contribute the hooks and/or more links, he/she is very welcome to contribute!

[theodcr]

I am also interested in automatic conversion using a pre-commit hook. AFAIK such a hook would work by first calling git commit without option, which would execute the hook without commiting any file. Then the generated .py files can be commited using git commit again.

So @lebigot the solution you tried might actually be working fine.

I may be able to find time to toy with this idea.

[mwouts]

I briefly discussed this with a Git grandmaster. What we want here seems to be made of two separate elements:

• a _pre-commit hook_ that generates or updates the .py file, and adds it to the commit, whenever a .ipynb file is part of the commit.
• plus, a _merge driver_ that is able to merge two conflicting .ipynb files. I see that nbdime already provides such a merge driver, we should try that one first.

In the following days I will experiment in these two directions. Let's see what we can do...

[mwouts]

Thanks @theolamayo for your interest! Please try that, and let us know - I saw that there are examples of pre commit hooks directly in the .git folder. Here the difficulty is that we create/update a file that was not initially part of the commit. I think it is feasible, but that may go a bit further than the base example.

[theodcr]

I wrote a very simple hook that converts all ipynb files found in a specific directory. It works by running git commit without giving a message (generated files must then be staged and commited). You can find it at https://gist.github.com/theolamayo/f0648c913c5dd2e1c4d4e9e5a0dfa324
The hook could also auto git-add the generated files.

About converting only staged notebooks (and maybe then unstaging the notebooks so that they aren't commited), the https://github.com/gitpython-developers/GitPython library could be useful.

And indeed, nbdime is probably the way to go for merging notebooks.

[mwouts]

I have just contributed a pre-commit mode to Jupytext command line. It will add the python representation (or the markdown one, or even multiple representations) of every staged notebook to the current commit.

@lebigot , @theolamayo , could you please have a look the documentation, and test the release candidate? The RC is available on pip. Please let me know if the hook works as intended. Thanks!

pip install jupytext==0.8.6rc1

[theodcr]

I tested it quickly and it works as expected, thank you!
Note: in my ideal workflow, notebooks would actually not be committed, only the python representations. This can be achieved by adding git reset HEAD **/*.ipynb to the pre-commit hook.

[mwouts]

Thanks @theolamayo for testing! And for the tip. We should include it in the readme. Do you want to add it below the pre commit hook example?

[theodcr]

Okay I will create a PR.

[mwouts]

Thanks everyone. This has been made available with version 0.8.6.