Jupytext: Per-project configuration

[grst]

I started wondering if a per-project configuration of jupytext would be possible using a config file.

For instance, there could be a .jupytext file in the root of each of my git repositories that defines, for instance, what format I want to pair with and which metadata to filter.

My use-case is, that for a certain project I work with Rmd files, in another I would prefer to store the documents as plain python scripts.

This would have the additional benefit, that the settings could be shared along with the code in the repository to achieve consistent behavior across collaborators.

[mwouts]

Hello @grst , this is certainly a great idea! I also note that at #238 we discussed a similar subject (global configuration). Do you know how we could implement that? I mean, do you have recommendations on which Python tools/package we could use to read these configuration files?

[grst]

How about TOML?

And then check all parent directories for a .jupytext file (similar to git finding a .git directory).

[mwouts]

Thanks, indeed TOML seems to be a great fit for this. And I agree, we could indeed check the parent directories.

Now what, in your opinion, should be in the .jupytext file? Should we mirror the options that are currently used in the contents manager? Also, do you agree that the options in the local .jupytext files should override those of the CM in Jupyter?

[grst]

In my opinion

• The .jupytext file should be completely optional (and fall back to defaults or a global configuration file if it doesn't exist)
• The .jupytext file should take precedence over the global configuration
• It should be possible to set all options that can be set in the contents manager or the metadata also in the .jupytext file.

Concerning global configuration: For now, the contents manager serves as a global configuration file. You could consider deprecating the contents manager in favor of a global .jupytext config file. The location of this file should adhere to the "XDG Base Directory Specifications" (see this stackoverflow answer).

I would prefer such a global configuration file over the contents manager for the sake of consistency with the local configuration files and because TOML is arguably nicer to write than a plain python config file.

[JCHHeilmann]

This would be extremely useful to me. The fact that the configuration can't come from inside a "project directory" right now is the reason I'm hesitating to use Jupytext in my current project.
The ideal scenario in my opinion is always that a tool is configured and up and running after a developer cloned a project and installed the dependencies. This is not possible with Jupytext at the moment (or is it? 😅).

I'm looking forward to this being added.

[matthewbegun]

Would this effect command line operation as well or only the Jupyter integration?

[mwouts]

Hello @JCHHeilmann ,

This would be extremely useful to me.

Yes! This is the next item on the list...

The ideal scenario in my opinion is always that a tool is configured and up and running after a developer cloned a project and installed the dependencies.

👍

This is not possible with Jupytext at the moment (or is it? 😅).

Well, since you're asking I think you are looking for a configuration that differs from the default? So the answer is probably no. Currently the only place where you can configure Jupytext is the Jupyter configuration file. Unless you are willing to use pre-commit hooks, see Jupytext as a Git pre-commit hook and Using Jupytext with the pre-commit package manager in the docs.

And, @matthewbegun

Would this effect command line operation as well or only the Jupyter integration?

It would affect both - as discussed with @grst above, we think that the local configuration file should have a higher priority than the Jupyter one.

[JCHHeilmann]

@mwouts That's great to hear.

For now I'm putting the config in the path related to the projects conda environment. That keeps it seperate, but it requires manual steps when setting up.

[mwouts]

Hello everyone. I've been working on this recently, and I have a PR more or less ready at #508.
The documentation is here: https://github.com/mwouts/jupytext/blob/jupytext_config_file/docs/config.md

It seems to work well, but I would like to get your advice on one point...

My question: do you think I can really search in every parent directory for a Jupytext configuration file?

May that cause issues if we don't have permission to read files in parent folders? Or, may that slow down the Contents Manager, which is going to search and read the config file for every notebook listed (not even opened) in the UI? Should I use a cache (for how long, etc.)? Thanks!

[grst]

Hi @mwouts, this is great progress!

Here are a few thoughts:

My question: do you think I can really search in every parent directory for a Jupytext configuration file?
May that cause issues if we don't have permission to read files in parent folders?

I don't think that would be an issue. You would just catch an exception and not find the file (which is legitimate as you don't have permission to read it)

Or, may that slow down the Contents Manager, which is going to search and read the config file for every notebook listed (not even opened) in the UI?

I'm not sure if I got that right: The contents manager searches for config files for every single file that shows up in, e.g. the File Browser tab of jupyterlab even if the files are not opened? In that case it might indeed become an issue if there are hundreds of files, even if searching the config file by itself is not expensive.

Should I use a cache (for how long, etc.)?

If the contents manager indeed runs on each listed file, a cache might be the only viable option.
But then maybe just store it for the time usually needed to render the file tree, i.e. a second or so?
Maybe there's a better solution, though...

---

I tried to find out how git handles this. While I didn't find any proper documentation about its behavior searching the .git directory, I found these two env variables which give us a hint. Maybe it's worth implementing something similar for jupytext to cope with slow file systems:

GIT_CEILING_DIRECTORIES

This should be a colon-separated list of absolute paths. If set, it is a list of directories that Git should not chdir up into while looking for a repository directory (useful for excluding slow-loading network directories). It will not exclude the current working directory or a GIT_DIR set on the command line or in the environment. Normally, Git has to read the entries in this list and resolve any symlink that might be present in order to compare them with the current directory. However, if even this access is slow, you can add an empty entry to the list to tell Git that the subsequent entries are not symlinks and needn’t be resolved; e.g., GIT_CEILING_DIRECTORIES=/maybe/symlink::/very/slow/non/symlink.

GIT_DISCOVERY_ACROSS_FILESYSTEM

When run in a directory that does not have ".git" repository directory, Git tries to find such a directory in the parent directories to find the top of the working tree, but by default it does not cross filesystem boundaries. This environment variable can be set to true to tell Git not to stop at filesystem boundaries. Like GIT_CEILING_DIRECTORIES, this will not affect an explicit repository directory set via GIT_DIR or on the command line.

(from https://git-scm.com/docs/git)

[mwouts]

Thank you @grst ! That is very helpful.

I have added a (one minute) cache for the current directory in the contents manager, and also implemented JUPYTEXT_CEILING_DIRECTORIES.

I'll merge the updated PR when the tests pass, and publish a RC with this feature soon.

[grst]

I'm a bit skeptical regarding a one minute cache.

What if I'm 'debugging' the config file, trying to find e.g. the right pairing for my project?
In that case I would have to wait a minute after each change. Or even worse, if I didn't know about the cache, I would have the feeling that something is behaving 'randomly' because, at first, the change doesn't take effect, and then suddenly it does...

[mwouts]

Good point... Do you think 1 or 10 seconds would be better? Alternatively/in addition, I'll have a look and see if we can use the cache only when get and save are called with content=False, and make sure that we use the latest config when actually reading or saving a notebook.

[grst]

I'll have a look and see if we can use the cache only when get and save are called with content=False

that would probably be the cleanest solution.

A pragmatic solution could be to set the cache to 1 sec. In that case it's quite unlikely to affect anyone.

[mwouts]

Good news - it was easy to do both, i.e. use the cache only when content=False, and for one second maximum, see https://github.com/mwouts/jupytext/pull/508/commits/0f0baf408c457b73ecc650043895591d7a6a51d5.

I've merged the PR. I'll close the issue now, but will come back to let you know when the corresponding RC is available.

[mwouts]

The rc is available:

pip install jupytext==1.5.0rc1

Please give it a try, and let me know if it works as expected!

[grst]

Sure, i'll have a look the next days!

On Fri, 22 May 2020 at 18:59, Marc Wouts [e-mail] wrote:

The rc is available:

pip install jupytext==1.5.0rc0

Please give it a try, and let me know if it works as expected!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mwouts/jupytext/issues/384#issuecomment-632804875,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABVZRVYHSJNJLBGH6IU25ILRS2VPFANCNFSM4JM6G6RA
.

[grst]

I gave it a quick try -- seems to work splendidly :)