Jupyter-book: Document how the execution context is determined.

[akhmerov]

Is your feature request related to a problem? Please describe.
I might be mistaken, but right now it seems that the notebooks are executed by cache with a different path being set, which makes it hard for the users to import other modules that they have prepared.

A link to the documentation page where you see an issue.

https://jupyterbook.org/content/execute.html

Describe the solution you'd like

To help the users, I propose to document the execution path, and preferably also explain how to enable e.g. importing python modules that exist in the same repo.

[choldgraf]

Good point - this is a little bit unclear to me as well, to be honest haha. I don't see a clear explanation of this either in jupyter cache (e.g. https://jupyter-cache.readthedocs.io/en/latest/using/api.html#execution) or in MyST NB (https://myst-nb.readthedocs.io/en/latest/use/execute.html).

A few questions like:

• What's the folder where execution happens?
• What if the code depends on relative paths for loading data, loading scripts, etc?

Perhaps @chrisjsewell can quickly write up what users can expect and then I am happy to integrate it into the docs?

[akhmerov]

Just realized that this is largely a duplicate of #704, but phrased as a documentation question.

[chrisjsewell]

Perhaps @chrisjsewell can quickly write up what users can expect and then I am happy to integrate it into the docs?

So there is the capability to add "assets" to staged notebooks in jupyter-cache
(see https://jupyter-cache.readthedocs.io/en/latest/using/cli.html#staging-notebooks-for-execution), but this functionality has not yet been implemented in myst-nb: https://github.com/executablebooks/MyST-NB/issues/131

What's the folder where execution happens?

There is a pending PRs to allow for setting the cwd for execution: https://github.com/executablebooks/MyST-NB/pull/184 for the auto execution, but for jupyter-cache the execution cwd is set to the temporary folder: https://github.com/executablebooks/jupyter-cache/blob/54bc0c107e08fc5fc324bf6f3624870558f29d12/jupyter_cache/executors/basic.py#L163

What if the code depends on relative paths for loading data, loading scripts, etc?

Really the aim of jupyer-cache was that the user should not decide where execution happens, it should be decided by the program. This allows for the execution to be isolated, either in a temporary directory, a docker container, or even another server. In this sense jupyter-cache should be "notified" in some way where dependant external data/scripts are then it can decide how to deal with them.

[choldgraf]

So, from a documentation standpoint, is this correct for _current behavior_?

• If executing with auto (and once https://github.com/executablebooks/MyST-NB/pull/184 is merged) notebooks will execute in the folder where they exist and relative paths will work
• If executing with cache then the notebook will execute in a temporary folder, and relative paths will not work

[chrisjsewell]

That’s the long and the short of it. Although obviously I would like to make Jupyter cache work in myst-nb with relative paths at some point

[choldgraf]

cool - yeah we can mention that relative paths w/ jupyter-cache is something we're working on, just wanna nail down the current behavior so folks know what to expect

[akhmerov]

Hmm, does that mean that the only way to replicate this behavior of executing with nbclient would be to send to cache the complete build directory as context? Is that a desired behavior?

[chrisjsewell]

Hmm, does that mean that the only way to replicate this behavior of executing with nbclient would be to send to cache the complete build directory as context?

Don't understand what you mean; only the executed notebook gets cached

[akhmerov]

Perhaps my point was offtopic to this issue. I mean that if the behavior proposed in executablebooks/MyST-NB#184 was replicated with jupyter-cache, then with every notebook sent to execution to cache, the complete contents of the book source would need to be sent to cache as assets every time.

[choldgraf]

Yeah it sounds like that is true. I wonder if we could have an option with jupyter cache for "copy assets" or not? E.g., what if somebody includes a 2GB file with their book (I don't think this is unrealistic), it means that file would need to be copied each time it changes or is built for the first time?

[chrisjsewell]

Well yeh, there're pros and cons for different use cases

[choldgraf]

What do you think are the main "cons" of running execute with cwd=path/to/original/notebook.ipynb within jupyter_cache? Is the worry that people will alter their book's file structure etc?

[akhmerov]

The logic for finding and storing additional files that were produced by the evaluation (if any) becomes somewhat more complicated.

[chrisjsewell]

Yep things like that, and reproducibility, e.g. if you are relying on relative files, which then aren't committed to the repository or get moved. At the very least, I think it would be good to encourage having some sort of manifest for notebook assets that clearly specify what that notebook requires to run.
As mentioned, I definitely want to look into integration with containers etc at some point; to execute the notebooks in a reproducible environment.
As we speak actually I am using the https://molecule.readthedocs.io test framework, which is pretty nice, in that it fires up a docker container to run all your tests in.

[chrisjsewell]

running locally can certainly be an option, but it shouldn't be the only option

[choldgraf]

makes sense - what I had done with the old jupyter book was basically say "any assets your book needs should exist within your "content" folder and relative paths will only work within that folder". I wonder if we could do something similar with jupyter-cache, and then when MyST-NB uses it, it keeps the same file structure that exists within the book's content folder (either keeps all the files, or keeps only ones that are explicitly given in a manifest-style file?)

[chrisjsewell]

This should be closed in v0.8.0 😄, but feel free to re-open/open another issue if anything is outstanding: https://jupyterbook.org/reference/_changelog.html#v0-8-0-2020-09-01