Jupyter-book: Allow users to open notebooks created in MyST in Colab and Binder directly

[MaxGhenis]

Is your feature request related to a problem? Please describe.
MyST md files that get translated to ipynb files can't currently be opened in Colab or Binder, as the links point to the source md file. I reported this with respect to a Jupyterbook documentation page in #724, and there's a workaround for Binder described in the MyST Markdown and launch buttons callout box. But I'd like to offer Colab without creating ipynb files in my main sources, and it seems like this should be unnecessary when Jupyterbook does so automatically.

Describe the solution you'd like
For notebooks created via MyST, point Colab and Binder links to the generated ipynb files on GitHub, rather than the md files they currently point to.

Describe alternatives you've considered
I may use jupytext to generate the files, but I prefer MyST source for version control.

Additional context
If this is impossible, it may be worth not rendering the Colab/Binder links for MyST files.

[chrisjsewell]

point Colab and Binder links to the generated ipynb files on GitHub

Well the problem here, as you note, is that if you only have the markdown file stored in your repository, then Colab/Binder has no .ipynb file to point towards. To the best of my knowledge they can't actually point towards the folder created by jupyterbook (at least for Binder, I don't have any experience yet with Colab).
BUT are you aware that, if you have jupytext[myst] (which is jupytext + myst-parser=0.8) installed in your Binder environment, then the MyST markdown files can be opened directly as notebooks? This conversion is automatic for the "classic" notebook, but currently requires "open with" for jupyter lab: see https://github.com/executablebooks/jupyter-book/pull/737
I imagine this should also work for Colab?

[MaxGhenis]

Well the problem here, as you note, is that if you only have the markdown file stored in your repository,

Can't it point to the _build/html/_sources/... version? e.g. here's the ipynb version of a md file I'm using as source.

[chrisjsewell]

Can't it point to the _build/html/_sources/... version?

Hmm yeh the problem is where to find these files. They are not normally uploaded to the main repository.
By what method do you actually publish your book?
We would have to work out some configuration to point towards the actual location the built documentation is located, which
would then depend on how you publish your book (in a gh-pages branch, on a service like netlify or ReadTheDocs, etc).

[MaxGhenis]

I use gh-pages.

[choldgraf]

right now, the assumption is that the binder / colab links etc point to the source files rather than the generated files. It might be possible to have them point to the generated files but this would require some changes to the link generation, and to the MyST-NB package. I think in the meantime, we should proriotize workflows like what @chrisjsewell describes. Either:

• Install Jupytext alongside the environment so that you can open markdown files as notebooks, or
• Commit both markdown and ipynb files that are synchronized to your repo, and that way the ipynb link could work

Right now Colab does not support opening a markdown file as a notebook - unfortunately the Colab infrastructure is almost entirely different from jupyter notebooks / lab, so them supporting this use-case depends on their teams deciding to implement the feature.

[dlukes]

@MaxGhenis Reading the logic for add_hub_urls(), there's a possibility of a hacky workaround exploiting the repository config, which is used to build the URLs.

You would have to put something like this in your _config.yml:

repository:
  url: ...
  path_to_book: path/to/_sources
  branch: gh-pages-branch-with-_sources-dir

Plus you'd have to keep .ipynb counterparts alongside your .md files in your worktree (synchronized via Jupytext and kept out of version control via .gitignore) so as to trigger this condition, which will make the link point to the file with the .ipynb extension instead of .md.

One downside of this workaround that I can think of is that it will break your _suggest edit_ links (they will point to the generated .ipynb on your gh-pages branch instead of the source .md).

Another option is to patch sphinx_book_theme/launch.py in the project's virtualenv. If you add the patch to version control and apply it as part of the dependency installation process, it's fairly painless. Unlike the previous workaround, if you get it right, it shouldn't break anything. The patch should look something like this:

--- a/launch.py 2020-10-14 00:29:31.762426000 +0200
+++ b/launch.py 2020-10-14 00:35:14.322426000 +0200
@@ -61,7 +61,8 @@
         # Skip the rest because the repo_url isn't right
         return

-    branch = _get_branch(config_theme)
+    # Hardcode your gh-pages branch.
+    branch = "master"

     # Construct the extra URL parts (app and relative path)
     notebook_interface_prefixes = {"classic": "tree", "jupyterlab": "lab/tree"}
@@ -76,15 +77,12 @@
         )
     ui_pre = notebook_interface_prefixes[notebook_interface]

-    # Check if we have a non-ipynb file, but an ipynb of same name exists
-    # If so, we'll use the ipynb extension instead of the text extension
-    if extension != ".ipynb" and Path(path).with_suffix(".ipynb").exists():
-        extension = ".ipynb"
+    extension = ".ipynb"

     # Construct a path to the file relative to the repository root
-    book_relpath = config_theme.get("path_to_docs", "").strip("/")
-    if book_relpath != "":
-        book_relpath += "/"
+    # book_relpath should point to the _sources directory generated by
+    # JupyterBook.
+    book_relpath = "_sources/"
     path_rel_repo = f"{book_relpath}{pagename}{extension}"

     # Now build infrastructure-specific links

@choldgraf I still think exploring an "official" solution might be worth it, GitHub Pages is one of the two hosting solutions documented by the JupyterBook project, and in this case, the assumption that the _sources directory is available in a git repo somewhere holds.

But the solution doesn't have to be GitHub Pages specific -- there could be repository url, path_to_book and branch overrides for generating the launch links, that would provide ample flexibility I think.

[dlukes]

I have a sketch of what an "official" solution could look like in https://github.com/dlukes/sphinx-book-theme/commit/9593f96ce2a4cf658a5c144afcb8071b4591714e. If there's interest, I'm happy to submit a PR, document the feature etc. :)

It's the simple yet flexible solution mentioned above, based on a repository_overrides key to the launch_buttons settings in _config.yml:

launch_buttons:
  notebook_interface: jupyterlab
  # optional overrides of the repository config which only apply to the launch buttons
  repository_overrides:
    url: https://github.com/example-org/example-repo
    path_to_book: path/to/_sources
    branch: gh-pages-branch
    # forces an ipynb extension, irrespective of the actual one
    force_ipynb: true