Repo2docker: Add an environment variable during the build process.

[travigd]

Proposed change

I'd like to see an environment variable to the tune of REPO2DOCKER=1 (or maybe just the current version of repo2docker?). The reason for this is to enable some specific setup for a Julia package if it's being built as part of repo2docker, specifically for Binder (see this GitHub issue for more context).

Alternative options

Status quo.

Who would use this feature?

How much effort will adding it take?

All of about twenty seconds (add a line to the base buildpack class).

Who can do this work?

I'm happy to make a PR if this sounds okay.

[manics]

Having it set to the current version seems a good idea, but see what everyone else thinks.

For your particular use-case why is it specific to running in binder? Wouldn't it affect anyone who installs the package in any local environment too?

[travigd]

The short version is that the package provides integrations with a number of frontends (including an electron binding, web server, and Jupyter). We're also a relatively common dependency, so we opted to not install the Jupyter extension by default, instead requiring users to run WebIO.install_jupyter_nbextension() (as an aside, this helped somewhat with issues about finding the "right" Jupyter to install to, especially for Jupyter Lab).

You can run the install_jupyter_nbextension() command inside of the Binder environment, but it also installs a server extension and (as far as I can tell) it's not possible to restart the notebook in that environment.

[betatim]

Making the repo2docker version available to the processes inside the container sounds reasonable.

We already add LABELs to the image with similar content https://github.com/jupyter/repo2docker/blob/c98e6ace78e032328baeef179520c46624da8e91/repo2docker/app.py#L687-L691

One thing that will make this trickier is that you need to find a balance between making the env variable available early enough to build steps but late enough so that you only invalidate the cache that actually needs invalidating (just because the version of repo2docker has changed doesn't mean a large number of layers have become invalid).

[travigd]

One thing that will make this trickier is that you need to find a balance between making the env variable available early enough to build steps but late enough so that you only invalidate the cache that actually needs invalidating (just because the version of repo2docker has changed doesn't mean a large number of layers have become invalid).

That's a great point! I only suggested the version as the value so that it's semantically meaningful, but it might make more sense to just set it equal to 1 (kind of like how you have CI=1 in CI environments) and not expose the version for exactly that cache invalidation problem. Another solution might just be to have minor versions (e.g. 1.2 instead of 1.2.3) since that changes less often.

Otherwise, I think the latest it could be set (and still work for my own personal goals) is right before installing packages.

[betatim]

I'm not mad keen. I think asking users to install the extension explicitly is more in line with our goal of trying to reduce the amount of magic/special casing. The goal of repo2docker is to automate best practices (not invent new ones). One thing this means is that we'd want it to be possible for someone to look at a repo that repo2docker can build and more or less quickly figure out what commands they'd have to type by hand to achieve the same goal. Packages/installers behaving differently when run inside a docker build by repo2docker (but not other docker builds) seems counter to that desire. So I think it would be better to require people to install the extension explicitly via a postBuild. This is a common pattern for other notebook extensions. The other common pattern is that installing the extensions package also installs the extension.

What do you think?

[travigd]

That's fair, and that's how we resolved our issue in the original issue on the WebIO repo. That being said, I'd still much rather have this. WebIO is the underpinning for Julia's version of ipywidgets (Interact.jl) and so I'd really rather have it installed automatically in repo2docker instead of asking users (who have probably never even directly heard of WebIO) to put it in their post-install.

The reason we don't install the server extension automatically is that WebIO is a pretty common package that's pulled in by lots of other packages that are orthogonal to jupyter and we've had a few complaints there (about touching unexpected files when you're just trying to install the Juno atom extension).

That being said, I don't think it's a great practice to encourage wildly different install processes when running under repo2docker. I think that can be a strong convention/warning rather than absolutely impossible though.

EDIT (Addendum): I would be 112% in your camp if it was easy to get things working once the repo has been built, but the nature of installing a server extension makes this impossible once the notebook is running. I don't think this is likely to lead to many issues about reproducibility because if you don't install the extension, WebIO will render a link to an FAQ about how to install the extension (it's all of one function to run) which makes it very easy to realize "Oh I need to add this extra step."

[manics]

Could you create a second package e.g. WebIO-binder that installs WebIO and runs the extra step?

[travigd]

That's not really any better for this setup, it's pretty much the same as adding a postinstall.

[travigd]

Bump. Any update on whether or not this can be included?

[betatim]

I don't think anyone has thought about this any further or worked on it.

[travigd]

The actual implementation seems relatively simple, it just needs to be agreed upon what needs to be... well... implemented.

I think I'm in favor of setting REPO2DOCKER=1 during builds and I'd be happy to take a stab at a PR.