Repo2docker: Support for quilt data packages

[meffij]

I contribute to an open-source project, Quilt, that provides a platform for managing datasets in a reproducible and performant way. I'm trying to figure out how to add support for the quilt.yml data package dependency format. It's like requirements.txt for datasets. Quilt has a CLI which can install packages from quilt.yml. I was wondering how requirements.txt is handled, and if I could use a similar process to add support for quilt.yml.

See binder issue #416.

Thanks!

[betatim]

Welcome @meffij! I am not a quilt user so excuse my potentially silly/simple questions :)

What does quilt support look like for you? From what I see there is quilt the python (and R and other languages?) package as well as quilt.yml. The python library looks like it allows you to load "quilt packages" that contain data. I couldn't (quickly) find quilt.yml does. Do you use it to specify a specific data package that is "the default" for the repository it is in?

Right now I am thinking all that a user needs to do is install the quilt package and then use it from their notebook(s)? Maybe we can preload the data so it is part of the repo already during the build phase?

This is more for my education: quilt itself is open-source, https://quiltdata.com/ is a commercial hoster, is there a community owned hoster as well? I am thinking along the lines of anaconda and conda-forge.

[meffij]

quilt.yml is like requirements.txt for data dependencies. The documentation for quilt.yml is here towards the bottom of the page. So I think doing something similar to requirements.txt makes sense here.

There currently aren't any community hosts, but we're very interested in enabling community members. If you know of anyone who might be interested, please have them email us at [email protected].

[yuvipanda]

Thanks for filing this, @meffij!

I think a good first step is to (as @betatim suggested), just put quilt in requirements.txt, and then use a postBuild script in a repo to fetch the data and bake it into the built image. IMO, I think that is the right thing to do for fetching data - putting it into postBuild for data that's immutably baked into the build image or the upcoming #196 for data that needs to be fetched every time the user code is run.

I think step 1 is to maybe make a nice example via this?

[akarve]

postBuild is doable and we can put an example together.

I believe that @ellisonbg and @choldgraf may have had something different in mind in BinderHub Issue 416? Gentlemen please advise if you prefer the postBuild route (in which case there's no PR needed for repo2docker?), or the "automatic install process", which would install and download the data dependencies as the container is building.

In the postBuild case we will need to address a race condition between installing the data packages and the using notebooks that depend on that data packages.

[ellisonbg]

The other option would be to put the needed logic into the actual build
process in the repo2docker code base. The advantage of that is that from
the users perspective, all they would have to do is put a quilt.yml file in
the repo and it would "just work". The postBuild approach would require all
users to learn how to add the needed lines to the postBuild script. The
disadvantage of the repo2docker built-in approach is that someone would
have to update the repo2docker version of quilt each time it is released.

On Thu, Jan 25, 2018 at 10:21 AM, Aneesh Karve [e-mail]
wrote:

postBuild is doable and we can put an example together.

I believe that @ellisonbg https://github.com/ellisonbg and @choldgraf
https://github.com/choldgraf may have had something different in mind
in BinderHub Issue 416
https://github.com/jupyterhub/binderhub/issues/416? Gentlemen please
advise if you prefer the postBuild route (in which case there's no PR
needed for repo2docker?), or the "automatic install process", which would
install and download the data dependencies as the container is building.

In the postBuild case we will need to address a race condition between
installing the data packages and the using notebooks that depend on that
data packages.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jupyter/repo2docker/issues/199#issuecomment-360554245,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABr0OXXa4T0nWQ4Ajkr3r3glf9j5ZFhks5tOMYNgaJpZM4RrvWD
.

--
Brian E. Granger
Associate Professor of Physics and Data Science
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
[email protected] and [email protected]

[yuvipanda]

In general, I'm a fan of keeping the core of repo2docker as small as
possible, given the constraints we have on long term maintenance. As such,
right now I would recommend going the postBuild route. It happens during
build time, before the notebook starts, so there should be no race
conditions.

I think we could promote that into an official binder example when you've a
good example working!

On Thu, Jan 25, 2018 at 10:21 AM, Aneesh Karve [e-mail]
wrote:

postBuild is doable and we can put an example together.

I believe that @ellisonbg https://github.com/ellisonbg and @choldgraf
https://github.com/choldgraf may have had something different in mind
in BinderHub Issue 416
https://github.com/jupyterhub/binderhub/issues/416? Gentlemen please
advise if you prefer the postBuild route (in which case there's no PR
needed for repo2docker?), or the "automatic install process", which would
install and download the data dependencies as the container is building.

In the postBuild case we will need to address a race condition between
installing the data packages and the using notebooks that depend on that
data packages.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/jupyter/repo2docker/issues/199#issuecomment-360554245,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAB23sZe1Le6Zo9432Qc9PN-ycyV3RJWks5tOMYNgaJpZM4RrvWD
.

--
Yuvi Panda T
http://yuvi.in/blog

[choldgraf]

I'm +1 on getting a working version with postBuild that we can include in binder-examples. That can also serve as inspiration if it makes sense to build it into repo2docker at some point

[yuvipanda]

I think part of the confusion is that we don't have clear written docs on
'how to get support for X into repo2docker', along with 'when to get
support for X into repo2docker'. I'll try to write something up over the
next few days. Apologies for any confusion in the meantime!

On Thu, Jan 25, 2018 at 10:31 AM, Chris Holdgraf [e-mail]
wrote:

I'm +1 on getting a working version with postBuild that we can include in
binder-examples. That can also serve as inspiration if it makes sense to
build it into repo2docker at some point

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/jupyter/repo2docker/issues/199#issuecomment-360557075,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAB23ob0grCoESsMJ1CL1bOg3WDArrm4ks5tOMh2gaJpZM4RrvWD
.

--
Yuvi Panda T
http://yuvi.in/blog

[betatim]

(Hey @akarve, a short heads up: we try and keep things gender neutral ("Gentlemen please advise...") as the people who make decisions around here are of all kinds.)

[meffij]

Here's an example of Quilt on Binder!
https://github.com/quiltdata/data2binder

[choldgraf]

by the way @meffij , regarding your earlier question about using the parameters in the Binder link in order to influence the behavior of your Binder. One option is to create something like a JupyterLab extension where people could input their username etc, and which would then modify the notebook accordingly. Some others have looked into that (e.g. Globus was using this approach to solve a similar problem, I believe)

[akarve]

Does it make sense to include data2binder in binder examples?

[choldgraf]

yep, I'll throw one together

[ellisonbg]

Thanks!

On Fri, Jan 26, 2018 at 4:35 PM, Chris Holdgraf [e-mail]
wrote:

yep, I'll throw one together

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jupyter/repo2docker/issues/199#issuecomment-360942900,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABr0EJgkO4YZKgl7lH65E-JDZwHYKEGks5tOm9FgaJpZM4RrvWD
.

--
Brian E. Granger
Associate Professor of Physics and Data Science
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
[email protected] and [email protected]

[choldgraf]

I forked / made very small edits to the other quilt repo, it now lives here! https://github.com/binder-examples/data-quilt

[yuvipanda]

I'm going to close this now since we have an example! I opened https://github.com/jupyter/repo2docker/issues/201 to discuss when and how to add new buildpacks!

Thanks everyone!