Repo2docker: Standardized description of data imports

[chrisgorgo]

It would be worth considering a standardized config file for staging web-accessible data that would. For
example data.yml

- src: s3://openneuro.org/ds000113
  dst: /data
  recursive: True

S3, FTP, HTTP and other protocols could be supported. This config file would be read by a script called withing Dockerfile to stage necessary data for the notebook to run. What do people think?

Related to #199.

[betatim]

Does the tool to read and act upon these files already exist and what does its adoption/usage look like? The philosophy of repo2docker is to adopt an existing community standard/best practice where ever possible.

We have started collecting examples for getting data into your binder. My feeling from the discussions there and talking with people about it is that in the medium size category everyone has their own favourite/discipline specific way of fetching data. How about adding some more examples to that repo to show how to fetch a file from S3 with rclone or a similar tool. I think with a good example that is as easy/hard to use as a new file format and doesn't add a new maintenance burden for repo2docker.

What do you think?

[choldgraf]

I agree this would be super useful! If nothing else like this exists perhaps we could work with folks in, e.g., the WholeTale project to define some standards

[betatim]

What would be the advantage of

- src: s3://openneuro.org/ds000113
  dst: /data
  recursive: True

over adding in postBuild a command like rclone s3://openneuro.org/ds000113 /data?

If I had to vote today I'd vote for creating examples of typical ways of fetching medium sized data (via rclone, wget, yet-another-tool) over creating a new file format, a new tool and having to evangelise it to get adoption.

[nuest]

One advantage of structured information that I see (or better: that we have thought about in o2r, see https://doi.org/10.5281/zenodo.1478542) is proxying and whitelisting. As a maintainer of a BinderHub, I might allow only specific data sources...

• to which I have very fast access
• which I trust not to disappear before myself
• which I might cache

[betatim]

I don't think the decision of what network access policy a BinderHub operator implements (or not) should factor into this discussion. You might only allow access to certain hosts/ports/protocols but that doesn't motivate why repo2docker should invent and maintain a new tool and file format that (currently) has zero users. Especially because you can use repo2docker on your laptop where you are free to do as you please.

It would be great if a tool to read and act on these files existed and had wide adoption. I am not convinced repo2docker is the project to make that happen. Maybe someone should create it outside of this project and then we can adopt it?

[chrisgorgo]

Other usecase I can think of is being smart about staging input data that
already we're staged for other notebooks - so you would reduce data
transfer from outside networks.

I wonder is a simple rclone based solution could be added to repo2docker
with the ability of jupeterhub and similar application to override it and
deal with satisfying providing requested data on their own optimal way.

On Tue, Nov 6, 2018, 4:46 AM Tim Head <[email protected] wrote:

I don't think the decision of what network access policy a BinderHub
operator implements (or not) should factor into this discussion. You might
only allow access to certain hosts/ports/protocols but that doesn't
motivate why repo2docker should invent and maintain a new tool and file
format that (currently) has zero users. Especially because you can use
repo2docker on your laptop where you are free to do as you please.

It would be great if a tool to read and act on these files existed and had
wide adoption. I am not convinced repo2docker is the project to make that
happen. Maybe someone should create it outside of this project and then we
can adopt it?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/jupyter/repo2docker/issues/460#issuecomment-436240437,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAOkp5LdwMM72yFgXrV2HPiOOlD9UCZwks5usYS7gaJpZM4YNGJt
.

[ltetrel]

Hi everyone,

Just wondered if any people had time to work on this project ? @chrisfilo ?
I am open to contribute to this kind of tool, because we would need this for a CONP project.

[choldgraf]

hello! 👋 I think at this point it seems like there is not a "standard" that everybody uses already, and to that extent it might be best to start documenting patterns that people use for data transfer first, and then consider building a standard around this in the future if we think it's worth it. What's the current limitation that you're not able to achieve with, for example, a postBuild file?

[ltetrel]

Hi @choldgraf
Yeah I think a postBuild file would do the job actually for a POC.

I was just wondering if there is any tools that is doing it automatically that is more "user firendly" using a data_requirement.txt. There should be also a "standard way" to store this data on a server like a shared /DATA folder. But it is not really your job :laughing:

[choldgraf]

I totally agree it'd be awesome to have some standards around this - definitely worth revisiting the question as we see more specifics in how people are moving data in/out with repo2docker / binder

[ltetrel]

So I started to work on this little proof of concept on what repo2data tool should look like, what do you think ?

https://github.com/SIMEXP/Repo2Data

It would be nice if we could all collaborate on it. @choldgraf @chrisfilo

Thanks,

[yuvipanda]

I have seen a lot of uptake for https://github.com/ContinuumIO/intake in various PANGEO communities - @mrocklin @jhamman etc might know more. I also <3 your experiment, @ltetrel!

I think repo2docker should stay out of the 'describe your data' problem, and focus only on environments. We should make it easy for folks to use whatever method they want, however.

[choldgraf]

very cool @ltetrel ! Thanks for putting in a prototype :-)

let's try to tackle this with documentation for the time being - we can write up some examples of using different tools for bringing data in (e.g. using repo2data, or using intake). Happy to try giving repo2data a whirl.

What do you think @ltetrel ?

[jhamman]

Thanks for the ping @yuvipanda.

@ltetrel - would really encourage you to take a look at the intake package for this use case. Their docs are pretty good: https://intake.readthedocs.io. I'd be happy to point you in the right direction in this regard should you have questions.

[betatim]

It would be great to extend https://github.com/binder-examples/getting-data with examples using other methods of fetching data. In particular https://github.com/binder-examples/getting-data/issues/3

[ltetrel]

Thanks guys for the update, I am happy to see that many people are interrested :)

I _think_ repo2docker should stay out of the 'describe your data' problem, and focus only on environments. We should make it easy for folks to use whatever method they want, however.

I am totally agree. Maybe some people would like the option to also fetch the data also from some libs . So we could use your requirement.txt to install the library, and then have a field in the json to do the actual command (ex: 'cmd' = 'tf.keras.datasets.mnist.load_data(path='mnist.npz')')

let's try to tackle this with documentation for the time being - we can write up some examples of using different tools for bringing data in (e.g. using repo2data, or using intake). Happy to try giving repo2data a whirl.

I am afraid I did not get your point, do you mean we need to create a documentation in binder for repo2data or intake ?

would really encourage you to take a look at the intake package for this use case. Their docs are pretty good: https://intake.readthedocs.io. I'd be happy to point you in the right direction in this regard should you have questions.

I will have a look at it thanks !

It would be great to extend https://github.com/binder-examples/getting-data with examples using other methods of fetching data. In particular binder-examples/getting-data#3

For sure when we will finalize this tool we need to provide some examples :)

[choldgraf]

I am afraid I did not get your point, do you mean we need to create a documentation in binder for repo2data or intake ?

Yep, basically I'm just saying "rather than building technical assumptions and dependencies within repo2docker, we'll make sure to highlight these tools for data import/export/versioning in the documentation and examples we put together"

[chrisgorgo]

Unfortunately, I did not have time to work on this. As a side note, I think
that listing different use cases/examples would help scope the feature. One
can also try a more data-driven method and use
https://www.kaggle.com/github/github-repos/home to see what data intake
methods people use in the wild.

On Fri, Feb 15, 2019 at 9:40 AM Chris Holdgraf [e-mail]
wrote:

I am afraid I did not get your point, do you mean we need to create a
documentation in binder for repo2data or intake ?

Yep, basically I'm just saying "rather than building technical assumptions
and dependencies within repo2docker, we'll make sure to highlight these
tools for data import/export/versioning in the documentation and examples
we put together"

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jupyter/repo2docker/issues/460#issuecomment-464135956,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAOkp6Gx5VAlkJ8H5HEKo9QyuFmehyu0ks5vNvD4gaJpZM4YNGJt
.