Repo2docker: Feature: "repo2docker reproducible" standard

Created on 13 Jun 2018 · 6Comments · Source: jupyterhub/repo2docker

Reproducibility of research

Scientific journals might want to make it easy for their reviewers, so they might enforce that the submission to the journal containing code should make it easy for others to rerun the code execution. If a scientific journal want to encourage reproducibility of code execution, what should the actually ask for?

Idea: Journals could encourage the submitted code to at least be repo2docker reproducible!

The core of the idea is that there should be something easy to measure to verify that the author of a scientific paper have put in valuable effort to support the reproduction of code execution.

@jzf2101 made an excellent analysis of submissions to NIPS 2017. This work inspired the thought that some form of concrete reproducibility standard could help raise the bar of published research while at the same time increase research's value to others as it would become more accessible.

Action point

Perhaps the repo2docker CLI could help confirm if a repo meet a certain standard relevant as a measure for research reproducibility?

329 - Feature: repo inspection summary --- It would be relevant to know how repo2docker acts on the repo, for debugging and transparency of those attempting to make their repo "repo2docker reproducible".

enhancement discussion reproducibility

Source

consideRatio

👍1

Most helpful comment

I like the idea!

I've been thinking we should make an example of how to use repo2docker in your travis/circleCI setup to build your repo and have a travis like badge for your repo that says "r2d is green".

One tricky thing is how to tell if the repo does more than build, how would you determine that it "runs the science code"? For that we either need a file that specifies a command to run or ask users to configure their travis setup so that it runs a command inside the container.

betatim on 13 Jun 2018

👍2

All 6 comments

I like the idea!

I've been thinking we should make an example of how to use repo2docker in your travis/circleCI setup to build your repo and have a travis like badge for your repo that says "r2d is green".

betatim on 13 Jun 2018

👍2

@betatim Ah yes I agree the code execution part will be a very complex matter, I figure it could be nice to speak of different levels of compliance for a code repository.

repo2docker finds a requirements.txt or similar.
repo2docker ensures that it could build the image
repo2docker / or other service ensures that that some code can run (But what code should run?)

Having some r2d green badge even though it only meant verification of 1. and 2. could be very valuable by itself though, it could be something concrete for a journal to request of the author of a submitted paper for example.

consideRatio on 13 Jun 2018

This is great @consideRatio and what I was trying to get at in the write-up. I think @choldgraf was talking about having some badge as well .