The-turing-way: Ideas for reproducibility chapter

[pherterich]

Detailed description

92 is a first draft at a chapter on reproducibility currently mainly including the Stodden definitions (they are referenced in most papers I've read so far), a reference to @KirstieJane 's adoption of them (assuming that's what we as a project mean when we say reproducibility)

I can add more content but I might go off topic so input would be welcome:

• we can provide some disciplinary definitions and issues but that might be leading to far away from the core data science topic and could be easily covered by further reading. On the other hand I think some examples would be useful to get people interested and "prove" that this is relevant to them.
• there are some reproducibility guidelines/manifestos out there already, some more disciplines specific than others. Would an overview be useful as it might make clear why we have certain chapters included and use it also as a bit of a discussion why we include some things in addition because these guidelines often are too high level to be actually useful?

Any ideas on this are welcome so I can get that chapter into a reasonable state and get that into master.

Update: Book skeleton notes say "Definition of Reproducibility/ How does reproducibility overlap with Open"
so I will try and highlight that part more in my next update

[r-j-arnold]

Currently there isn't explicitly a section in #92 for "Why reproducibility is important". It could get bundled with "what is reproducible science" or "How this will help you/ why this is useful" section. Either would work, though my personal preference would be to give it its own heading.

Wherever it is, it could be a good idea to divide it into "Why reproducibility is important for the researcher" and "Why reproducibility is important for science"

For the researcher:

• Often makes things easier and quicker in the long run when you need to revisit past work
• Makes it easier for people to collaborate with you if they can reproduce your work, and then add to it
• Your science will be better quality, hopefully leading to more citations/reputation.career benefits (one would hope)

For science:

• How confident can science be of a result if the result can't be reproduced? Answer: not at all
• If we make it easy for others to check our results they're more likely to do it. Catching mistakes early can stop entire fields wasting time going down the wrong paths
• Making work reproducible makes it much easier for others to reuse it, saving valuable time

[pherterich]

Thanks @r-j-arnold - all valid points that I'm trying to include now.

[jzf2101]

FWIW http://lorenabarba.com/gallery/reproducibility-pi-manifesto/ by @labarba includes a lot of documentation on their standards

[labarba]

Thanks for tagging me, @jzf2101 — I have new content since posting the Reproducibility PI Manifesto in 2012. Maybe this post is useful:
http://lorenabarba.com/blog/barbagroup-reproducibility-syllabus/
Cite as: Barba, Lorena A. (2017): Barba-group Reproducibility Syllabus. figshare. Paper. https://doi.org/10.6084/m9.figshare.4879928.v1

Regarding terminologies, I have a comprehensive literature review in this preprint:
Terminologies for Reproducible Research (9 Feb 2018), arXiv preprint: https://arxiv.org/abs/1802.03311

I'm member of the National Academies Committee on Reproducibility and Replicability in Science, which has been working for over a year and is about to send the report to external review.
I can tell you what the definition of reproducibility will be in that NASEM report:

Reproducibility: Obtaining consistent results using the same input data, computational steps, methods and code, and conditions of analysis.

[labarba]

I see that Chapter_Reproducibility.md is citing the ICERM Report for definitions. Do note that that report contains internal contradictions (perhaps due to the multiple authors contributed uncoordinated content) and is probably a poor source for terminology.

Some examples:

On [p.2] :

Both in the workshop and in this report the terms ‘reproducible research’ and ‘reproducibility’ most often refer to the ability to recreate computational results from the data and code used by the original researcher.

The above appears to be consistent with the Claerbout/Donoho/Peng terminology.

On the same page, it says:

… distinct from […] ‘repeatability,’ when an experiment is conducted independently from first principles.

Now, this seems in contradiction of Peng (Science, 2011), where the term used for independent confirmation of the scientific findings is ‘replication.’ However, it is not a clear statement, because it just says that “an experiment is conducted”—it does not make a requirement that the scientific findings be consistent with the previous study.

Page 2 references Appendix A for a taxonomy. There, it says:

Replicable Research. Tools are made available that would allow one to duplicate the results of the research, for example by running the authors’ code to produce the plots shown in the publication.

The above definition, as far as I can see, parallels that for “reproducible research” on p.2—i.e., using the author-provided code and data to recreate the results—but the adjective has been swapped to “replicable.” I actually believed this to be a typo, at first.

But below that, it says:

Open or Reproducible Research. Auditable research made openly available. This comprised [sic] well-documented and fully open code and data that are publicly available that would allow one to (a) fully audit the computational procedure, (b) replicate and also independently reproduce the results of the research, and (c) extend the results or apply the method to new problems.

This appears to only add the requirement of open code and data. Which is strange, because in the definition of p.2 for ‘reproducible research’—recreate computational results from the data and code used by the original researcher—open access could be implied. It says that “tools are made available”: how could you recreate the results from the author’s code and data otherwise? The only difference I can see is that the first case allows for closed-source, but available software.

Then later:

Confirmable Research: The main conclusions of the research can be attained independently without the use of software provided by the author.

This is called ‘replication’ by Peng and others.

—.

As far as I know, no one else uses these various terms, as listed in the ICERM Report Appendix.
Other parts of the report use the terms in imprecise ways, too.

[p.14, Appendix E]

Tools to aid in reproducible research. A substancial portion of the workshop focused on tools to aid in replicating past computational results …

The distinction between ‘replicable’ and ‘reproducible’ that seems to be introduced in the definitions of the Appendix (the first could be closed-source, as long as it’s available) is now lost in this sentence. It also happens in other parts of the text.

I therefore concluded that this report did not pay detailed attention in its attempt to define terms. It also came after Peng’s Science paper (2011), whose distinction between ‘reproducible research’ and ‘replication’ has been adopted by many others since.

Also, it's hardly valuable to consider a version of reproducible research where the code is available, but closed source. An important observation by Donoho was that if two implementations fail to give the same results: _“The only way we’d ever get to the bottom of such a discrepancy is if we both worked reproducibly and studied detailed differences between code and data.”_
This calls for open-source research software.

[labarba]

This recent report tackles the issue of contradicting terminologies:

Toward a Compatible Reproducibility Taxonomy for Computational and Computing Sciences
Michael A. Heroux, Lorena A. Barba, Manish Parashar, Victoria Stodden and Michela Taufer (October 2018)
https://cfwebprod.sandia.gov/cfdocs/CompResearch/docs/SAND2018-11186.pdf

Abstract:
Reproducibility is an essential ingredient of the scientific enterprise. The ability to reproduce results builds trust that we can rely on the results as foundations for future scientific exploration. Presently, the fields of computational and computing sciences provide two opposing definitions of reproducible and replicable. In computational sciences, reproducible research means authors provide all necessary data and computer codes to run analyses again, so others can re-obtain the results (J. Claerbout et al., 1992). The concept was adopted and extended by several communities, where it was distinguished from replication: collecting new data to address the same question, and arriving at consistent findings (Peng et al. 2006). The Association of Computing Machinery (ACM), representing computer science and industry professionals, recently established a reproducibility initiative, adopting essentially opposite definitions. The purpose of this report is to raise awareness of the opposite definitions and propose a path to a compatible taxonomy.

[pherterich]

Thank you @labarba - this is incredibly useful. I will read through all your literature suggestions and make sure they get integrated where suitable.

[KirstieJane]

Just wanted to +1 @pherterich’s comment - thank you @labarba for the amazing resource. Thank you @jzf2101 for linking us up ✨✨✨

Our goal for this book is VERY MUCH to reuse/quote/cite (with attribution) all the amazing work that already exists. The goal is to build something brief for people who don’t care about reproducible research but have lots of links to kickass resources once we have whetted their appetite to learn more!

It’s an absolute delight to have your comments in this thread, please do send us any other resources you wish more people read, and we’ll try to incorporate them.

[labarba]

This may be of interest, as well:

— Lorena A. Barba. Praxis of Reproducible Computational Science. Authorea. October 11, 2018. DOI: 10.22541/au.153922477.77361922

This preprint is queued to appear published in IEEE Computing in Science and Engineering soon.

[KirstieJane]

Sitting in a talk by Florian Markowetz and just want to make sure we've captured the following:

• Paper on "Five selfish reasons to work reproducibly": https://genomebiology.biomedcentral.com/articles/10.1186/s13059-015-0850-7
• Video of his talk in 2016: https://youtu.be/Is15CMVPHas
• Slides of his talk in 2019: https://osf.io/a8wq4/

[pherterich]

Paper is covered @KirstieJane (I still need to go over it as my first attempt at including it wasn't the best writing) but I'll make sure I add the other parts too.
@r-j-arnold should we divide the "Useful resources" section in the chapter by type and have e.g. videos clearly singled out for those who learn better through other ways that are not reading?