Joss-reviews: [PRE REVIEW]: Caliban: Docker-based job manager for reproducible workflows

[whedon]

Submitting author: @sritchie (Sam Ritchie)
Repository: https://github.com/google/caliban
Version: 0.2.5
Editor: @diehlpk
Reviewers: @lukasheinrich, @arokem
Managing EiC: Arfon Smith

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Author instructions

Thanks for submitting your paper to JOSS @sritchie. Currently, there isn't an JOSS editor assigned to your paper.

@sritchie if you have any suggestions for potential reviewers then please mention them here in this thread (without tagging them with an @). In addition, this list of people have already agreed to review for JOSS and may be suitable for this submission (please start at the bottom of the list).

Editor instructions

The JOSS submission bot @whedon is here to help you find and assign reviewers and start the main review. To find out what @whedon can do for you type:

@whedon commands

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Software report (experimental):

github.com/AlDanial/cloc v 1.84  T=0.26 s (377.9 files/s, 72607.4 lines/s)
--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
Python                           37           2799           3919           7182
reStructuredText                 40            860           1166           1441
Markdown                          6            248              0            803
YAML                              6             13             23            130
TeX                               1              4              0             83
make                              2             19             13             64
JSON                              2              0              0             40
Dockerfile                        1             15             37             37
Bourne Again Shell                1              6              3             29
DOS Batch                         1              8              1             26
Bourne Shell                      2              7             34             10
--------------------------------------------------------------------------------
SUM:                             99           3979           5196           9845
--------------------------------------------------------------------------------


Statistical information for the repository '2378' was gathered on 2020/06/22.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Ambrose Slone                   32         11878           5846           52.16
Sam Ritchie                     80         12487           3439           46.87
Vinay Ramasesh                   4           322             10            0.98

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Ambrose Slone              5619           47.3          3.3               10.29
Sam Ritchie                8224           65.9          5.1               20.65
Vinay Ramasesh              195           60.6          4.7               13.33

[whedon]

Reference check summary:

OK DOIs

- 10.1145/2889160.2891057 is OK

MISSING DOIs

- https://doi.org/10.1109/cvpr.2009.5206848 may be missing for title: ImageNet: A Large-Scale Hierarchical Image Database

INVALID DOIs

- None

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[arfon]

:wave: @sritchie - Thanks for your submission to JOSS. As described in our blog post announcing the reopening of JOSS, we're currently working in a "reduced service mode", limiting the number of papers assigned to any individual editor.

Since reopening JOSS last month we've had a very large number of papers submitted and as such, yours has been put in our backlog that we will be working through over the coming weeks and months.

Thanks in advance for your patience!

[arfon]

Also, while we're waiting for an editor to be assigned, you might want to look at fixing up your citations - the BibTeX key you're using in your paper.md seem to be different from the entries in your paper.bib.

You can ask Whedon to regenerate the paper proof with the command @whedon generate pdf

[sritchie]

Thanks for the note, @arfon! This was a bibtex misunderstanding on my part, as I'm sure you guessed. Let's try this again.

[sritchie]

@whedon generate pdf

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[sritchie]

terrytangyuan seemed like he might be an excellent reviewer, or arokem, given their interest in machine learning tooling.

[diehlpk]

@arfon I would be interested to be the editor for this. My group is working on something similar to make our application reproducible.

[danielskatz]

@whedon assign @diehlpk as editor

[whedon]

OK, the editor is @diehlpk

[danielskatz]

@diehlpk - no one will ever complain if you volunteer to edit :)

And you can just claim the paper by issuing the same command I did, or even @whedon assign me as editor

[sritchie]

Thanks so much, @diehlpk!

[diehlpk]

@diehlpk - no one will ever complain if you volunteer to edit :)

And you can just claim the paper by issuing the same command I did, or even @whedon assign me as editor

Ok, will do. I just did not want to overpass the Managing EiC.

[diehlpk]

Hi @terrytangyuan or @arokem or @kmoham6 or @stevenrbrandt or @rtohid would one of you be interested to review this paper?

[terrytangyuan]

@diehlpk Where did you find my username? I thought I am already removed from the list of reviewers since I am an editor now.

[diehlpk]

terrytangyuan seemed like he might be an excellent reviewer, or arokem, given their interest in machine learning tooling.

@terrytangyuan The author @sritchie suggested you as a reviewer and I normally ask the suggested reviewers and try to find additional ones.

[terrytangyuan]

I see. No, I don't have bandwidth to review at this point. Thanks for mentioning.

[diehlpk]

@cboettig Would you be interested to review this paper?

[diehlpk]

@TheChymera Would you be interested to review this paper?

[TheChymera]

@diehlpk thank you for thinking of me :) While I am indeed interested in new work on software reproducibility, it is my constant experience that Docker is an incredibly unfitting and shoehorned solution for this. It's a deployment system, and has nothing to do with environment reproducibility as much as copy-paste-ability. I am afraid I could not offer a competent review beyond just disagreeing very strongly with the fundamental premise and the ex ante choice of technologies.

[sritchie]

@TheChymera I actually agree with you, as far as raw Docker is concerned; it may not be clear, but I built this as a reaction to the dozens of copy-pasted Dockerfiles that end up infesting projects that attempt to use this tool to give themselves reproducibility guarantees.

Caliban is an attempt to implement a declarative build system that compiles a user's project on top of a stable base image. It uses Docker to manage caching and the stable base image, but relies more on tools like conda and virtualenv to provide reproducibility guarantees.

I didn't see a way (or a need) to get away from Docker, because Docker really is the only way to ship an environment from a local machine to a Cloud environment. All of the Cloud providers either have you ship a Docker image, or they attempt to reproduce the environment for you, which inevitably breaks if you're developing on a Mac and shipping to Linux.

So, instead of going down that bad road, we built a tool that piggybacks on tools you already have to use — some declaration of requirements, and a directory of code — to dynamically generate the usually-copy-pasted code. The fact that this lets you locally execute the exact same image and environment that the cloud provider will execute is really valuable, imo.

Maybe you're the perfect reviewer! Especially if you have some alternative road to the goal, here.

[diehlpk]

@whedon assign @lukasheinrich as reviewer

[whedon]

OK, @lukasheinrich is now a reviewer

[diehlpk]

@sritchie Could you recommend some more potential reviewers (without the @ in front of their GitHub handle)? We still need one more reviewer to go ahead.

[arokem]

✋ I'd be interested in reviewing this

[diehlpk]

@whedon add @arokem as reviewer

[whedon]

OK, @arokem is now a reviewer

[diehlpk]

@whedon start review

[whedon]

OK, I've started the review over in https://github.com/openjournals/joss-reviews/issues/2403.