Joss-reviews: [REVIEW]: MemCNN: a Framework for Developing Memory Efficient Invertible Neural Networks

[whedon]

Submitting author: @silvandeleemput (Sil C. van de Leemput)
Repository: https://github.com/silvandeleemput/memcnn
Version: 0.3.3
Editor: @terrytangyuan
Reviewer: @GregaVrbancic, @sepandhaghighi
Archive: 10.5281/zenodo.3353604

Status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/54b718341b2fef3d97275722d58796ef"><img src="http://joss.theoj.org/papers/54b718341b2fef3d97275722d58796ef/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/54b718341b2fef3d97275722d58796ef/status.svg)](http://joss.theoj.org/papers/54b718341b2fef3d97275722d58796ef)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@GregaVrbancic & @sepandhaghighi, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @terrytangyuan know.

✨ Please try and complete your review in the next two weeks ✨

Review checklist for @GregaVrbancic

Conflict of interest

• [x] As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Version: 0.3.3
• [x] Authorship: Has the submitting author (@silvandeleemput) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Authors: Does the paper.md file include a list of authors with their affiliations?
• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

Review checklist for @sepandhaghighi

Conflict of interest

• [x] As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Version: 0.3.3
• [x] Authorship: Has the submitting author (@silvandeleemput) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Authors: Does the paper.md file include a list of authors with their affiliations?
• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @GregaVrbancic, @sepandhaghighi it looks like you're currently assigned to review this paper :tada:.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[sepandhaghighi]

@whedon check references

[whedon]

Attempting to check references...

[whedon]

```Reference check summary:

OK DOIs

• None

MISSING DOIs

• https://doi.org/10.1109/cvpr.2016.90 may be missing for title: Deep Residual Learning for Image Recognition
• https://doi.org/10.1109/cvprw.2009.5206848 may be missing for title: ImageNet: A Large-Scale Hierarchical Image Database
• https://doi.org/10.1007/978-3-642-35289-8_27 may be missing for title: Training deep and recurrent networks with hessian-free optimization
• https://doi.org/10.1109/iccv.2017.244 may be missing for title: Unpaired image-to-image translation using cycle-consistent adversarial networks

INVALID DOIs

• None
  ```

[sepandhaghighi]

review in progress, would say "accept with minor revisions".
added 4 issues, after those are closed, this should be ready to accept
silvandeleemput/memcnn/issues/22
silvandeleemput/memcnn/issues/21
silvandeleemput/memcnn/issues/20
silvandeleemput/memcnn/issues/19

[sepandhaghighi]

@whedon check references

[whedon]

Attempting to check references...

[whedon]

```Reference check summary:

OK DOIs

• 10.1007/978-3-319-46493-0_38 is OK
• 10.1109/cvpr.2016.90 is OK
• 10.1109/cvprw.2009.5206848 is OK
• 10.1007/978-3-642-35289-8_27 is OK
• 10.1109/iccv.2017.244 is OK

MISSING DOIs

• None

INVALID DOIs

• None
  ```

[sepandhaghighi]

another suggestion ;-)
silvandeleemput/memcnn/issues/23

[sepandhaghighi]

@silvandeleemput Thanks for your efforts!
@terrytangyuan Looks good to me

[terrytangyuan]

Thanks @sepandhaghighi. @GregaVrbancic Any update on the remaining items in the review checklist?

[GregaVrbancic]

Hi @silvandeleemput @terrytangyuan, sorry for a bit of a delay, I was swamped at work. I finally had time to go through the remaining items from the checklist.

In addition to the already addressed remarks from @sepandhaghighi, @silvandeleemput please take a look at this issue: https://github.com/silvandeleemput/memcnn/issues/24

[GregaVrbancic]

@terrytangyuan I'm satisfied with the changes made. It looks good to me for publishing.

[terrytangyuan]

Thank you!

@silvandeleemput - At this point could you make a new release of this software that includes the changes that have resulted from this review. Then, please make an archive of the software in Zenodo/figshare/other service and update this thread with the DOI of the archive? For the Zenodo/figshare archive, please make sure that:

• The title of the archive is the same as the JOSS paper title
• That the authors of the archive are the same as the JOSS paper authors

I can then move forward with accepting the submission.

[silvandeleemput]

@sepandhaghighi @GregaVrbancic Thank you both for your time and effort in reviewing this work.

@terrytangyuan I have created a new release of MemCNN (1.0.0) with the changes from the review included and I have uploaded it to Zenodo and saved it as an unpublished upload. The DOI of the archive is: 10.5281/zenodo.3353604. The title and the authors of the submission are the same as in the paper and should match your requirements.

[terrytangyuan]

@whedon set 10.5281/zenodo.3353604 as archive

[whedon]

OK. 10.5281/zenodo.3353604 is the archive.

[terrytangyuan]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[terrytangyuan]

@whedon set 0.3.3 as version

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[whedon]

OK. 0.3.3 is the version.

[terrytangyuan]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

Check final proof :point_right: https://github.com/openjournals/joss-papers/pull/869

If the paper PDF and Crossref deposit XML look good in https://github.com/openjournals/joss-papers/pull/869, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.
@whedon accept deposit=true

[terrytangyuan]

@openjournals/joss-eics Since this is the first accepted paper I am editing, could you help double check? Also, it looks like @whedon accept deposit=true can only be done by editors-in-chief so I am handing this over to you now.

[kyleniemeyer]

@terrytangyuan sure thing. This is the right point to tag us; I do the following:

• Double check that all the review items have been checked off and issues resolved.
• For us to publish, we should see that the archive DOI resolves and that the metadata is correct; @silvandeleemput, can you publish the latest version? We are done with any code changes.
• I next go through the paper to look for any typos, errors in references/citations, or anything else that stands out. Another thing I typically notice is incomplete affiliation info (should generally be something like department, institution, city, country—what you'd see in any journal article).

Also, an explicit @whedon check references check at the end is helpful, but is not a perfect tool.

The AEiCs will always do these checks when you hand it off to us, but it's certainly helpful for you to take care of any such issues before it gets to us.

[kyleniemeyer]

@whedon check references

[whedon]

Attempting to check references...

[whedon]

```Reference check summary:

OK DOIs

• 10.1007/978-3-319-46493-0_38 is OK
• 10.1109/cvpr.2016.90 is OK
• 10.1109/cvprw.2009.5206848 is OK
• 10.1007/978-3-642-35289-8_27 is OK
• 10.1109/iccv.2017.244 is OK

MISSING DOIs

• None

INVALID DOIs

• None
  ```

[silvandeleemput]

@kyleniemeyer @terrytangyuan The Zenodo code archive is now published (MemCNN version 1.0.0) and can be found here: https://zenodo.org/record/3353604#.XT6VqOgzaUk

[kyleniemeyer]

Thanks @silvandeleemput; can you merge this PR that corrects some references? https://github.com/silvandeleemput/memcnn/pull/25

I'll be looking through the paper itself next.

[silvandeleemput]

@kyleniemeyer Thanks! I have merged the PR.

[kyleniemeyer]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[kyleniemeyer]

Hi @silvandeleemput, for the paper, I've made some edits to in-text citations in https://github.com/silvandeleemput/memcnn/pull/26.

My only other requested change is for you to add, from our list of "what should my paper contain?":

• A summary describing the high-level functionality and purpose of the software for a diverse, non-specialist audience.

Thanks!

[silvandeleemput]

@kyleniemeyer We are unsure what exactly is meant with a "non-specialist audience" since it is very likely that people finding this work are at least familiar with neural networks, Python, and/or PyTorch, which should be sufficient to understand the purpose of the library from the paper.

If we are to add the requested change, what section of the paper do you recommend for this?

[kyleniemeyer]

@silvandeleemput We just ask that all papers try to have a sentence or two at the beginning explaining the purpose in a way such that a reader from a non-expert background could understand (in a shallow way) what the application is.

In your case, you might add one or two sentences at the very beginning explaining classification problems and/or neural networks for a general reader.

[silvandeleemput]

@kyleniemeyer Thank you for clarifying your recommendation. We have added a high-level explanation about neural networks to the summary. Additionally, we have also added a missing arxiv URL to the bibliography so that all works have either an URL or DOI.

[kyleniemeyer]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[kyleniemeyer]

@silvandeleemput looks great to me!

[kyleniemeyer]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

Check final proof :point_right: https://github.com/openjournals/joss-papers/pull/872

If the paper PDF and Crossref deposit XML look good in https://github.com/openjournals/joss-papers/pull/872, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.
@whedon accept deposit=true

[kyleniemeyer]

@whedon accept deposit=true

[whedon]

Doing it live! Attempting automated processing of paper acceptance...

[whedon]

🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦

[whedon]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

1. Check final PDF and Crossref metadata that was deposited :point_right: https://github.com/openjournals/joss-papers/pull/873
2. Wait a couple of minutes to verify that the paper DOI resolves https://doi.org/10.21105/joss.01576
3. If everything looks good, then close this review issue.

4. Party like you just published a paper! 🎉🌈🦄💃👻🤘

   Any issues? notify your editorial technical team...

[kyleniemeyer]

Congrats @silvandeleemput on your submission's publication in JOSS! Thanks to @GregaVrbancic and @sepandhaghighi for reviewing, and @terrytangyuan for editing!

[whedon]

:tada::tada::tada: Congratulations on your paper acceptance! :tada::tada::tada:

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](http://joss.theoj.org/papers/10.21105/joss.01576/status.svg)](https://doi.org/10.21105/joss.01576)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.01576">
  <img src="http://joss.theoj.org/papers/10.21105/joss.01576/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: http://joss.theoj.org/papers/10.21105/joss.01576/status.svg
   :target: https://doi.org/10.21105/joss.01576

This is how it will look in your documentation:

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: http://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://numfocus.salsalabs.org/donate-to-joss

[silvandeleemput]

@kyleniemeyer @terrytangyuan Hi thank you both very much for accepting our work and for helping us to publish our paper!

There is just a minor issue regarding the citation information on the linked DOI url: my surname is missing the "C." and has the "van de" in camel case while it should be all in lowercase (as it is written in the paper). Similarly, for the third author "Bram van Ginneken" the "van" should be all in lowercase. Can this be corrected?

See the corrected bib file below:

@article{vandeLeemput2019MemCNN,
  journal = {Journal of Open Source Software},
  doi = {10.21105/joss.01576},
  issn = {2475-9066},
  number = {39},
  publisher = {The Open Journal},
  title = {MemCNN: A Python/PyTorch package for creating memory-efficient invertible neural networks},
  url = {http://dx.doi.org/10.21105/joss.01576},
  volume = {4},
  author = {Sil C. {van de} Leemput and Jonas Teuwen and Bram {van} Ginneken and Rashindra Manniesing},
  pages = {1576},
  date = {2019-07-30},
  year = {2019},
  month = {7},
  day = {30},
}

[kyleniemeyer]

@arfon can you make this correction manually? Also, any idea about how to avoid such issues in the future? This isn't the first surname issue we've had recently.

[arfon]

@arfon can you make this correction manually? Also, any idea about how to avoid such issues in the future? This isn't the first surname issue we've had recently.

Done. I'm afraid the name parsing library we use doesn't handle surnames names like this very well so fixing is non-trivial.

[silvandeleemput]

@kyleniemeyer @arfon Thanks for the manual correction.