Joss-reviews: [REVIEW]: crystal-torture: A crystal tortuosity module

Created on 8 Mar 2019  Â·  66Comments  Â·  Source: openjournals/joss-reviews

Submitting author: @connorourke (Conn O'Rourke)
Repository: https://github.com/connorourke/crystal_torture
Version: v.1.0.08
Editor: @katyhuff
Reviewers: @mkhorton, @katyhuff
Archive: 10.5281/zenodo.3254426

Status

status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/c3d8e702ecfee04f16a0ad6f14d96419"><img src="http://joss.theoj.org/papers/c3d8e702ecfee04f16a0ad6f14d96419/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/c3d8e702ecfee04f16a0ad6f14d96419/status.svg)](http://joss.theoj.org/papers/c3d8e702ecfee04f16a0ad6f14d96419)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@vahtras & @mkhorton, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @katyhuff know.

✨ Please try and complete your review in the next two weeks ✨

Review checklist for @mkhorton

Conflict of interest

Code of Conduct

General checks

  • [x] Repository: Is the source code for this software available at the repository url?
  • [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
  • [x] Version: v.1.0.08
  • [x] Authorship: Has the submitting author (@connorourke) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

  • [x] Installation: Does installation proceed as outlined in the documentation?
  • [x] Functionality: Have the functional claims of the software been confirmed?
  • [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

  • [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
  • [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
  • [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
  • [x] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
  • [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

  • [x] Authors: Does the paper.md file include a list of authors with their affiliations?
  • [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • [x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
accepted published recommend-accept review
Source
picture whedon

Most helpful comment

@kyleniemeyer - this looks to be fixed now.

@mkhorton, @katyhuff - many thanks for your reviews and to @katyhuff for editing this one too ✨

@connorourke - your paper is now accepted into JOSS :zap::rocket::boom:

picture arfon on 24 Jun 2019
👍3 🎉1

All 66 comments

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @vahtras, it looks like you're currently assigned as the reviewer for this paper :tada:.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands
whedon picture whedon on 8 Mar 2019 
Attempting PDF compilation. Reticulating splines etc...
whedon picture whedon on 8 Mar 2019

:point_right: Check article proof :page_facing_up: :point_left:

whedon picture whedon on 8 Mar 2019

@vahtras This submission is waiting on your review. What is your status?

katyhuff picture katyhuff on 28 Mar 2019

@mkhorton - It looks like this review may have stalled mid-stream. Are there installation issues holding up progress? Thank you!

katyhuff picture katyhuff on 28 Mar 2019

Hi @katyhuff, thanks for following up/for the reminder! There were installation and test issues (see https://github.com/connorourke/crystal_torture/issues/3 and https://github.com/connorourke/crystal_torture/issues/4) but these have been resolved and I am now continuing with the review.

mkhorton picture mkhorton on 28 Mar 2019

@connorourke, just a small note that your PyPI formatting doesn't seem to be working correctly, see here. @katyhuff, regarding the "version number" list item, does JOSS require a formal GitHub release that matches the PyPI version, or is this simply a check to make sure that the version being reviewed from the repository matches the version in PyPI?

mkhorton picture mkhorton on 28 Mar 2019

@connorourke the only other checklist item missing is a guideline for contributions.

From a functionality point of view, I'm happy to recommend this software, and all functionality seems to be present and correct. I think this will be a very useful tool for the community (and, indeed, having studied percolation myself in the past, wish I had had this tool available!)

From a code point of view, I would have a few recommendations that, under my understanding of the JOSS guidelines, have no bearing on the review but I include here just as suggestions:

  • Code generally looks to be of a good quality, but is not PEP8 compliant. I would recommend passing it through a tool such as Black to re-format appropriately.

  • The correct usage of this code requires explicitly running graph.torture() before other commands such as graph.minimal_clusters can be run. I would strongly recommend adding appropriate error messages so that if a user tries to run graph.minimal_clusters before graph.torture() it will give a hint that they need to do this (currently it gives 'NoneType' object is not iterable).

  • The paper/examples would be greatly improved if they generated a known-true example from literature. From my understanding, the output will be very dependent on the size of supercell used, and extrapolation of multiple supercell values will be needed to get correct values for an infinite cell? This has no bearing on this code itself which, as far as I can tell, correctly calculates clusters and tortuosity for the cells provided, but is relevant for the broader application of this code.

  • There is a concept in pymatgen of transformations including, for example, an OrderDisorderTransformation to create ordered (not random) supercells. The more idiomatic (from a pymatgen point of view) way to construct the supercell for your spinel example would be to first create a disordered structure, i.e. to replace Mg with {'Mg':0.9, 'Li': 0.05, 'Al': 0.05}. To create a random supercell would then require creating a new RandomOrderDisorderTransformation class or similar (something pymatgen is definitely in need of). I only mention this to say that using site property labels to perform the doping is perfectly fine, but there might be a better/more idiomatic way of doing it that would also be useful for the broader community of people who use pymatgen.

  • A cut-off radius approach for calculating the graph is not sufficient for all structures. There are always edge cases; this is especially true for, for example, the output of MD simulations. There's a whole collection of different NearNeighbor classes in pymatgen that determine whether two sites should be 'connected', and also a StructureGraph class to construct graphs using these strategies. However, in your case, I think the use of cut-off radius makes sense because it's fast (the pymatgen classes NearNeighbor classes are probably too slow for your use case). I only mention this point just because I've encountered issues myself before related to cut-off radii approaches that ultimately led to incorrect analysis outputs.

In summary however, I'm very excited about this code, and really look forward to see how it develops going forwards. I will add an additional comment to this thread to formally complete the review once all checklist items are completed.

mkhorton picture mkhorton on 28 Mar 2019

@mkhorton,

Thanks for taking the time to do such a thorough review of the code, and for the useful suggestions. I'll make the changes you suggested, sort out the PyPi formatting issue and cross off the final checklist items asap.

I'd agree that the examples would benefit from a known-true example being included - and you are right that the results are dependent upon the size of supercell being used. We are currently preparing a publication which uses the code, and this is a point which we discuss within it - once it is published we will update the examples in the repo accordingly.

Thanks!

connorourke picture connorourke on 4 Apr 2019

Great! Looking forward to seeing it :)

Just to re-iterate, the additional comments don't have a bearing on my JOSS review (as I understand the JOSS guidelines). It's really the user's responsibility if they correctly extrapolate with supercell size etc. As long as the code itself works, is tested, documented, etc. and fulfills the other JOSS criteria, which it looks like it does, then I'm happy to recommend publication. The known-true example would be good however.

mkhorton picture mkhorton on 4 Apr 2019

@mkhorton - I've added some brief community guidelines (as well as sorting out the other issues you pointed out. Thanks for the tip on Black - found that very useful)

connorourke picture connorourke on 30 Apr 2019

@katyhuff @vahtras,

I was wondering if you guys have any update on the status of this review?

Thanks!

connorourke picture connorourke on 30 Apr 2019

Thank you for your patience @connorourke .

Reviewer @vahtras : Can you please confirm that you still intend to conduct this review?

katyhuff picture katyhuff on 9 May 2019

I will begin a detailed review, just in case. It should go quickly as I've already read the paper and looked over the repository.

Review checklist for @katyhuff

Conflict of interest

Code of Conduct

General checks

  • [x] Repository: Is the source code for this software available at the repository url?
  • [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
  • [x] Version: Does the release version given match the GitHub release (v1.0.3)?
  • [x] Authorship: Has the submitting author (@connorourke) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

  • [x] Installation: Does installation proceed as outlined in the documentation?
  • [x] Functionality: Have the functional claims of the software been confirmed?
  • [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

  • [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
  • [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
  • [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
  • [x] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
  • [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

  • [x] Authors: Does the paper.md file include a list of authors with their affiliations?
  • [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • [x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
katyhuff picture katyhuff on 9 May 2019

@whedon generate pdf

katyhuff picture katyhuff on 4 Jun 2019 
Attempting PDF compilation. Reticulating splines etc...
whedon picture whedon on 4 Jun 2019

:point_right: Check article proof :page_facing_up: :point_left:

whedon picture whedon on 4 Jun 2019

@whedon check references

katyhuff picture katyhuff on 4 Jun 2019 
Attempting to check references...
whedon picture whedon on 4 Jun 2019

```Reference check summary:

OK DOIs

  • 10.1038/s41598-018-27851-x is OK

MISSING DOIs

INVALID DOIs

  • None
    ```
whedon picture whedon on 4 Jun 2019

@connorourke I apologize again for the delay in this process. In my review I have found a couple of issues:

  • We require that the paper should "clearly state what problems the software is designed to solve and who the target audience is". I found your statement of need less than clear. While the physics and methodological background is very important for understanding the statement of need, perhaps a very concise, clear statement of need (and audience) could be added at the beginning of the paper to satisfy this requirement.
  • All of the references need to have DOIs. This paper has one, among a long list of references. Please see the above whedon response regarding the DOIs that are missing from your references.

Once these two issues are solve, we can move forward with this submission. Thank you for your patience.

katyhuff picture katyhuff on 4 Jun 2019

@whedon remove @vahtras as reviewer

katyhuff picture katyhuff on 4 Jun 2019

OK, @vahtras is no longer a reviewer

whedon picture whedon on 4 Jun 2019

@whedon add @katyhuff as reviewer

katyhuff picture katyhuff on 4 Jun 2019

OK, @katyhuff is now a reviewer

whedon picture whedon on 4 Jun 2019

👋 @connorourke - Any news on the needed changes?

danielskatz picture danielskatz on 18 Jun 2019

Thanks for the reminder @danielskatz. Yes - I'll make the required changes toward the end of this week.

connorourke picture connorourke on 18 Jun 2019

@katyhuff @danielskatz - the latest commit should address the paper changes requested in https://github.com/openjournals/joss-reviews/issues/1306#issuecomment-498647832

connorourke picture connorourke on 21 Jun 2019
👍2

@whedon generate pdf

katyhuff picture katyhuff on 21 Jun 2019 
Attempting PDF compilation. Reticulating splines etc...
whedon picture whedon on 21 Jun 2019

:point_right: Check article proof :page_facing_up: :point_left:

whedon picture whedon on 21 Jun 2019

@whedon check references

katyhuff picture katyhuff on 21 Jun 2019 
Attempting to check references...
whedon picture whedon on 21 Jun 2019

```Reference check summary:

OK DOIs

MISSING DOIs

  • None

INVALID DOIs

  • None
    ```
whedon picture whedon on 21 Jun 2019

Thank you @mkhorton for your review -- we couldn't do this without you.
Thank you @connorourke for a strong submission and for engaging actively in the review process! I have looked over the paper, double-checked all the DOI links, and have conducted a high-level review of the code itself. Everything looks ship-shape to me. At this point, please double-check the paper yourself, review any lingering details in your code/readme/etc., and then make an archive of the reviewed software in Zenodo/figshare/other service. Please be sure that the DOI metadata (title, authors, etc.) matches this JOSS submission. Once that's complete, please update this thread with the DOI of the archive, and I'll move forward with accepting the submission. Until then, now is your moment for final touchups!

katyhuff picture katyhuff on 21 Jun 2019

I've archived the code in Zenodo - DOI: 10.5281/zenodo.3253650

Thanks for taking the time to review @katyhuff & @mkhorton.

connorourke picture connorourke on 24 Jun 2019

@connorourke No problem, glad to see this code available to use!

mkhorton picture mkhorton on 24 Jun 2019

@whedon set 10.5281/zenodo.3253650 as archive

katyhuff picture katyhuff on 24 Jun 2019

OK. 10.5281/zenodo.3253650 is the archive.

whedon picture whedon on 24 Jun 2019

@connorourke : The metadata in the DOI must match the JOSS metadata. Currently, the title of your DOI is "connorourke/crystal_torture: JOSS reviewed release" and the JOSS submission title is "crystal-torture: A crystal tortuosity module". You should be able to edit the zenodo metadata.. Could you please be sure that the DOI metadata (title, authors, etc.) matches this JOSS submission and let me know when that's complete?

katyhuff picture katyhuff on 24 Jun 2019

Hi @katyhuff - the metadata should match now. Thanks

connorourke picture connorourke on 24 Jun 2019

@connorourke I see there's a new version, and the subtitle has changed in that version, but the title seems to have remained the same. This is clear from the heading on the landing page as well as the title in the bibtex export.

image

@misc{conn_o_rourke_2019_3254426,
  author       = {Conn O'Rourke and
                  Benjamin Morgan},
  title        = {connorourke/crystal_torture: JOSS review release},
  month        = jun,
  year         = 2019,
  doi          = {10.5281/zenodo.3254426},
  url          = {https://doi.org/10.5281/zenodo.3254426}
}
katyhuff picture katyhuff on 24 Jun 2019

Hi @katyhuff - metadata should now be correct on version:
https://doi.org/10.5281/zenodo.3254426

Let me know if there any more issues.

connorourke picture connorourke on 24 Jun 2019
👍1

@whedon set 10.5281/zenodo.3254426 as archive

katyhuff picture katyhuff on 24 Jun 2019

OK. 10.5281/zenodo.3254426 is the archive.

whedon picture whedon on 24 Jun 2019

@connorourke : Is v1.0.08 the desired version?

katyhuff picture katyhuff on 24 Jun 2019

Hi @katyhuff, yes v1.0.08 is the correct version. There have been a few changes from the review, so the version has bumped.

Thanks

connorourke picture connorourke on 24 Jun 2019
👍1

@whedon set v.1.0.08 as version

katyhuff picture katyhuff on 24 Jun 2019

OK. v.1.0.08 is the version.

whedon picture whedon on 24 Jun 2019

@whedon accept

katyhuff picture katyhuff on 24 Jun 2019 
Attempting dry run of processing paper acceptance...
whedon picture whedon on 24 Jun 2019

```Reference check summary:

OK DOIs

MISSING DOIs

  • None

INVALID DOIs

  • None
    ```
whedon picture whedon on 24 Jun 2019

PDF failed to compile for issue #1306 with the following error:

sh: 54: Syntax error: newline unexpected
Looks like we failed to compile the Crossref XML

whedon picture whedon on 24 Jun 2019

@openjournals/jose-eics This is ready to accept, but I'm not sure what this crossref xml error is about ^^ .

katyhuff picture katyhuff on 24 Jun 2019
👍1

Hmm I can't see any issues with the paper that are obvious to me—it compiled ok a few days ago. @arfon ?

kyleniemeyer picture kyleniemeyer on 24 Jun 2019

@whedon generate pdf

katyhuff picture katyhuff on 24 Jun 2019 
Attempting PDF compilation. Reticulating splines etc...
whedon picture whedon on 24 Jun 2019

:point_right: Check article proof :page_facing_up: :point_left:

whedon picture whedon on 24 Jun 2019

Ah, so it appears the issue is just with CrossRef.

kyleniemeyer picture kyleniemeyer on 24 Jun 2019

@whedon accept

arfon picture arfon on 24 Jun 2019 
Attempting dry run of processing paper acceptance...
whedon picture whedon on 24 Jun 2019

```Reference check summary:

OK DOIs

MISSING DOIs

  • None

INVALID DOIs

  • None
    ```
whedon picture whedon on 24 Jun 2019

PDF failed to compile for issue #1306 with the following error:

sh: 54: Syntax error: newline unexpected
Looks like we failed to compile the Crossref XML

whedon picture whedon on 24 Jun 2019

Just a heads up that I'm working on this. It looks like Crossref is having some issues right now and I'm processing the acceptance manually.

arfon picture arfon on 24 Jun 2019

@kyleniemeyer - this looks to be fixed now.

@mkhorton, @katyhuff - many thanks for your reviews and to @katyhuff for editing this one too ✨

@connorourke - your paper is now accepted into JOSS :zap::rocket::boom:

arfon picture arfon on 24 Jun 2019
👍3 🎉1

:tada::tada::tada: Congratulations on your paper acceptance! :tada::tada::tada:

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](http://joss.theoj.org/papers/10.21105/joss.01306/status.svg)](https://doi.org/10.21105/joss.01306)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.01306">
  <img src="http://joss.theoj.org/papers/10.21105/joss.01306/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: http://joss.theoj.org/papers/10.21105/joss.01306/status.svg
   :target: https://doi.org/10.21105/joss.01306

This is how it will look in your documentation:

DOI

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

whedon picture whedon on 24 Jun 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

whedon picture whedon  Â·  135Comments
whedon picture whedon  Â·  116Comments
whedon picture whedon  Â·  175Comments
whedon picture whedon  Â·  119Comments
whedon picture whedon  Â·  112Comments