Joss-reviews: [REVIEW]: Comet Time Series (CometTS) Visualizer

[whedon]

Submitting author: @jshermeyer (Jacob Shermeyer)
Repository: https://github.com/CosmiQ/CometTS
Version:
Editor: @Kevin-Mattheus-Moerman
Reviewers: @zhampel, @rmsare, @jjmcnelis
Archive: 10.5281/zenodo.3418091

Status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/2e38056740935063b95551d23334a1c1"><img src="http://joss.theoj.org/papers/2e38056740935063b95551d23334a1c1/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/2e38056740935063b95551d23334a1c1/status.svg)](http://joss.theoj.org/papers/2e38056740935063b95551d23334a1c1)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@zhampel, and @rmsare please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @Kevin-Mattheus-Moerman know.

✨ Please try and complete your review in the next two weeks ✨

Review checklist for @zhampel

Conflict of interest

• [x] As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Version:
• [x] Authorship: Has the submitting author (@jshermeyer) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Authors: Does the paper.md file include a list of authors with their affiliations?
• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

Review checklist for @rmsare

Conflict of interest

• [x] As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Version:
• [x] Authorship: Has the submitting author (@jshermeyer) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Authors: Does the paper.md file include a list of authors with their affiliations?
• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

Review checklist for @jjmcnelis

Conflict of interest

• [x] As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Version:
• [x] Authorship: Has the submitting author (@jshermeyer) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Authors: Does the paper.md file include a list of authors with their affiliations?
• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @rhiever, it looks like you're currently assigned as the reviewer for this paper :tada:.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[Kevin-Mattheus-Moerman]

@rhiever, @zhampel this is where the review process takes place. There is information and check boxes at the top of this issue to guide you through the review process. Let me know if you have any questions.

[zhampel]

@Kevin-Mattheus-Moerman I must disclose an affiliation I have with the author @jshermeyer of this submission. We are employees of the same parent organization 'IQT', though we have affiliations with separate independent laboratories within IQT. I leave it to you to decide whether I proceed as a reviewer.

[Kevin-Mattheus-Moerman]

@zhampel thanks for disclosing that. Can you provide a link to that organization IQT?

[zhampel]

@Kevin-Mattheus-Moerman Sure thing: https://www.iqt.org

[Kevin-Mattheus-Moerman]

@whedon add @dmittman as reviewer

[whedon]

OK, @dmittman is now a reviewer

[Kevin-Mattheus-Moerman]

@zhampel thanks for mentioning your shared affiliation. I do not feel this is serious enough to exclude you from the review process. I'm confident that, now that we have three reviewers, and since review takes place in the open, we'll have a balanced review process. Thanks.

[Kevin-Mattheus-Moerman]

@rhiever, @zhampel, @dmittman thanks again for acting as reviewers. You all have a set of tick boxes at the top of this issue which will guide you through the review process (which you can tick if you've accepted this invitation). Please let me know if you have any questions. :rocket: :robot:

[Kevin-Mattheus-Moerman]

@jshermeyer can you work on adding community/contributing guidelines (e.g. CONTRIBUTING.md)? I also recommend adding a code of conduct document (e.g. COC.md, you can find a good standard template here: https://www.contributor-covenant.org/). I recommend you link to the contributing guidelines in the readme and to the code of conduct in the contributing guidelines.

See also these resources:
https://help.github.com/articles/setting-guidelines-for-repository-contributors/
https://opensource.guide/code-of-conduct/

Examples:
https://github.com/atom/atom/blob/master/CONTRIBUTING.md
https://github.com/rails/rails/blob/master/CONTRIBUTING.md

[zhampel]

@jshermeyer
Here are my initial comments on the paper.

Paper

General comments:

• I think it would be better to intro something like ‘user defined polygon identifying a region of interest (ROI)’ and use ROI for the remainder of the paper.
• Q: Any limits to ‘user defined area of interest’?
• Figures: please provide descriptive captions to the three figures.

Specific correction suggestions:

• I feel like the paper title could be smooth like this: 'Comet Time Series Visualizer: CometTS'
• that we call -> called
• (CometTS) -> (CometTS)
• a times series of satellite imagery -> time series satellite imagery.
• -> The tool aims to enable research into population estimation, land use detection, or natural disaster monitoring using a variety of data types.
• series of satellite imagery -> series of satellite images
• variation), -> variation)
• visualizes -> provides a visualization of
• from user drawn polygons. -> from a user defined region of interest.
• define GIS as geographic information systems
• can only be used to analyze individual -> are limited to analysis of individual
• CometTS then produces -> CometTS produces
• within the polygon -> region of interest
• of most relevance -> most relevant
• standard statistics -> standard statistical measures / standard test statistics

[jshermeyer]

@Kevin-Mattheus-Moerman
Added a CONTRIBUTING.MD. Code of conduct seems overkill, if I need to post a code of conduct calling for basic civility in my little codebase I think the world is doomed.

@zhampel
Thanks for the notes. Will make some edits when another review comes in.

_I think it would be better to intro something like ‘user defined polygon identifying a region of interest (ROI)’ and use ROI for the remainder of the paper._

Agreed, will make this change then when other reviews come back.

_Q: Any limits to ‘user defined area of interest’?_

Just computational strength, the larger the polygon the more slowly the processing. I believe CometTS will not like polygons that are larger than the imagery you provide, but it can tolerate and ignore nodata/ blank space/ masked areas.

_Figures: please provide descriptive captions to the three figures._
Good idea.

[Kevin-Mattheus-Moerman]

@zhampel, @dmittman, can you provide an update on where we are in the review process? @rhiever when can we expect your contribution? Thanks!

[dmittman]

Apologies for the delay @Kevin-Mattheus-Moerman. I've reviewed the reviewer guidelines, checked the conflict of interest statement and took a look through the submitted PDF and repo. I find the material just outside my area of expertise, being more about satellite imagery analysis with a time-series component than pure time-series visualization. Perhaps @lewismc (https://github.com/lewismc) might be a more appropriate reviewer than I.

[Kevin-Mattheus-Moerman]

@dmittman Thanks for your reply. I understand. Thanks for taking the time to check out this project, and for recommending another reviewer.
@lewismc would you be interested in reviewing this work for JOSS?

[lewismc]

@Kevin-Mattheus-Moerman yes I would. Please give me a deadline for reviewing. I've just returned from some vacation.

[Kevin-Mattheus-Moerman]

@whedon add @lewismc as reviewer

[whedon]

OK, @lewismc is now a reviewer

[Kevin-Mattheus-Moerman]

@whedon remove @dmittman as reviewer

[whedon]

OK, @dmittman is no longer a reviewer

[Kevin-Mattheus-Moerman]

Great @lewismc thanks for agreeing to help. I've added you as a reviewer. It would be great to get this review done in about 2 weeks. Does that work?

[zhampel]

@Kevin-Mattheus-Moerman Sorry for the delay. Yes, I've nearly finished through the code, and will be posting my review by early next week.

[lewismc]

@Kevin-Mattheus-Moerman please point me at what it is you want me to review. Thank you

[Kevin-Mattheus-Moerman]

@lewismc the review focuses on the software and the short paper. The instructions and tick boxes (see the top of this issue) will guide you through the process. Let me know if you still have questions. You can also consult our review criteria here: https://joss.readthedocs.io/en/latest/review_criteria.html.

[zhampel]

@jshermeyer here are my comments regarding the structure of CometTS's code. I found your instructions easy to follow with the tools you have built. Your code does work as advertised, though I do think there are a changes that should be made to the formatting of the code, including commenting and other coding standards that should be upheld. Furthermore, I think your work could be made much stronger as an extensible project, by migrating the functionality of your python notebooks to a more API-like structure. I would also like to point out that a significant hurdle to properly reviewing the project is the lack of easy access to example datasets (you do point this out), which is beyond the scope of you as the author. To this end, the open source community encourages steps like CometTS to making these tools openly available indeed.

Code Review

General

• MacOS fails to pip install due to gdal. Needs brew install gdal prior to pip install -r Requirements.txt. Works on my MacOS Mojave laptop.

• DownThemAll add-on link points to a non-existent page.

• Ensuring a proper directory structure is clear for the example you gave, but is that the case with other datasets? Not being a geospatial data expert, is there a resource that provides guidance on proper structuring, or perhaps a universal industry standard?

• Had difficulty with GDAL install (who doesn’t?). Perhaps best to include gdal and subsequent deps on gdal at the end of the requirements list, so that all other packages install first. This may assist in debugging install.

• Also recommend including link to building GDAL with python-bindings in Installation section. Had success using following: http://trac.osgeo.org/gdal/wiki/BuildingOnUnix

• Please include a python style .gitignore file that excludes the CometTS python notebooks.

Code Specifics

• Recommend reformatting for python3 compatibility

• Recommended to import functions explicitly, instead of * for ease of tracking

• Header description for each fn

• Inputs and outputs description

• All input argument declarations should be initialized to None, ‘’, etc

• Run flake8 to format code properly, will make it easier to read too ;)

• Multi_studyAreas_FullStats.csv is in same dir as notebooks. Plot_Results.ipynb looks elsewhere for it. Consider moving it into the example data dir. Yet, when trying to use this example csv data, I get an error that there is no key count. Did work on my generated csv data though.

• For plotting, should have more scalable y-axis, as I chose a rather small city (Santa Fe, NM) to test

• Difficult to debug errors, esp wrt formatting data issues

• What are the temp*.vrt files? Are they part of the project?

• Need a set of tests for function verifications

[Kevin-Mattheus-Moerman]

Best wishes for 2019 everybody.

[Kevin-Mattheus-Moerman]

@jshermeyer can you reply to @zhampel 's comments?

[Kevin-Mattheus-Moerman]

@rhiever, @lewismc can you provide an update on the review process? Let me know if you need help. Also please let us know in case you are no longer able to assist in this review. Thanks.

[jshermeyer]

@Kevin-Mattheus-Moerman Thanks for staying on this. I'll do a full reply to all comments once all parties are done reviewing.

[Kevin-Mattheus-Moerman]

@rhiever has indicated he is no longer able to review this work

[Kevin-Mattheus-Moerman]

@whedon remove @rhiever as reviewer

[whedon]

OK, @rhiever is no longer a reviewer

[Kevin-Mattheus-Moerman]

@lewismc please let me know if you are able to proceed with this review. We could really use your help. Thanks.

[Kevin-Mattheus-Moerman]

@lewismc can you give an update on this review? Are you still able to help us? Thanks!

[Kevin-Mattheus-Moerman]

@lewismc Please can you inform us if you are still able to do this review. Thanks

[Kevin-Mattheus-Moerman]

I've e-mailed @lewismc to check if he is still able to review. Apologies for the delay encountered so far.

[Kevin-Mattheus-Moerman]

@jshermeyer although I will continue to try to contact @lewismc I suggest you start addressing comments by @zhampel to avoid further delays.

[Kevin-Mattheus-Moerman]

@jshermeyer in particular can you comment on whether you agree with @zhampel to migrate "... the functionality of your python notebooks to a more API-like structure..."?

[Kevin-Mattheus-Moerman]

@jshermeyer :point_up:

[labarba]

👋 It looks like we're waiting here for the author, @jshermeyer, to come back with responses to how they are addressing the reviewer comments.

It also looks like the second reviewer, @lewismc, has gone MIA (unresponsive to multiple mentions here, plus emails from the editor). I suggest we proceed to a recommendation from the first reviewer and make the acceptance decision at that point.

[jshermeyer]

@jshermeyer here are my comments regarding the structure of CometTS's code. I found your instructions easy to follow with the tools you have built. Your code does work as advertised, though I do think there are a changes that should be made to the formatting of the code, including commenting and other coding standards that should be upheld. Furthermore, I think your work could be made much stronger as an extensible project, by migrating the functionality of your python notebooks to a more API-like structure. I would also like to point out that a significant hurdle to properly reviewing the project is the lack of easy access to example datasets (you do point this out), which is beyond the scope of you as the author. To this end, the open source community encourages steps like CometTS to making these tools openly available indeed.

Thanks to Zig @zhampel for his helpful and thoughtful comments. Also thanks to the editors @Kevin-Mattheus-Moerman @labarba for pushing this on, I apologize for being an absentee on this. I've gradually added onto Comet to address all of his concerns.

_A brief overview:_
-Code totally reformatted to meet pep8 standards and reorganized for simplicity
-Now has a command line API for those who want to ignore the notebooks (would still recommend notebooks for easy plotting/viz).
-Installable via pip/conda
-Dockerized
-Pytests included for sensitive functions
-Added support for autoregressive integrated moving average modeling for future trend forecasting
-Includes an example test dataset of NPP VIIRS monthly composites over San Juan, PR.

Specifics:

MacOS fails to pip install due to gdal. Needs brew install gdal prior to pip install -r Requirements.txt. Works on my MacOS Mojave laptop.

Updated to have gdal install at the end. Also include support with docker and a conda environment for mac users. Would not recommend brew installing gdal, things get ugly fast when using brew/pip/conda.

DownThemAll add-on link points to a non-existent page.

Still works for old firefox, unfortunately isn't supported any more. I'll leave it up to an end user to get their own data. I now include sample VIIRS imagery for play.

Ensuring a proper directory structure is clear for the example you gave, but is that the case with other datasets? Not being a geospatial data expert, is there a resource that provides guidance on proper structuring, or perhaps a universal industry standard?

This is a standard for a few time-series methods, i.e. CCDC/YATSM- https://github.com/ceholden/yatsm

Had difficulty with GDAL install (who doesn’t?). Perhaps best to include gdal and subsequent deps on gdal at the end of the requirements list, so that all other packages install first. This may assist in debugging install.

Done.

Also recommend including link to building GDAL with python-bindings in Installation section. Had success using following: http://trac.osgeo.org/gdal/wiki/BuildingOnUnix

Multiple choices for installing GDAL now available.

Please include a python style .gitignore file that excludes the CometTS python notebooks.

Notebooks are pretty core for this project, but project is now pip/conda installable with an option for CLI only so this seems redundant.

Code Specifics

Recommend reformatting for python3 compatibility

Done.

Recommended to import functions explicitly, instead of * for ease of tracking

Done.

Header description for each fn/ Inputs and outputs description

Now have comments that describe inputs/outputs + github readme

All input argument declarations should be initialized to None, ‘’, etc

Some of these I like to keep to give a user what an example input would look like.

Run flake8 to format code properly, will make it easier to read too ;)

Done

Multi_studyAreas_FullStats.csv is in same dir as notebooks. Plot_Results.ipynb looks elsewhere for it. Consider moving it into the example data dir. Yet, when trying to use this example csv data, I get an error that there is no key count. Did work on my generated csv data though.

Code structure has been totally reorganized

For plotting, should have more scalable y-axis, as I chose a rather small city (Santa Fe, NM) to test

I've included a few helper notebooks for plotting only. A user will have to have a bit of background with matplotlib for tweaking some of the plotting features.

Difficult to debug errors, esp wrt formatting data issues

Not really sure how to make this easier, probably will require more iterations. Have added pytests to hopefully alleviate some stress.

What are the temp*.vrt files? Are they part of the project?

Temp outputs from some functions, can be ignored. I likely should clean these up but could be helpful for debug.

Need a set of tests for function verifications

Included for a good chunk of the code that could be sensitive. Also include test data.

[Kevin-Mattheus-Moerman]

@labarba thanks for helping here. In relation to:

It also looks like the second reviewer, @lewismc, has gone MIA (unresponsive to multiple mentions here, plus emails from the editor). I suggest we proceed to a recommendation from the first reviewer and make the acceptance decision at that point.

Agreed, however, although I do not feel there is evidence for a clear conflict of interest here (in fact the reviewer has been very thorough and detailed requirements), the reviewer @zhampel did point out to me that "...the author, Jake Shermeyer, is a member of a sister lab of my organization.". Hence I would prefer to find a replacement reviewer.

[Kevin-Mattheus-Moerman]

@chrismattmann @dmittman I believe you pointed out you might be able to review this work over at https://github.com/openjournals/joss-reviews/issues/866. Would greatly appreciate if you would be able to help (save the day! :rocket: ) with this review? The review has already started but we now need at least one additional/replacement reviewer.

[Kevin-Mattheus-Moerman]

@jshermeyer can you check if you need to update your paper to reflect all the changes you've made? Please run @whedon generate pdf here to regenerate your paper here.

[jshermeyer]

No changes required.

[Kevin-Mattheus-Moerman]

@chrismattmann @dmittman :wave: do you think you can help review this work?
@danielskatz do you know these potential reviewers personally? We've had a reviewer drop out so we need a replacement. Thanks.

[danielskatz]

I sent an email in case they don't see the github notification, but it's also just 4:30 am in California ...

[dmittman]

I find the material just outside my area of expertise, being more about satellite imagery analysis with a time-series component than pure time-series visualization.

[danielskatz]

And @chrismattmann replied to me by email that he doesn't have time currently

[danielskatz]

@turmon, @darth-pr - would either of you be able to step in and help with a review of this submission & this software for JOSS? (Or perhaps suggest someone else who might?)

See JOSS review criteria in case you are not aware of how JOSS works...

[turmon]

Maybe Kyo Lee in my group? He's a climate/remote sensing model analysis person, who contributes to an open source regional climate model system.

https://orcid.org/0000-0003-3754-3204

Huikyo.[email protected]

Mike

On May 20, 2019, at 6:24 PM, Daniel S. Katz [e-mail] wrote:

@turmon, @darth-pr - would either of you be able to step in and help with a review of this submission & this software for JOSS? (Or perhaps suggest someone else who might?)

See JOSS review criteria in case you are not aware of how JOSS works...

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[danielskatz]

Thanks Mike - Does Kyo has a github account? I'll let @Kevin-Mattheus-Moerman follow-up with him, as the editor.

[turmon]

I was not able to find a GH account, but I can't rule it out.
Mike

On May 21, 2019, at 8:44 AM, Daniel S. Katz [e-mail] wrote:

Thanks Mike - Does Kyo has a github account? I'll let @Kevin-Mattheus-Moerman follow-up with him, as the editor.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[Kevin-Mattheus-Moerman]

:wave: @lubiavinhas @mloning @TonyBagnall @gilbertocamara @rolfsimoes @pedro-andrade-inpe @ammaciel @jgrss @slandersson @mmann1123 @avanetten @aftimg would you be interested in reviewing this submission (about time-series analysis of satellite image data) for the Journal of Open Source Software (JOSS)? The review process, which takes place in this GitHub issue, focuses largely on the software and a short paper.
Let me know if you have questions.

[Kevin-Mattheus-Moerman]

@whedon remove @lewismc as reviewer

[whedon]

OK, @lewismc is no longer a reviewer

[Kevin-Mattheus-Moerman]

:wave: @ceholden @PontusOlofsson @ZachTRice @dgketchum @rmsare would you be interested in reviewing this submission (about time-series analysis of satellite image data) for the Journal of Open Source Software (JOSS)? The review process, which takes place in this GitHub issue, focuses largely on the software and a short paper.
Let me know if you have questions.

[usethedata]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[rmsare]

@Kevin-Mattheus-Moerman I can complete this review within the next 2 weeks. Is that okay?

[Kevin-Mattheus-Moerman]

@rmsare yes thanks!

[Kevin-Mattheus-Moerman]

@whedon add @rmsare as reviewer

[whedon]

OK, @rmsare is now a reviewer

[TonyBagnall]

hi, don't have time or expertise to review this, sorry.

[Kevin-Mattheus-Moerman]

@rmsare thanks for your help. I've created a set of tick boxes at the top of this issue which will guide you through the review process. Please also check out our review guidelines. Let me know if you have any questions.

[Kevin-Mattheus-Moerman]

@TonyBagnall okay, thanks for your reply.

[usethedata]

@whedon add @jjmcnelis as reviewer

[whedon]

OK, @jjmcnelis is now a reviewer

[Kevin-Mattheus-Moerman]

@jjmcnelis thanks for joining as reviewer. Would you be able to update your GitHub profile e.g. such that it lists your affiliation (and/or a link ORCID profile if possible). Thanks. Let me know if you have questions.

[rmsare]

@jshermeyer I am reviewing your submission, and I had a question about your changes to the project.

I would like to suggest some reorganization, but it looks like a bit of that has been implemented in @zhampel's fork of CometTS. Do you plan to incorporate any of these changes, e.g. to directory structure?

Thanks for clarifying - it will help us to provide more manageable reviews.

[rmsare]

Thanks for submitting this, @jshermeyer! Here is my initial review (based on the master branch of CosmiQ/CometTS).

Overall, it looks like a useful set of tools for quick time series analysis of satellite imagery that is appropriate for JOSS. Nice work.

While the code functions as advertised, the codebase and documentation could be improved with substantial changes that will make it easier to use. I believe implementing many of these changes will ensure that CometTS is a usable, extensible software package rather than an analysis utility.

I was able to install in a conda environment with PyPI.

Documentation

The code is well-commented, but function-level documentation is not quite sufficient (see review criteria). I also think the README could be streamlined and the example notebooks improved with some instructions. Someone interested in CometTS should be able to scan the example notebooks and documentation to quickly see how things work.

• [x] Improve the README and add separate documentation (#12)
• [x] Make notebooks more user friendly (#11)
• [x] Add doc strings (#10)
• [x] Document testing to satisfy review criteria (#9)

Once you add docstrings you might consider auto-generating API documentation with Sphinx and ReadtheDocs (file an issue with the docs tag?).

Functionality

• The main notebooks are functional. They don’t run under Jupyter lab due to changes to the widgets extension, but they do run as standalone notebooks.
• The command line utilities are functional and run with real data.
• I agree with @zhampel that refactoring to an API design would be a good idea. This might be a good thing to plan out in an issue with the refactor or enhancement tags.

Paper

The manuscript is written well, but describes the CometTS workflow rather than making a case for the research software contribution. Consider adding two or three sentences to introduction providing more context on satellite timeseries analysis and why CometTS is a good alternative to other open-source offerings.

Many of the details in the final paragraph could be summarized in a shorter sentence like “CometTS output includes user-specified statistics such as mean, median, and quartiles, and the package offers masking functionality to remove clouds and snow from the area of interest.” Then the remaining space could outline key research use cases in more detail.

The statement of need should state the intended audience, which appears to include social scientists, earth scientists, and/or geospatial intelligence analysts.

• [x] Add audience to statement of need
• [x] Expand introduction, including appropriate citations to related work
• [x] Edit manuscript to remove workflow details in favor of suggested research use cases
• [x] Remove ® symbol from “Python ®”

Minor issues

• [x] Cleanup: See issues #7, #8
• [ ] Version: The Zenodo archive and paper should correspond to a tagged release on Github (#6)

Recommendation: Could be accepted after moderate enhancements

[Kevin-Mattheus-Moerman]

@rmsare thanks for this very thorough review!

[Kevin-Mattheus-Moerman]

@jshermeyer can you comment on the points raised by @rmsare. Thanks.

[jjmcnelis]

@jjmcnelis thanks for joining as reviewer. Would you be able to update your GitHub profile e.g. such that it lists your affiliation (and/or a link ORCID profile if possible). Thanks. Let me know if you have questions.

@Kevin-Mattheus-Moerman Happy to help. ORCID and Org are now listed on my profile. I'll complete my initial review before the business week opens on Monday. Please let me know if that doesn't agree with your expected schedule for reviewers and I'll up the urgency.

[jshermeyer]

Thanks @rmsare for the thorough review. Good recommendations that I'm going to gradually start addressing. I don't have as much time for this repo as I used to as this project was more of a focus about a year ago. Will see how it goes, may take some time.

-Jake

[jjmcnelis]

@jshermeyer I enjoyed working through your example notebooks. I had a few hangups but nothing major. Please give me a day or so to open issues with specific recommendations. I will update this comment with specifics.

Key recommendations:

1. Acknowledge in the documentation whether CometTS is
   a. a workflow for processing/visualizing raster time series, or
   b. tools for workflows; i.e. it's unclear how many of the functions are reliant on outputs from other pieces.
2. Acknowledge the key features that distinguish CometTS from other processing/visualization tools for raster time series, e.g. TIMESAT.

@Kevin-Mattheus-Moerman Is 1a is compatible with JOSS submission requirements?

General

• Maybe add an ignore file, will open issue: __pycache__ __pycache__
• I was able to install in a conda environment.
• I wasn't able to install in a Python 3 virtual environment because of the common GDAL path hang-ups.

Functionality

• Are raster formats other than GeoTIFF supported?
  I don't see any reference to other formats than GeoTIFF. Display a list of supported formats prominently on the README. Support for other GDAL-supported formats will be straightforward to implement if you choose to refactor as an API as zhampel and rmsare recommend. The metadata stored internally by HDF and others eliminates the need for a config file like the one generated by CSV_lt.py.
• Test and indicate in the README which imagery sources and formats are compatible with CometTS.
  The VIIRS example is cool, but I think it represents an edge case for most VIIRS users. VIIRS is distributed in HDF for most standard products, not GeoTIFF like in the example. The documentation suggests that users of "overhead imagery" are the intended audience, but many of the relevant datasets are distributed in other formats.

Documentation

• Docker, requirement, and environment files and relevant install instructions are included.
• The code could benefit from inline documentation. Adding docstrings.
• Provide examples for more data sources. It's not clear how transferable the VIIRS workflow example is to other data sources.
• Cut and paste the demonstration from the README into a notebook or notebooks, as recommended by rmsare. The notebooks need more detail.
• The Jupyter notebooks need more detail - closer to what you provide in the README - and some explanation of the code. The whole code pasted into Jupyter notebook doesn't pass muster as an example or documentation.

Paper

My feelings about the content of the paper are pretty well covered above. No problems with the grammar/writing style but I can send a marked up copy if you need that level of detail.

[Kevin-Mattheus-Moerman]

@jshermeyer apologies this review has taken so long. We initially had great difficulty finding suitable (and suitably active) reviewers.

Since you've indicated that addressing the points raised may take a while, I've paused this review for now. We can resume it again at any point, when you are ready.

Would you be able to give me your best (and most realistic) estimate as to when you think you'll have addressed the points raised?

Let me know if you think you will not be able to implement these changes. If that is the case we could instead retract this submission (and resubmision in the future s also an option).

Thanks.

[Kevin-Mattheus-Moerman]

@zhampel, @rmsare, @jjmcnelis I've paused this review for now since @jshermeyer has indicated that addressing the issue raised may take a significant amount of time.
@jjmcnelis thanks also for your review comments. You mentioned you might be able to open specific issues related to your comments. Please do, so @jshermeyer can work on these too.

Once @jshermeyer indicates they are ready to resume the review I'll remove the paused label and will call upon the reviewers again.

[jshermeyer]

Thanks @jjmcnelis for another strong review. I think this process will make a much stronger toolset when it's all said and done.

I've update the docstrings which I think was the most cumbersome thing to add. Next on my list is documenting the notebooks a bit better. After that's adjusting the readme and responding to some of the more minor comments as proposed by reviewers. Seems like it could be done in a day or two if I can steal a few hours.

My timing on this is quite ambiguous and my availability to work on this will decline rapidly starting next week. If I can get to it over the next 2 weeks I have a feeling I could get it done by then. Otherwise this may bleed for awhile.

@Kevin-Mattheus-Moerman Am I correct in assuming no further reviewers will be added? I'd hope to address everything in one shot and then move forward to a decision.

[Kevin-Mattheus-Moerman]

@jshermeyer thanks for the update. Correct, no more reviewers will be added (I had to recruit new/more reviewers halfway through the review process because one was no longer responsive and one indicated a (minor) conflict of interest). So yes it looks like you are looking at the full set of changes/comments now. Thanks for working on this now. Looking forward to the next updates.

[jshermeyer]

@Kevin-Mattheus-Moerman, @zhampel, @rmsare, @jjmcnelis Thanks for the reviews and constructive feedback. I've worked to incorporate all of your suggestions throughout this process into the repository in one way or another. As I stated previously I think the process has been worthwhile and I’m appreciative for your time to make CometTS a better and more usable package.

What I've worked on
Specific items that I've changed in the code, readme, or paper are all tracked in the issues: https://github.com/CosmiQ/CometTS/issues

Specifically issue 15 tracks the minor comments from all reviewers and aggregates them in one location.

Specific responses
A few things throughout the review likely require a direct response from me, I’ll run through them here.

1. Acknowledge in the documentation whether CometTS is
   a. a workflow for processing/visualizing raster time series, or
   b. tools for workflows; i.e. it's unclear how many of the functions are reliant on outputs from other pieces.

I think CometTS is both a package with unique functionality, a workflow, and provides tools for workflows for processing and visualizing raster time series. So all of the above.

1. Acknowledge the key features that distinguish CometTS from other processing/visualization tools for raster time series, e.g. TIMESAT.

Key differences are:
1) the ability to handle arbitrarily sized areas of interest (polygons) vs. individual pixels for visualization.
2) Anomaly detection and auto-regressive integrated moving average forecasting
3) Python / jupyter notebook deployment

Are raster formats other than GeoTIFF supported?
I don't see any reference to other formats than GeoTIFF. Display a list of supported formats prominently on the README. Support for other GDAL-supported formats will be straightforward to implement if you choose to refactor as an API as zhampel and rmsare recommend. The metadata stored internally by HDF and others eliminates the need for a config file like the one generated by CSV_lt.py.
• Test and indicate in the README which imagery sources and formats are compatible with CometTS.
The VIIRS example is cool, but I think it represents an edge case for most VIIRS users. VIIRS is distributed in HDF for most standard products, not GeoTIFF like in the example. The documentation suggests that users of "overhead imagery" are the intended audience, but many of the relevant datasets are distributed in other formats.

Have updated the readme to address this. Any GDAL format raster is supported. Any OGR format vector file is supported. All metadata unfortunately is not created equally and no standard exists for what is contained in each file (although one is being worked on). I think an ad-hoc approach to extracting at least the date information from each raster file is still the best method and lends flexibility to an end user. CometTS will auto-extract extent and projection if it is appropriately defined in the rasters. HDF has no standard metadata format, and is really a poor choice by NOAA to use. In fact, some of their recent data has no projection or extent information embedded in each file. When you don’t have that from the start, you’re setting people up to fail. If you have experience getting these data to play nicely out of the box with a GIS, I’d love to hear about it.

• Provide examples for more data sources. It's not clear how transferable the VIIRS workflow example is to other data sources.

This I really can’t do. I’ve built the package, now it’s up to end users to run with it, and get creative with their scientific pursuits. I’ve written a full paper regarding tracking electrification rates in Puerto Rico following Hurricane Maria using CometTS (now cited in the paper). Additionally, I’ve shown separate examples with Landsat (transferable to any multi-spectral imagery with some tweaking) and using VIIRS for estimating seasonal population migrations.

Future Things
If/when the package is approved for JOSS, I intend to update the zenodo code as well as the code on pypi/conda. Want to make sure there are no more modifications or recommendations before I do this.

[rmsare]

@jshermeyer Thanks for your response. I will be able to look through your changes in more detail this weekend.

@Kevin-Mattheus-Moerman, I will add a comment here when that's done and follow up in review issues where appropriate.

[rmsare]

@Kevin-Mattheus-Moerman Looking through the changes, my major comments have been addressed by @jshermeyer and I think the notebooks and code have been made much more accessible.

Right now, I feel like the code and packaging meets the standards of JOSS, but it is up to the author to justify CometTS as a novel research tool. As @jjmcnelis asked above, could editors clarify when a workflow-oriented tool like this submission is defined as a "minor utility"?

The paper should be edited a bit for clarity (#16). In particular, it claims CometTS provides "new" functionality (averaging, cloud masking, etc.) that is available in other open source software like SITS.

[Kevin-Mattheus-Moerman]

@jjmcnelis can you review the latest changes and @jshermeyer 's response to your points.

@jjmcnelis @rmsare To answer your question yes this package is considered more than a "minor utility", and it is clear it may be valuable for scientific research applications. A typical "minor utility" is a package doing something quite trivial or simple and which does not influence scientific research and the findings much or at all. Therefore one would likely not feel the need to cite such a minor utility. In this case however the package does influence the science as it provides both visualization and processing tools and workflows.

[Kevin-Mattheus-Moerman]

@zhampel can you finalize the review on your end? Are you satisfied with the changes made? Thanks

[Kevin-Mattheus-Moerman]

@rmsare @jjmcnelis you can tick the version box (this tick box will be removed in future review issues as it often leads to confusion), the reviewed version will be set appropriately before the acceptance stage.

[jjmcnelis]

@Kevin-Mattheus-Moerman Apologies for the radio silence over the last week.

@jshermeyer I'm satisfied with the improvements to the repo. My main critiques about the README content have all been addressed:

1. Outlined basic steps for the primary use case (check - displayed prominently atop README.)
2. Supported formats (check - would've been fine with "All GDAL/OGR compatible formats are supported.", but it's nice to be reminded that I need to update my GDAL bookmarks :)
3. Replacing the details in the README with the three (?) additional time series examples better communicates your intent in developing this software. And the plots are fantastic. Clear and information rich, with concise descriptions of the phenomena captured by the data and depicted through CometTS.

Your updates to the paper help to clarify CometTS's place among the other tools that provide some similar functionality.

I apologize for not giving you more specific feedback during my initial review. Several commitments came calling at once. @rmsare Thanks for your diligence in documenting the review comments in the issues on CometTS repo!

[Kevin-Mattheus-Moerman]

@zhampel can you finalize the review on your end? Are you satisfied with the changes made? Thanks

[zhampel]

@Kevin-Mattheus-Moerman I see that the author has addressed my earlier comments as well as those of the other reviewers. The documentation has been improved, and I think users will appreciate the addition of sample data for testing purposes. The final two boxes have been checked. Thanks.

[Kevin-Mattheus-Moerman]

@zhampel thanks a lot!

[Kevin-Mattheus-Moerman]

@jjmcnelis thanks for the update too

[Kevin-Mattheus-Moerman]

@jshermeyer @rmsare all boxes are ticked. However is it correct to say some work remains related to https://github.com/CosmiQ/CometTS/issues/16 raised by @rmsare?

[Kevin-Mattheus-Moerman]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[rmsare]

@Kevin-Mattheus-Moerman @jshermeyer Yes, I think the claims in the paper need to be revised a bit to describe what's new in CometTS, and some small edits remain. Otherwise the package is in good shape, and I'd be happy to recommend an updated submission for acceptance.

[Kevin-Mattheus-Moerman]

@jshermeyer can you give an update on progress in terms of replying/dealing with @rmsare comments? It looks like we are nearly there.

[jshermeyer]

Hi Kevin,

@Kevin-Mattheus-Moerman https://github.com/Kevin-Mattheus-Moerman Updated
the text to make it seem as if "Cloud and snow masking" were not new
functionality. If there is any further specific changes required for the
text let me know, otherwise I am satisfied with it.

-Jake

On Sat, Aug 24, 2019 at 5:30 AM Kevin Mattheus Moerman <
[email protected]> wrote:

@jshermeyer https://github.com/jshermeyer can you give an update on
progress in terms of replying/dealing with @rmsare
https://github.com/rmsare comments? It looks like we are nearly there.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openjournals/joss-reviews/issues/1047?email_source=notifications&email_token=AGNX2AOT2JWN55T3PTTYNULQGD5TLA5CNFSM4F672LN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5B4QTQ#issuecomment-524535886,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGNX2AI72FIL23AHRZAM5KDQGD5TLANCNFSM4F672LNQ
.

[rmsare]

@jshermeyer, Thanks for making that change. It's nearly there, but some small edits are required for clarity and a typo.

I've noted the three changes to be made in a comment to this issue.

[Kevin-Mattheus-Moerman]

@jshermeyer :point_up: see above. It looks like we are close. Can you work on these minor sounding issues?

[Kevin-Mattheus-Moerman]

Thanks @zhampel for your contribution. I've just unticked the "conflict of interest" box for you since you indicated to me in an email that "Jake Shermeyer, is a member of a sister lab of [your] organization". I'm reiterating that here for the record.

[rmsare]

My last comments were addressed, and I am comfortable recommending the paper be accepted @Kevin-Mattheus-Moerman. Thanks for your patience, and congratulations @jshermeyer

[zhampel]

@Kevin-Mattheus-Moerman When I brought this up, I was informed that there was in fact no conflict of interest and thus not a problem during the first round of reviews, hence the box was checked.

[danielskatz]

Let's go back and have this box be checked - there is a conflict here, but we've recorded it and waived it for the purpose of this review. (We probably need to rephrase this criteria slightly, which I will bring up outside this review thread.)

[Kevin-Mattheus-Moerman]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[Kevin-Mattheus-Moerman]

@whedon check references

[whedon]

Attempting to check references...

[whedon]

```Reference check summary:

OK DOIs

• 10.5281/zenodo.267110 is OK
• 10.5281/zenodo.1324419 is OK
• 10.1117/12.2325585 is OK
• 10.1016/j.cageo.2004.05.006 is OK
• 10.5281/zenodo.1974065 is OK

MISSING DOIs

• None

INVALID DOIs

• None
  ```

[Kevin-Mattheus-Moerman]

@jshermeyer looks good. We are ready for the next steps. Can you please:

• [ ] Can you deposit the release (version) of the reviewed software in Zenodo? Once you have archived the software there please report the DOI of the archived version here. Please ensure the authors and title on the Zenodo archive match those of the paper.
• [ ] Can you list the version number? (is it still v1.0 or has it changed)?

thanks

[Kevin-Mattheus-Moerman]

@jshermeyer :wave: let me know if you have questions about depositing and archived version on Zenodo. Thanks

[jshermeyer]

New version on Zenodo: https://zenodo.org/record/3418091
Version number is 1.2.0 which incorporates all the edits and comments from reviewer. This version has also been pushed to pypi.

Thanks!

[kthyng]

Hi @jshermeyer! I see that the title in this submission and the zenodo archive just barely don't match... can you update the Zenodo archive to match exactly?

[jshermeyer]

@kthyng Good catch, fixed.

[kthyng]

@whedon set v1.2.0 as version

[whedon]

OK. is the version.

[kthyng]

@whedon set 10.5281/zenodo.3418091 as archive

[whedon]

OK. 10.5281/zenodo.3418091 is the archive.

[kthyng]

@jshermeyer:
Edits for paper:

• [ ] Spacing issues throughout. Make sure there are spaces next to parentheses, this is coming up around references mostly but other times too.
• [ ] ROI is used before being defined
• [ ] obligatory spelled wrong first page 2nd paragraph
• [ ] Figure 3 caption appears to be set up incorrectly.
• [ ] The line above Figure 3: What is this? Is it trying to link to its own DOI? That is out of date and shouldn't be in the references anyway, it is on the left side of the page on the first page of the paper.

[kthyng]

Another edit, @jshermeyer: your citation e-sensing/sits:2018 is not formatted correctly. In particular at least the authors in the author list need to be separated by "and" as in your other entries. There may be other problems too, but you can see in your reference list and when you cite it inline that it is not working correctly.

[jshermeyer]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[jshermeyer]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[jshermeyer]

@kthyng I think I have fixed all of the issues you flagged (among a few others). Let me know if there's anything else you notice. Thanks!

[kthyng]

Another edit, @jshermeyer: your citation e-sensing/sits:2018 is not formatted correctly. In particular at least the authors in the author list need to be separated by "and" as in your other entries. There may be other problems too, but you can see in your reference list and when you cite it inline that it is not working correctly.

@jshermeyer This doesn't appear to have changed yet.

[jshermeyer]

Please send me the appropriate format exactly, I added an “and” between the second to last and final author. I’m not sure what you’re looking for.

On Sep 26, 2019, at 12:45 PM, Kristen Thyng [e-mail] wrote:

Another edit, @jshermeyer: your citation e-sensing/sits:2018 is not formatted correctly. In particular at least the authors in the author list need to be separated by "and" as in your other entries. There may be other problems too, but you can see in your reference list and when you cite it inline that it is not working correctly.

@jshermeyer This doesn't appear to have changed yet.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[kthyng]

Here under "example paper.bib", the first example paper shows how multiple authors (each author) should be separated by "and" instead of a semi-colon. You can check to make sure it worked properly and there aren't other errors by how it looks when cited (shouldn't take up a whole line in the paper) and how it looks in the references (some of the authors are being abbreviated to a single letter I think?).

[jshermeyer]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[jshermeyer]

@kthyng thanks, looks like that worked

[kthyng]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

```Reference check summary:

OK DOIs

• 10.5281/zenodo.267110 is OK
• 10.1117/12.2325585 is OK
• 10.1016/j.cageo.2004.05.006 is OK
• 10.5281/zenodo.1974065 is OK

MISSING DOIs

• None

INVALID DOIs

• None
  ```

[whedon]

Check final proof :point_right: https://github.com/openjournals/joss-papers/pull/992

If the paper PDF and Crossref deposit XML look good in https://github.com/openjournals/joss-papers/pull/992, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.
@whedon accept deposit=true

[kthyng]

@whedon accept deposit=true

[whedon]

Doing it live! Attempting automated processing of paper acceptance...

[whedon]

🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦

[whedon]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

1. Check final PDF and Crossref metadata that was deposited :point_right: https://github.com/openjournals/joss-papers/pull/993
2. Wait a couple of minutes to verify that the paper DOI resolves https://doi.org/10.21105/joss.01047
3. If everything looks good, then close this review issue.

4. Party like you just published a paper! 🎉🌈🦄💃👻🤘

   Any issues? notify your editorial technical team...

[kthyng]

Woohoo congratulations @jshermeyer!!! Thanks so much to @zhampel, @rmsare, @jjmcnelis for reviewing, and to @Kevin-Mattheus-Moerman for editing!

[whedon]

:tada::tada::tada: Congratulations on your paper acceptance! :tada::tada::tada:

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.01047/status.svg)](https://doi.org/10.21105/joss.01047)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.01047">
  <img src="https://joss.theoj.org/papers/10.21105/joss.01047/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.01047/status.svg
   :target: https://doi.org/10.21105/joss.01047

This is how it will look in your documentation:

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://numfocus.salsalabs.org/donate-to-joss

[jshermeyer]

Thanks all @zhampel, @rmsare, @jjmcnelis, @Kevin-Mattheus-Moerman, @kthyng !

This is presently giving me a 404:
https://joss.theoj.org/papers/10.21105/joss.01047

Assuming it will auto-update given a bit of time?

[kthyng]

Weird! It is not giving me a 404. Is it still?

[jshermeyer]

Working on another browser, all good! I must just have to clear my cookies or something.