Joss-reviews: [REVIEW]: AutoFunc: A Python package for automating and verifying functional modeling

[whedon]

Submitting author: @AlexMikes (Alex Mikes)
Repository: https://github.com/AlexMikes/AutoFunc
Version: v0.2.1
Editor: @gkthiruvathukal
Reviewers: @cmccomb, @e-dub, @srmnitc
Archive: Pending

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/14f9eb408a9cb91c28332fcac1bce505"><img src="https://joss.theoj.org/papers/14f9eb408a9cb91c28332fcac1bce505/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/14f9eb408a9cb91c28332fcac1bce505/status.svg)](https://joss.theoj.org/papers/14f9eb408a9cb91c28332fcac1bce505)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@cmccomb & @e-dub, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @gkthiruvathukal know.

✨ Please try and complete your review in the next six weeks ✨

Review checklist for @cmccomb

Conflict of interest

• [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Contribution and authorship: Has the submitting author (@AlexMikes) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
• [x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• [x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @e-dub

Conflict of interest

• [ ] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [ ] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [ ] Repository: Is the source code for this software available at the repository url?
• [ ] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [ ] Contribution and authorship: Has the submitting author (@AlexMikes) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• [ ] Installation: Does installation proceed as outlined in the documentation?
• [ ] Functionality: Have the functional claims of the software been confirmed?
• [ ] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [ ] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [ ] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [ ] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [ ] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [ ] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• [ ] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [ ] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [ ] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [ ] State of the field: Do the authors describe how this software compares to other commonly-used packages?
• [ ] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• [ ] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @srmnitc

Conflict of interest

• [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Contribution and authorship: Has the submitting author (@AlexMikes) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• [ ] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [ ] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [ ] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [ ] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
• [x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• [ ] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @cmccomb, @e-dub it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Reference check summary:

OK DOIs

- 10.1115/1.1625400 is OK
- 10.1115/1.1289637 is OK
- 10.5281/zenodo.3694300 is OK

MISSING DOIs

- https://doi.org/10.1007/s00163-001-0008-3 may be missing for title: A functional basis for engineering design: Reconciling and evolving previous efforts

INVALID DOIs

- None

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[cmccomb]

@AlexMikes will you please enable issues in your repository? Once you've done so I'll put my review notes in as issues. For the time being here they are:

1. Contribution and authorship: four authors are listed on the work, but only one author has made commits in the software. How did the other authors contribute to this software?
2. Example Usage and Functionality Documentation: There are a number of (really cool!) example scripts, but the functionality in these is largely not demonstrated or discussed in the README. Could you add a list of example scripts as a way of highlighting and documenting functionality?
3. Community guidelines: no guidance on how to report issues on problems with the software, or to seek support.

[AlexMikes]

@cmccomb Thanks for being a reviewer and taking the time to give great feedback! I just enabled issues, should I reply here or after you've put your notes in the issue?

I'll work on adding documentation for the example files and community guidelines in the README.

[cmccomb]

@AlexMikes will you please enable issues in your repository? Once you've done so I'll put my review notes in as issues. For the time being here they are:

1. Contribution and authorship: four authors are listed on the work, but only one author has made commits in the software. How did the other authors contribute to this software?
2. Example Usage and Functionality Documentation: There are a number of (really cool!) example scripts, but the functionality in these is largely not demonstrated or discussed in the README. Could you add a list of example scripts as a way of highlighting and documenting functionality?
3. Community guidelines: no guidance on how to report issues on problems with the software, or to seek support.

@AlexMikes its my pleasure! Really cool piece of software. I've added the notes as issues in your repo. I believe the preference is to handle discussion there.

1. https://github.com/AlexMikes/AutoFunc/issues/1
2. https://github.com/AlexMikes/AutoFunc/issues/2
3. https://github.com/AlexMikes/AutoFunc/issues/3

[AlexMikes]

Thank you @cmccomb for the excellent suggestions. I have completed these and responded to the issues individually.

[cmccomb]

Thank you @cmccomb for the excellent suggestions. I have completed these and responded to the issues individually.

Thanks @AlexMikes! My review is complete, and I recommend accept, @gkthiruvathukal .

[gkthiruvathukal]

@cmccomb, This is great news.

@e-dub, Will you be able to complete your evaluation soon?

[e-dub]

@cmccomb, This is great news.

@e-dub, Will you be able to complete your evaluation soon?

I would like to start my review but am unable to edit the review checklist.
@gkthiruvathukal: could you check this for me?

[gkthiruvathukal]

@openjournals/joss-eics @e-dub is having trouble to edit the review checklist (and shows as a proper reviewer). I think we've seen this happen in the past but don't recall exactly how we resolved it.

[gkthiruvathukal]

@e-dub You can still start working on identifying issues and opening issues in the submissions project repo and linking those issues here.

[kthyng]

@whedon re-invite @e-dub as reviewer

[whedon]

OK, the reviewer has been re-invited.

@e-dub please accept the invite by clicking this link: https://github.com/openjournals/joss-reviews/invitations

[kthyng]

@e-dub See if this helps, maybe you didn't accept the invitation to review previously.

[gkthiruvathukal]

@kthyng Thanks for helping to sort this out!

@e-dub Can you let us know when you are able to complete your review?

[gkthiruvathukal]

@e-dub Just a gentle nudge to find out when we can expect your review? I know this is a busy time of year for many, including me.

[gkthiruvathukal]

@e-dub ?

[kthyng]

@gkthiruvathukal Unless you can track down an email address for @e-dub to try another contact venue, this may be a good time to find a replacement reviewer.

[AlexMikes]

Thanks, @gkthiruvathukal and @kthyng

Let me know if I can do anything to help move this along

[gkthiruvathukal]

@kthyng Yes, I'll try to get someone else before the weekend. I've been dealing with a rather intense phase at work. I almost went into "out of office" mode as I thought I could work through this.

@AlexMikes What could help is to suggest more reviewers. I think I've tried all of your initial suggestions. Let me know if you have any others in mind from our reviewer list.

[AlexMikes]

I suggest @trallard as an additional reviewer

[gkthiruvathukal]

Hi @trallard, Are you willing to contribute a review for this JOSS submission?

[kthyng]

@gkthiruvathukal yeah sorry :( Don't want to bug people but yet here I am doing so!

@AlexMikes more suggestions == better :)

[trallard]

Hi @gkthiruvathukal at the moment I do not have the bandwidth to contribute to this. I have other reviews that I need to move forward.

[gkthiruvathukal]

Thanks, @trallard. I know the feeling. The work just isn't stopping now.

@AlexMikes if you can suggest a couple more, it would help me right now. Thanks!

[AlexMikes]

Thanks @trallard. I suggest @moorepants and @galessiorob as potential reviewers

[arfon]

@gkthiruvathukal if at all possible we should avoid asking JOSS editors to review (@trallard and @galessiorob) as they're already busy with other JOSS work.

[moorepants]

I'm currently reviewing another JOSS paper. Once that is done I can likely take on future ones. So I decline.

[gkthiruvathukal]

@arfon I agree 100%. I'll try the other suggestion.

[gkthiruvathukal]

It looks like @galessiorob is also a member of joss-editors. So we might need a few more suggestions, @AlexMikes. We'll get there soon, I hope!

[AlexMikes]

Ok, how about @nnadeau or @srmnitc?

[gkthiruvathukal]

@nnadeau and @srmnitc, are either of you willing to serve as a reviewer for this JOSS submission? Would really appreciate your help!

[nnadeau]

@gkthiruvathukal unfortunately, I do not have time right now :(

[srmnitc]

@gkthiruvathukal @AlexMikes this module looks interesting and I could find some time to help with the review. However, before I start I would like to actually comment on a major issue. It is great that the software has already been used in publications, however, I do not find any documentation except what is on the README file. I really find this very limiting and would be one blocking issue that I would raise!

[AlexMikes]

Thanks, @srmnitc!

I agree that documentation is important. Every module has NumPy formatted docstrings that explain the functionality and all functions have docstrings that explain the functionality as well as the parameters with their expected type and purpose in the function and the returns with their type and explanation. What else would you suggest for documentation?

[srmnitc]

@AlexMikes Yes, the docstrings do look very well written! I was thinking more about using readthedocs or github pages to host the documentation. Readthedocs then would build the docs including docstrings. It would also allow you to have more details about installing, tests, examples, and so on. Although not required, you could also have interactive examples which can be set up using mybinder. I would be happy to raise some issues in detail if I am the reviewer.

[gkthiruvathukal]

@srmnitc I am interpreting your last comment as being willing to serve as a reviewer. Can I add you so we can get started?

[srmnitc]

@gkthiruvathukal Yes, sorry about the confusion. I am happy to be reviewer.

[gkthiruvathukal]

@whedon add @srmnitc as reviewer

[whedon]

OK, @srmnitc is now a reviewer

[gkthiruvathukal]

Thanks, @srmnitc. You should be all set.

[srmnitc]

@gkthiruvathukal Sorry to bother, but I do not seem to have a checklist. Could you please help with this?

[AlexMikes]

@AlexMikes Yes, the docstrings do look very well written! I was thinking more about using readthedocs or github pages to host the documentation. Readthedocs then would build the docs including docstrings. It would also allow you to have more details about installing, tests, examples, and so on. Although not required, you could also have interactive examples which can be set up using mybinder. I would be happy to raise some issues in detail if I am the reviewer.

Thanks @srmnitc, yes please open this as an issue along with anything else you find.

[danielskatz]

@gkthiruvathukal Sorry to bother, but I do not seem to have a checklist. Could you please help with this?

added

[srmnitc]

@gkthiruvathukal I have finished my first round of review and opened some issues in the repository. At the moment, it is not possible for me to check many of the boxes, but that should be possible once the issues are cleared. Overall, it seems like a well-written and useful module.

However, I have to raise some concerns regarding the scholarly effort. The module itself is fairly small, with about 1500 lines or so including the example scripts and test scripts. But the author mentions in the accompanying paper that the module has been used in publications already, which is of course wonderful and warrants publication. The DOIs for these publications are missing at the moment, once those are available I would be happy to go through them.

[AlexMikes]

@gkthiruvathukal When I went to look at the issues from @srmnitc, I found some issues from @e-dub from September 18th. So even though he does not seem to be responding here, he still opened some issues. Has he been removed as a reviewer for a lack of communication? Most of the issues he opened are reflected in the issues from @srmnitc, can they be removed since he is not a reviewer?

[gkthiruvathukal]

@AlexMikes Yes, but I'd make one amendment to this. In software engineering, we generally don't delete issues. We simply mark them as resolved. What you could do is make a comment for each of these issues. Since they are duplicates (or near duplicates), it should be easy to link to the other issue in the comments field and mark it closed. It is my understanding we don't usually remove a reviewer, especially if they did some work toward the review but perhaps experienced challenges that prevented them from finishing it. With COVID-19, many have found themselves unable to finish work. It is even taking me a lot longer to bring submissions to proper closure, so I also want to thank you for being so great during the entire process!

Once you've worked through all the issues and the two active reviewers sign off, we'll be able to move toward acceptance.

[AlexMikes]

Thanks, @gkthiruvathukal. I commented on each issue, linked them to a similar issue from @srmnitc (if applicable), and closed them. They are linked to your previous post if you want to check them out.

[AlexMikes]

Thanks, @srmnitc for your thoughtful and thorough review! You had great suggestions and insight that helped improve the quality of this software package. I made comments on each issue that you opened. There is one specific issue (Missing DOIs) that I would like your feedback on how you suggest I proceed. The rest of them I hope to have addressed sufficiently, but please comment where appropriate if you suggest further changes.

[srmnitc]

@AlexMikes Thanks for your work on the issues. I have been able to check some, but others still remain unchecked. The reasons have been updated in the corresponding issues.

@gkthiruvathukal Thanks for the chance to review this submission. As I have mentioned in a previous comment, there are still two main issues that concern me.

• I am unable to verify the scholarly effort in this work as the mentioned references are without DOIs (see here). From the perspective of the code, it is rather a short module, as such I still remain unable to review the scholarly effort.

• Although I am able to check some of the review boxes, the overall quality of documentation, tests are not comparable to the recent JOSS publications. This lack of documentation, in my opinion, severely limits the potential usage. The author has also mentioned that there will be no further work in this direction. As such no further improvements or maintenance to the code is immediately visible.

Because of these reasons, I cannot recommend the acceptance in JOSS at the moment.

[gkthiruvathukal]

@srmnitc I have followed up on the DOI issue in the relevant thread. This appear to be a resolvable issue.

@AlexMikes Can you comment on the documentation and testing issue and how you plan to address it?