Joss-reviews: [REVIEW]: SALib: An open-source Python library for Sensitivity Analysis

[whedon]

Submitting author: jdherman (Jonathan Herman)
Repository: https://github.com/SALib/SALib
Version: v1.0.0
Editor: @kyleniemeyer
Reviewer: @stijnvanhoey
Archive: 10.5281/zenodo.233103

Status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/431262803744581c1d4b6a95892d3343"><img src="http://joss.theoj.org/papers/431262803744581c1d4b6a95892d3343/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/431262803744581c1d4b6a95892d3343/status.svg)](http://joss.theoj.org/papers/431262803744581c1d4b6a95892d3343)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer questions

Conflict of interest

• [x] As the reviewer I confirm that there are no conflicts of interest for me to review this work (such as being a major contributor to the software).

  General checks

• [x] Repository: Is the source code for this software available at the repository url?

• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Version: Does the release version given match the GitHub release (v1.0.0)?

• [x] Authorship: Has the submitting author (jdherman) made major contributions to the software?

  Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?

• [x] Functionality: Have the functional claims of the software been confirmed?

• [x] Performance: Have any performance claims of the software been confirmed?

  Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?

• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g. API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?

• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

  Software paper

• [x] Authors: Does the paper.md file include a list of authors with their affiliations?

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] References: Do all archival references that should have a DOI list one (e.g. papers, datasets, software)?

[whedon]

Hello human, I'm @whedon. I'm here to help you with some common editorial tasks for JOSS.

For a list of things I can do to help you, just type:

@whedon commands

[kyleniemeyer]

@whedon commands

[whedon]

Here are some things you can ask me to do:

# List all of Whedon's capabilities
@whedon commands

# Assign a GitHub user as the reviewer of this submission
@whedon assign @username as reviewer

# List the GitHub usernames of the JOSS editors
@whedon list editors

# List of JOSS reviewers together with programming language preferences and domain expertise
@whedon list reviewers

# Change editorial assignment
@whedon assign @username as editor

# Set the software archive DOI at the top of the issue e.g.
@whedon set 10.0000/zenodo.00000 as archive

# Open the review issue
@whedon start review

:construction: Important :construction:

This is all quite new. Please make sure you check the top of the issue after running a @whedon command (you might also need to refresh the page to see the issue update).

[kyleniemeyer]

Hi @quaquel I can't seem to assign you, but here's the review issue.

[stijnvanhoey]

Installation checked for both python 2.7 and 3.5. Able to run the minimal example in both cases.

[quaquel]

on what OS did you check it? I have acces macOS, windows 7, windows 10, and if necessary windows server 2012.

[stijnvanhoey]

Installed on Ubuntu 16.04 within conda environments.

[kyleniemeyer]

Hi @stijnvanhoey @quaquel, any updates on your reviews? Thanks!

[arfon]

I forgot to update this thread with the compiled paper, here it is: 10.21105.joss.00097.pdf

[quaquel]

I have successfully installed it on both mac and windows, under both python 2.7 and python 3.5.

[quaquel]

I have used the various sampling techniques successfully. The claimed functionality is thus working.

[kyleniemeyer]

Thanks @quaquel! Could you check off the remaining reviewer questions at the top? Please note that in addition to the functionality, there are questions about the documentation and paper (which @arfon shared a link to a few comments ago).

@stijnvanhoey your comments are still also welcome!

[stijnvanhoey]

In general, I would like to congratulate the authors on the current work. This library is nicely structured, well documented and easy to setup. The issues raised require minor adaptations.

Some comments:

• As no statement is made about performance, this is already checked as such. Still, the current contributing guidelines information is giving an advice about potential benefits by vectorizing functions, I suggest to make an issue from this, cfr. my issue https://github.com/SALib/SALib/issues/116
• With regard to the community guidelines and current setup of the paper, I'm wondering if the descriptions and package could not be adapted in order to formalize and simplify the potential extension with new methods (what is the workflow, what kind of tests,..). Also, the current paper is focusing on the 6 currently implemented methods, but when this would increase, this kind of information will be outdated (as well as the paper). How does this work? Should the paper be updated at that moment?

[kyleniemeyer]

Thanks @stijnvanhoey—both to you and @quaquel, what about the "Example usage" question above?

@jdherman: it looks like some issues/concerns have been raised regarding the testing, documentation, and community guidelines. Could you resolve/respond to these?

[jdherman]

Thanks @stijnvanhoey and @kyleniemeyer. I'm at a conference right now but I will resolve the issues as soon as possible (hopefully next week).

[kyleniemeyer]

Hi @jdherman, just wanted to ping you on these changes (happy new year!). (That said there's no deadline or anything—the paper will remain "under review" indefinitely until you make the changes.)

[jdherman]

Hi @kyleniemeyer and @stijnvanhoey -- we have updated the contributing guidelines, docs, and testing. Thanks for the suggestions, and please see what you think when you have time. Here is the pull request (now merged):
https://github.com/SALib/SALib/pull/120

[kyleniemeyer]

Thanks @jdherman!

@stijnvanhoey could you take a look at the changes, and let me know if they satisfy your outstanding issues?

[stijnvanhoey]

Thanks @jdherman and @willu47 for the work and revisions of the guidelines and documentation. I do like the scope and structure of the package. Tests and examples are available for the user as well. All issues are closed and checked, so I'm perfectly fine with publication.

Just some minor remark, the link to the advanced readme on the github page does not work: http://salib.github.io/SALib/README-advanced.md

[willu47]

Thank you @stijnvanhoey for the comments. It has been a productive (and enjoyable!) process to go through.

Regarding the broken links, I also spotted these yesterday and have made a new issue to prompt a fix. I'll get around to tidying up the page asap.

Thanks again. 👍

[kyleniemeyer]

@stijnvanhoey 🚀

@willu47 sounds great! It seems this is the only outstanding issue, so when you've made that change please let us know. Also create an archive for the software, and report it here—then we'll be done!

[jdherman]

Thanks again @kyleniemeyer and @stijnvanhoey , this has helped a lot.

I fixed the broken links on the landing page (see here) as well as a few other bugs.

This is all wrapped up in a new version number, v1.0.2, which is also updated on PyPI.

[kyleniemeyer]

@jdherman ⚡️ Can you give a new DOI associated with this version? That's the last bit we need.

[jdherman]

It's still in the process of being activated, but according to Zenodo it looks like it will be: 10.5281/zenodo.233103

[kyleniemeyer]

Thanks @jdherman, I'll hold off closing this until the DOI is available via https://doi.org/10.5281/zenodo.233103 (it still isn't resolving for me)

[jdherman]

Ok thanks. I'm not sure how long it should take. @willu47 can you check if I did something wrong? It looks like the new DOI was created for the release:
https://zenodo.org/account/settings/github/repository/SALib/SALib

but still doesn't resolve after 4 hours. How long does it usually take?

[kyleniemeyer]

Yeah, that's a bit strange- when I upload something to Zenodo and it gives me a DOI, it generally resolves pretty much immediately, I think.

That second link is not publicly viewable, although something that has been published/released on Zenodo will have a URL there in addition to the DOI (what the DOI resolves to). Perhaps the release hasn't actually been published, and is just a draft?

[jdherman]

Hm still not resolving. This is a screenshot of the zenodo page, looks like it's published

Maybe it just takes more time. Sorry about this!

[arfon]

Hm still not resolving. This is a screenshot of the zenodo page, looks like it's published

My guess is that this is an issue with Zenodo. @jdherman - might be worth emailing the Zenodo folks if this isn't fixed tomorrow.

[willu47]

That's frustrating. I have sent Zenodo support an e-mail to check up on the DOI. It should point to this Zenodo entry but there seems to be something wrong with Zenodo as @arfon suggested. Fingers crossed it will get fixed asap. I have checked and rechecked @jdherman & everything seems fine.

[willu47]

Okay, DOI LIVE! 🚀 🎸 . I received a quick reply back from Zenodo and they manually created the doi link. There was some kind of one-off error which has now been resolved.

[kyleniemeyer]

@whedon set 10.5281/zenodo.233103 as archive

[whedon]

OK. 10.5281/zenodo.233103 is the archive.

[kyleniemeyer]

Alright @arfon good to go! Great job @willu47 and @jdherman, thanks for figuring that out.

[jdherman]

Great, thanks Will!!

[arfon]

Thanks for your review @stijnvanhoey and for editing this one @kyleniemeyer ✨

@jdherman - your paper is now accepted into JOSS and your DOI is http://dx.doi.org/10.21105/joss.00097 ⚡️ 🚀 💥