Joss-reviews: [REVIEW]: emcee v3: A Python ensemble sampling toolkit for affine-invariant MCMC

[whedon]

Submitting author: @dfm (Daniel Foreman-Mackey)
Repository: https://github.com/dfm/emcee
Version: v3.0.2
Editor: @xuanxu
Reviewer: @benjaminrose, @mattpitkin
Archive: 10.5281/zenodo.3543502

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/d759becba430a700633694389bd021a4"><img src="https://joss.theoj.org/papers/d759becba430a700633694389bd021a4/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/d759becba430a700633694389bd021a4/status.svg)](https://joss.theoj.org/papers/d759becba430a700633694389bd021a4)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@benjaminrose & @mattpitkin, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @xuanxu know.

✨ Please try and complete your review in the next two weeks ✨

Review checklist for @benjaminrose

Conflict of interest

• [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Contribution and authorship: Has the submitting author (@dfm) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
• [x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• [x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @mattpitkin

Conflict of interest

• [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Contribution and authorship: Has the submitting author (@dfm) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
• [x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• [x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @benjaminrose, @mattpitkin it looks like you're currently assigned to review this paper :tada:.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[xuanxu]

@dfm @mattpitkin, @benjaminrose : this is the review thread for the paper. All of our communications will happen here from now on.

Both reviewers have checklists at the top of this thread with the JOSS requirements. As you go over the submission, please check any items that you feel have been satisfied. There are also links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention openjournals/joss-reviews#1864 so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for reviews to be completed within about 2-4 weeks. Please let me know if any of you require some more time. We can also use Whedon (our bot) to set automatic reminders if you know you'll be away for a known period of time.

Please feel free to ping me (@xuanxu) if you have any questions/concerns.

[benjaminrose]

@dfm, it looks via conda-forge emcee is only at version 2.2.1. Do you have a plan to update conda-forge to version 3?

[dfm]

@benjaminrose: really? It looks like 3.0.0 to me: https://anaconda.org/conda-forge/emcee and that's what I get with:

conda install -c conda-forge emcee

Is that not what you get?

[benjaminrose]

@dfm After further testing, it appears to be some conflict with Python 3.8 and conda. When I switched to an environment running Python 3.7.3 (or install via pip) everything works as expected. This is odd, but appears to not be an issue with emcee.

[dfm]

Ah yes - I think that conda-forge is just getting up to speed on 3.8. Should be there soon! Thanks!

[benjaminrose]

Overall, this is a great software package and I am eager to see it published in JOSS. I do have a few things to discuss before the paper is finalized.

• Authors - I fully support the full list of possible authors on https://github.com/dfm/emcee/issues/316. There has been plenty of time for maintainers to request authorship so I am ok with publishing as is or including anyone from that issue.
  
  
  
  • But to be forthright, I work at the same institution as some of the potential co-authors. After reading the JOSS COI policy, I don't think that is an issue nor do I feel unable to make an impartial evaluation of emcee. I just wanted to be candid.
  
  
• Performance - The new parallelization examples are great, but I was unable to get MPI to work on my machine.
• Docs
  
  
  
  • Installation - Installation, as controllable by this project, is good.
  
  
  • API documentation - The deprecated parameters are documented as such at runtime. Is there a way to add something like "all other parameters are deprecated" in the online docs?
  
  
  • Community Guidlines - Contributing.md is great. In the documents, there is just a one sentance about using GitHub issues. I think it would be helpful to add a link to both the users mailing list and the Contribuing.md file in the offical docs.
  
  

I think this is a great project and once we finalize the authors and the best way to inform users about contributing, I will be happy to endurance this paper for publication.

[dfm]

@benjaminrose: Thanks for this feedback! Do you think that the MPI issues were a problem with emcee/schwimmbad or something else? If the former, please open an issue and let's get it figured out!

I've updated the docs to include some words about deprecations and a bit more info about contributing.

As for authors: the current master branch version of the manuscript has all of the authors from the issue, but I'll make one more call for co-authorship requests before finalizing the list if that's ok!

Let me know if there's anything else and thanks again!

[benjaminrose]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[benjaminrose]

@dfm these changes look great! I think my MPI issues stem mostly from my lack of experience and not an issue with emcee.

[mattpitkin]

Sorry for my delay. I'll be able to complete my review by the end of the week.

[mattpitkin]

Hi @dfm, would you be able to update the references to include more DOIs and a link to the kombine github page or doc page? I think the following are correct:

• Goodman & Weare 10.2140/camcos.2010.5.65
• Ter Braak 10.1007/s11222-006-8769-1
• Ter Braak & Vrught 10.1007/s11222-008-9104-9

[mattpitkin]

Installation works well, but the conda install gives v3.0.0, whereas pip gives v3.0.1. I'm just curious as to why the conda-forge feedstock is not autoupdating to find the latest PyPI version?

[mattpitkin]

The documentation for emcee is excellent, and is a great resource. All the examples work exactly as expected (I know it's to be expected, but setting the Numpy random seed means I get spookily identical plots when running the example codes). Just a couple of points, one bigger than the other:

• minor point: in the "What about shorter chains?" section it's worth noting that the example requires having autograd installed.
• slightly bigger point: could an example be added that shows using a different Move. Maybe something where the walk move, or differential evolution, given shorter autocorrelation times than the stretch move?

[dfm]

@mattpitkin: Thanks! I've added those DOIs and a link for kombine. I also merged the pull request to conda-forge so it'll be up to date shortly.

[dfm]

Those points are also both good. I'll add those too. Thanks!

[mattpitkin]

Could you add a link to the CONTRIBUTING.md file in the "How to Use This Guide" section of the docs?

[dfm]

Could you add a link to the CONTRIBUTING.md file in the "How to Use This Guide" section of the docs?

It is in the introduction of the latest version of the docs: https://emcee.readthedocs.io/en/latest/, but adding it there would also be good.

[mattpitkin]

Could you add a link to the CONTRIBUTING.md file in the "How to Use This Guide" section of the docs?

It is in the introduction of the latest version of the docs: https://emcee.readthedocs.io/en/latest/, but adding it there would also be good.

Ah, yes. Very good. I was just looking at the stable version.

[mattpitkin]

When the Moves example is added I'm happy to sign off the review. All boxes are ticked.

[mattpitkin]

I was just trying to run the tests (based on emcee-3.0.2.dev21+gba05c7f), after installing from source, and I get a range of errors starting:

(emcee_pip) matthew@work:~/repositories/emcee$ pytest -v src/emcee/tests
============================= test session starts ==============================
platform linux -- Python 3.7.5, pytest-5.2.2, py-1.8.0, pluggy-0.13.0 -- /home/matthew/miniconda2/envs/emcee_pip/bin/python
cachedir: .pytest_cache
rootdir: /home/matthew/repositories/emcee
collected 0 items / 14 errors                                                  

==================================== ERRORS ====================================
___________ ERROR collecting src/emcee/tests/integration/test_de.py ____________
ImportError while importing test module '/home/matthew/repositories/emcee/src/emcee/tests/integration/test_de.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
src/emcee/__init__.py:22: in <module>
    from .emcee_version import __version__  # isort:skip
E   ModuleNotFoundError: No module named 'emcee.emcee_version'

Have you come across this before?

[mattpitkin]

It looks like an emcee_version.py file is expected in src/emcee, but isn't present. Should this be getting created during the setup?

[dfm]

Yes - there was a typo in the source install docs! I just fixed it, but the basic change is that you need to run:

pip install -e .

or

python setup.py develop

so that that file is written in place. Good catch!

[mattpitkin]

Great, it's worked now and all tests pass :smile: :tada:

[dfm]

@mattpitkin: I've added the note about autograd and a tutorial demonstrating the moves interface. Let me know if you have any comments or questions!

[mattpitkin]

@mattpitkin: I've added the note about autograd and a tutorial demonstrating the moves interface. Let me know if you have any comments or questions!

Great, that looks very good to me.

@xuanxu - I'm happy to sign-off the review.

[xuanxu]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[xuanxu]

@whedon check references

[whedon]

Attempting to check references...

[whedon]

```Reference check summary:

OK DOIs

• 10.1007/s11222-008-9104-9 is OK
• 10.1007/s11222-006-8769-1 is OK
• 10.2140/camcos.2010.5.65 is OK
• 10.1086/670067 is OK

MISSING DOIs

• None

INVALID DOIs

• None
  ```

[xuanxu]

OK @dfm, everything looks good, here are the next steps:

• Please release a new version from the current master so it includes all the changes made during the review process
• Then archive that latest release in Zenodo
• Check the Zenodo deposit has the correct metadata: title and author name should match the paper; you may also add your ORCID.

Once you do that please report here the Zenodo DOI.

[dfm]

Thanks all!!

The new DOI is: 10.5281/zenodo.3543502

I've also pushed the update to PyPI and it'll be on conda-forge in the next few days.

[xuanxu]

@whedon set v3.0.2 as version

[whedon]

OK. v3.0.2 is the version.

[xuanxu]

@whedon set 10.5281/zenodo.3543502 as archive

[whedon]

OK. 10.5281/zenodo.3543502 is the archive.

[xuanxu]

Everything looks ready.
Pinging @openjournals/joss-eics for final processing

[labarba]

@dfm -- A few small items to address:

1. Originally, emcee was implementation >> an implementation (missing article), or better yet "implemented" (verb form rather than nominalization)
2. Foreman-Mackey, et al. (2013) is missing a journal title
3. Both Ter Braak references need caps protection {} in the article title

[dfm]

@labarba: Thanks! Good catches. I've updated the manuscript and BibTeX. Let me know if you have any other suggestions. Should I make another release with these changes?

[dfm]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[labarba]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

```Reference check summary:

OK DOIs

• 10.1007/s11222-008-9104-9 is OK
• 10.1007/s11222-006-8769-1 is OK
• 10.2140/camcos.2010.5.65 is OK
• 10.1086/670067 is OK

MISSING DOIs

• None

INVALID DOIs

• None
  ```

[whedon]

Check final proof :point_right: https://github.com/openjournals/joss-papers/pull/1105

If the paper PDF and Crossref deposit XML look good in https://github.com/openjournals/joss-papers/pull/1105, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.
@whedon accept deposit=true

[labarba]

@whedon accept deposit=true

[whedon]

Doing it live! Attempting automated processing of paper acceptance...

[whedon]

🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦

[whedon]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

1. Check final PDF and Crossref metadata that was deposited :point_right: https://github.com/openjournals/joss-papers/pull/1106
2. Wait a couple of minutes to verify that the paper DOI resolves https://doi.org/10.21105/joss.01864
3. If everything looks good, then close this review issue.

4. Party like you just published a paper! 🎉🌈🦄💃👻🤘

   Any issues? notify your editorial technical team...

[labarba]

Congratulations, @dfm, your JOSS paper is published! 🚀

Sincere thanks to our editor: @xuanxu, and the reviewers: @benjaminrose, @mattpitkin — we couldn't do this without you 🙏

[whedon]

:tada::tada::tada: Congratulations on your paper acceptance! :tada::tada::tada:

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.01864/status.svg)](https://doi.org/10.21105/joss.01864)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.01864">
  <img src="https://joss.theoj.org/papers/10.21105/joss.01864/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.01864/status.svg
   :target: https://doi.org/10.21105/joss.01864

This is how it will look in your documentation:

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://numfocus.org/donate-to-joss