Joss-reviews: [REVIEW]: carl: a likelihood-free inference toolbox

[whedon]

Submitting author: @glouppe (Gilles Louppe)
Repository: https://github.com/diana-hep/carl
Version: v0.2
Editor: @arfon
Reviewer: @betatim
Archive:

Status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/26a9ffd9e7b98b1911d89d2ceb268f37"><img src="http://joss.theoj.org/papers/26a9ffd9e7b98b1911d89d2ceb268f37/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/26a9ffd9e7b98b1911d89d2ceb268f37/status.svg)](http://joss.theoj.org/papers/26a9ffd9e7b98b1911d89d2ceb268f37)

Reviewer questions

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Version: Does the release version given match the GitHub release (v0.2)?

• [x] Archive: Does the software archive resolve?

  Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?

• [x] Functionality: Have the functional claims of the software been confirmed?

• [ ] Performance: Have the performance claims of the software been confirmed?

  Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is? diana-hep/carl#53

• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the functionality of the software documented to a satisfactory level (e.g. API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?

• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support diana-hep/carl#53

  Software paper

• [x] Authors: Does the paper.md file include a list of authors with their affiliations?

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is? diana-hep/carl#53
• [ ] References: Do all archival references that should have a DOI list one (e.g. papers, datasets, software)?

[whedon]

Hi @glouppe. Many thanks for submitting to JOSS!

The next step is to look for a couple of reviewers - if you have any suggestions please add them here. :rocket:

@jakevdp - this do you have any suggestions?

[cranmer]

maybe @dfm @betatim

[whedon]

Compiled paper PDF: paper.pdf

[cranmer]

Thanks @whedon would be nice if arxiv showed up in bib. should we edit bib or change style for latex?

[arfon]

@cranmer This is the command that we're running to compile from Markdown to PDF: https://github.com/openjournals/whedon/blob/master/lib/whedon/processor.rb#L64 and this is the latex template: https://github.com/openjournals/whedon/blob/master/resources/latex.template

Any suggestions/modifications that would make arXiv show up very welcome 😄

[betatim]

As a reviewer should I do things beyond what is listed in the initial comment on this PR or is that it? If it is only those questions I can review this and use this opportunity to declare that I am a good friend of @glouppe but don't think this creates a conflict of interest.

[arfon]

If it is only those questions I can review this and use this opportunity to declare that I am a good friend of @glouppe but don't think this creates a conflict of interest.

Thanks for the heads up.

@betatim - basically we just need you to work through the checklist under Reviewer questions. If anything isn't clear please shout.

Also, there are some additional notes here (http://joss.theoj.org/about#reviewer_guidelines) but the review checklist should represent these guidelines.

Also, thanks so much for helping out!

[betatim]

UI question: how do I tick off things from the list? I think technically it is impossible to organise for me to be able to edit @whedon's comment. Had a scan around openjournals/whedon to try and find the raw MD but couldn't find it sad 🐼.

[betatim]

• no information for contributing or seeking help.
• In the PDF the reference needs a arxiv ID/DOI, this is a template issue I assume?

---

edit: should all be in the first comment now

[betatim]

A few "trivial" comments are included inline. I haven't yet tried to answer the "functionality" questions.

re: statement of need in the PDF. It states "support inference in the likelihood-free setup, including density ratio estimation algorithms, parameterized supervised learning and calibration procedures". This could be improved by stating a few concrete problems, that your average researcher can recognise as problems they have, to which it is the solution. Right now I think you have to already know that carl can solve your problem to realise that it solves your problem.

The first paragraph of the PDF is a bit longer than what is in the README so I would copy/sync them once a few specific problems are added to it.

Otherwise everything looked good to me.

[glouppe]

Thanks Tim! Changes proposed are https://github.com/diana-hep/carl/pull/53

Note: this back and forth between two github repos is not the most convenient :s :hamster:

[cranmer]

Thanks Tim

[arfon]

UI question: how do I tick off things from the list? I think technically it is impossible to organise for me to be able to edit @whedon's comment. Had a scan around openjournals/whedon to try and find the raw MD but couldn't find it sad 🐼.

@betatim - this is great point (that I didn't consider until now) - you need to be able to edit the issue too :-) I've added you to the joss-reviewers team which means you should be able to update the original issue.

Archive: Does the software archive resolve? Comment: see openjournals/whedon#1

This should should be fixed :soon:

Also, thanks for the rapid review :zap:

[betatim]

I've moved everything to the first comment. Most of the missing ☑️ are being dealt with in diana-hep/carl#53. Then there is the missing DOI in the generated PDF, do you take care of that @arfon?

[arfon]

Then there is the missing DOI in the generated PDF, do you take care of that @arfon?

Yep, tracking that in https://github.com/openjournals/joss/issues/56

[glouppe]

I made further changes at https://github.com/diana-hep/carl/pull/53 (now merged). Is it possible to regenerate the PDF?

[whedon]

Updated paper PDF: paper.pdf

[cranmer]

I don't see what the default bib style is in the wehedon resources:

https://github.com/openjournals/whedon/blob/71a1a35a21cce942378229892976ddc4c26e57e8/resources/latex.template#L98

but it looks like \bibliographystyle{utphys} might give a nice format for the arxiv link in the reference

http://tex.stackexchange.com/questions/243636/mendeley-and-arxiv-citation-style

[whedon]

Updated paper PDF: paper.pdf (now including the archive DOI and software repository address)

[betatim]

Installing works, automated tests pass, and ran some of the examples.

There are no performance claims (as in runs faster than X) so I don't think that reviewer question applies.

Only outstanding issue is rendering of references in the pdf.

[arfon]

@betatim - awesome thanks.

Only outstanding issue is rendering of references in the pdf.

@cranmer @betatim how do you feel about this rendering? paper.pdf

I've not managed to get the utphys biblio-style working but have managed to format the reference more generically to get the arXiv URL included:

@article{Cranmer:2015-llr,
      author         = "Cranmer, Kyle and Pavez, Juan and Louppe, Gilles",
      title          = "{Approximating Likelihood Ratios with Calibrated
                        Discriminative  Classifiers}",
      eprinttype = {arxiv},
      version     = {2},
      eprint      = {http://arxiv.org/abs/1506.02169v2},
      url         = {http://arxiv.org/abs/1506.02169v2},
      date        = {2016-03-18},
}

[betatim]

LGTM. I use the arXiv ID/DOI as a simple way to find the article in google, if you directly link me there even better (appeals to the efficient German in me)

[cranmer]

Sorry I missed this earlier. LGTM too.

[arfon]

Ok then I think we're done here. Congrats @glouppe and @cranmer on our first accepted paper!

I'll finish up the editorial pieces this evening.

🚀

[cranmer]

Woo hoo!

We can provide a testimonial on the review process if you want.

Congratulations to you and the JOSS team. 🎉

[glouppe]

Great, thank you all :)

[betatim]

Whoop! 🏁

Our work here is done, closing the issue.

[betatim]

Actually, where do we click to toggle the badge from under review to published?

[arfon]

Actually, where do we click to toggle the badge from under review to published?

@betatim - I have to do this (for now).

[cranmer]

I'm so excited!

[arfon]

This is now accepted and live on JOSS!

• http://joss.theoj.org/papers/10.21105/joss.00011
• http://dx.doi.org/10.21105/joss.00011

[danielskatz]

It seems like the reference is still broken - there's no DOI or URL or way to know where the reference was published or is available

[arfon]

It seems like the reference is still broken - there's no DOI or URL or way to know where the reference was published or is available

Which reference sorry?

[betatim]

Compare https://github.com/openjournals/joss-papers/blob/master/joss.00011/10.21105.joss.00011.pdf (from the journal page) to the last PDF you posted in this issue. The only reference in the PDF has a link to the arxiv in the version you posted here but not in the journal version

[arfon]

Compare https://github.com/openjournals/joss-papers/blob/master/joss.00011/10.21105.joss.00011.pdf (from the journal page) to the last PDF you posted in this issue. The only reference in the PDF has a link to the arxiv in the version you posted here but not in the journal version

Gotcha. Not sure what happened there. Will investigate.

[arfon]

OK, this is fixed in the online PDF now. The PR to fix the bibtex is here: https://github.com/diana-hep/carl/pull/56