Joss-reviews: [REVIEW]: A Julia package for bilevel optimization problems

[whedon]

Submitting author: @matbesancon (Mathieu Besançon)
Repository: https://github.com/matbesancon/BilevelOptimization.jl
Version: v0.2.1
Editor: @poulson
Reviewer: @HaoZeke, @vissarion
Archive: 10.5281/zenodo.3249188

Status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/78ebab36f82fc977eb78b989aeb2f972"><img src="http://joss.theoj.org/papers/78ebab36f82fc977eb78b989aeb2f972/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/78ebab36f82fc977eb78b989aeb2f972/status.svg)](http://joss.theoj.org/papers/78ebab36f82fc977eb78b989aeb2f972)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@HaoZeke & @vissarion, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @poulson know.

✨ Please try and complete your review in the next two weeks ✨

Review checklist for @HaoZeke

Conflict of interest

• [x] As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Version: v0.2.1
• [x] Authorship: Has the submitting author (@matbesancon) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Authors: Does the paper.md file include a list of authors with their affiliations?
• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

Review checklist for @vissarion

Conflict of interest

• [x] As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Version: v0.2.1
• [x] Authorship: Has the submitting author (@matbesancon) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [ ] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Authors: Does the paper.md file include a list of authors with their affiliations?
• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @HaoZeke, it looks like you're currently assigned as the reviewer for this paper :tada:.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[HaoZeke]

Hi, @matbesancon I've initiated the review process and hope to finish in the next couple of days. I will be opening pull requests and issues on the repository tracker, but I might also ask for clarifications here (typically with matters pertaining to the paper itself). In addition, for the paper.md, I request that a copy to make changes on hackmd.io is made available to me and @vissarion since it is not appropriate to make changes to it via pull requests.

[HaoZeke]

@whedon check references

[whedon]

Attempting to check references...

[whedon]

```Reference check summary:

OK DOIs

• http://doi.org/10.1007/s00446-006-0020-y is OK
• http://doi.org/10.1109/TPWRS.2019.2892607 is OK

MISSING DOIs

• https://doi.org/10.1007/978-1-4613-0307-7_12 may be missing for title: Bilevel optimization: theory, algorithms and applications
• https://doi.org/10.1137/15m1020575 may be missing for title: JuMP: A modeling language for mathematical optimization
• https://doi.org/10.1287/trsc.35.4.345.10433 may be missing for title: A bilevel model for toll optimization on a multicommodity transportation network
• https://doi.org/10.1016/j.ejor.2019.01.059 may be missing for title: Toll Caps in Privatized Road Networks

INVALID DOIs

• None
  ```

[HaoZeke]

@matbesancon, please update the paper.md file to fix the missing DOIs.

[matbesancon]

The hackmd can be found there:
https://hackmd.io/s/B1e2UUNIE#

[matbesancon]

@HaoZeke the first DOI seems invalid (the preprint does not correspond to the article suggested), I'm fixing the others in https://github.com/matbesancon/BilevelOptimization.jl/pull/9

[matbesancon]

@whedon check references

[whedon]

Attempting to check references...

[whedon]

```Reference check summary:

OK DOIs

• http://doi.org/10.1137/15m1020575 is OK
• http://doi.org/10.1287/trsc.35.4.345.10433 is OK
• http://doi.org/10.1007/s00446-006-0020-y is OK
• http://doi.org/10.1016/j.ejor.2019.01.059 is OK
• http://doi.org/10.1109/TPWRS.2019.2892607 is OK

MISSING DOIs

• https://doi.org/10.1007/978-1-4613-0307-7_12 may be missing for title: Bilevel optimization: theory, algorithms and applications

INVALID DOIs

• None
  ```

[matbesancon]

missing DOI still invalid

[HaoZeke]

Yeah, we'll just have to let that one go. It does not seem to correspond to any DOI. Even the ISSN number does not seem valid. @matbesancon

[matbesancon]

yup I just saw it is considered as a book and not article on google scholar now, I'll update the bib

[matbesancon]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[matbesancon]

@whedon check references

[whedon]

Attempting to check references...

[whedon]

```Reference check summary:

OK DOIs

• http://doi.org/10.1137/15m1020575 is OK
• http://doi.org/10.1287/trsc.35.4.345.10433 is OK
• http://doi.org/10.1007/s00446-006-0020-y is OK
• http://doi.org/10.1016/j.ejor.2019.01.059 is OK
• http://doi.org/10.1109/TPWRS.2019.2892607 is OK

MISSING DOIs

• None

INVALID DOIs

• None
  ```

[poulson]

:wave: @HaoZeke, just checking in as it's been about a week since any comments were provided. I know that reviewing requires a significant amount of your (volunteered!) time and appreciate your commitment on this.

[HaoZeke]

Sorry I was hospitalized in a road accident. I'll be discharged today but have some appointments with an orthodontist... I might take a few days before I get home and am able to finish.

---

From: Jack Poulson [e-mail]
Sent: Thursday, March 7, 2019 8:00 AM
To: openjournals/joss-reviews
Cc: Rohit Goswami; Mention
Subject: Re: [openjournals/joss-reviews] [REVIEW]: A Julia package for bilevel optimization problems (#1278)

👋 @HaoZeke, just checking in as it's been about a week since any comments were provided. I know that reviewing requires a significant amount of your (volunteered!) time and appreciate your commitment on this.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[matbesancon]

sorry to read that, great recover! :/

[poulson]

@HaoZeke I am so sorry to hear about the accident but am glad to hear you are okay. Please excuse the distraction of my message; your recovery is obviously much more important.

[HaoZeke]

Thank you @matbesancon and @poulson for your kind wishes and for waiting so long. I have finally recovered enough to complete this review. I believe everything is in order.. Sorry the final signing off was delayed so much.

[matbesancon]

Absolutely no trouble, glad to hear you're doing good

[poulson]

:wave: @vissarion -- it's been a little while and the other reviewer has provided their feedback. Would you be able to provide your input some time in the near future?

[vissarion]

@poulson sorry for being inactive here, I can do it by the end of next week if this is not an issue.

[vissarion]

Hello, and again sorry for the late review. It is now complete. I raised an issue here: https://github.com/matbesancon/BilevelOptimization.jl/issues/14 regarding documentation.

[matbesancon]

thanks, on it today

[vissarion]

@matbesancon thanks for addressing the issue on documentation. I closed https://github.com/matbesancon/BilevelOptimization.jl/issues/14 and finished my review check-list.

[matbesancon]

@vissarion thanks for the discussion, I will adapt the paper to also include the signatures of build_blp_model

[matbesancon]

@whedon generate pdf from branch doc-joss

[whedon]

Attempting PDF compilation from custom branch doc-joss. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[matbesancon]

@whedon generate pdf from branch doc-joss

[whedon]

Attempting PDF compilation from custom branch doc-joss. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[matbesancon]

@whedon generate pdf from branch doc-joss

[whedon]

Attempting PDF compilation from custom branch doc-joss. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[matbesancon]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[labarba]

👋 @HaoZeke, @vissarion — it looks like this submission is ready for you to take a second look now, and possibly make a recommendation?

[HaoZeke]

@labarba, having taken a second look, I believe all previous concerns raised were addressed and I would recommend that this be accepted.

[poulson]

Thank you, @HaoZeke!

@vissarion: you still haven't checked the 'Performance' box. It would be great to get a response from you on this and whether or not you feel this submission is ready for acceptance.

[vissarion]

@poulson I recommend acceptance. However, I am not clicking the box because, as far as I understand, there are no performance claims by the submitted software to confirm.

[kyleniemeyer]

Hi @poulson, checking on on this review—it looks like both reviewers have given their ok.

[poulson]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[poulson]

@matbesancon Would you mind fixing the following typo/grammatical/citation/clarification issues?

"at the two level" -> "at the two levels"

"with possibly a disjoint feasible set" -> "with a possibly disjoint feasible set"

"solving method" -> "solver"

"for such case" -> "for such cases"

"the two principal being" -> "with the two primary approaches being"
[Citations needed for what the two approaches roughly are.]

Clarification: "into a regular one"?
[Citation of Special Ordered Sets of type 1]

"forcing at least one of them to 0" -> "forcing at least one of them to zero"

"m the JuMP model" -> "m: the JuMP model,", etc., adding an "and" in the second-to-last item.

"Where CM is the complementarity" -> "where CM is the..."

"Each arc has an initial cost, the lower-level" ->
"Each arc has an initial cost, and the lower-level"

"This problem can entirely be modeled using the framework presented above, a" ->
"This problem can be entirely modeled using the framework presented above, using a composite datatype defined in the package for holding all required data,"

Add commas and an "and" in the final list, such as: "The initial matrix of arc costs init_cost,"

"Even though BilevelOptimization.jl is thought for" -> "Even though BilevelOptimization.jl is typically for"

After doing so, could you register a DOI on zenodo and post it here?

[danielskatz]

👋 @matbesancon - this review is waiting on you to make changes and then to archive the repo.

[matbesancon]

apologies to all for the delays! I'll edit the paper with @poulson's comments

[matbesancon]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[matbesancon]

I'm registering a new version to trigger the DOI creation from Zenodo

[matbesancon]

The version of the software is v0.2.1, the corresponding DOI is 10.5281/zenodo.3249188

[poulson]

@whedon commands

[whedon]

Here are some things you can ask me to do:

# List all of Whedon's capabilities
@whedon commands

# Assign a GitHub user as the sole reviewer of this submission
@whedon assign @username as reviewer

# Add a GitHub user to the reviewers of this submission
@whedon add @username as reviewer

# Remove a GitHub user from the reviewers of this submission
@whedon remove @username as reviewer

# List of editor GitHub usernames
@whedon list editors

# List of reviewers together with programming language preferences and domain expertise
@whedon list reviewers

# Change editorial assignment
@whedon assign @username as editor

# Set the software archive DOI at the top of the issue e.g.
@whedon set 10.0000/zenodo.00000 as archive

# Set the software version at the top of the issue e.g.
@whedon set v1.0.1 as version

# Open the review issue
@whedon start review

EDITORIAL TASKS

# Compile the paper
@whedon generate pdf

# Compile the paper from alternative branch
@whedon generate pdf from branch custom-branch-name

# Remind an author or reviewer to return to a review after a
# certain period of time (supported units days and weeks)
@whedon remind @reviewer in 2 weeks

# Ask Whedon to accept the paper and deposit with Crossref
@whedon accept

# Ask Whedon to check the references for missing DOIs
@whedon check references

[poulson]

@whedon set v0.2.1 as version

[whedon]

OK. v0.2.1 is the version.

[poulson]

@whedon set 10.5281/zenodo.3249188 as archive

[whedon]

OK. 10.5281/zenodo.3249188 is the archive.

[poulson]

Congratulations, @matbesancon, I have recommended the paper for publication! (@openjournals/joss-eics)

[kyleniemeyer]

Hi @matbesancon, a few little things to fix before we publish:

• In your Zenodo archive, the summary/description says "A changelog is being generated, please check back in a few minutes."
• I made a couple of copy-edits to the paper in https://github.com/matbesancon/BilevelOptimization.jl/pull/20, could you merge that?

[matbesancon]

hi @kyleniemeyer thanks, I've updated the Zenodo description field, I responded on the repo for the edit, I'm not sure about the square bracket thing

[kyleniemeyer]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[kyleniemeyer]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

```Reference check summary:

OK DOIs

• 10.1137/15m1020575 is OK
• 10.1287/trsc.35.4.345.10433 is OK
• 10.1007/s00446-006-0020-y is OK
• 10.1016/j.ejor.2019.01.059 is OK
• 10.1109/TPWRS.2019.2892607 is OK
• 10.1007/978-3-319-33461-5_9 is OK

MISSING DOIs

• None

INVALID DOIs

• None
  ```

[whedon]

Check final proof :point_right: https://github.com/openjournals/joss-papers/pull/808

If the paper PDF and Crossref deposit XML look good in https://github.com/openjournals/joss-papers/pull/808, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.
@whedon accept deposit=true

[kyleniemeyer]

@whedon accept deposit=true

[whedon]

Doing it live! Attempting automated processing of paper acceptance...

[whedon]

🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦

[whedon]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

1. Check final PDF and Crossref metadata that was deposited :point_right: https://github.com/openjournals/joss-papers/pull/809
2. Wait a couple of minutes to verify that the paper DOI resolves https://doi.org/10.21105/joss.01278
3. If everything looks good, then close this review issue.

4. Party like you just published a paper! 🎉🌈🦄💃👻🤘

   Any issues? notify your editorial technical team...

[kyleniemeyer]

Congrats @matbesancon on your submission's publication in JOSS! Thanks to @HaoZeke and @vissarion for reviewing, and @poulson for editing.

[whedon]

:tada::tada::tada: Congratulations on your paper acceptance! :tada::tada::tada:

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](http://joss.theoj.org/papers/10.21105/joss.01278/status.svg)](https://doi.org/10.21105/joss.01278)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.01278">
  <img src="http://joss.theoj.org/papers/10.21105/joss.01278/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: http://joss.theoj.org/papers/10.21105/joss.01278/status.svg
   :target: https://doi.org/10.21105/joss.01278

This is how it will look in your documentation:

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: http://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://numfocus.salsalabs.org/donate-to-joss

[matbesancon]

Thanks all for the reviews!

On Mon, Jul 1, 2019, 17:39 whedon [e-mail] wrote:

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the
following code snippets:

Markdown:

HTML:

reStructuredText:

.. image:: http://joss.theoj.org/papers/10.21105/joss.01278/status.svg

:target: https://doi.org/10.21105/joss.01278

This is how it will look in your documentation:

[image: DOI] https://doi.org/10.21105/joss.01278

We need your help!

Journal of Open Source Software is a community-run journal and relies upon
volunteer effort. If you'd like to support us please consider doing either
one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add
  your name to the reviewer list here:
  http://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here:
  https://numfocus.salsalabs.org/donate-to-joss

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openjournals/joss-reviews/issues/1278?email_source=notifications&email_token=AB2FDMUCOGBQMWD2R6KHI4DP5IQLXA5CNFSM4G2FIVO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY6Q2DI#issuecomment-507317517,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB2FDMTVLEBLDFZOW2BZEWLP5IQLXANCNFSM4G2FIVOQ
.