Joss: Handling multiple reviewers

[Kevin-Mattheus-Moerman]

Our current review pipeline handles a single reviewer. We need to upgrade the process to handle N editor assigned reviewers. Perhaps this requires an adjustment of the current tick box style form. Are multiple tick boxes required, one for each reviewer per point, or do we use multiple forms, one for each reviewer? What does the reviewer system look like for multiple reviewers?

For multiple reviewers, should we collect reviewer feedback as editors first and then present it to the authors? E.g. what if reviewer 1 comments "_Please implement A_", followed by reviewer 2 who comments "_Whatever you do please never implement A_". If reviewer 1's comment comes first and reviewer 2's comment several days later, perhaps the authors will have started implementing A already. So do we need to formalize the way the editor recommends how the authors address the reviewers comments?

[labarba]

I would continue the review conversations openly on the Issue thread, rather than separately with the editor. If reviewers disagree on something, that is part of the conversation. The editor might make a decision to favor one of the reviewers on the issue of disagreement. But doing it all publicly has value.

[kyleniemeyer]

I want to bump this issue, as I think it's quite important to support >1 reviewer—I think in most cases, we should want more than one person to review. It shouldn't be too hard to have a list of names, rather than just one.

[labarba]

When adding a second reviewer, is it possible to add a second checkbox to each item in the Review Checklist?

• [ ] [ ] … I guess not

Maybe a nested item?

• [ ] item 1, reviewer 1
  
  
  
  • [ ] reviewer 2
  
  

[danielskatz]

or a second checklist for reviewer2 would also work and would be more likely to be possible in GItHub

[iljah]

A separate issue in joss-reviews would a good way to start at least I think. This way both reviewers can do their work without accidentally messing with wrong check boxes, etc. Perhaps rename the title of issue from [REVIEW] to [REVIEW1], [REVIEW2], etc.

[kyleniemeyer]

@iljah One of the strong features of JOSS we don't want to lose is the single review issue, where the review is more of a conversation between the reviewer(s), editor, and author. Moving to two review issues would simplify the check box question, but lose that combined discussion.

@arfon @parrish any updates on handling more than one reviewer with whedon?

[arfon]

@arfon @parrish any updates on handling more than one reviewer with whedon?

:wave: @parrish - would you be able to take a look at this. Here's the code for handling the reviewer assignments: https://github.com/openjournals/whedon-api/blob/master/whedon.rb#L223-L249

Basically I think we need Whedon to be able to:

Respond to a command to assign multiple reviewers

• Currently the command is @whedon assign @person as reviewer
• We probably want to modify this to allow for something like: @whedon assign @person1 and @person2 as reviewer
• We might also want to think about how Whedon would handle a request to add a reviewer when there is already one set (currently this just clobbers the current reviewer assignment). Should the request to add a reviewer (when there is already one set) assume that this is a second reviewer?

Make sure we handle the GitHub API pieces

• Reviewers are added as external collaborators to the joss-reviews repository so they can update the review checklist (https://github.com/openjournals/whedon-api/blob/master/whedon.rb#L235). We'd need to make sure that multiple reviewers are invited in this way.
• Reviewers are assigned to the review issue (https://github.com/openjournals/whedon-api/blob/master/whedon.rb#L245-L249) - we probably want both reviewers as assignees.

Render the checklist multiple times

• If we have multiple reviewers then we probably want multiple checklists. This probably needs handling in the joss codebase.

[parrish]

:wave: I took a quick pass at this in openjournals/whedon-api#24.

I added @whedon add @person as reviewer and @whedon remove @person as reviewer.

assign still replaces all reviewers with the given user, but maybe it makes more sense to remove it now?

[labarba]

My first reaction is to leave assign for (human) backwards compatibility (we're used to it!).

[leeper]

I'd leave add and assign (both are add without replacing) as synonyms but leave the assign formulation out of the commands list as a nudge to use add rather than assign in the future.

[labarba]

Well ... interesting ... if they are going to be synonyms, why not just leave assign as the command for assigning first _and_ second reviewer?

[labarba]

Also, assign used to _replace_ the reviewer, if one was previously assigned. Now, we have add to get a second assignment. What if we need to _remove_ a reviewer? Say, they just didn't respond and it's been two months, or they recanted their commitment.

[labarba]

Take that back! ☝️
I read now that we have remove.

[arfon]

👋 I took a quick pass at this in openjournals/whedon-api#24.

⚡️, thanks @parrish. This is looking great. I've got a test deploy hooked up to the https://github.com/openjournals/joss-reviews-testing repository if you want to try it out some more.

We just need support on the joss side now 😬

[arfon]

We just need support on the joss side now 😬

@parrish - I've taken a stab at this here: https://github.com/openjournals/joss/pull/389

[arfon]

This is now supported. @whedon supports the following commands related to reviewer assignments:

# Assign a GitHub user as the sole reviewer of this submission (also clobbers existing assignments)
@whedon assign @username as reviewer

# Add a GitHub user to the reviewers of this submission
@whedon add @username as reviewer

# Remove a GitHub user from the reviewers of this submission
@whedon remove @username as reviewer

You can see an example review here: https://github.com/openjournals/joss-reviews/issues/645 - multiple review checklists are created for each reviewer.

[Kevin-Mattheus-Moerman]

:tada: great stuff. @arfon are multiple tick box sets also added or does that remain manual?

[arfon]

🎉 great stuff. @arfon are multiple tick box sets also added or does that remain manual?

This is done by Whedon now.

[Kevin-Mattheus-Moerman]

@arfon Just now? Earlier today it did not create multiple checklists over at https://github.com/openjournals/joss-reviews/issues/671#, so I manually added them. Unless it is slow and I was not patient enough. I think I waited a couple of minutes.
Or does the process only work when both reviewers are added in the pre-review issue, because I assigned reviewer 1 in the pre-review issue and added reviewer 2 in the review issue.

[arfon]

@Kevin-Mattheus-Moerman - this only works at the pre-review stage (i.e. multiple reviewers assigned at the pre-review stage results in multiple checklists)

[Kevin-Mattheus-Moerman]

@arfon :clap: okay. Thanks

[labarba]

Question --- can we run the command @whedon add @username as reviewer in the REVIEW issue to get a new checklist added therein?

[Kevin-Mattheus-Moerman]

@labarba I think not. As @arfon said it works for the pre-review issue. If reviewers join during the review stage i think we currently have to add a list manually.

[arfon]

@labarba I think not. As @arfon said it works for the pre-review issue. If reviewers join during the review stage i think we currently have to add a list manually.

Right. I'd really rather not add code to Whedon to do this. Given how the GitHub API works, it would be pretty complex logic to add a new checklist when one has already been created.

[labarba]

That's fine. I didn't realize this, and I started the review for a couple of papers that I would like to have a second reviewer for. I guess I'll just have to copy-and-paste a new checklist for the second reviewer.

But, to clarify, it's not possible to @whedon add @username as reviewer in the REVIEW issue and have the reviewer listed in the header, even if the checklist has to be added manually?

[arfon]

But, to clarify, it's not possible to @whedon add @username as reviewer in the REVIEW issue and have the reviewer listed in the header, even if the checklist has to be added manually?

Yeah, I think @whedon ignores this. As an admin on the repo, you can edit (and add) this too.