Pkp-lib: Add option to select "Reviewers from this submission previous review rounds" in Round 2

[amandastevens]

In OJS 3.1, when you opened a second round of review and added reviewers, there was a checkbox to select "Reviewers from this submission previous review rounds." It allowed you to quickly find the reviewers from the previous review round. When the reviewer selection grid was improved in OJS 3.1.1, (Issue #2894) this option disappeared. The feature is used often by editors and it would be helpful to have it back.

[asmecher]

@amandastevens, are you sure this disappeared between 3.1.0 and 3.1.1-x?

[amandastevens]

Here is a screenshot of the select reviewer form for round 2 in 3.1.1-4 (from the test drive site) and that option isn't there. Is there is a way to make it appear that I missed?

[mfelczak]

Hi @asmecher, confirming that the hosted client that reported this issue was previously running OJS 3.1.0.

[asmecher]

Ah, I see, this used to be a search filter. @NateWr, I suspect this is something you could add without a lot of work...

[NateWr]

I'd like to do this when a new review round is created. So we can list the reviewers from the last round with a checkbox and ask them which should be assigned:

• [ ] Alec Smecher
• [ ] Michael Felczak
• [ ] Amanda Stevens

In previous iterations, what happened when you selected "Reviewers from this submission previous review rounds." Did you go through the assignment process for each one one-by-one, or did you set due dates for all of them together? Did you send an email, etc?

[asmecher]

I think this was simply a search filter, i.e. you'd need to go through the process with each reviewer and assign them one by one.

[NateWr]

Seems like a 98% of the time kind of situation. They're usually going to want the same reviewers. So I'm leaning towards just pinning the previous reviewers to the top of the selection list then.

[openacademia]

Pinning the previous reviewers to the top of the selection list sounds great, especially if it is somehow clear that they indeed are the previous reviewers. "This reviewer has previously reviewed a previous version of this submission" or something like that?

Thanks so much for working on this! It will be great to get this functionality back and I like Nate's idea of pinning the previous reviewers to the top. //Emma

_(Edit: I moved your side note into a separate issue #4312)_

[forgive38]

Hi,
our editors need this functionality too.

Thank you in advance

[PeterfordUP]

Hi there,
Has there been any movement on this since the last update? We're happy to assign some resource to it, and begin working/assisting, but we don't want to replicate work that's already being done somewhere else.
Best,
Peter Ford (PM, Ubiquity Press)

[NateWr]

Hi @PeterfordUP, no there hasn't been any work on this. Reviewing the discussion, it looks like there are two approaches being discussed:

1. When creating a new round, add checkboxes to the form which allow the editor to select which reviewers to re-assign. This is my preferred solution, however, editors will also need to be able to customize the email that is sent to each reviewer. If you're up for it, this is in my opinion the _best_ approach.

2. Add a filter to the reviewer selection list which shows only the reviewers from the last review round. This will involve working mostly with the REST API and QueryBuilders, which is probably a little bit easier.

3. My idea of pinning the previous reviewers to the top of the list. This strikes me as something that will be challenging to do well with the way that ListPanels work now. I'd recommend avoiding it.

Finally, it's also possible to do 1 and 2, which would probably provide the best overall experience, allowing editors to get at past reviewers even if they forgot to assign them when they created the round.

Do either of these approaches sound workable for you? Did you have an alternate solution in mind? Happy to discuss and review work on this front and also happy to give some advice on where to start. :+1:

[openacademia]

I agree, either option 1 or 2 (or a combo) would be great!

(Regarding the email: There is already an email template for second/third/fourth/etc. review rounds, the "Review Request Subsequent" and as always, the editor can edit the email prior to sending.)

[PeterfordUP]

Hi @NateWr,

Thanks for this - we're looking at options 1 and/or 2. I do want to ask about your thoughts on the forced pairing of reviewer _assignment_ and reviewer _invitation_?

At the moment, this seems a bit problematic, as there are a number of situations where reviewers need to be assigned, but invitations are not ready to be sent out (in OJS 2 this functionality might have been split?). With the auto-assignment of the previous round's reviewers, this problem would be compounded, as all reviewers would be invited at the same time, at the moment of assignment, and the individual emails would not be customisable before sending.

Our thoughts are that implementing option 1 would not be feasible without the splitting of the _assigning_ and _inviting_ functions, as it forces too much to happen at once, without control. Option 2 would be acceptable under this condition, however, as a pre-applied 'reviewed in previous rounds' filter on the 'add reviewer' page would still assign and invite in one action, as is current, but the emails could still be customised because the assignment would be done one reviewer at a time.

I think the situation could hinge on the pairing of assigning and inviting reviewers, in short, so your thoughts would be really helpful on that.

Best,
Pete (PM, Ubiquity Press)

[NateWr]

Thanks @PeterfordUP. I haven't heard a request to split assignment and email before. I'm curious about the use-case here. Can you say a little more about why editors want to do this?

I suspect, since I haven't heard it before, it's not a common need. But maybe other community members will correct me.

I'd recommend moving forward with option 2 on the premise that assigning and emailing will happen at the same time. What we often have is an checkbox to skip the email when performing an action (see editorial actions). This is sometimes used if the editor has contacted the reviewer independently, but we discourage this since it means that editorial activity isn't captured in the activity log.

This could be added into the assign reviewer workflow if you'd like. However, you would need some way for editors get a review link to the reviewer. Without an easy way to retrieve the link editors will not be able to send a useful email to reviewers later.

[PeterfordUP]

Hi @NateWr, thanks for coming back to me. In our experience, some of the reasons for splitting _assigning_ and _inviting_ are:

• Editors may want to have the list of reviewers available to be edited or reviewed by other editors before invitation;
• Editors may wish to prepare the reviewer list as one action, but not have time to compose the correct invitation emails to each reviewer in that same work session;
• Editors may be uncertain about whether a reviewer is a good choice or not, but wish to make a note of it, and review it at a later date before inviting;

Generally, the use case revolves around capturing the names and details of reviewers, without making a final decision to invite. We would much prefer that this coordination and work process happened within the system, rather than outside of it. We want for editors to be able to consider and discuss potential reviewers, without resorting to making paper notes or Word documents, which happens in the case where only reviewers that are definitely being invited can be added to (or even have their details recorded) in OJS.

We agree very strongly that contact between editors and reviewers should be captured by the system, which can in some cases be partly eased by the splitting of the two tasks. If an editor is mostly sure about a particular reviewer, but is either busy or wants to have more time to think about it, we don't want to force them to choose between:
-adding the reviewer, skipping the email, and sending the invitation outside of the system; or
-not adding the reviewer because that would commit them to inviting, and instead making their notes outside the system.

However, we understand that splitting the two tasks is quite a large change to the current workflow, which is no small decision to make. If the split can't happen, or can't happen any time soon, we're much more happy to only implement option 2, as this fits our use case, and avoids the potential consequences of bulk assignment without control over invitation.

[NateWr]

Thanks @PeterfordUP that's helpful context. I think that tying this into the reviewer selection and assignment process is going to raise some issues because the system doesn't have a way of thinking about a review being _staged_ for assignment. A reviewer assignment has implications in the system:

• It can effect a status message in the submissions list like "Waiting for reviewers to be assigned".
• It changes the count of active reviews for reviewers.
• In the future, when we provide reviewer statistics, it will impact things like time to turn around a review, etc.

I wonder if the use case would be better filled by drawing on the discussions tool and adding some enhancements. I'm thinking that an editor who wanted feedback on a reviewer could start a discussion with another editor and include the reviewer details. Perhaps a simple plugin could be made that provides a page for each reviewer, so that a URL could be dropped in and the discussion participant could go straight to a page with details on that reviewer (maybe a user profile + review history).

This page could be a nice starting point for providing a reviewer with a page about their own reviewing statistics (https://github.com/pkp/pkp-lib/issues/3534). Perhaps making that page accessible to editors, and giving them an easy way to link to it in a discussion, would address your use case?

[PeterfordUP]

Hi @NateWr, thanks for providing this. You make a very good point about the difficulty of tying this into reviewer selection - I'll look into the 'reviewer page' feature idea in the future, including the issue you've linked to, so thanks for this!

I'll post when we have an update on developing Option 2 of this issue.
Best,
Pete

[pmangahis]

A hosting client would be keenly interested in having a feature that will indicate to them whether reviewers who were already invited to a previous review round.