Joss: Anonymous reviewers

[cMadan]

In a submission I was reviewing, @mdnunez made a recommendation that we should have the option of reviewers being anonymous (https://github.com/openjournals/joss-reviews/issues/55#issuecomment-248471265). I hadn't thought about this consideration before, and I don't think our current infrastructure allows for this really... what do others think?

[arfon]

This would be very difficult with our current setup. While I'm sympathetic to the idea I actually think that the lack of anonymity is a positive feature of our review process.

[mdnunez]

Anonymity may not work with the current github structure. However I think anonymity will be important for the quality of reviews. Otherwise the reviews may be a lot less critical, leading to ultimately worse repositories. Anonymity may also help whenever there is an eventual heated discussion between the author and the reviewer. Even if JOSS doesn't agree with either of those points, JOSS may attract more reviewers if there is an anonymous option.

[kyleniemeyer]

I agree (with @arfon)—this being a journal of open-source software, I think the openness of the review process matches the ... openness... of the open-source software community. (There must be a better way to phrase that using "open" less.)

Of course, I am also sympathetic to people feeling that they need their review to be anonymous.

[mdnunez]

I think the open review push by the open-source community is an artifact of the open-source community not wanting to be criticized by the non-open source community. I'm not convinced that open reviews are beneficial in the long term however.

[labarba]

I have not seen any evidence of anonymity being a factor that adds quality to peer review. On the contrary, I have seen plenty of evidence (anecdotal, granted) of anonymity being detrimental, resulting in superficial, conflicted or just ugly refereeing.

Open peer review is a feature, not a bug.

[danielskatz]

I agree with @labarba that open peer review is a feature, specifically one that we have chosen for JOSS. I don't know that there's any proof that it's better, but I feel like it's part of the experiment that we are trying.

[labarba]

A quick search landed me here:
http://jp.senescence.info/thoughts/open_peer_review.html

Quote:

This coward behavior is nearly exclusive of science. In the arts and even in politics criticism is open and serves an important role in making ideas stronger (Brown, 2003).

Ha!

[mdnunez]

Until there is research, it remains an open question.

[labarba]

_Open peer review: a randomised controlled trial_
ELIZABETH WALSH, MAEVE ROONEY, LOUIS APPLEBY, GREG WILKINSON
The British Journal of Psychiatry Jan 2000, 176 (1) 47-51; DOI: 10.1192/bjp.176.1.47
http://bjp.rcpsych.org/content/176/1/47.full

_Method_ Reviewers for the British Journal of Psychiatry were asked whether they would agree to have their name revealed to the authors whose papers they review; 408 manuscripts assigned to reviewers who agreed were randomised to signed or unsigned groups. We measured review quality, tone, recommendation for publication and time taken to complete each review.

_Results_ A total of 245 reviewers (76%) agreed to sign. Signed reviews were of higher quality, were more courteous and took longer to complete than unsigned reviews. Reviewers who signed were more likely to recommend publication.

[mdnunez]

Signed reviews were of higher quality, were more courteous and took longer to complete than unsigned reviews. Reviewers who signed were more likely to recommend publication.

This sounds like a trade-off to me. Signed reviews were of a higher quality but took longer to complete. Signed reviews were also more likely to recommend publication. This is only good if JOSS' publication model is accept everything unless there are serious problems (which it may be?)

[kyleniemeyer]

@mdnunez I would say that that actually is JOSS's model—we consider all research software that satisfies the JOSS requirements to be capable of eventually being published, even if that takes significant time/revision (from http://joss.theoj.org/about#reviewer_guidelines):

What happens if the software I'm reviewing doesn't meet the JOSS criteria?

We ask that reviewers grade submissions in one of three categories: 1) Accept 2) Minor Revisions 3) Major Revisions. Unlike some journals we do not reject outright submissions requiring major revisions - we're more than happy to give the author as long as they need to make these modifications/improvements.

[tracykteal]

I still haven't come to a conclusion in my thinking about anonymous peer review in journals. However, I think the process here for JOSS is different than standard journals. The review process is collaborative, and there are clear guidelines for acceptance. The reviewer's role is to help the author meet the JOSS criteria and communicate clearly. I think there is less potential for fear in these types of interactions, and open and transparent communication is an important part of the process.

[mdnunez]

I think the downside is that software will be more likely to get pushed that _does not meet_ JOSS standards. But how often this occurs is not something that is currently known.

[cMadan]

I think it's up to the editors to confirm that software meets the JOSS standards before publication. If the editor is unsure based on their own judgment and that of the first reviewer, they could solicit an additional reviewer (and/or discuss with other editors).

[kyleniemeyer]

Yes, what @cMadan said. Ultimately we (the editorial board) are the final "gatekeepers", not the reviewers.

If a reviewer is concerned about something that they feel uncomfortable discussing openly in the issue, perhaps one thing we could do is either make our contact info more obvious, or add easy access through (e.g.) Gitter, to send a private message.

[karthik]

JOSS is not a regular journal. @kyleniemeyer and others have pointed out the differences between our model and that of a traditional journal. As long as a paper checks off the JOSS criteria, we accept it. Otherwise papers go into minor revision or on hold until major problems get fixed. This is part of the editorial board's job.

Reviewers are welcome to privately email concerns to the handling editor.

[arfon]

Thanks for the feedback everyone. I'm going to label this as a wontfix as I think we're happy with what we currently have.