Joss: Flag borderline reject submissions that move to Review

[labarba]

1. Our process of open peer reviews is incompatible with giving reviewers the decision power to reject submissions. The accept/reject decisions should always be made by the editors.

2. Most of the time, we would like to flag out-of-scope or minor submissions in the Pre-Review stage, as this is much cleaner than doing it after it goes to Review. But, in some circumstances, the deeper inspection made by reviewers is needed to make a determination.

3. “Quantity of code” is not a universal measure, and we can’t use it blindly to reject submissions as minor. But _most of the time_ it is a good measure. We need a bar below which all submissions are inspected for being possibly “minor.” Say, 1,000 lines of code. It doesn’t mean we reject if it doesn’t pass that bar, it means that research use and scholarship content needs to stand out for submissions under this bar to move forward.

4. Submissions that were borderline and discussed in the Pre-Review stage as possibly out-of-scope or minor should be _flagged_ as such if they move to Review. This can ease the burden on reviewers: we tell them, look, we think this should be rejected, but are giving it a chance to prove research need: can you find reasons to defend it? So the reviewer is not asked to reject; they have to be enthusiastic for us to “revert” our Pre-Review assessment to reject. If they are just indifferent, the Pre-Review assessment stands.

[arfon]

Thanks for summarizing this @labarba.

We need a bar below which all submissions are inspected for being possibly “minor.” Say, 1,000 lines of code.

I think this is a reasonable level to trigger some kind of default process such as the handling editor labelling the issue as flagged as minor or something similar. (It would be great if we could figure out how to get Whedon to do this too).

It doesn’t mean we reject if it doesn’t pass that bar, it means that research use and scholarship content needs to stand out for submissions under this bar to move forward.

👍👍 👍

So the reviewer is not asked to reject; they have to be enthusiastic for us to “revert” our Pre-Review assessment to reject. If they are just indifferent, the Pre-Review assessment stands.

This definitely sounds like it could potentially streamline our decision-making process as an editorial team, especially if the semantics of having the flagged as minor label means that unless demonstrated otherwise, the submission is assumed to be rejected post-review unless good reasons are found to defend it by reviewers.

One question I have based on this proposal:

• Do you think we need one or two labels here? We want to both flag an incoming submission as potentially a minor submission, plus we want to be able to denote something that was discussed at pre-review stage and deemed possibly out of scope. Are these two the same? (Possibly not).

/ cc @openjournals/joss-editors for additional thoughts.

[kthyng]

I think this is a reasonable way forward to improve our guidelines and process for processing submissions.

[arfon]

I think we've made a bunch of progress on this over the past few weeks and I'm wondering if the progress is sufficient to close this issue?

Summarizing some of the updates:

• _Substantial scholarly effort_ is now a defined submission requirement and is also included in the review process.
• We've matured our thinking such that we have written a public blog post capturing our thoughts.
• We've adding a label (query-scope) and review process for editors to vote on submissions (https://github.com/openjournals/joss/pull/764).
• The query-scope label is persisted between the pre-review and review issue.

So I think everything is in place to do this now. Would you agree?