Addons-server: Modify Points System for Add-on Reviewers

[caitmuenster]

Part of Issue #6454. Add the following reviewer behaviors and points value to the add-on reviewer points system:

REVIEWED_EXTENSION_HIGHEST_RISK -- 140
REVIEWED_EXTENSION_HIGH_RISK -- 120
REVIEWED_EXTENSION_MEDIUM_RISK -- 90
REVIEWED_EXTENSION_LOW_RISK -- 0
CONTENT_REVIEW -- 10

Dependencies:

• [x] Post review going live
• [x] Determining ranges for highest risk, high risk, medium risk, and low risk - issue #6475

[jvillalobos]

Discussed with @kewisch earlier about how to set the ranges for post-reviewed add-ons. This is how we define the ranges and how many points they get when reviewed (confirmed approved or rejected):

• REVIEWED_EXTENSION_HIGHEST_RISK: add-ons with weight > 150 get 140 points.
• REVIEWED_EXTENSION_HIGH_RISK: add-ons with 150 >= weight > 100 get 120 points.
• REVIEWED_EXTENSION_MEDIUM_RISK: add-ons with 100 >= weight > 20 get 90 points.
• REVIEWED_EXTENSION_LOW_RISK: add-ons with 20 >= weight get no points.

I'm assuming need info and super review remain the same in the post-review world.

[caitmuenster]

Works for me!

[jvillalobos]

Making this a P2 since volunteer reviewers are being moved to the new system and the points program is an important incentive for them.

[caitmuenster]

Let's reduce the point value for content review to 10 points. That way, 12 content reviews will equal 1 review of a high risk add-on.

[rctgamer3]

REVIEWED_EXTENSION_LOW_RISK: add-ons with 20 >= weight get no points.

Why zero points? We still have to check the source code and compare it with the last human-reviewed version. Giving it less points than simple user reviews seems a bit strange.

[caitmuenster]

The reason why we assigned 0 points to extensions with the lowest risk factors is that we don't believe these should be reviewed by reviewers at all, especially since so many other extensions will be in the higher risk categories.

If there comes a point when there are few add-ons to review -- or if there's a need to review the lowest risk extensions -- we can re-visit this and assign points.

[caitmuenster]

@jvillalobos, is it possible to retroactively add points to reviewers who onboarded to the new system before updated points were available?

[diox]

It's possible from a technical standpoint if we do want to do it.

[caitmuenster]

That would be terrific. Thanks, @diox.

[kewisch]

Just a side note on the implementation side, I suspect that "reject multiple versions" is just implemented as a reject operation on each version. If an add-on with N approved versions is rejected, I suspect that will give the reviewer N times the points mentioned above for just one "reject multiple versions". We should figure out if that is acceptable or if there needs to be some limitation.

[diox]

Just a side note on the implementation side, I suspect that "reject multiple versions" is just implemented as a reject operation on each version.

It's not (it would send multiple emails etc, that'd be weird). I was planning to only award points once for this action.

[kewisch]

Great, thanks for confirming!

[diox]

is it possible to retroactively add points to reviewers who onboarded to the new system before updated points were available?

I split this in a different issue, #6570.

[ValentinaPC]

I've checked this, for the 2 actions that should award points :

• Confirming an auto-approval
• Rejecting multiple versions
  using AMO-dev FF56(Win 7).

Here are my results:
Start points: 125,623

1. REVIEWED_EXTENSION_HIGHEST_RISK: add-ons with weight > 150 get 140 points
   
   
   
   • https://addons-dev.allizom.org/en-US/editors/review/rtl-addon-submission (w170) -confirm approval
   
   
   • https://addons-dev.allizom.org/en-US/editors/review/codechange01 (w166) - reject last vers
   
   

1. REVIEWED_EXTENSION_HIGH_RISK: add-ons with 150 >= weight > 100 get 120 points
2. https://addons-dev.allizom.org/en-US/editors/review/postreview02-ahole (w140) - confirm approval

3. https://addons-dev.allizom.org/en-US/editors/review/chrome-extensions-test (w110) - rej only vers

4. REVIEWED_EXTENSION_MEDIUM_RISK: add-ons with 100 >= weight > 20 get 90 points

5. https://addons-dev.allizom.org/en-US/editors/review/readability-new (w100) - confirm approval

6. https://addons-dev.allizom.org/en-US/editors/review/webext-to-many-reasons (w70) - reject

7. REVIEWED_EXTENSION_LOW_RISK: add-ons with 20 >= weight get no points

8. https://addons-dev.allizom.org/en-US/editors/review/permission-nativecocojambo (w20) - confirm approval
9. https://addons-dev.allizom.org/en-US/editors/review/borderify1 (w0) - rej

Points after these reviews: 126,443
126,443 - 125,623 = 820
Total reward points according to above reviews is 700.

@diox:
If my math is correct, _seems like there is a difference of 120_ between what should be the total reward points and what it is.
Maybe the fact that I picked some boundary values (at 3. - w100 and 4. w20) could be the problem?

Also, I've seen that "CONTENT_REVIEW -- 10" - this means that for any type of content review action (confirm/reject) and any weight that add-on had, the award points are always 10?

Verified for Content Review actions - 4 cases, as follows:
Start points: 126,443 points.
a. https://addons-dev.allizom.org/en-US/editors/review-content/test-source-file (w170) - approve - points increased with 10 - 126,453
b. https://addons-dev.allizom.org/en-US/editors/review-content/webext-setinterval (w40) - reject - points increased with 10 - 126,463
c. https://addons-dev.allizom.org/en-US/editors/review-content/axl4j3fhl (w0) - approve - points increased with 10 - 126,473
d. https://addons-dev.allizom.org/en-US/editors/review/axl4egwb2 (w0) - reject - points stay the same

@diox: is case d. expected?

Thanks and sorry for the long reply :)

[diox]

Maybe the fact that I picked some boundary values (at 3. - w100 and 4. w20) could be the problem?

Yeah, that's my fault, I mishandled the boundaries. The rewards for weight == 150, weight == 100, weight == 20 are all in the wrong tier. So you got 120-90 + 90-0 = 120 extra points.

Also, I've seen that "CONTENT_REVIEW -- 10" - this means that for any type of content review action (confirm/reject) and any weight that add-on had, the award points are always 10?

Yes

d. https://addons-dev.allizom.org/en-US/editors/review/axl4egwb2 (w0) - reject - points stay the same

@diox: is case d. expected?

No, that sounds bad :( It's super weird, there are no reviewer scores attached to this add-on in the database :(

[diox]

@ValentinaPC : for https://addons-dev.allizom.org/en-US/editors/review/axl4egwb2 the log seem to indicate that you rejected it "normally", without doing a content review. That's why you didn't get any points (it acted as a post-review, so with weight 0 you don't get points).

[ValentinaPC]

@diox : so, if weight 0, shouldn't the approve content action have the same impact as rejection - no increase at the total reword points? (see c.)

[diox]

Content related actions are always 10, never depend on the weight - the rationale is that we do want content review of low weight add-ons, but we don't care that much about having a full post review for them.

[ValentinaPC]

So, what happens at c. and d. is correct?

[diox]

Yeah. It's just the points awarded at the various boundaries for post-review actions that are wrong.

[ValentinaPC]

Verified the boundary weight values: 100 and 20 and points are incremented as expected.
Statistics:
Before: 126,763 points
a. https://addons-dev.allizom.org/en-US/editors/review/webext-simplegrr (w100) - confirm approve (90 points)
b. https://addons-dev.allizom.org/en-US/editors/review/promptable3 (w20) - reject (no points)
After: 126,853 points

Also, I've check again the reward points for content review and obtained:

1. https://addons-dev.allizom.org/en-US/editors/review-content/validationhanging (w0) - reject action - increments total points with 10 - which was not the case before
2. https://addons-dev.allizom.org/en-US/editors/review-content/content-scripting (w0) - approve content - increments total points with 10 - as before

What do you think about 1?

[diox]

It's working as expected. Like I said above, in your previous round of testing you did a regular post-review and not a content-review, and that's why you had an unexpected reward. In this case you did a content review and you got the expected content review reward.

[ValentinaPC]

Oh, I see, I was sure I did a content review action in both cases. :)

This can be marked as verified fixed based on the above checks.
Thanks for your help, Mat!

[spoji]

Let's reduce the point value for content review to 10 points. That way, 12 content reviews will equal 1 review of a high risk add-on.

I can see one problematic about this. Those 0 point reviews will just stand there forever in the queue as reviewers will only review those that are "worth" something. This is a good thing in a way as we do want reviewer spending time on the top on the list.

Hiding those 0 points reviews from the queue could be a possibility to fix this issue. However content reviewers would still need to see them somehow to prevent spam and do proper content reviews on them.

[wagnerand]

I am in favor of hiding the "0 risk" entries, and maybe even others.

The content-review queue is a different queue.

[kewisch]

See also #6472 which we might consider reopening once we've made that decision from a product perspective.