Coq: PR review should be incentivized

[ejgallego]

After the latest changes in the merging process, it seems that the main problem we are facing is that some maintainers/developers are not meeting their review / assignee obligations while not being MIA.

This is creating a worse-than-linear loss of time for the authors of PR, and IMO requires action in order to restore fairness.

Incentives in an open system are a very hard problem, so I can't support a concrete action plan yet. A typical incentive scheme leading to an equilibrium under reasonable assumptions would be that developer X should have met their review obligations before his own PRs can get reviewed / merged.

For example, following this scheme the merge script could check the number of reviews pending and emit a warning if X has pending reviews for more than 2 weeks. Of course this warning would be overridable.

[Zimmi48]

While I'm willing to look at a solution to this issue, I don't agree with the example solution proposed. I would first start by considering to have @coqbot ping people when an action from them is warranted. As for incentivizing, this is a difficult question:

• GitHub recently introduced a new way to put forward the variety of one's contributions on one's profile (see e.g. https://github.com/Zimmi48). This shows that they are also thinking about this question.
• Only the two engineers of the consortium are actually paid (in part) to contribute to the development so they are the only ones who could have contractual obligations to contribute to the review process. For the rest, I'd say the deal must be that developers who are part of the official dev team and participate to the decision process must do their share of reviews. People who do little reviews should be considered external contributors, even if they have contributed for very long.

[ejgallego]

Thanks for the thoughts @Zimmi48, a couple of comments:

• I am not sure having coqbot ping people is going to add a lot over what we have. Developers can easily see their pending reviews so that pings would look a bit like noise. On the other hand a ping such as "your own PRs won't be considered until you clear your review queue seems way more effective".
• I would keep meta considerations such as contracts etc... outside of the mechanism. IMHO this should work for any person who is a maintainer.
• a way to think about this is in terms of "harm", "payoff", and "utility". For example, by not doing a review you are causing a "harm" to the person who wrote the PR, how this harm is computed will determine many possible strategy choices.

[gares]

I think tools could provide a list of PRs with no review ranked by how long they have been waiting. The whole point is to replace (temporarily one hopes) the inactive people, not to make them even more inactive.

This is also why I don't think trading reviews for merges can be named incentive, it looks more like a punishment (it is not something extra, that is what an incentive is). Punishing someone that has no time/motivation today is a very effective way to avoid he becomes active tomorrow.

[SkySkimmer]

it seems that the main problem we are facing is that some maintainers/developers are not meeting their review / assignee obligations while not being MIA.

Are they really less MIA on their own PRs than on reviews?

[ejgallego]

Are they really less MIA

I mentioned MIA as IMHO it should be handled differently than what we would like to see here.

The whole point is to replace (temporarily one hopes) the inactive people, not to make them even more inactive.

I think the idea here is more about having a system that _incentivizes_ people try to tackle first the tasks that do the most _common_ good. Reviews is one of such cases. Of course people is free not to follow the incentive structure, but then the gain should be lesser.

This is also why I don't think trading reviews for merges can be named incentive, it looks more like a punishment (it is not something extra, that is what an incentive is).

I am using incentive in the sense of MD, see for example http://theory.stanford.edu/~tim/f16/f16.html . In this case a punishment is just a negative incentive if you want, I prefer not to see it as a "punishment", and IMO it is wrong to see it that way. Agents have free will, and the incentive structure should tell them what's the best path, if they don't follow it, well, it is normal they get a lower payoff.

Think of it this way, many mechanisms do work by having you pay proportionally to the harm you do to others. If you can do unlimited harm without payback, then obviously you are not going to have an incentive not to so. That's the whole idea.

[ejgallego]

@ppedrot just pointed out that 2/3 of currently open PRs are without an assignee.