Vscode: Community participation in issue triaging

[chrmarti]

https://github.com/Microsoft/vscode/issues/34432#issuecomment-413285072

There are several approaches, this issue is for discussing and adding to these.

@V-ed suggests having a way for the community to flag an issue as a duplicate of another, but not immediately close it. Instead the bot would label the issue community-triaged and after a few days close it.

Another approach would be to immediately close the issue (e.g., duplicates, questions) and add a label community-triaged, so the team could still query for these issues, but they also immediate disappear from our regular queries.

The bot could also label new issues by new reporters as first-issue (and post a welcome message) to help spot returning reporters. We could then think about weaving that into the community triage, but generally I prefer to go with less rules/process.

[cleidigh]

@chrmarti
another typical tag the community could probably do is "needs more info"
this could result in faster additions by the original poster
just a thought...

[chrmarti]

We could start with a simple recipe for the most active members of the community. They could:

Check our "inbox" for new issues and see if any can be helped with:

• If it is a duplicate of an existing issue, mark it as such with /duplicate #12345.
  
  
  
  • Optionally let the bot search for duplicates (if it hasn't done so already) with /findDuplicates.
  
  
• If it needs more information, mark it as such with /needsMoreInfo and maybe explain what is missing.
• If it is a bug that reproduces, /confirm it and add any additional info that might help the dev team reproduce it.

This would be with the commands as they work today, just augmented with an allow-list of community members. Once we have some experience with this we can adapt and extend the approach.

/cc @usernamehw @Microsoft/vscode

[octref]

@chrmarti We should add a guide in our wiki for the community that's helping us triaging. Other than what you described, we can add things like:

• Ping the OP to try if the bug happens with no extension
• For perf issues, send OP to the perf wiki so they can start profiling and sending info back
• For issues using other languages, point them to the issue posting guideline and ask OP to use English

These would greatly help as they make the back-and-forth communication happen early, so when a team member is on an issue he would get all info he needs.

[cleidigh]

@chrmarti
Another factor that I run into with issue tagging is that almost none of the labels have descriptions. I know the core team can probably recite the labels in their sleep, those of us, even if we been working on the code for a while, cannot. I often flip a coin on some issues with some similar labels, would be great to be more precise. This would also be important for people marching duplicates.

The "feature-request" might be another consideration for community tagging, I realize this can sometimes require knowing each and every feature, but I think it still could be helpful.

[chrmarti]

@cleidigh I'd rather start without allowing too much labelling since the team uses the labels also for confirming an issue as valid. I take it you are already doing some issue triage? Does the proposed approach align with what you do?

@usernamehw Are you interested in giving the proposed approach a try? Does it align with what you already do? Of course this would be all for fame an honor and only when you have the time. (You were mentioned as being active on many issues when we talked about this in the team.)

@octref Here's the updated recipe (which I will move to a Wiki page):

Check our "inbox" for new issues and see if any can be helped with:

• If it needs more information, mark it as such with /needsMoreInfo and maybe explain what is missing:
  
  
  
  • Does it reproduce with all extensions disabled (--disable-extensions or Reload Window With Extensions Disabled)?
  
  
  • For performance issues ask them to follow https://github.com/Microsoft/vscode/wiki/Performance-Issues.
  
  
  • For native crashes ask them to follow https://github.com/Microsoft/vscode/wiki/Native-Crash-Issues.
  
  
  • For non-English reports, ask them to describe the issue in English.
  
  
• If it is a duplicate of an existing issue, mark it as such with /duplicate #12345.
  
  
  
  • Optionally let the bot search for duplicates (if it hasn't done so already) with /findDuplicates.
  
  
• If it is a bug you can reproduce, /confirm it and add any additional info that might help the dev team reproduce it.

[octref]

LGTM

[V-ed]

Another candidate for community issue triaging could be issues that are caused by extensions / issues that should be created in the related extension's repository.

[usernamehw]

I'm in.

Couple of questions:

Can bot monitor upvotes on comments? Example: If someone marks issue as a duplicate and gets 3(n) upvotes - the bot should then close the issue. (Possibly closing with 1 upvote from the OP)

---

For non-English reports, ask them to describe the issue in English.

Maybe there could be a list of people who can translate %LANGUAGE% and using
/needsTranslation German will ping someone who can translate German.

---

Should /confirm include OS and vscode version? Example: /confirm Win10 1.28.2 7f3ce96ff4729c91352ae6def877e59c561f4850

---

Can issue be a duplicate of already answered https://stackoverflow.com question?

---

Would it be possible to put invalid tag for empty issues? Example: #62697.

---

Is it possible somehow for bot to transform +1 comments to This comment was marked as spam/offtopic? Maybe with the help of users too. Like the last 6 comments there: https://github.com/Microsoft/vscode/issues/1477#issuecomment-384495917

[V-ed]

Can bot monitor upvotes on comments? Example: If someone marks issue as a duplicate and gets 3(n) upvotes - the bot should then close the issue. (Possibly closing with 1 upvote from the OP)

I really like this idea, and would even add "Possibly closing with 1 upvote from the OP _or a member from Microsoft_"!
It makes alot of sense to have multiple confirmation when it comes to community triaging - with much power comes great responsibilities after all, and humans are not responsible at all, so peer review is a nice way to solve this!

[chrmarti]

Great @usernamehw! Welcome to the team!

I like the suggestions, some would require additional coding on the bot's side, some we could configure today. Before we extend the proposal, I would like to test it in its current simple form and continue the discussion on additions to that.

I'll add @usernamehw and @cleidigh (who already has the powers, but it will be nice for visibility) to the allow-lists for /needsMoreInfo, /duplicate, /findDuplicates and /confirm.

On some questions:

• /confirm won't complain about additional information, so you could add that already. 👍
• /duplicate doesn't check what you write after it, so you can also link to StackOverflow. (We could also add /question to better capture this case.)

[cleidigh]

@chrmarti
I understand about limiting user tagging , what about getting descriptions on the labels within the system.
Even if users cannot tag themselves, everyone would now be able to recognize mislabeled issues
and could flag them for changing by a team member.

Concerning posts in English:
Maybe we could suggest a user translation from Google translate. That would at least give people some chance to consume the issue even if it's not perfect.

[chrmarti]

Understood, having descriptions on the labels makes sense.

Mentioning a machine translator also makes sense. I have added that to the proposal. The proposal is now on the Wiki: https://github.com/Microsoft/vscode/wiki/Community-Issue-Tracking

[octref]

@chrmarti Can we also add a section about community issue tracking on CONTRIBUTING.md? For some people who don't write a lot of code and wants to get involved in the VS Code community, this would be a great way for them to contribute. See https://github.com/Microsoft/vscode/pull/62799.

Also I find the message a bit too neutral. We are fortunate to have @cleidigh and @usernamehw willing to help us out (issue triaging is not fun). The message should have a more thankful tone, like this one:

Finally, should the wiki mention how another user could apply to be included in the list?

[cleidigh]

@chrmarti
@octref
saw the issue to add descriptions, cool.
maybe we could add a translate button on the reporter, that would make it really easy for OP.

• If users language != English, offer a translate button that keeps both the original and translation
  in the final issue reporter.
• perhaps issue reporter could also translate the title which is of course key for anyone else

Concerning users applying for the triage list, I would think running limited for a while might be a bit cleaner to not get swamped and have to turn people away. IMHO

[chrmarti]

@octref I don't know yet how we should onboard additional members from the community. I think ideally those interested would start helping out through comments, like @usernamehw, so they can get their feet wet and gain firsthand experience with our issue tracking (maybe @cleidigh is suggesting the same above). (Copied that Welcome sentence, thanks.)

[cleidigh]

FYI
@chrmarti @octref @usernamehw posted initial label descriptions: #62767

[cleidigh]

@chrmarti
What do you think about adding a translation feature directly into the issue reporter to improve its consistency?

[chrmarti]

@cleidigh I like the idea. Filed #63006 to track.

[chrmarti]

I have added the /unconfirm command for the community members helping with issue triage to indicate when they have looked into an issue, but could not reproduce it on their machine. The command adds the unconfirmed label like /confirm already adds confirmed. (A suggestion from @usernamehw.)

[chrmarti]

Changed to /confirmationPending and confirmation-pending to better reflect the intent. (https://github.com/Microsoft/vscode/issues/64604#issuecomment-445240588)

[usernamehw]

(We could also add /question to better capture this case.)

@chrmarti, Yeah, it would be nice to have /question in disposal.

[chrmarti]

@usernamehw I have just added /question. 👍

[usernamehw]

@chrmarti Is there no one else that has access to bot commands from the users? There are some active ones ,like @gjsjohnmurray. Maybe they should be able to use the bot...

[joaomoreno]

Also @IllusionMH

[gjsjohnmurray]

I'll be happy to receive these superpowers, and I promise to abide by the wiki guidelines.

[IllusionMH]

I'm ready to follow guidelines and happy to help as much as I can.

[isidorn]

@cleidigh @usernamehw @gjsjohnmurray and @IllusionMH now all you have permissions. Enjoy the superpowers!
Closing this issue and we can add more users as more people get interested.
Thanks everybody!