Vscode: Bot Proposal: "Needs Version Info"

[JacksonKearl]

Background

We get a large number of incoming issues that do not include important information such as the VS Code version the author was using when they encountered the issue.
For bugs, this information helps us know whether a bug we thought we fixed has come back, or if the user just isn't on a version that contains the fix.
Similarly for feature requests, it is helpful to know when a user is asking for a feature that has already been implemented whether they are unable to discover the existing feature (in which case we might consider increasing visibility), or if they're simply on a version that doesn't have the feature yet (in which case we ask them to upgrade).

We previously had a bot to request version information, but it had a couple issues and is currently disabled.

As evidence for the need of version information, querying the string "what version" against the set of vscode issues yields around 1,300 results. This is information that previously a team member had to request and wait for, but now will be requested immediately by the bot, which should then make the turnaround time much smaller.

Proposal

On new issue

If the issue author is a maintainer, do nothing.
Otherwise, if the issue has a vscode version string, do nothing.
Otherwise, trigger a comment to be created requesting more info (and possibly stating the latest stable version is XXX and users should be on it), and add the needs more info label.

On update to issue

If the only activity on the issue is the comment requesting version info, and the issue now contains version info, remove the needs more info label.

Potential issues

Some team members complained of the old bot that it created too much noise in the issue timeline and required them to do more work in label management as now they had to remove the needs more info label in addition to what other triage work they performed. By augmenting the bot to remove the label itself, some of this work will no longer be needed. Further, we could look into making the bot delete its comment, but I believe that should be avoided for the sake of transparency.

[jrieken]

I am not keeping track of this but I don't remember when I have asked for the version the last time. IMO this is totally overrated and add bureaucracy to the whole process

[Tyriar]

I ask for it a lot, OS version is especially important for me on Windows

This is my saved reply (more generic than just version):

It looks like your report is missing some important information, See this wiki page for more info on how to report actionable terminal issues.

But still, the bot seems a little overly aggressive imo.

[bpasero]

I am not sure I would need this. However, I often set the "needs more info" label when the description is slim. Maybe the bot should just do that?

[JacksonKearl]

@bpasero It's hard to know what "slim" means programatically. Maybe instead of this version approach we could look at training a model.

[JacksonKearl]

I put up a new Action that will clean up after itself (remove its comment and the label) once the issue has been modified to contain version info. It will also not flag feature requests, so long as they leave the template intact (I added a NEEDS_VERSION_INFO_IGNORE flag to a comment in the template and instructed users to not remove it).

This should hopefully address @jrieken and @isidorn complaints about extra work needed on the inbox tracker's side, as the bot will have no lasting impact for users that do add the version info.

As for @bpasero's suggestion we flag "slim" issues, the idea behind this is that containing version info is an indicator of the slimness of the issue. If this ends up not holding out we can look into other ways of programmatically determining if the issue needs more information.

Initial:

After edit:

Note this will only remove the label if it was the only thing to interact with the issue. If a maintainer asks for additional information, the label will not be removed on update to the issue.

[jrieken]

I put up a new Action that will clean up after itself (remove its comment and the label) once the issue has been modified to contain version info.

I think I wasn't clear: I am not interested in the version info, I don't want that the bot annoys users to add this. It makes the process for them more painful and I still have to ask my questions. I understand that this is important for some of us (like the exact windows version for terminal issues) but how is this a general issue? Yes, I also keep asking the same questions, like attaching CPU profiles, attaching snippets etc but I use saved replies for that I am not forcing that onto all issue authors nor on the whole team.

[isidorn]

Everything what Joh just said applies to me as well. I like my issues as clean as possible.

Idea: every auto assigned label can opt into this behavior. So for example if the bot figures out it is a terminal issue, Daniel can configure the bot that for temrinal issue the bot asks for the version number.

[jrieken]

Either the bot knows this per area or (and I believe that simpler) we have the same as we now have for "caused by extension", e.g a fast way to insert a saved reply. I always find it hard to find the right saved reply and I would very much like to type /r perf_ext and that would insert default replies for how to profile the extension host. Others might have /r win_ver to ask the user to insert the exact windows version etc

[Tyriar]

Daniel can configure the bot that for temrinal issue the bot asks for the version number.

I don't really want this either for the annoy problem. I don't think this can be solves because so many things do not need version info and currently my saved reply works just fine. Sure it would be nice to always have version info before I go to the notification but I don't think it's worth it to annoy most users about this.

Perhaps even better would just be the new label ~needs version info that acts much the same as ~needs more info, then I can use that instead of my saved reply. I haven't tried using it yet though. It's easier than remembering and typing out a command and can be done in fewer key strokes (l, vers, enter).

[jrieken]

out a command and can be done in fewer key strokes (l, vers, enter).

My struggle is that focus is too often in the comment box and I am too stupid to move out of there

[JacksonKearl]

I don't think it's worth it to annoy most users about this

This would only apply to the set of users that report an issue from the web UI rather than the internal flow, and then only those that disregard our asking for version info in the issue template. Over the past ~12 hours, that's been three people. One of which was a "we've put the information in your clipboard", who then pasted their clipboard on being notified by the bot (they might have pasted it regardless, but this at least nudges them in the direction of double checking). The other was an instance where version info really does help, as they later posted their version to be 1.4.3_x64, which suggests something strange regardless. The third just closed the issue themselves, not sure whats going on there but 🤷‍♀️

https://github.com/microsoft/vscode/issues/93478
https://github.com/microsoft/vscode/issues/93476
https://github.com/microsoft/vscode/issues/93449

Maybe we can wait and see how this plays out in real world use cases?

[jrieken]

https://github.com/microsoft/vscode/issues/93478

☝️ I think's that actually a good and a bad sample of how the bot works. Yes, if an issue only contains the placeholder message -> YES add a comment, if you ask me close the issue right away. But now, with the comment added I already now what this is, there is no more reason to wait for the user to add version info and anything alike

[JacksonKearl]

For #93478 the flows are:

With Bot:
Inbox tracker asks what language/extension they're using and then closes as caused by extension

Without bot:
Inbox tracker asks for them to paste
Waits for them to paste
Inbox tracker asks what language/extension they're using and then closes as caused by extension

It removes inbox tracker work and turnaround time.

[jrieken]

Inbox tracker asks what language/extension they're using and then closes as caused by extension

My definition of the inbox tracker role is to assign item to owners, e.g in this case the inbox trackers see formatting and assigns this to formatting-owner. That's it.

[JacksonKearl]

Okay, but that doesn't change the overall comparison. Its still less work for the inbox tracker and less turnaround time for the user with the bot

[jrieken]

Sure - not disagreeing on that. I say the version info has no value, it should not be the blocker here. The lack of any information should be.

[JacksonKearl]

I agree, like Ben earlier said we're really looking for issues which are "thin" in their descriptions. But absent a String#isThin() method, we're using presence of the version information we specifically request as an indicator. If this ends up not being a good indicator, we can try other things, like developing a NLP model or something. But if version number works 95% of the time, the time taken to develop and maintain such a model would be a waste.

We could potentially omit references to version number in the bot's comment to hopefully avoid people adding a version number to an otherwise sparse issue, instead making them look over the issue reporting guidelines to figure out what all they're missing.

[JacksonKearl]

Another case: https://github.com/microsoft/vscode/issues/93487

Would anyone not ask for their VS Code version as an initial step in debugging this?

[jrieken]

Would anyone not ask for their VS Code version as an initial step in debugging this?

That depends - you might have a dupe for this because there is a regression, you might know right away what this is because you refactored something in that area. There is no generic answer. The thing that's almost always true is that the feature are owner knows best

[JacksonKearl]

Sure the owner might know of a recent code change or ongoing issue, but in all the cases they don’t (and even if they do), it would still help to know if this is, for instance Insiders or Stable.

We still haven’t seen any examples of people being upset that they were asked to include version information. If those start appearing I think that’d be a good indication to reconsider.

[jrieken]

We still haven’t seen any examples of people being upset that they were asked to include version information.

👋 I am upset. And then, really? Is that the approach? We add pointless bureaucracy till people scream?

Don't get me wrong. I am all for useful issues and all against useless issues. Not english? Useless. No content? Useless. Thin? Useless. And while version information surely plays into this, it is not the only single thing. If I see an issue which says "with latest insiders format on save is broken" then that's a good issue. I know what to do and how follow up, no reason to send the author into stupid circles.

Detecting a good issue is hard, using a regex to fish for a version number is easy but that doesn't make it a good check for declaring if an issue is good or not.

[JacksonKearl]

Inclusion of the string “insiders” does count as a version, just by the way.

Right now the argument is we should either hoist more work onto the issue tracker or put much more work into detection of good issues because otherwise users might be upset, or this might not be a good enough indicator. All I’m saying is that we should validate those hypotheses before taking on those tasks.

[jrieken]

but it's main responsibility is to route issues to the actual feature area owner.

⬆️ from https://github.com/microsoft/vscode/wiki/Issue-Tracking#inbox-tracking-and-issue-triage. The inbox tracker doesn't need to ask these questions, I don't know under what assumption you are doing this?

All I’m saying is that we should validate those hypotheses before taking on those tasks.

Agree, but don't do it in the field on real people. How about looking at issues that weren't actionable (closed as needs more info) because the version info was missing?

[Tyriar]

This bot effort is all about reducing the effort it takes to work with issues as we scale. And while version info isn't _always_ important, it is _often_ important and can change the amount of time it takes to action an issue from 5+ minutes to locally verify or multiple back-and-forths into 10 seconds if they're on an old version for example. Version info is also an example of something that is very easy to fetch, relatively easy to scan for and most developers have the expectation that they would need to include version information in a bug report anyway

We could "solve" this for the majority of cases and very rarely need to ask for version info again by having the bot identify issues that lack it, and keeping the issue out of the triage queue until version info is provided.

I'm all for adding the little more friction for user reporting when it can save us more time and make dealing with notifications less of a chore. It is extremely easy for the user to get their version info, we could have the bot use a picture that points out how to get it. Note also we could have the bot be quite permissive/optimistic and still pass it on to the inbox tracker/area owner if the user commented and the bot couldn't detect a version.

---

Extending this, if we do spot a version and it's several months old, the bot could also keep that out of the inbox tracker queue and make a comment like this:

It looks like you're on an old version of VS Code, could you try updating to see if it still reproduces and comment with the results?

Then when the user comments it goes into the inbox tracker queue. All these little things we could be doing will save us each a lot of time in aggregate, and that time saved will double when we have double the users.

[JacksonKearl]

This has been disabled for now.

[alexdima]

I'm late to the party, but here are my 2 cents:

I think it is fair to assume that an incoming issue is either coming from latest stable or from latest insiders.

So what I do is: regardless of the version in the issue, I just try to reproduce on master. If I cannot reproduce, then I tell the people I cannot reproduce and show them a screenshot or a screen recording of me trying to reproduce and failing and inviting them to reproduce in insiders.

[chrmarti]

Those using GitHub's predefined replies might be interested in using /needsMoreInfo as part of your reply (on a separate line). That will add the label and a comment from the bot.

[JacksonKearl]

Moving to a model where the bot first categorizes issues into feature areas, then is able to request additional info based on them.