Sprints: Automatically label BCD PRs

[ddbeck]

User story

As an mdn/browser-compat-data maintainer, I want to install a GitHub bot that will automatically assign relevant labels to pull requests based on the files each PR modifies, so I can stop wasting time assigning labels manually.

This would also prove the feasibility of further work in the area of automation for BCD PRs (e.g., laying the ground work for a real attempt at #1097).

Acceptance criteria

• [x] Bot is deployed
• [x] Bot has assigned one or more labels

Tasks

• [x] Write a comment on this issue describing the proposed behavior for a BCD bot and its deployment
• [x] Get feedback from one or more BCD maintainers, including @Elchi3
• [x] Build the bot
• [x] Deploy it

[jpmedley]

I'm curious what labels you're talking about. Sometimes I can't tell for sure from an email notification that a PR changes a Chrome value. I'm wondering if this will help me with that in any way.

[ddbeck]

@jpmedley I'm starting out with something simple, to assign the existing "data" labels based on file paths. I wanted to start with something low stakes before pinging real people.

But if that goes well, I'd like to expand the automation for reviewers, to request reviewers based on browser or feature hierarchy (e.g., you could get an automatic review request for any change that modifies Chrome data; MDN folks could get tagged for features that are in their areas of expertise).

[ddbeck]

OK, here's a detailed plan of what I hope to build for this user story and a couple of unanswered questions. I'd love to have your feedback, @Elchi3!

Intro

I've been experimenting with GitHub's Probot Node library to automate managing BCD PRs. Now that I've played with the API a bit, I think I have a decent idea of what v1 for a BCD bot would look like.

In short, I want to deploy a bot that will respond to PR creation or modification events by assigning labels according to a configuration file in the repository.

Configuration

Configuration will be stored in the BCD repository as .github/labels.yml. The file will contain a list of labels and matching file patterns, like this:

"data:css 🎨":
  - "css/**"
"data:api 🐇":
  - "api/**"
infra 🏗:
  - "*.js"
# etc.

Each top-level key is a label. Each label entry contains a list of one or more file globbing patterns that match to files in the repo.

We'll only ever read the configuration from master because Probot's best practices advise against trusting PRs as a source for configuration. Plus it's inconvenient to do otherwise with the APIs provided by Probot and GitHub.

Behavior

Each time anyone opens a new PR or pushes a change to an existing open PR, the BCD bot will do these things:

1. Check for the configuration file in master. If there's no config, it will silently do nothing.
2. For each label, check if any of the patterns match files changed in the PR. If the pattern matches, apply the label to the PR.
3. If the file list consists of 300 files (the maximum returned by the GitHub API), leave a comment warning that we reached the maximum number of files, so it's possible that not all applicable labels have been applied.

You may have noticed that something doesn't happen: it doesn't _remove_ labels if a PR is modified such that a label no longer applies. On its face it doesn't seem that complicated, but I can imagine a few scenarios where you could get in a fight with the bot (e.g., manually applying a label on a WIP PR, which the bot dutifully removes after your next push). Plus, as a first attempt at automating BCD workflows, I'd like to avoid doing anything potentially destructive.

Deployment

There's probably some robust serverless setup we could use, but I figured I'd just do the easiest thing that could possibly work, which is to follow the Glitch deployment instructions in the Probot docs. I expect this would be a temporary measure until GitHub Actions become generally available in November, at which time we could follow Probot's GitHub Actions instructions.

Open questions

I haven't thought of everything. There are a few things I have been waffling on already:

• Is it a good idea to frame this as a general BCD bot, rather than a generic file labeler bot? On the one hand, we'll probably want to extend this for other BCD-specific purposes (e.g., applying labels or reviewers based on affected feature identifiers or browsers). On the other hand, we may want this specific functionality on other MDN projects (e.g., stumptown-content).
• If we do make this a general BCD bot, what should we call it? I worry that seeing activity from "bcd-bot" might be a little cryptic for first-time contributors to BCD.
• Where should this code live? I've already written a bit of code in a private repo. I could open that repo up and transfer ownership to the mdn org. Or we could put the automation in mdn/browser-compat-data directly, treating it as a monolithic repo for all things BCD. I can imagine a few arguments in favor of each, though again this depends on how we want to scope the bot generally.

[jpmedley]

MDN folks could get tagged for features that are in their areas of expertise

Can peers also get this?

[ddbeck]

Yeah, I figure if anyone wants to be automatically tagged for review, the more the merrier. I was just thinking of examples where someone had already expressed an interest.

[Elchi3]

Love this! Thanks so much, Daniel! :+1: I can imagine this being a great start into exploring some more automation opportunities.

There's probably some robust serverless setup we could use, but I figured I'd just do the easiest thing that could possibly work, which is to follow the Glitch deployment instructions in the Probot docs. I expect this would be a temporary measure until GitHub Actions become generally available in November, at which time we could follow Probot's GitHub Actions instructions.

This sounds like a good plan to me! Definitively looking forward to using GitHub actions here.

Is it a good idea to frame this as a general BCD bot, rather than a generic file labeler bot? On the one hand, we'll probably want to extend this for other BCD-specific purposes (e.g., applying labels or reviewers based on affected feature identifiers or browsers). On the other hand, we may want this specific functionality on other MDN projects (e.g., stumptown-content).

I think having this specific to BCD for now is okay. We will learn from that and if there is demand in the future, we can re-purpose it to be generally useful for MDN projects.

If we do make this a general BCD bot, what should we call it? I worry that seeing activity from "bcd-bot" might be a little cryptic for first-time contributors to BCD.

"compat-data-bot"? Or something entirely made up? In other projects bots often have friendly names like "highfive" or similar.

Where should this code live? I've already written a bit of code in a private repo. I could open that repo up and transfer ownership to the mdn org. Or we could put the automation in mdn/browser-compat-data directly, treating it as a monolithic repo for all things BCD. I can imagine a few arguments in favor of each, though again this depends on how we want to scope the bot generally.

I have no strong opinion about this one. Is it a lot of code? if not, I think we can have it inline and don't have to set up an entire new repo with CoC and readme etc etc.

[ddbeck]

Thanks for this feedback, @Elchi3! A couple follow ups:

I think having this specific to BCD for now is okay. We will learn from that and if there is demand in the future, we can re-purpose it to be generally useful for MDN projects.

OK, great. Reading your comment, it seems obvious in retrospect that making it a general purpose tool is a premature optimization.

If we do make this a general BCD bot, what should we call it? I worry that seeing activity from "bcd-bot" might be a little cryptic for first-time contributors to BCD.

"compat-data-bot"? Or something entirely made up? In other projects bots often have friendly names like "highfive" or similar.

Looking again at a recent test, GitHub puts a "Bot" label next to its activity, so I'll go with something obvious like "compat-data-helper", unless or until someone comes up with a cute name.

I have no strong opinion about this one. Is it a lot of code? if not, I think we can have it inline and don't have to set up an entire new repo with CoC and readme etc etc.

Oh, the CoC and readme and issue management… I didn't really think about that. I don't think it's too big to stuff into the main BCD repo. Looking at my experimental code now, it's only ~120 lines of code spread across a few files (assuming you ignore the GitHub API test fixtures, which are probably going to exceed 10,000 lines).

OK, unless there are any other issues, I'm going to proceed with this!

[Elchi3]

Sounds good to me, Daniel!

[ddbeck]

I've opened mdn/browser-compat-data#4794 to check in the bot code. We're getting pretty close to finishing this, but it may need a little more time in the next sprint to deploy. I'll update again here tomorrow (the last day of the sprint) with details.

[ddbeck]

OK, we've merged the code into BCD and I'm working on deployment now. I expect this will be _done_ done in the next 24 hours or so, so we don't need to account for this in the next sprint.

[ddbeck]

The bot is deployed now and seems to be working as intended (example: https://github.com/mdn/browser-compat-data/pull/4813#event-2626246442). A few issues cropped up after initial deployment, such as:

• https://github.com/mdn/browser-compat-data/pull/4807
• https://github.com/mdn/browser-compat-data/pull/4810
• https://github.com/mdn/browser-compat-data/pull/4812

But this is done now! 🎉

[Elchi3]

Thanks a lot for driving this, Daniel! :) Well done! :tada: