Venus: implement a lightweight issue triage process for stage 2 (trusted collaborators)

[phritz]

Likely end state some time relatively soon around stage 2: implement a wrangler role that rotates through go-filecoin members on a ~weekly basis. Scope of this role for the week that one plays it is likely to include:

• triage incoming issues: ensure they have repro instructions, confirm, label, and prioritize
• ensuring build breakages get fixed
• notice and file bugs for flaky tests

We need to decide how much if any of this (or some other process) we want to do for stage1. Possibly one person just plays this role for starters, and is responsible for documenting what's involved and then onboarding the next person.

Edit: we are not doing anything for stage 1

[phritz]

In separate channel jbenet expressed a requirement that we want to funnel questions/pings into a canonical process/mechanism in order to help protect why and nicola's ability to focus (ie, to prevent them from getting all the pings directly).

[phritz]

Proposal

Goals

• ensure we have a healthy, strong community with whom we are engaged
• ensure we have operational coverage for interrupty things

Non-goals:

• pagers
• anything outside of regular work hours
• sinking a ton of time into it

Note that the roles below should eventually be combined into one role that is rotated across the team. The roles are separated out for now because we as we're getting testing and the cluster off the ground it makes sense to specialize the information.

Test Sheriff

Who: @acruikshank
What:

• ensures that master CI stays green and has zero test flakiness. "Ensures that tests get fixed" doesn't mean "fixes all tests".
• notices master CI flakiness, files an issue when we have a flake, ensures flakiness issues are prioritized and ensures people are actually working on them
• accumulates knowledge about test failures, noticing patterns that enable them to make recommendations for systematic improvements

Cluster Warden

Who: @frrist
What:

• receives and responds to problems in and alerts from the cluster (that it didn't start, that it fell out of consensus, an ERROR occurred, etc); responsible for doing initial triage on problems and filing well-scoped issues of appropriate priority to address any problems; ensures that these issues get prioritized and people actually work on them
• notices what's hard about debugging filecoin in the cluster and files Candidate issues to make improvements (ie, responsible for making their own life easier).
• collects essential information about running and debugging the cluster in one place so that when this role expands to other people they have access to the right information

Interrupts

Who: this role rotates weekly through all of go-filecoin
What:

• backstop for any interrupt-driven or periodic responsibilities of the team. The other two roles likely collapse into this role as they mature.
• triages issues filed by the community. These issues will show up in the Inbox. They do some light validation, respond to the author (very important), and prioritize appropriately. If the issue is a P0 or a P1 ensures that someone picks the work up. Triage should happen about 1/day.
• responds to community conversations in discord and IRC. This doesn't mean other people can't respond, this means that Interrupts is the backstop who will respond if no one else does. Relays any conversations of team/project-wide significance to the team.
• creates documentation, tools, or processes to make this role easier, eg curates template responses. While within this role Interrupts should prefer to do this work of improving the role and not farm it out.

TODOs

• @anoemi to propose more specifics for the community-based responsibilities
• pick a oncall/rotation helper so we can manage the interrupts schedule
• figure out an interrupts on-boarding strategy, likely training provided by anaoemi on community and riding along with the previous Interrupts
• encode this all in github

Thoughts? Comments? Suggestions? Clarifications? Questions?

[mishmosh]

Two thumbs up from me on the proposal above.

Let's keep an eye on whether weekly is the right rotation cadence. These are a lot of new skills & routines. For each go-filecoin member to pick them up and put them down might require 1-2 days warmup time, which is a big fraction of a week. Maybe it's fine.

One way to help everyone get better at being The Interrupted could be borrowed from hospital triage: a shift change report, which we could do in writing or video.

[dignifiedquire]

👍 generally. I suspect these will expand, rather than fold into a single role, in the future. But lets cross that bridge when we come to it.

[anthrocypher]

Doing Shifts as The Interrupt

Per @phritz's message above, here's a closer look at the shape of dev shifts on community channels (aka Interrupts). Let's start with requirements and non-requirements.

Requirements:

• Community members who engage reliably get helpful responses
• Conflict and trolling is addressed quickly
• Teammates know when they are on shift as The Interrupt
• Teammates on shift have means of getting input when things look fishy

Non-requirements:

• Weekend coverage
• Any dev becoming the only person handling community interactions, and getting buried by it

What do shifts look like?

I recommend starting with week-long shifts, because that avoids mid-week context switching. I will actively solicit feedback and adjust as needed.

Each week, The Interrupt starts off doing a shift change report, conferring with last week's on-shift dev via an internal Slack channel (probably called #filecoin-moderation). This is the time to share any incidents (_"There were several folks spreading FUD on IRC late last week"_) or developments (_"So-and-so from the community is working on a PR for X. Keep an eye out and see if they need any help!"_).

From there, The Interrupt spends their week monitoring three different channels:

Chat (IRC/Discord): The Interrupt remains in chat for the duration of their workday, observing, and stepping in when needed. The purpose of chat is for people to get help with Filecoin. If someone posts a question and receives no response, The Interrupt ensures they get one. Chat is ephemeral, so The Interrupt watches for situations where someone should post on the forum instead. For example, if someone asks a question The Interrupt has seen before, or a community member wants to submit a PR but first needs to understand a piece of Filecoin's high level architecture, those are good times to ask them to post on discuss.filecoin.io. Finally, when conflicts or trolling emerge, The Interrupt makes sure things don't escalate, and incidents don't destroy the chan's signal to noise ratio. _Chat monitoring will likely be the most time-intensive part of the week for the on-shift dev._

Forum (discuss.filecoin.io): The Interrupt checks the forum at the start of their workday and catches up on any new threads. The forum is for higher level question and discussions. It serves a similar purpose to chat, but is used when asking and answering a question will create a useful artifact for future community members. Building up a body of information on discuss.filecoin.io saves time for our future selves. You can see stebalien's activity on on discuss.ipfs.io to get a good idea of what being The Interrupt on the forum will look like. You can review the existing categories on discuss.filecoin.io to see how we think posts will be bucketed.

GitHub: The Interrupt reviews the state of GitHub at the start and the end of their day, watching for any new issues or PRs from the community. The Interrupt's job on GitHub is to help Filecoin code get written by the community and ensure contributors have a good experience. They do this by getting community members the help they need to make PRs, keeping their PRs from getting stale, stepping in early if someone's work is out of scope, and redirecting high level questions/debates to discuss.filecoin.io.

If at any point during The Interrupt's shift, a situation arises that they're not sure how to respond to, they can go to #filecoin-moderation for an extra set of eyes. I will be available to deliver feedback and guidance at the end of each shift.

[phritz]

OK, we're implementing this. TODOs

• [x] add roles to docs
• [x] add role instructions to docs
• [x] find an rotation management tool
• [x] implement rotation
• [x] schedule @anoemi to present to the team in person