Sig-release: Async Release Team Project

[rawkode]

I've been on the release team since 1.15 and I've learnt a lot about the processes we have in place to deliver a version of Kubernetes. The weekly update for the release team happens consistently at 1700 GMT.

For our US team members, this is 0900 (West), 1200 (East)
For our APAC team members, this is 2300 (India), 0200 (Japan), 0400 (Australia)

Here's a breakdown of geographic distribution for 1.15 through 1.17:

1.15

Total: 44

US: 60% (25)
EU: 25% (11)
APAC: 15% ( 8)

1.16

Total: 38

US: 60% (23)
EU: 16% ( 6)
APAC: 24% ( 9)

1.17

Total: 34

US: 58% (20)
EU: 21% ( 7)
APAC: 21% ( 7)

---

With a synchronous update schedule / video call - bringing in shadows and leads from Europe can be difficult, especially those with families to tend to from 1800, and completely impossible for those from APAC.

I'd like to lead a team within SIG-Release to discuss, plan, and execute a remote-first-async release team process to expand our geographical reach for newcomers all over the world.

[rawkode]

ATTN @justaugustus @tpepper @calebamiles @claurence @guineveresaenger

[justaugustus]

/assign @rawkode
cc: @kubernetes/sig-release
/milestone v1.18
/priority important-soon
/kind feature

[alenkacz]

I actually really love the face to face meetings but I as I led CI signal this iteration I can confirm, that staying 6-6:30PM every day at the end of iteration is pretty hard.

Oh and just to be clear here, as much as I struggled with the timezones this iteration, I see a great value in having face to face meetings at least once, later even more times a week. Potentially a solution could be really structuring the teams to cover multiple timezones, potentially maybe even having meetings alternating between timezones?

[nikopen]

Awesome idea and an unavoidable eventuality given the upcoming dissolution of the release team as we know it.

100% support the cause, I'd love to see async comms being the norm.

[rawkode]

Now that the 1.17 release is out the door, I'll begin to put together my initial proposal and request for comments.

[alejandrox1]

Latest update on this: with 1.18 beginning, there has been no opposition to making the video call part of the release team meetings optional - still required for shadows who can attend to gain some experience giving the updates, discuss, etc.
However, we re treating as of now the meeting notes as the single source of truth.
Any and all important information should be included in the release team meeting notes and I (RT lead) make it a goal to go through the notes after end of day on Monday (the day of the release team meeting) to make sure all updates are correct.

For heavier discussion a GitHub issue and a reference to it within the meeting notes is also a requirement.

[alejandrox1]

@rawkode any updates on this?

[alejandrox1]

comment to everyone (specially non-US RT members): Please feel free to add your proposals for how we can move forward

[tpepper]

I like the meta idea here.

We need more discussion around what is needed for a workable decision making process in a fully async world.

The regular release team meetings are not solely about status read outs, and that portion certainly should be more async. The meetings' highest value IMO is in broader conversation around the evolving status, actively questioning "what if", having more eyes on spotting and getting ahead of potentially brewing issues and ultimately making decisions on what action is needed to proactively drive towards a successful release day. As it stands, as a project broadly we are not consistently great about reading all status, writing clear and fully detailed issues/PRs/commits/comments, "thinking out loud" about possible worries, and proactively drawing in all the right stakeholders for a discussion in GitHub (versus aiming for quick lgtm/approve/close/merge). This means the release team operates in an already lossy environment. Dropping the regular cross-all-issue, cross-all-PRs, cross-all-teams checkpoints could make it even more lossy, although that isn't the only possible outcome. We can do this better. It is a cultural improvement we should aim to make.

[xmudrii]

I like the idea and I agree that we should try to improve how we handle such stuff. I've been on the release team from 1.14 until 1.16, so my feedback is based on that experience. Also, I'm based in Europe, so my timezone is Central European (CET/CEST).

I agree that the highest value of the meetings is the conversation. We have had came up with many great ideas and IMO doing that offline, on Slack or GitHub, would not bring as good experience as doing that at the meeting. It would be much slower and the quality would not be that good.

But the key to successful meetings and good conversations is having a good meeting attendance. It is important that both leads and shadows can attend the meetings. For shadows, it's (usually) not required, but attending the meetings allows you to learn the decision making flow in an interactive way (which is important), but also to have your voice heard.

I'm aware that we can't have a meeting that's going to work for all timezones. But I still think we can do a little bit better job there:

• We should try to schedule meetings early morning US time/afternoon EU time.
  
  
  
  • IMO, 9 am PT was not the greatest slot, but it was good enough. It's still very bad for APAC folks, but at least it's (hopefully) good for both US and EU folks. Anything later then that is quite hard to make for EU folks. After a whole day of work/studying/whatever, having to stay until late to attend the meeting is not always possible. You'd have to take from your rest/family/friends time and that leads to burnout quite quickly.
  
  
  • Another slot to consider is 8 am PT. SIG-Release meetings are Monday 8 am and IMO attendance is quite good.
  
  
  • We have tried to hold EU friendly meetings before and IMO that didn't work well. Most of those meetings were about reporting and most of the conversation was left for the main meeting.
  
  
  • I'm aware that this is a complex topic. It depends on many factors, including what is going to work for the current leads and the current release team.
  
  
• We should reduce the number of meetings (in burndown):
  
  
  
  • I think we can agree that reporting can be done async and that the biggest value is the conversation. From my experience, having meetings 3 or 5 times a week is not really useful. At least 2 out of 5 meetings a week boils down to just reporting. We rarely have to make a decision each day, and even if that happens, I believe such a decision could be made on Slack or GitHub.
  
  
  • I'd propose that we have 2 meetings a week in early burndown and 3 meetings a week in late burndown. On-demand, we can always schedule another meeting, but it's relatively rare that we need all 5 meetings a week.
  
  

I believe that there are many other ways to solve this. This is what I had on the mind so far and I wanted to share my experience. I'm sure we can improve this and I'm also sure we'll eventually find a good way for that. :slightly_smiling_face:

[rawkode]

I was thinking we'd write a small application that allowed the release team Gmail alias members to upload a small video to a specific Google Drive folder, created for each week of the release

After a certain cut off date or signal from the release team lead, the app would merge the videos together and publish to YouTube.

[jtslear]

I'd like to propose another option that does not include video calls. Given that, the majority of the time, thus far that I've seen, is spent simply providing status updates, I propose that the release team is provided a special issue with a large template that can be filled out by any member that includes the necessary status updates. Use the below as an example:

One GitHub issue per week (I'll call this the tracking issue). In this tracking issue, all current release team members and shadows are tagged, and from here we can drive any necessary conversations. Since this is mostly a status update, we should be able to trigger a tracking issue creation at 00:00UTC Mondays with a goal of having this closed out by Tuesday 23:59UTC. This provides all teams roughly 2 days to complete the necessary updates and allows anyone to quickly read what is important to them and concentrate on any discussion points that do not to fall into a specific issue that may already be open. Though the goal would be that discussion points are brought up, and then continued in the appropriate issue; potentially creating one if necessary.

I imagine the template of this tracking issue is a repeat of the Google Doc, but due to this being inside of a GitHub, we can tag people a little easier and more quickly link issues/PR's inside of this tracking issue.

Doing this will add more stress to the current Release Team lead, which for 1.18 is @alejandrox1. I would imagine the need to ensure accurate information posted would be up to release team leads, but we will need someone to manage this issue. Perhaps if we go this route, parts of it can be automated as much as possible, where the only human interaction comes from questions about the status and discussion points. The ultimate goal would be that all status updates are made by EOD Monday, and all discussions are completed by, EOD Tuesday, after which the issue is closed. Any discussion items that come from this tracking issue should either have it's own issue for further discussion as needed.

---

The reason for this specific proposal removes the need for video calls. This will undoubtedly make conversation more difficult and is less personal overall, however, this is the trade-off when comparing sync vs async communications. Anytime a video call is scheduled, this is an interruption to the daily life flow of any member, and horribly unfair to persons where calls interfere with their personal schedule. Trying to find a time where calls can include the majority, is not solving the problem of allowing persons outside of that time range to be included and is therefore a very exclusive method of executing this meeting.

Members that are forced to watch recordings do not have the same way to contribute to conversations as persons who make it to the meetings. With this proposal, we keep all communications unified, inside of issues or PR's, throughout the release process.

---

I do not disagree with the above points, I simply would like to provide an alternate solution that is more accessible in general, despite not fulfilling all needs. Please keep in mind, 1.18 is my first time as a shadow, and therefore, I've not yet seen a full release cycle.

[saschagrunert]

I like the idea of having a tracking issue in GitHub for status updates. This would also help us to search for information in GitHub rather than the Google Docs.

I still think that we need room for open discussions and other topics than just status reports. For that I would stick to the video call. Once a week should be fine in my point of view, but maybe on alternating times.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[LappleApple]

This topic came up as a priority during the recent RT prioritization sessions around process improvements. One breakout group expressed strong interest in having at least one session (~45 mins) to resolve this topic as well as a related one: establishing continuity of work once a RT disbands, to ensure functioning/proper handoffs (+h/t @justaugustus, who also brought this up in a chat recently.)

To summarise (hopefully accurately!) the needs and options identified above so far (can think of this as a potential draft agenda for such a discussion):

• structuring the teams to cover multiple timezones, potentially maybe even having meetings alternating between timezones? (@alenkacz; +1 @saschagrunert)
• discussion around what is needed for a workable decision making process in a fully async world...broader conversation (+1 from @xmudrii) around the evolving status, actively questioning "what if", having more eyes on spotting and getting ahead of potentially brewing issues and ultimately making decisions on what action is needed to proactively drive towards a successful release day (@tpepper)
• important that both leads and shadows can attend the meetings (@xmudrii)
• reduce the number of meetings (in burndown) (@xmudrii)
• could write an application allowing the release team Gmail alias members to upload a small video to a specific Google Drive folder
• to remove the need for using video calls for status updates, release team can be provided a special issue with a large template/tracking issue in GitHub that can be filled out by any member that includes the necessary status updates (+1 @saschagrunert); someone needs to manage the template/issue (@jtslear)

FWIW, on the last point I've created such templates and agree that using them for status updates is often preferable to sharing them in the moment/out loud, but also agree with the points expressed here about making space and time for thoughtful conversation.

As for the handoffs: some of the motivations behind this idea include (h/t Stephen):

• giving release team members the opportunity to keep building knowledge about relevant topics after the cycle ends, and to notice patterns over time as they go along
• Is an issue representing broader failures? Different systems?
  Try to build people into leads by giving them time+space to identify and drive bigger/higher-impact systems efforts

Everyone on this thread is invited to LMK if the face-to-face session is of interest—it's not scheduled yet.

[LappleApple]

cc @palnabarun @divya-mohan0209 @bai @mkorbi @harshanarayana, as you noted during the prioritisation breakouts that we should prioritise this item

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[mkorbi]

/cc @LappleApple
just wanted to left my 5ct:

• Maybe we should think about a heliocentric release meeting model, means ether having a US/EU and a US/APAC meeting per week, or every second week the one and the other
• strongly utilize the issues and project boards to document everything
• rethink the way of doing the release: coming closer to the release we increase meetings from 1 to 5; I personally don't find all of them helpful, most of them only take 10-15 min. Why not do the following:
  Having a release team meeting until code freeze once peer week e.g. Monday and after code freeze in addition Thursday. Besides this, each team should set up their own regular meeting as it is easier to find in a small-time a good time then for a big team. Each team (enhancements, comms, docs, release note, signal) also has its own Agenda (not part of the release team agenda). This is to easier follow the subject per team. I find the agenda in a doc for the whole release train not really user-friendly.
  With this, the release lead team can easier follow what's going on per subteam, and for example, if there are 4 people in the lead, each can have an eye on 2 subteams.

That model overall we will have less big team meetings but potentially a higher engagement and condensed information status.

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.