Sig-release: Investigate automation to facilitate tracking spreadsheets

[spiffxp]

From 1.13 retro:

• We attempted automated Issue and test failure tracking this cycle, though we ran into account quota issues etc. Irrespective of how useful it was this release, this is something we can definitely improve and stabilize in future cycles. [aishs]

• Zach Arnold in 1.12 trialed an AirTable integration that did some successful tracking.
• Need something that interacts with GitHub APIs, and not HTML scraping.
• This is an opportunity here for an AirTable / API / App developer (ie: doesn’t require kubernetes-internals know-how...Good First Issue, perhaps internship?)
• Using spreadsheets doesn’t work well, is highly manual.
• 1.13’s attempts were important R&D, we need to continue the exploration toward an eventual solution
• TODO: write up what was done, what is required, make a spec for a coder to implement. @jberkus can do it for CI, need somebody else to add for issues/PRs. @guineveresaenger could mention requirements at GitHub, which might be interested in the feature need. @spiffxp will drive/delegate in 1.14

[spiffxp]

/milestone v1.14
/kind design
/assign @spiffxp @jberkus @nikopen @tpepper @AishSundar
I'm assigning everyone listed in the "Action Items" section, please feel free to unassign yourself if you don't have the bandwidth / interest for this.

I've poked a tiny little bit at scraping GitHub things into AirTable, see this google doc (all can comment)

[justaugustus]

/assign @spiffxp @jberkus @nikopen @tpepper @AishSundar
/help

[k8s-ci-robot]

@justaugustus:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/assign @spiffxp @jberkus @nikopen @tpepper @AishSundar
/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[nikopen]

@spiffxp @justaugustus After a lot of brainstorming with my 1.14 triage team I've come up with a series of proposals which will negate the need for a spreadsheet/airtable mechanism. It will be mostly kanban/project board automation which is super easy to create + simple github queries with revamped labels. I'll present them sometime soon, either Retro or sig-release meeting

On spreadsheets themselves, the verdict is that maintenance of external tooling is cumbersome and not easy for the longer run. A simple dynamic webpage that queries github and has some state for notes would be good but not super useful in general

[justaugustus]

/priority important-longterm
/milestone v1.15

[spiffxp]

/milestone clear
/assign @guineveresaenger
this sounds like what you're looking to use project board automation for

I got as far as https://github.com/spiffxp/gitable/tree/breaking-changes to use something to scrape github into airtable. I ended up abandoning because at the time the free version of airtable wasn't amenable to charting/aggregating. Also, either the api or golang bindings didn't support auto-creation of multiple-choice options (which meant lots of manual updating as new GitHub labels got added)

eg:

source .envrc # put AIRTABLE_APIKEY, AIRTABLE_BASEID, AIRTABLE_TABLE, GITHUB_TOKEN in env
gitable --once --autofill --orgs kubernetes --milestone v1.16 --type issue

populated: https://airtable.com/shrDIwOSIJ4RgaUYk

[justaugustus]

/milestone v1.16

Who's owning this at the moment? @mrbobbytables ?

[mrbobbytables]

I've done a little bit of automation to improve the enhancements spreadsheet. I can look at doing similar things for the others as a first pass.

/assign

[tpepper]

Another area of spreadsheets based status tracking:
https://github.com/kubernetes/sig-release/pull/782

[josiahbjorgaard]

/assign josiahbjorgaard

[mrbobbytables]

/milestone v1.17

[mrbobbytables]

I'll limit this to the changes made in the Enhancement tracking sheet and let @josiahbjorgaard fill in the details for the bug triage sheet.

There is room for further improvement (called out below), but it is fairly limited without some greater KEP workflow changes.

---

Enhancement tracking across releases

Related issue: https://github.com/kubernetes/enhancements/issues/618

The [Enhancement tracking sheet] has a [data] sheet that serves as the source of
truth for Enhancement tracking and contains up-to-date information on overall KEP
state as of the 1.17 release. It uses the GitHub issue number as a key to fetch
the Enhancement issue name and generate the associated link. Updates are triggered
by calling a script via _Enhancement_ -> _Update Enhancement Issues_ drop down.

Fields:

• Issue - GitHub issue for the Enhancement to be tracked.
• Name - Name + link to GitHub Enhancement issue.
• Responder - the person last actively responding to Enhancement team member
  inquiries on the KEP tracking issue.
• SIG - Owning SIG for the KEP.
• KEP - No KEP, link to KEP PR, or link to merged KEP.
• KEP State - none, provisional, implementable, etc.

These fields are less likely to change between releases and are referenced by
other sheets to auto-populate their own fields.

All but Name and SIG are provided by Enhancement team members. SIG is
only auto-populated if the KEP link itself is provided.

Potential further automation

• Automatically add new issues to data sheet. If the GitHub issue template is
  updated to include the label kind/kep or a new enhancement tracking specific
  label, they could be added to the data sheet when _Update Enhancement Issues_
  is run. This might be moot because every issue requires some level of triage
  by the Enhancement team members.
• Responder could be replaced by those assigned to the issue, or assigned in
  the KEP, but it is RARE that the people assigned or referenced in the KEP
  are the ones actively responding on the Enhancement issue. This usually means
  an Enhancement team member go through the issue from bottom to top hunting
  down the last person to actually respond to the issue to ask regarding its
  current state.

Notable hinderance for further automation:

• Parent Enhancement issues are frequently out of date and do not follow consistent
  formatting. Using a regex / checking labels for SIGs gets very inconsistent and
  generally out of date results.
• KEPs / KEP metadata are frequently inconsistent and out of date.
• Some Enhancement issues are created as a _request_ for a feature and are not
  actually proposed or being handled by a SIG.

Enhancements sheet improvements

Multiple small improvements were made to the primary Enhancement tracking sheet.

• General information is automatically populated based off of the GitHub issue
  number and pulled from the [data] sheet.
• KEPs that are Removed from the milestone, or Deferred are moved to the
  Removed from milestone sheet by means of the _Enhancements_ ->
  _Remove Enhancements from the Milestone_ drop down. To move back from the
  Removed from Milestone sheet, the Enhancement status is set to Tracked and
  use the _Enhancements_ -> _Track Removed Enhancement_ dropdown.
• Proposal and KEP State are color coded indicating if a KEP is "At Risk"
  
  
  
  • Proposal
  
  
  • KEP is merged
  
  
  • KEP PR in flight
  
  
  • No KEP or PR found
  
  
  • KEP State
  
  
  • Implementable or Implemented
  
  
  • Provisional
  
  
  • none or invalid
  
  

Potential further improvements

• Break out docs content to their own sheet. This would be a larger workflow
  change for docs team, but the Enhancements sheet itself has been very
  overloaded. It might make sense to break out Enhancements tracking from docs
  tracking and have a slimmed down report page that is along the lines of:
  
  
  
  • Enhancement Name
  
  
  • Enhancement Implementation Status (Tracked, At Risk etc)
  
  
  • Enhancement Docs Status (No PR, Merged etc)
  
  

Stat tracking improvements

• Global KEP stats based on: No KEP, KEP PR in flight, Has KEP
• Break down of Enhancements that are At Risk by SIG.

[guineveresaenger]

Asking for input from CI Signal @alenkacz, Docs @daminisatya and Bug Triage @josiahbjorgaard.

Let's get tracking sheets all onto the same workflow.

[daminisatya]

+1

[josiahbjorgaard]

I'll do similar in the near future, but for what it's worth bug triage is not as advanced as enhancements. I have the feeling that sheets is not scalable, though.

[mrbobbytables]

Here is a WIP version with the separation of docs and enhancements with a dashboard for a quick 'at a glance' view
https://docs.google.com/spreadsheets/d/1Wyb6P8SVVG-LySEAYYokwkoVbgEsgyoFPW2XHxyYxFY/edit?usp=sharing

[guineveresaenger]

TODO: add information on how this works to Enhancements/Docs Handbook; and then this is done!

[alejandrox1]

/milestone v1.18

[alejandrox1]

@jeremyrickard @droslean @smourapina @VineethReddy02 @evillgenius75 @karenhchu could you all please take a look at this issue and post whether your team would benefit by having some spreadsheet with enhancements-like type automation (please see discussion above).
If you don't need it then please comment that as well. Wanna see what needs to be done for this issue :smile:

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.