Enhancements: [Process improvement]: Make an easy-to-navigate website to find and query KEPs

[LappleApple]

This issue documents the roadmap/plan for a key 1.20 process-related objective.

What Is the Website

• An up-to-date index with links to tracking issues and KEPs
• Displays KEPs similar to how Python displays PEPs, with more detail
• Would be part of the Contributor website

Why Do We Need It

• Helps fulfil needs raised by @jberkus's Make KEPs transparent issue

Who's Needed, Who's Helping, Who's Informed

Needed:

• _SIG contributors_ for Step 1, to update KEP metadata
• _Anyone with experience working with Hugo data templates_ for steps 2-4

Helping:

• @palnabarun has volunteered to work on this
• @sftim will look for “Good First Issues”

Informed:
/sig docs
/sig contributor-experience
/sig architecture

How Do We Get There: Implementation Steps and Notes

• [ ] 1. SIGs get their metadata happy/valid, because the site will read from this metadata
• [ ] 2. Programmatically generate KEP metadata dump for Hugo (kepify, jq/yq etc)
  
  
  
  • For Netlify/Kepify, maybe an NPM wrapper that then runs Hugo
  
  
  • Kepify = Netlify: will either be Go or bash (if a script to produce it)
  
  
  • Once the metadata is consistent, we can render it all in a table quite easily
  
  
  • We don't have to surface the KEP directly yet, just the metadata and links back to the actual thing
  
  
• [ ] 3. Automate updating of data on merge (prow job dump, a JSON blob somewhere, etc)
• [ ] 4. Integrate data dump into Hugo
  
  
  
  • Hugo can query for data at site build time, including from public HTTP APIs via Hugo Data Templates
  
  

Related Action Item: Preparing for 1.21

• @kikisdeliveryservice will survey not-updated KEPs + their enhancement status and use results to assert that updates should be required for inclusion in a 1.21 release via kepval or something else; we'll make this blocking in 1.21
  
  
  
  • We have the most leverage when people want to be in the milestone
  
  
  • people are updating at our request now because they want to be in the milestone
  
  

Potential Next Iterations/Milestones

• Using the metadata, capture which release a KEP went into, at what stage
  
  
  
  • Making it so that docs contributors no longer have to use git bisect against k/k to work out this kind of detail
  
  
• Linking to Feature Gates
  
  
  
  • SIG Docs contributors would like to use this effort as groundwork for a future enhancement: tracking all Feature Gates and having their status available as an API that can be consumed by https://kubernetes.io/; graduating a feature automatically, updating documentation pages that mention it.
  
  
• Supports Social media guidelines for Enhancements (holding this until the website is running)

[palnabarun]

/area enhancements

[gkarthiks]

@LappleApple I am interested in this. Please ping me on Slack.

[jberkus]

A site designed after the PEP page will fulfill most of the transparency requirements. The one additional feature we'll eventually need that PEP doesn't have is to have lists of:

• current work tied to each KEP (issues and PRs)
• planned deliverables for the KEP, by release no.

Copying the PEP page would be a good first step, though, and way more accessible than what we have now.

[LappleApple]

Some user stories drafted in June 2020 for KEPs tooling, based on Make KEPs Transparent and misc. conversations:

• As a Kubernetes contributor, I would like to find useful information (including SIG) about current KEPs in an automated and searchable way, so that I can consider contributing to efforts that the community deems valuable.
• As a SIG Lead, I would like to find out in an automated and searchable way what are the current active KEPs in my SIG and their statuses, so that I can track responses to PR reviews after those PRs have been merged and see which KEPs are making/missing release cycle timelines.
• As a new Kubernetes contributor, I would like to know which KEPs in any SIG touch on, or even subsume, a proposed feature, so that I can dedicate my time effectively.
• As a KEP owner or stakeholder, I would like to find (in a searchable and automated way) issues and PRs that are blocking any current KEPs, so that I can move them along.
• As a member of ContribEx, I would like to find the latest KEPs so that I can promote them for review/commentary.
• As a member of ContribEx, I would like to see active KEPs in a website format to make them more transparent and discoverable.

[kbhawkey]

Hi @LappleApple .
I may have some time to pitch in with the templating piece.

[sftim]

OK, here's a mockup of a future https://k8s.dev/resources/enhancements/:

What do we need to render that? The minimum is that for each KEP we have a bit of metadata.
For the mockup the minimum you need is like this:

path: "/keps/sig-contributor-experience/1553-issue-triage"
title: Issue Triage Workflow and Automation
kep-number: 1553

or

title: Implement maxUnavailable for StatefulSets
path: "/keps/sig-apps/20190226-maxunavailable-for-statefulsets.md"

(you _don't_ need a KEP number - but not having it makes them hard to sort!)

If those files end up in data/generated/enhancements before site build, the mockup adds them to the KEP list.

https://github.com/kubernetes-sigs/contributor-site/compare/master...sftim:20201017_kep_list_mockup shows the Hugo & Markdown changes I made for the mockup.

---

How it could work:

• add a new GitHub Action to this repo that generates a _single_ JSON document listing all KEPs
  
  
  
  • I'm thinking to GitHub Pages
  
  
  • doesn't have to be perfect, so long as title and path are available
  
  
• use Hugo's ability to fetch remote data, so that https://k8s.dev/ fetches that JSON during build
  and uses it to render a KEP list.
• https://k8s.dev/ builds on a timer, so new KEPs get picked up automatically within a few hours
  
  
  
  • extra credit: have a build to this repo trigger a notification to rebuild https://github.com/kubernetes-sigs/contributor-site/
  
  

[sftim]

Questions about that mockup?

[mrbobbytables]

I'd go the table formatting route, but we should be able to get at a minimum from the metadata:

• title
• status
• owning-sig

Once the enhancement structure is in place, we can get the kep # (and associated tracking issue).

I'm thinking to GitHub Pages

GitHub pages would require an additional commit any time a kep is updated =/ I think we could provision a gcs bucket and dump the content there as a postsubmit job.

[palnabarun]

@sftim -- that mockup looks great.

I have a few ideas that I was experimenting with related to preparing the data for the website.

One question I had though, do the KEP markdown files need to have certain Hugo metadata in order to be rendered by Hugo during the site generation phae?

I think we could provision a gcs bucket and dump the content there as a postsubmit job.

@mrbobbytables -- Putting the JSON into a GCS bucket sounds great. 👍🏽

I will get back with more updates by next Monday as I don't have free cycles to work on this during the work week. :(

[mrbobbytables]

One question I had though, do the KEP markdown files need to have certain Hugo metadata in order to be rendered by Hugo during the site generation phae?

We don't want to render the keps themselves in hugo (at this point). We do want to provide a list of them from a single page.

As someone that spent many hours trying to fix rendering of them in hugo (the original goal of the contrib site was to list keps). It very much is not a path forward without significant effort in fixing paths, links and more. =/

[palnabarun]

We don't want to render the keps themselves in hugo (at this point). We do want to provide a list of them from a single page.

Okay. That clears the doubt. Thank you! 😃

[sftim]

provision a gcs bucket and dump the content there as a postsubmit job

Something I've thought of doing as a place to dump generated JSON: use the repo wiki (which is actually a Git repo), and have the GitHub Action update a JSON file in that repo every time someone merges to the main branch in the main repo.

Depends what we think about GitHub Actions though!

[kbhawkey]

:eyes:

[kbhawkey]

Hi. I don't know whether you need more help.
Do you have a program to create the Markdown files from the KEP data?
I'd like to see what a list of KEPs looks like. Is there an existing site?

[palnabarun]

Hi everyone :wave:, I was experimenting with a change in kepctl to output the list of KEPs in YAML/JSON. My changes are in this tree and I shared a sample output here.

Broadly, the tasks to complete this look like the following:

• [ ] Make kepctl query be able to output YAML/JSON, say keps.yaml. (#2139)
• [ ] Create a GCS bucket (let's say kubernetes-enhancements-artifacts) to store the keps.yaml.
  This bucket should be writeable only by the workload in the next item. It may be read by anyone at the moment since the only thing we are storing at the moment is keps.yaml which is public information.
• [ ] Create a Prow postsubmit job for kubernetes/enhancements to generate the keps.yaml after every merge to master and upload the same to the kubernetes-enhancements-artifacts GCS bucket.
• [ ] Implement the KEP table in kubernetes-sigs/contributor-site which will get populated at build time by the contents of keps.yaml.
• [ ] Create a Prow periodic job for kubernetes-sigs/contributor-site to rebuild the website.

Let me know what do you think of the plan.

[kbhawkey]

Hello @palnabarun .
Do you want to publish/republish the "chart" to the contributor site every time keps.yaml is changed or
could you batch the updates (small changes)?
How often during a release is this data updated? Does the information remain on the contributor site after the release?
Do you plan on showing historical data from previous releases?

[sftim]

BTW I looked again and JSON is going to be easier to work with with YAML, for the Kubernetes contributor website.