Test-infra: Clean up the experiments directory.

Created on 7 Sep 2019 · 14Comments · Source: kubernetes/test-infra

What should be cleaned up or changed:
The experiments directory has grown a bit out of control and contains a lot of code that is not experimental and would benefit from better organization. I see a mix of config generators, version bumping tools, setup utilities, metric scripts, etc.
It would be great to move some of these tools into semantically scoped directories (existing or new). Similarly there are some directories at the repo root that might be better as subdirectories.

/help
cc @Katharine @spiffxp @stevekuznetsov @chases2

help wanted kincleanup lifecyclrotten

Source

cjwagner

❤1

Most helpful comment

Might as well list them here and give our opinions for what to do with each, then use this as a tracker

stevekuznetsov on 9 Sep 2019

👍2

All 14 comments

@cjwagner:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

What should be cleaned up or changed:
The experiments directory has grown a bit out of control and contains a lot of code that is not experimental and would benefit from better organization. I see a mix of config generators, version bumping tools, setup utilities, metric scripts, etc.
It would be great to move some of these tools into semantically scoped directories (existing or new). Similarly there are some directories at the repo root that might be better as subdirectories.

/help
cc @Katharine @spiffxp @stevekuznetsov @chases2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot on 7 Sep 2019

Might as well list them here and give our opinions for what to do with each, then use this as a tracker

stevekuznetsov on 9 Sep 2019

👍2

+1
/assign

chases2 on 9 Sep 2019

A couple of these folders are old, but most of them seem to be seeing active use.

Summary (Freshness == Last Substantive Edit)

|Folder                 |Contents                           |Freshness |
|-----------------------|-----------------------------------|----------|
|add-hook               |Go Program (no doc)                |May       |
|autobumper             |Go Program (no doc)                |September |
|aws-stockout           |Go Program: ???                    |2018 June |
|cherrypicker           |Link to Other Directory            |2018 March|
|ci-janitor             |Go Program (no doc)                |March     |
|config-forker          |Go Program: Handles Prow Annotation|August    |
|config-rotator         |Go Program: Handles Prow Annotation|May       |
|coverage               |Go Program: Measures Test Coverage |2018 April|
|dummybenchmarks        |Go Program (no doc)                |April     |
|image-bumper           |Go Program: Deployment Utility     |May       |
|logviewer              |Go Program: ???                    |April     |
|maintenance            |Python Scripts: ???                |March     |
|manual-trigger         |Go Program (no doc)                |August    |
|resultstore            |Go Program (no doc)                |June      |
|service-account-creator|Go Program (no doc)                |August    |
|slack-oncall-updater   |Go Program (no doc)                |August    |
|tracer                 |Go Program (no doc)                |August    |

There's one piece of low-hanging fruit, and two directories from 2018 that should be reevaluated.

chases2 on 10 Sep 2019

And for the individual scripts, I included my suggestions for sensible-sounding subdirectories.

|Script                                      |Apparent Function                                        |Freshness|Move To                                                |
|--------------------------------------------|---------------------------------------------------------|---------|-------------------------------------------------------|
|bilge.py                                    |Cancels Duplicate Tests in Prow                          |2017 Feb |Delete                                                 |
|bump_e2e_image.sh                           |Bump Kubekins                                            |April    |"Deployments"                                          |
|dump_kind_coverage.sh                       |Dumps logs from a kind node                              |2018 Oct |"Debug"                                                |
|find-merge-conflicts.sh                     |Checks unmergable PRs to see why they're unmergable      |2018 Apr |"Git Utils"                                            |
|find_developers.py                          |Randomly selects K8s Developers                          |2017 Jan |Delete                                                 |
|flakedetector.py                            |Counts flaky tests using Prow data                       |2017 Aug |???                                                    |
|generate_tests.py and generate_tests_test.py|Generates Tests based on a Prow Config & a utility config|July     |"TestGrid Utils"> "generate_tests" (and/or to Prow Job)|
|get_job_pods.py                             |Print every job and the pod it ran on                    |2018 Aug |"Debug"                                                |
|graphql_issue_example.py                    |Runs GitHub v4 API "Get Issues"                          |2017 Aug |"Git Utils"                                            |
|kind-conformance-image-e2e.sh               |??? (@BenTheElder)                                       |Aug      |???                                                    |
|kind-coverage-dump.sh                       |Dump logs from a kind node                               |Feb      |"Debug"                                                |
|migrate_testgrid_tabs.py                    |Turns TestGrid Config into Prow Annotations              |July     |"TestGrid Utils"                                       |
|parse_build_log.py                          |Parses logs (which?) for overlapping tests (?)           |2018 Feb |"Debug"                                                |
|pod.yaml                                    |K8s Pod Config for "k8s-e2e-test"                        |2017 June|???                                                    |
|prepare_release_branch.py                   |Runs some release branch tasks                           |June     |"Deployments"                                          |
|test-configmap.txt                          |File with three lines                                    |March    |???                                                    |
|test_config.yaml                            |The configuration file for generate_tests.py             |September|Wherever generate_tests.py ends up                     |

chases2 on 10 Sep 2019

👍1

In general a lot of these tools are in experiment/ because they aren't really built for re-use or production grade, however they are useful and it tends to be helpful to have most of them already checked in rather than constantly re-implementing them ...

Details for a few of these:

|maintenance |Python Scripts: ??? |March |

experiment/maintenance has a readme explaining:

This directory contains hacky scripts / tools used to maintain the Kubernetes Prow deployment.

Each of those scripts contains usage internally and have been used intermittently.
They target stable tools / formats etc. and do not need regular substantive changes, however they're not production grade and low effort has been put into reuse by third parties.

Also noting that most(?) of the sub directories of experiment/ have READMEs like this..

EG:

|aws-stockout |Go Program: ??? |2018 June |

Has one explaining what it is and why it exists. This particular one appears not to be in use however.

|kind-conformance-image-e2e.sh |??? (@BenTheElder) |Aug |??? |

This is for testing the kubernetes/kubernetes e2e image (used by sonobuoy in particular). It is roughly equivalent to kubetest if it were only built for usage in our CI and only for this particular job. It's just some glue a Kubernetes CI job uses at the moment.

@johnSchnake also nominally owns this and possibly @dims.

|kind-coverage-dump.sh |Dump logs from a kind node |Feb |"Debug" |

That'd be @Katharine. It's probably out of date currently.

|generate_tests.py and generate_tests_test.py|Generates Tests based on a Prow Config & a utility config|July |"TestGrid Utils"> "generate_tests" (and/or to Prow Job)|

This has nothing to do with Testgrid ..?

|slack-oncall-updater |Go Program (no doc) |August |

This updates the @oncall alias in slack. @Katharine owns this.
This could maybe move to slack-infra /shrug

BenTheElder on 17 Sep 2019

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 16 Dec 2019

/remove-lifecycle stale

chases2 on 3 Jan 2020

@chases2 -- @matthyx mentioned that this might be done now that https://github.com/kubernetes/test-infra/pull/15578 has merged?

Will leave you to close it out if that's the case.

justaugustus on 21 Jan 2020

ping @chases2

matthyx on 28 Feb 2020

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 28 May 2020

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot on 27 Jun 2020

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

fejta-bot on 27 Jul 2020

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close