Test-infra: 1.12+ should not exclude kubectl with skip regex for Conformance tests

[BenTheElder]

https://github.com/heptio/kube-conformance/commit/c4eb36ad07a5d7d7e0a47bb6535f787d589b729a

we need to:

• update master / 1.12 jobs
• update docs in test-infra, and the community developer guide, possibly elsewhere.

probably inform sig-cloud-provider members that publish continuous signal to testgrid with kubetest. cc @spiffxp @dims @Katharine

[spiffxp]

/area conformance

ref: https://github.com/heptio/kube-conformance/issues/29

[spiffxp]

I would much rather complete "pulling the skiplist upstream", aka stripping the Conformance tag from any tests that are skipped by this regex, such that focusing on Conformance is all that's needed

[spiffxp]

/area job
/sig testing
/sig gcp
/sig architecture
/priority important-soon
/kind cleanup

[spiffxp]

ref: https://github.com/kubernetes/kubernetes/pull/67901 to drop a [Flaky] test

[spiffxp]

ref: https://github.com/kubernetes/kubernetes/issues/62161 the umbrella issue in k/k for Serial / Flaky / Slow... at present I am only interested in Flaky. Serial and Slow may be necessary evils.

[spiffxp]

ref: https://github.com/kubernetes/test-infra/pull/9104 where we introduced the skiplist in the first place, this can be considered closed once we've eliminated it

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[spiffxp]

/remove-lifecycle stale
/milestone v1.14
Keeping this open as the "pulling the skiplist upstream" issue re: https://github.com/kubernetes/test-infra/issues/9697#issuecomment-427067895 unless I find a better issue for it

[spiffxp]

I stripped the skip regex from the conformance image we are hoping to publish as part of this release https://github.com/kubernetes/kubernetes/pull/74808#issuecomment-468860534

We'll compare the list of tests that runs to the list of tests our jobs run. If they match, let's drop the regex from 1.14

[spiffxp]

/milestone v1.15
/assign
I'm pretty sure this is either done, or can be done, I need to verify list of tests

I would like to ensure conformance tests are prevented from using these tags ever again

[BenTheElder]

this bit me submitting the kind 1.14 results (they were accepted though! I amended the next day ❗️), we've removed the skip from kind everywhere (except the parallel / non-serial runs), there might still be something out there using them incorrectly, but those were the last ones I know about.

[spiffxp]

Verified the list of tests for v1.15 and v1.14 by using the kind and kind-image jobs, eg:

• https://testgrid.k8s.io/conformance-all#kind-image,%201.14%20(dev)&sort-by-name=
• https://testgrid.k8s.io/conformance-all#kind,%20v1.14%20(dev)&sort-by-name=

[spiffxp]

I opened https://github.com/kubernetes/kubernetes/pull/78710 to enforce the regex as a gate to promotion to conformance

[spiffxp]

Opened https://github.com/kubernetes/test-infra/pull/12887 to strip the regex from as many cargo cults as I could find

[spiffxp]

/close
No complaints since #12887 merged, meant to close this a few weeks ago but forgot

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

/close
No complaints since #12887 merged, meant to close this a few weeks ago but forgot

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.