Test-infra: Kubernetes CI Policy: merge-blocking jobs must run in dedicated cluster

[spiffxp]

Part of https://github.com/kubernetes/test-infra/issues/18551

Why this is necessary:

• we believe declaring Guaranteed Pod QOS jobs may not be defended against Best Effort or Burstable Pods hogging all resources on the same node
• a cluster that only has Guaranteed pods is far more likely to respect resource requirements

Decisions to make:

• For all of the jobs being migrated to k8s-infra-prow-build, we're going to use the same node pool as everything else. To pin to a dedicated node pool will require much more boilerplate (or possibly augmenting prow's preset feature). If after migrating everything we find we _do_ need a dedicated nodepool, then we'll pay the added cost.
• Jobs that can't be migrated quickly will remain in the default google.com-owned k8s-prow-builds cluster until such time as they can be migrated. They will continue to compete for resources until they have migrated, and we will be reliant on test-infra-oncall to provide any visibility into the behavior of such jobs.

Jobs to migrate:

• These jobs don't need any e2e projects
  
  
  
  • [x] https://github.com/kubernetes/test-infra/issues/18846 - pull-kubernetes-dependencies
  
  
  • [x] https://github.com/kubernetes/test-infra/issues/18811 - pull-kubernetes-integration
  
  
  • [x] https://github.com/kubernetes/test-infra/issues/18848 - pull-kubernetes-typecheck
  
  
  • [x] https://github.com/kubernetes/test-infra/issues/18849 - pull-kubernetes-verify
  
  
• These jobs use kind, also don't need e2e projects
  
  
  
  • [x] https://github.com/kubernetes/test-infra/issues/18850 - pull-kubernetes-conformance-kind-ga-only-parallel
  
  
  • [x] https://github.com/kubernetes/test-infra/issues/18812 - pull-kubernetes-e2e-kind
  
  
  • [x] https://github.com/kubernetes/test-infra/issues/18812 - pull-kubernetes-e2e-kind-ipv6
  
  
• These jobs can use standard gce-project e2e projects
  
  
  
  • [x] https://github.com/kubernetes/test-infra/issues/18852 - pull-kubernetes-e2e-gce
  
  
  • [x] https://github.com/kubernetes/test-infra/issues/18851 - pull-kubernetes-node-e2e
  
  
  • [x] https://github.com/kubernetes/test-infra/issues/18813 - pull-kubernetes-e2e-gce-ubuntu-containerd
  
  
  • [x] https://github.com/kubernetes/test-infra/issues/18853 - pull-kubernetes-e2e-gce-network-proxy-grpc
  
  
  • [x] https://github.com/kubernetes/test-infra/issues/18854 - pull-kubernetes-e2e-gce-network-proxy-http-connect
  
  
• These jobs may need scalability-project
  
  
  
  • [x] https://github.com/kubernetes/test-infra/issues/19067 - pull-kubernetes-e2e-gce-100-performance
  
  
• These jobs need to be migrated off of RBE
  
  
  
  • [x] https://github.com/kubernetes/test-infra/issues/19073 - pull-kubernetes-bazel-build
  
  
  • [x] https://github.com/kubernetes/test-infra/issues/19070 - pull-kubernetes-bazel-test
  
  

TODO:

• [x] https://github.com/kubernetes/test-infra/pull/18576 - write a test that enforces this policy, but logs instead of errors
• [x] determine gcp project / boskos requirements, provision
  
  
  
  • [x] https://github.com/kubernetes/test-infra/issues/18552 - migrate away from "k8s-infra-gce-project" and stick to "gce-project" for less config munging
  
  
  • [x] https://github.com/kubernetes/k8s.io/issues/1078 - gce-project
  
  
  • [x] https://github.com/kubernetes/k8s.io/issues/851 - scalability-project
  
  
• [x] migrate jobs that require special projects
• [ ] declare we are finished (or done enough to move on)

[spiffxp]

sig testing
/sig release
/wg k8s-infra
/area release-eng
/area jobs

[spiffxp]

I'm trialing migration of three different kinds of jobs to get a feel for whether the rest can be migrated over in a similar fashion, or whether more preparation is needed:

• https://github.com/kubernetes/test-infra/issues/18811 - integration (no e2e project)
• https://github.com/kubernetes/test-infra/issues/18813 - ubuntu-containerd (e2e project)
• https://github.com/kubernetes/test-infra/issues/18812 - kind

Based on how these go I'll break out other TODO's into help wanted issues

[ZhiFeng1993]

/cc

[spiffxp]

OK, I've broken out everything I think can be done right now, tagged as help-wanted, and added to the CI Policy Improvements - Next Priority column

The remaining TODO's need some unblocking work

[spiffxp]

We may need to roll some jobs back or adjust some limits. Started bumping into a quota we're having difficulty raising https://github.com/kubernetes/k8s.io/issues/1132#issuecomment-678389503

[spiffxp]

IP quota has been bumped to a reasonable level. I'm now less concerned about trying to mitigate or undo migration of the jobs that have been migrated thus far.

[spiffxp]

https://github.com/kubernetes/k8s.io/issues/1187 trying to improve cluster's I/O capacity, I'm wary of moving over pull-kubernetes-bazel jobs until this is addressed

[spiffxp]

https://github.com/kubernetes/k8s.io/issues/1231 we're starting to hit the max nodepool size of 90 so I'd like to increase to 150

[LappleApple]

Heya: +1 if we can get this updated with the checkbox marked for #18854

[LappleApple]

In the write-up, under

_TODO: determine gcp project / boskos requirements, provision_

This item is now closed:
_kubernetes/k8s.io#851 - scalability-project_

Is that TODO now complete?

Also wondering if the other items in the TODO section:

TODO: migrate jobs that require special projects
TODO: declare we are finished (or done enough to move on)

need an update?

[spiffxp]

@LappleApple https://github.com/kubernetes/test-infra/issues/19073 is the last remaining issue

[spiffxp]

Closed out the last remaining issue https://github.com/kubernetes/test-infra/issues/19073#issuecomment-756848275

All merge-blocking jobs run in k8s-infra-prow-build. Last thing to do is switch job config tests to fail instead of warn to prevent future violations of this policy.

[spiffxp]

Opened https://github.com/kubernetes/test-infra/pull/20416 to switch job config tests