Test-infra: release-blocking jobs must run in dedicated cluster: ci-kubernetes-build-fast

[spiffxp]

What should be cleaned up or changed:

This is part of #18549

To properly monitor the outcome of this, you should be a member of [email protected]. PR yourself into https://github.com/kubernetes/k8s.io/blob/master/groups/groups.yaml#L603-L628 if you're not a member.

NOTE: I am not tagging this as "help wanted" because it is blocked on https://github.com/kubernetes/k8s.io/issues/846. Here is my guess at how we could do this:

• create a duplicate job that pushes to the new bucket writable by k8s-infra-prow-build
• ensure it's building and pushing appropriately
• update a release-blocking job to pull from the new bucket
• if no problems, roll out changes progressively
  
  
  
  • a few more jobs in release-blocking
  
  
  • all jobs in release-blocking that use this job's results
  
  
  • a job that still runs in the "default" cluster
  
  
  • all jobs that use this job's results
  
  
• rename jobs / get rid of the job that runs on the "default" cluster

It will be helpful to note the date/time that PR's merge. This will allow you to compare before/after behavior.

Things to watch for the job

• https://prow.k8s.io/?job=ci-kubernetes-build-fast
  
  
  
  • does the job start failing more often?
  
  
  • does the job start going into error state?
  
  
• https://testgrid.k8s.io/presubmits-kubernetes-blocking#ci-kubernetes-build-fast&graph-metrics=test-duration-minutes
  
  
  
  • does the job duration look worse than before? spikier than before?
  
  
• https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&job=ci-kubernetes-build-fast
  
  
  
  • do more failures show up than before?
  
  
• https://prow.k8s.io/job-history/gs/kubernetes-jenkins/pr-logs/directory/ci-kubernetes-build-fast
  
  
  
  • (can be used to answer some of the same questions as above)
  
  
• metrics explorer: CPU limit utilization for ci-kubernetes-build-fast for 6h
  
  
  
  • is the job wildly underutilizing its CPU limit? if so, perhaps tune down (if uncertain, post evidence in this issue and ask)
  
  
  • (it will probably be helpful to look at different time resolutions like 1h, 6h, 1d, 1w)
  
  
• metrics explorer: Memory limit utilization for ci-kubernetes-build-fast for 6h
  
  
  
  • is the job wildly underutilizing its memory limit? if so, perhaps tune down (if uncertain, post evidence in this issue and ask)
  
  
  • (it will probably be helpful to look at different time resolutions like 1h, 6h, 1d, 1w)
  
  

Things to watch for the build cluster

• prow-build dashboard 1w
  
  
  
  • is the build cluster scaling as needed? (e.g. maybe it can't scale because we've hit some kind of quota)
  
  
  • (it will probably be helpful to look at different time resolutions like 1h, 6h, 1d, 1w)
  
  
• prowjobs-experiment 1w
  
  
  
  • (shows resource consumption of all job runs, pretty noisy but putting this here for completeness)
  
  
• https://monitoring.prow.k8s.io/d/wSrfvNxWz/boskos-resource-usage?orgId=1

Keep this open for at least 24h of weekday PR traffic. If everything continues to look good, then this can be closed.

/wg k8s-infra
/sig testing
/area jobs

[spiffxp]

/sig release
FYI @kubernetes/release-engineering

[spiffxp]

https://github.com/kubernetes/test-infra/pull/19487 created a canary job

Next steps to take are spelled out in https://github.com/kubernetes/k8s.io/issues/846#issuecomment-705722852

Is the job working? If so, it's time to switch other jobs to use the new path

[spiffxp]

https://testgrid.k8s.io/sig-release-master-informing#build-master-fast-canary&graph-metrics=test-duration-minutes&width=5 is green, and its durations are comparable to https://testgrid.k8s.io/sig-release-master-blocking#build-master-fast&graph-metrics=test-duration-minutes&width=5

[spiffxp]

https://github.com/kubernetes/test-infra/pull/19620 updated kubetest to add a --extract-ci-bucket flag

https://github.com/kubernetes/test-infra/pull/19632 updates a job to use the new kubetest without the flag to verify nothing breaks

Next PR will add --extract-ci-bucket k8s-release-dev to migrate a job to use the results of the canary job added in https://github.com/kubernetes/test-infra/pull/19487

[spiffxp]

/help

Now that https://github.com/kubernetes/test-infra/pull/19634 has confirmed the new bucket works, remaining steps are:

• wait for https://github.com/kubernetes/test-infra/pull/19631 to merge or bump kubekins in jobs to be migrated (see #19632)
• any jobs that have --extract=ci/latest-fast should have --extract-ci-bucket=k8s-release-dev added (see #19634)
• conform migrated jobs work
• update canary to match old job name and annotations, delete old job

[cpanato]

@spiffxp I will update the ci-kubernetes-build-fast job

[justaugustus]

/reopen

Reopening this to keep an eye on it, given https://github.com/kubernetes/test-infra/pull/19841.
/assign @justaugustus @cpanato

[k8s-ci-robot]

@justaugustus: Reopened this issue.

In response to this:

/reopen

Reopening this to keep an eye on it, given https://github.com/kubernetes/test-infra/pull/19841.
/assign @justaugustus @cpanato

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[spiffxp]

Opened https://github.com/kubernetes/test-infra/pull/19871 to enforce jobs should be using the k8s-release-dev bucket if they're using ci/latest-fast; caught one outlier

I'm ready to call this done unless there are any remaining concerns

[spiffxp]

/close
There have been no concerns raised

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

/close
There have been no concerns raised

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.