Test-infra: setup release 1.9 branch

[BenTheElder]

The branch was cut tonight, I'm working on getting all the right configs updated and figured it would be useful to collect all the PRs involved against this issue for next time.

/area jobs
/area testgrid

[BenTheElder]

Derailed for the moment by looking into https://github.com/kubernetes/kubernetes/issues/55917, I suspect that one will not have a great simple fix, it looks like multiple PRs changing the versioning logic went in recently.

[enisoc]

ref https://github.com/kubernetes/sig-release/issues/30

[BenTheElder]

Moving the checklist here and breaking up the PR, we decided to do it piece-meal and put in the build job(s) first:

To setup release-1.9:

Testgrid:

• [x] create sig-release-1.9-all dashboard
• [x] create sig-release-1.9-blocking dashboard
• [x] [WIP] add entries for all release-1.9 jobs to the above dashboards
• [x] add 1.9 jobs to any other appropriate dashboards
• [x] [WIP] sig dashboards for sig relatedrelease-1.9 jobs
• [x] move appropriate jobs into the blocking dashboard (~ match release-1.8)

Jenkins:

• [x] add ci-kubernetes-build-1.9, bump stableN for build jobs
• [x] add kubernetes-test-go-release-1.9
• [x] add kubernetes-verify-release-1.9
• [x] update kubernetes-kubemark-5-gce-last-release trigger job to ci-kubernetes-build-1.9
• [ ] take a second pass to check for any remaining jobs

Prow:

• [x] ensure presubmits will run against release-1.9 branch the same as existing master
• [x] add 1.9 versions of bazel ci/periodic jobs
• [x] add 1.9 versions of kubeadm ci/periodic jobs
• [x] add 1.9 versions of device-plugin / gpu jobs
• [ ] add 1.9 versions of federation jobs (punting to sig-multicluster, see comments below)
• [x] setup 1.9 versions kubectl-skew jobs

Other TODO:

• [x] generate jobs where possible (@krzyzacy) #5573
• [ ] [WIP] move jobs to Prow where possible, xref #3778
  
  
  
  • Need to add more resources to prow-builds, also https://github.com/kubernetes/test-infra/pull/5561
  
  
  • #5585
  
  

/area jobs

[BenTheElder]

@shashidharatd @marun what's the deal with kubernetes-federation-build-1.8 etc.?

[BenTheElder]

Complete list of merged PRs involved (will keep up to date):

• Add release 1.9 build (what it says) https://github.com/kubernetes/test-infra/pull/5567
• add relase-1.9 verify and test-go (also fix dashboard grouping, but apparently not PR title spelling) https://github.com/kubernetes/test-infra/pull/5569
• Branch bikeshed (generate jobs) https://github.com/kubernetes/test-infra/pull/5573 by @krzyzacy
• update kubernetes-kubemark-5-gce-last-release trigger to 1.9 instead … (update kubemark ci job trigger and kubernetes version) https://github.com/kubernetes/test-infra/pull/5574
• add missing config.json entries for test-go and verify jobs for relea… https://github.com/kubernetes/test-infra/pull/5580
• add release 1-9 ci/periodic bazel jobs (also kubeadm) https://github.com/kubernetes/test-infra/pull/5576
• Add stable1-beta (currently 1.8-1.9) upgrade/skew jobs https://github.com/kubernetes/test-infra/pull/5609 by @krzyzacy
• add gpu jobs for 1.9 https://github.com/kubernetes/test-infra/pull/5616
• Add kops-beta job https://github.com/kubernetes/test-infra/pull/5635 by @krzyzacy
• Add kubelet and scalability suite for beta, and add to 1.9 dashboard https://github.com/kubernetes/test-infra/pull/5649 by @krzyzacy
• add gpu-p100 job to release 1.9 https://github.com/kubernetes/test-infra/pull/5661
• disable integration tests ci-kubernetes-bazel-test to match presubmits https://github.com/kubernetes/test-infra/pull/5664
• Release 1.9 bazel patchup https://github.com/kubernetes/test-infra/pull/5680
• fix incorrect flag in ci-kubernetes-test-go-1.9 https://github.com/kubernetes/test-infra/pull/5684
• kick kubeadm upgrade job out of release-1.9-blocking https://github.com/kubernetes/test-infra/pull/5724
• fix release 1.9 bazel build job branch https://github.com/kubernetes/test-infra/pull/5728

Semi related PRs:

• migrate pull-kubernetes-cross to prow https://github.com/kubernetes/test-infra/pull/5585
• Change dashboard names to use 1.y instead of 1-y https://github.com/kubernetes/test-infra/pull/5648 by @spiffxp
• add bazel jobs to release-master-blocking https://github.com/kubernetes/test-infra/pull/5654
• disable integration tests ci-kubernetes-bazel-test to match presubmits https://github.com/kubernetes/test-infra/pull/5664

[shashidharatd]

@shashidharatd @marun what's the deal with kubernetes-federation-build-1.8 etc.?

now federation is moved to k/f repo and the ci build for head is here https://k8s-testgrid.appspot.com/sig-multicluster-federation-gce#build

To create a federation-build release against 1.9 we need to use the jobs which are running against the master branch. Also there are some specific requirements(like increased quota) for federation e2e jobs.

Definetely the federation build is a non-blocker for k8s 1.9 release. We still need to conclude the federation release timelines, whether it should be inline with k8s releases or should have independent release which works for the corresponding k8s release. We will discuss this in our SIG meeting on tuesday.

/cc @kubernetes/sig-multicluster-pr-reviews

[BenTheElder]

I think this is done minus the federation jobs.
Edit and kops :^) (Fixed)

[spiffxp]

A few nits while attempting to reconcile with diff. In an ideal world, release-1.y-blocking would just be a copy of release-master-blocking with jobs pointed at different branches. Trying to understand why the differences exist. I'm hoping there are some common explanations which we could add in comment form someplace, eg:

• was added as a release-blocker during v1.8
• can only be run against release-1.y branches
• etc.

@kubernetes/sig-gcp-test-failures The gci-{gce,gke} names in sig-release-{master,1.8} are now just {gce,gke} in sig-release-1.9-blocking, can we strip the gci- prefix from the other names? There are both gce and gci-gce jobs in sig-release-1.7-blocking

~@kubernetes/sig-testing-test-failures These jobs in sig-release-1.9-blocking have no equivalent in sig-release-{master,1.8,1.7}-blocking. Should they be removed from release-1.9 or added to any of the others?~ these were added during the v1.9 dev cycle, #5654 ensure they're in master, and we're not backporting

• ~bazel-build-1.9~
• ~bazel-test-1.9~

@kubernetes/sig-cluster-lifecycle-test-failures These jobs in sig-release-1.9-blocking have no equivalent in sig-release-{master,1.8}-blocking. Should they be removed from release-1.9 or added to any of the others?

• gce-kubeadm-upgrade-1.8-1.9

@kubernetes/sig-multicluster-test-failures These jobs in sig-release-1.8-blocking have no equivalent in sig-release-{master,1.9,1.7}-blocking. Should they be removed from release-1.8 or added to any of the others?

• federation-build-1.8
• federation-test-1.8

~@kubernetes/sig-network-test-failures These jobs in sig-release-{master,1.9}-blocking have no equivalent in sig-release-{1.8,1.7}-blocking. Should they be added to any of the others?~ disregard, @spiffxp was blind

• ~gce-ingress-1.9~
• ~gke-ingress-1.9~

@kubernetes/sig-scalability-test-failures These jobs in sig-release-master-blocking have no equivalent in sig-release-{1.9,1.8,1.7}-blocking. Should they be added to any of the others?

• gce-scale-correctness
• gce-scale-performance

~@kubernetes/sig-auth-test-failures These jobs in sig-release-master-blocking have no equivalent in sig-release-{1.9,1.8,1.7}-blocking. Should they be added to any of the others?~ The goal is to fold these tests into the standard suite, and remove the gci-gce-audit job (ref: kubernetes/kubernetes#55953)... and the job is removed via #5690

• ~gci-gce-audit~

~@kubernetes/sig-scheduling-test-failures These jobs in sig-release-master-blocking have no equivalent in sig-release-{1.9,1.8,1.7}-blocking. Should they be added to any of the others?~ I'm going to claim this was added during the v1.9 dev cycle, and we've added it to the 1.9 board via #5661

• ~gke-device-plugin-gpu-p100~

~@kubernetes/sig-node-test-failures These jobs in sig-release-master-blocking have no equivalent in sig-release-1.9-blocking. Should they be added to any of the others?~ #5649 added kubelet-1.9

• ~kubelet~

misc:

• ~gci-gce-100 (test_group_name is ci-kubernetes-e2e-gci-gce-scalability, there is a gce-scalability job in 1.8 and 1.7)~ this has been added
• ~soak-gce-gci~ this was dropped from master via #5529

[krzyzacy]

For the ones I know:

• I have blown away all the 1.7-gke-'cvm' jobs
• @BenTheElder has added the bazel jobs, we'll need to add that to 1.9-blocking dash
• we don't create separate gce-scale jobs for release (they are expensive!) but we probably should have a gci-gce-100 for 1.9, I'll create it.
• I'll add a kubelet job
• Do we still want a soak job? I'll create one for sure.

[BenTheElder]

+1 to the comments on why something exists, I've at least tried to make sure all the changes we make for 1.9 show up https://github.com/kubernetes/test-infra/issues/5558#issuecomment-345392515

[yujuhong]

@kubernetes/sig-node-test-failures These jobs in sig-release-master-blocking have no equivalent in sig-release-1.9-blocking. Should they be added to any of the others?

kubelet

Yes, we have kubelet-1.8, and kubelet 1.7 jobs.

[krzyzacy]

added kubelet ones from https://github.com/kubernetes/test-infra/pull/5649

[BenTheElder]

@BenTheElder has added the bazel jobs, we'll need to add that to 1.9-blocking dash

These jobs block on presubmit, and are no more flaky than unit / non-bazel build. They're in the 1.9 blocking dash but we should have them in master blocking too.

[spiffxp]

has added the bazel jobs, we'll need to add that to 1.9-blocking dash

Yeah, they're there, I guess my question to @BenTheElder is, is there any reason they're blocking 1.9 but not blocking master?

lol ok question answered above, as a member of the release team I'll lgtm a pr that adds them based on history of passing against PR's

[BenTheElder]

I'm adding them to master. We have no reason to not block on these.

[BenTheElder]

Federation jobs for 1.9 have been punted to federation to sort out how they want them but "the federation build is a non-blocker for k8s 1.9 release" (since they moved out of k/k?) https://github.com/kubernetes/test-infra/issues/5558#issuecomment-345624723

[BenTheElder]

For sig questions about blocking tests in comment above (https://github.com/kubernetes/test-infra/issues/5558#issuecomment-346385580) I am asking the following sig members to respond with how they would like these issues handled :-)

• cc @enisoc @abgworrall for possible GCP job name bikeshed
• cc @luxas for kubeadm / sig-cluster-lifecycle jobs potentially missing from master and previous releases
• @shashidharatd discussed federation / sig-multicluster above though this isn't solved yet it shouldn't be a problem for now
• cc @MrHohn for sig-network jobs missing from previous releases
• not sure who to flag down for sig-auth, but the sigs have all been pinged at their test failures handle anyhow..
• cc @mindprince for sig-scheduling / GPU jobs only present in master blocking but not any of the release branches (including 1.9)

Please let us know what makes sense for these questions and I will make the necessary changes.

[mindprince]

These jobs in sig-release-master-blocking have no equivalent in sig-release-{1.9,1.8,1.7}-blocking. Should they be added to any of the others?
gke-device-plugin-gpu-p100

Yes. Let's add them to release-1.9-blocking. When I added that job, release-1.9-blocking didn't exist, so I only added it to master-blocking.

[BenTheElder]

Yes. Let's add them to release-1.9-blocking. When I added that job, release-1.9-blocking didn't exist, so I only added it to master-blocking.

Thanks, we'll do that :+1:

[MrHohn]

kubernetes/sig-network-test-failures These jobs in sig-release-{master,1.9}-blocking have no equivalent in sig-release-{1.8,1.7}-blocking. Should they be added to any of the others?

• gce-ingress-1.9
• gke-ingress-1.9

@BenTheElder I think we do have equivalent jobs, just that the naming format changed?

• https://k8s-testgrid.appspot.com/sig-release-1.8-blocking#gci-gce-ingress-1.8
• https://k8s-testgrid.appspot.com/sig-release-1.8-blocking#gci-gke-ingress-1.8
• https://k8s-testgrid.appspot.com/sig-release-1.7-blocking#gci-gce-ingress-release-1.7
• https://k8s-testgrid.appspot.com/sig-release-1.7-blocking#gci-gke-ingress-release-1.7

[spiffxp]

@MrHohn you're right, that was my bad, not sure how I missed those

[BenTheElder]

Thanks @MrHohn, with all of the jobs I don't have full track of every job 🙃

[BenTheElder]

This https://github.com/kubernetes/test-infra/issues/5663 / the associated PR https://github.com/kubernetes/test-infra/pull/5664 will be relevant now that https://github.com/kubernetes/kubernetes/pull/55279 is in.

[crassirostris]

@spiffxp Audit tests will be added to the default suite in gce/gke env by https://github.com/kubernetes/kubernetes/pull/55953

[spiffxp]

@crassirostris does this mean we should remove the audit-specific job once that PR merges?

[crassirostris]

@spiffxp Yes

[BenTheElder]

There are a few unanswered questions above, but the branch has been set up and running for some time (with a few fixes since then), I think we should close this.