Test-infra: Set up e2e testing using Bazel-built binaries

[ixdy]

The process keeps getting more involved as I investigate this, so I wanted to tally the steps necessary. (Help is welcome!)

Short-term goals:

1. One prow postsubmit job running a basic set of e2e tests using binary artifacts from a Bazel build. (I'm imagining basically just duplicating the ci-kubernetes-e2e-gce-etcd3 job.)
2. One prow presubmit job which builds everything with bazel and then runs e2e tests. It'd basically be the pull-kubernetes-e2e-gce job but using make bazel-release instead of make quick-release.

To support the postsubmit job, there are three options:

1. Update the existing build/test job to use release/push-build.sh instead of calling bazel build //:ci-artifacts directly. The bazel rule doesn't create the latest*.txt files we need.
2. Create a new "bazel-build-release" job which uses the kubernetes_build scenario; we'd need to extend the scenario to support make bazel-release. We'd also need to upload to a unique GCS path (either different bucket or different directory, or both).

For either of the above options, we'd also need to create the prow postsubmit job.
We can maybe repurpose/rename the existing prow-canary one. It'll need to grab the binaries from whatever GCS location we chose.

1. The last option is to treat CI like PR jobs and do a full build/push/test in the job itself. With sufficient caching this may not be too bad, but it's also the most unlike our existing workflows.

To support the presubmit job:

1. Update the kubernetes_e2e scenario and existing job configs to change --build from a boolean to a string arg. Existing jobs should use quick; the bazel one should use bazel.
2. Create a new presubmit e2e job using --build=bazel.

I'm pretty sure Bazel-built clusters are actually broken right now, possibly due to https://github.com/kubernetes/kubernetes/issues/45298, but I haven't yet been able to confirm. Getting CI going will help us debug the issues and hopefully prevent future breakages.

[ixdy]

Added complication: to use kubetest --build we need something like the kubekins-e2e image with Bazel installed. This seems to add ~600MB to the docker image (1.703GB instead of 1.1.156), which makes me a bit hesitant. (It also makes the docker build significantly slower, since we have to install the JDK + associated dependencies.)

We can probably create a separate image that's kubekins-e2e + bazel, though this adds complication to the e2e scenario. (We probably don't need half of the stuff in the kubekins-e2e image anymore, I'd bet...)

[spiffxp]

What remains to be done here?

[krzyzacy]

I think we haven't adopt bazel build for all the ci jobs yet

[ixdy]

yeah, CI jobs aren't using bazel-built binaries - only PR jobs.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[krzyzacy]

CI jobs will be move forward with https://github.com/kubernetes/test-infra/issues/5905, I'll close this as from the issue description the target was for presubmit jobs.