Test-infra: federation pull job is broken

[fejta]

/cc @madhusudancs

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/41942/pull-kubernetes-federation-e2e-gce/47/

W0227 16:20:45.759] + set_release_vars_from_local_gcs_stage
W0227 16:20:45.760] + local -r local_gcs_stage_path=/workspace/_output/gcs-stage
W0227 16:20:45.760] + KUBERNETES_RELEASE_URL=file:///workspace/_output/gcs-stage
W0227 16:20:45.760] ++ ls /workspace/_output/gcs-stage
W0227 16:20:45.760] ++ grep '^v.*$'
W0227 16:20:45.761] ls: cannot access /workspace/_output/gcs-stage: No such file or directory

Looks like pull-kubernetes-federation-e2e-gce.sh does not stage binaries, which the kubekins-e2e job expects to have happened when JENKINS_USE_LOCAL_BINARIES=y

[fejta]

The core problem here actually seems to be that we have a new kubernetes/hack/jenkins/build-federation.sh which does not call release/push-build.sh like build.sh does.

[fejta]

@rmmh @spxtr FYI this is going to require making bootstrap support multiple repos. I intend to make --repo --branch --pull allow specifying multiple instances of them. So the following will allow me to cline both kubernetes and release:

bootstrap.py --repo=k8s.io/kubernetes --branch=release-1.6 --repo=k8s.io/release --branch=master

[fejta]

Scenario-driven PR test is running: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/37868/pull-kubernetes-e2e-gce-canary/3/

[fejta]

Federation PR job is running: https://github.com/kubernetes/kubernetes/pull/37868#issuecomment-284917168

[fejta]

/assign @madhusudancs

Any ideas?

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/37868/pull-kubernetes-federation-e2e-gce/53/?log#log

W0308 17:44:18.529] /workspace/kubernetes/federation/cluster/common.sh: line 68: federation_contexts[@]: unbound variable
I0308 17:44:18.629] Cleaning Federation control plane objects
W0308 17:44:18.730] The connection to the server localhost:8080 was refused - did you specify the right host or port?
W0308 17:44:18.803] The connection to the server localhost:8080 was refused - did you specify the right host or port?
W0308 17:44:18.807] 2017/03/08 17:44:18 util.go:110: Step './federation/cluster/federation-down.sh' finished in 483.233896ms
W0308 17:44:18.807] 2017/03/08 17:44:18 none.go:41: Noop Down()
W0308 17:44:18.807] 2017/03/08 17:44:18 none.go:26: Noop Up()
W0308 17:44:18.807] 2017/03/08 17:44:18 util.go:108: Running: ./federation/cluster/federation-up.sh
W0308 17:44:18.852] Project: k8s-jkns-pr-bldr-e2e-gce-fdrtn
W0308 17:44:18.852] Zone: us-central1-f
I0308 17:44:18.952] Couldn't cleanup federation api objects
I0308 17:44:18.953] +++ [0308 17:44:18] Deploying federation control plane for e2e-federation in cluster federation-e2e-gce-us-central1-f
W0308 17:44:19.053] error: context "federation-e2e-gce-us-central1-f" does not exist
W0308 17:44:19.054] 2017/03/08 17:44:18 util.go:110: Step './federation/cluster/federation-up.sh' finished in 175.638616ms
W0308 17:44:19.054] 2017/03/08 17:44:18 util.go:108: Running: ./federation/cluster/federation-down.sh
W0308 17:44:19.054] Project: k8s-jkns-pr-bldr-e2e-gce-fdrtn
W0308 17:44:19.054] Zone: us-central1-f
I0308 17:44:19.181] Cleaning Federation control plane objects
W0308 17:44:19.282] /workspace/kubernetes/federation/cluster/common.sh: line 68: federation_contexts[@]: unbound variable

[madhusudancs]

I am debugging this. I still don't know the root cause, but it looks like the kubeconfig load isn't working. How do I try this with changes in my local test-infra repo?

[fejta]

/assign @krzyzacy
Any chance you can help out @madhusudancs with this tomorrow?

[krzyzacy]

sure, you should be able to trigger it locally by running

bootstrap.py --job=pull-kubernetes-federation-e2e-gce -more -flags

I'll need to try it out

[madhusudancs]

@krzyzacy thanks! This seems to be working so far.

[fejta]

/unassign
Should we /close?

[madhusudancs]

Yeah, let's close this. It is fixed. Thanks for making this happen!