Test-infra: Kubemark jobs failing due to jenkins unresponsiveness

[shyamjvs]

Many of our kubemark jobs are failing frequently due to jenkins:

• https://k8s-testgrid.appspot.com/sig-scalability-kubemark#kubemark-100
• https://k8s-testgrid.appspot.com/sig-scalability-kubemark#kubemark-500
• https://k8s-testgrid.appspot.com/sig-scalability-kubemark#kubemark-5000

E.gs from each:

• https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-kubemark-100-gce/5386/build-log.txt
• https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-kubemark-500-gce/9718/build-log.txt
• https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale/715/build-log.txt

There are many many msgs like these in the build-logs:

I1115 11:31:02.863] error dialing [email protected]:22: 'ssh: handshake failed: read tcp 172.17.0.4:51754->35.193.52.163:22: read: connection reset by peer', retrying
I1115 11:31:02.865] Nov 15 11:31:02.864: INFO: Error when trying to SSH to master machine. Skipping probe. error getting SSH client to [email protected]:22: 'ssh: handshake failed: read tcp 172.17.0.4:51792->35.193.52.163:22: read: connection reset by peer'

This is urgent and needs to be fixed ASAP.

/assign @krzyzacy
cc @kubernetes/sig-scalability-misc @porridge

[spiffxp]

Left hand meet right hand https://github.com/kubernetes/kubernetes/issues/55770

[krzyzacy]

😂
/shrug

[krzyzacy]

(I really have no clue, kubekins looks fine-ish?)
/assign @ixdy
toss a hot potato at our Jenkins master 😂 Jeff might know more

[ixdy]

I'm pretty sure this isn't a jenkins issue. The ssh username is jenkins because the tests are running as the jenkins user, but the tests are trying to connect to the kubetest master, not jenkins master.

Note how the IP address is different in each of the instances you linked:
error creating session to [email protected]:22
error creating session to [email protected]:22
error getting SSH client to [email protected]:22

You can also see that that IP address corresponds to the kubemark master, e.g. for the last failure:

I1114 14:49:07.392] NAME                           ZONE        MACHINE_TYPE    PREEMPTIBLE  INTERNAL_IP    EXTERNAL_IP      STATUS
I1114 14:49:07.392] kubemark-5000-kubemark-master  us-east1-a  n1-standard-64               10.142.15.211  104.196.172.201  RUNNING

[BenTheElder]

Side note: Xref #3778

[shyamjvs]

Closing this.. as the issue seems unrelated to test-infra. Filed the bug in k/k instead.
/close