Test-infra: jenkins isn't running presubmits?

[spiffxp]

Submit queue is timing out waiting on presubmits

https://submit-queue.k8s.io/#/history
eg:

crassirostris

52541: [fluentd-gcp addon] By default ingest audit logs in JSON format

Timed out waiting for PR 52541 to finish testing
Sep 18, 2017 6:06:39 AM
crassirostris

52541: [fluentd-gcp addon] By default ingest audit logs in JSON format

Running github e2e tests a second time.
Sep 18, 2017 4:06:39 AM
crassirostris

52541: [fluentd-gcp addon] By default ingest audit logs in JSON format

Requested and waiting for github e2e test to start running a second time.
Sep 18, 2017 4:06:08 AM
crassirostris

52565: [fluentd-gcp addon] Update Stackdriver plugin to version 0.6.7

Queued to run github e2e tests a second time.
Sep 18, 2017 4:05:58 AM

Prow isn't showing any results for presubmit jobs that run on jenkins:
http://prow.k8s.io/?type=presubmit&repo=kubernetes%2Fkubernetes

This is usually a sign that jenkins (or its agents) is down

[BenTheElder]

This has already been highlighted elsewhere, but /cc @krzyzacy for folllow up (maybe this can be closed now with a note about what happened?)

[krzyzacy]

Sorry, and update: @fejta rebooted failed VMs, and https://github.com/kubernetes/test-infra/pull/4641 will add the timeout for maintenance jobs.

Currently Jenkins is running fine, will still need to keep an eye on it.

[fejta]

Things to keep an eye on:

• Have we merged something recently? https://submit-queue.k8s.io/#/history
• Do batch results look sane? http://prow.k8s.io/?type=batch
• Have the daily and hourly tabs ran recently? https://k8s-testgrid.appspot.com/sig-testing

PS, we merged something within the past hour!

[krzyzacy]

ohhh I was misleaded by https://submit-queue.k8s.io/#/history, we probably want to highlight Merged there

[fejta]

This kind of message in #history is a strong signal that jenkins is sad:
Timed out waiting for PR 52561 to finish testing

These are signals everything is happy:
MERGED! or MERGED! (batch)

This is a signal that prow and jenkins are happy, but tests may or may not (depending on the PR) be healthy:
Second github e2e run failed.
Second run failing might mean: PR is bad/flaky, tests/product w/o PR is bad/flaky, something wrong with test-infra (bootstrap, kubetest, image, cluster-lifecycle, etc)

[krzyzacy]

Jenkins looks fine as of right now, I'll close for now.

[fejta]

Woo!

[krzyzacy]

maintenance-all-daily has been running (It fails on some nodes, probably need a separate issue?)

[fejta]

thanks for the diligent work Sen 🥂

[xiangpengzhao]

Found pull-kubernetes-e2e-gce-gpu — Error checking queue status. in https://github.com/kubernetes/kubernetes/pull/52478

Does the error has something to do with this issue?

[krzyzacy]

@xiangpengzhao that is already a prow job now. @BenTheElder ^^