Test-infra: Prow job started twice

[mborsz]

What happened:
The ci-kubernetes-e2e-gce-scale-performance job was started two times on 2018-11-16: at 21:40 CET, and 9:02 CET.

What you expected to happen:
Only single run to happen, the one at 9:02 CET which is specified in the job config.

How to reproduce it (as minimally and precisely as possible):

Please provide links to example occurrences, if any:

https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/258
https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/259

Anything else we need to know?:

/cc @wojtek-t

/kind bug

[BenTheElder]

this is almost definitely kubernetes evicting the test pod, #258 is not completed while #259 is.

this happens sometimes, see #10085

[mborsz]

Thanks for the response. Is there any way to prevent rerunning test in case of pod eviction? This test is sharing gce project with other tests, so if it doesn't finish in time, other test will run and will break both of them.

[BenTheElder]

I don't think we have any functionality for that ourselves, but k8s pods might, k8s is rescheduling the pod after evicting it. cc @cjwagner

[cjwagner]

Based on the plank code, k8s actually marks the pod as failed when it is evicted and plank is responsible for the rescheduling.
https://github.com/kubernetes/test-infra/blob/2946ed2b3cc74e8f1d91ee05687d209ac9b1b571/prow/plank/controller.go#L376-L378
It sounds like for some jobs we should instead be completing the ProwJob and setting it to kube.ErrorState. We could make this behavior configurable with an additional per job config field.

[BenTheElder]

SGTM, we'd definitely disable rescheduling for most scalability jobs. maybe failOnEviction: true? (naming is hard!)

[cjwagner]

errorOnEviction might be a bit more precise. I'll make a PR.