Test-infra: Support triggering specific e2e tests on PR merge that touches files in a particular directory

[yliaog]

The particular problem I'd like to solve is:

when a PR touches a file in kubernetes/cluster/gce/, trigger e2e tests for gci-gke before the PR is allowed to merge. Put it another way, gci-gke e2e tests will not run or block PR merge if the PR does not touch any file under the directory.

this could be a generally useful feature, IMHO.

[fejta]

See pull-kubernetes-cross and the run_if_changed config option: https://github.com/kubernetes/test-infra/blob/master/prow/config.yaml#L943

[BenTheElder]

Though this can't actually block merging the PR currently /cc @cjwagner
run_if_changed will get it running but previously when I tried to make pull-kubernetes-cross blocking we hit some kind of bug that caused it to block PRs that shouldn't run the test (and weren't)
xref: https://github.com/kubernetes/test-infra/issues/5069

[cjwagner]

pull-test-infra-gubernator is another blocking job that uses run_if_changed.
pull-kubernetes-cross is a k/k job that also uses it, is blocking, and runs on jenkins not kubernetes.
Have we ever seen this behavior on either of these jobs?

[BenTheElder]

pull-kubernetes-cross is not actually blocking AFAIK, test-infra runs on tide so all jobs that run are blocking right?

[cjwagner]

Yes, you're right it is not blocking, I read the wrong context.

I'm not sure that whether or not the job is blocking is related to the bug. AFAICT Prow never checks if a job is blocking.

Also, I don't think this is related to the submit-queue/mungegithub because MG never sets any status context besides the Submit Queue context.

[BenTheElder]

Wherever the bug actually is last time we made a run_if_changed job blocking we quickly reverted that because it was ultimately blocking lots of PRs that it shouldn't have.

[yliaog]

is there any conclusion? is it worth a try? it is kind of urgent as i'm investigating e2e test failures, and found 5 bugs one after another so far. Some bug was introduced before the other bugs uncovered were fixed. I think adding the presubmit check is probably the quickest way to get e2e tests green.

[BenTheElder]

run_if_changed works quite well, and we can report it on github, but adding it to the list of blocking jobs might not work out well immediately.
We punted on that for now since the cross job being blocking wasn't _that_ important and we're focusing on switching over to tide instead of the submit-queue (https://github.com/kubernetes/test-infra/issues/3866)

[cjwagner]

The job you are interested in is ci-kubernetes-e2e-gci-gke, right? If so the first step would be to make a presubmit version of this job and set the run_if_changed field appropriately.
We can work to make the job blocking and deal with any problems after that.

[yliaog]

Yes, ci-kubernetes-e2e-gci-gke is the one. I'll make changes as suggested.

[yliaog]

5523 merged. when can it be made blocking? the reason to ask is the gke e2e tests were broken again, likely due to changes under cluster/gce

[yliaog]

@cjwagner is there anything i need to do to make it blocking?

[BenTheElder]

see: #5584 this is being looked into

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[BenTheElder]

5523 merged. when can it be made blocking? the reason to ask is the gke e2e tests were broken again, likely due to changes under cluster/gce

I think there was a very explicit change in the past to not block on GKE tests, if you want a test to be blocking you should ideally float it through sig-testing and possibly sig-release. I'll defer to the leads on that here..