Test-infra: tide batches newest PRs instead of oldest

[rmmh]

This means the same pool can be repeatedly tested and fail while the oldest PRs are tested and merged serially. Batches and serial tests should operate on the same queue, not different ends!

[BenTheElder]

/kind bug
/cc @cjwagner @kargakis @stevekuznetsov

[cjwagner]

This problem was introduced when we switched from unbounded batches to bounded batches. I think that all we need to do is sort the PRs by increasing number before looping through them in pickBatch().

[stevekuznetsov]

Why did we bound batches? Why not do log(N) batches of size 1,2,4,...log(N) ?

[BenTheElder]

Why did we bound batches? Why not do log(N) batches of size 1,2,4,...log(N) ?

I would also like to know why we are bounding batches. This sounds more reasonable to me.

[cjwagner]

I agree that we shouldn't be bounding batches, their size should be dynamic. Either way the batch size won't be the entire pool though, so we will still need to sort the PRs.

Multiple batches sounds good to me, but will require a bit of a rework of Tide. Currently it is designed to work with one pending batch per subpool.

[stevekuznetsov]

We did it here: https://github.com/kubernetes/test-infra/pull/5953

Maybe we can just revert that?

[stevekuznetsov]

/cc @kargakis

[kargakis]

Unbounded batches is a separate issue although it caused this one. I am fine with reverting but we need to keep in mind that the bigger a batch is the higher chance of a test failure. Maybe a separate issue for enhancing the batch logic to trigger different sizes of batches?

This issue is fixed.

[fejta]

Which PR fixed the sorting issue?

[kargakis]

Which PR fixed the sorting issue?

https://github.com/kubernetes/test-infra/pull/7568