Spek: Guarantee tests run in order

[ebrowne72]

Looking at the implementation, tests are stored in a LinkedHashSet and are executed in the order they are defined. Can we rely on that? Will this change in Spek 2.x? Can we get something in the documentation which says whether we can rely on tests executing in order?

The reason I ask is that we have some code where we want to verify calls to a mock are done in order. It seems like each verification should be in a separate test, but we can't do that if they might be executed in a different order than we defined them.

[vanniktech]

We should try to randomize the order so tests are not making assumptions that others have run before.

[artem-zinnatullin]

No, please don't rely on that.

I'm also leaning towards randomization of execution order.

I'm trying to remember what language switched to random iteration order for Maps/Dictionaries because people relied on that, I think it was Python but quick googling didn't help

[egor-n]

@artem-zinnatullin it was Go:

Since Go 1 the runtime randomizes map iteration order, as programmers relied on the stable iteration order of the previous implementation.

Go maps in action – The Go Blog

[ebrowne72]

Then how about adding some way to specify that tests need to be run in a certain order? Maybe a thenIt() function?

[vanniktech]

Why would you want that?

[ebrowne72]

For example, to verify calls to a mock happen in a certain order.

on("foo called on bar") {
    val mockDependency = mock<Dependency>()
    val inOrder = inOrder(mockDependency)
    val bar = Bar(mockDependency)

    bar.foo()

    it("calls fred() on dependency") {
        inOrder.verify(mockDependency).fred()
    }

    thenIt("calls barney() on dependency") {
        inOrder.verify(mockDependency).barney()
    }
}

[vanniktech]

Simple solution: Don't use mocks :trollface: seriously though

[artem-zinnatullin]

Well, that's where our ideology diverts with @raniejade hehe

If we decide to keep on (action) and I think we will, inside on you indeed need a strict order because that's whole idea of on.

However in context (group), I believe order should be randomized by Spek to avoid people relying on it.

[punkisdead]

+1 for randomizing the order, or at least providing that as an option similar to how RSpec does.

I've run into weird failures where one test would depend on some state that was triggered in an earlier test.

I would also not test that mocks are being called in a certain order, that is not a very good test in my opinion. To paraphrase Martin Folwer, "interaction-based tests are ... more coupled to the implementation of a method." (https://www.martinfowler.com/articles/mocksArentStubs.html)

I would focus more on value based testing, or state based testing where needed and avoid interaction based testing if at all possible.

[raniejade]

Resolved by #497 , by default the execution order is unspecified for each scope. Guaranteed execution order is needed by the gherkin style.