Hypothesis: Stateful testing

[pckroon]

Hi all,

first off, thanks for making hypothesis, I've been using it only briefly now and I really like it so far.
I need a hand in setting up stateful testing, since it sound like The Right Thing To Do(c).

I have code that can serialize and deserialize certain objects, and I made a composite strategy to create these objects. In addition, I have some operations that I can perform on these objects as functions. What I want to do now is set up a stateful testing framework where 1) one of these objects is instantiated using my strategy, 2) 0 or more operations are performed on this objects, and 3) between all those steps an invariant is asserted. It would be awesome if I could also give different arguments to those functions in 2.

Unfortunately, I'm lost on how to do this. Here's my attempt so far, can someone give me a nudge in the right direction?

class StatefulTestExample(RuleBasedStateMachine):
    @initialize
    def setup(self):
        self.obj = ..?  # Create something from my_strategy

    @rule
    def step(self):
        project.function(self.obj)

    @rule
    @given(...)
    def step2(self, *args):
        project.function2(self.obj, *args)

    @invariant
    def write_read_cycle(self):
        out = serialize(self.obj)
        found = deserialize(self.obj)
        assert out == found

[Zac-HD]

You're doing well! I assume you found the docs, but here's a link just in case :smile:

• @rule is basically a copy of @given that does a few extra things - you have to call it with the strategies you want to pass as arguments for the method. @initialize is the same (it's just a rule that gets called first)
• @invariant() also needs to be called, as shown here
• To have your test runner actually execute the test, you usually need a final line like TestFoo = StatefulTestExample.TestCase

[pckroon]

Thanks for the help!
I currently have the following code, which seems to work.

class StatefulTestExample(RuleBasedStateMachine):
    @initialize(obj=my_strategy)
    def setup(self, obj):
        self.obj = obj

    @rule()
    def step(self):
        project.function(self.obj)

    @rule()
    def step2(self):
        project.function2(self.obj)

    @invariant()
    def write_read_cycle(self):
        if not hasattr(self, obj):
            # Apparently the invariant is checked *before* running setup
            return
        out = serialize(self.obj)
        found = deserialize(self.obj)
        assert out == found
TestFoo = StatefulTestExample.TestCase

However if I then run pytest it gives me a failing test with an assertion error, but it doesn't actually show what it did or with what. Running it with just python produces no output at all.

Edit: Scratch that, I just missed it.

[Zac-HD]

Can you share the exact output from pytest? It should show where the error arose.

[pckroon]

I just completely overlooked it.
Hypothesis also immediately showed it's full potential by unearthing an ungodly amount of bugs. So thanks, I guess :)