Hypothesis: Hypothesis doesn't recognize unittest.SkipTest exception

[coriolinus]

Narrative example of failure

I haven't had time to construct a minimal example, so I'll give a full example of failure. I've recently updated a test in my Django project which conditionally skips a test in cases which no longer make sense.

The test in question is defined in an abstract base class of a couple other test classes which specialize it further. One branch of one test no longer makes sense in the context of one of the specialization classes. I can't solve this with assume() or @unittest.skipIf() because neither has enough contextual data. Instead, I raise unittest.SkipTest() at runtime if the test should be skipped.

The issue is that Hypothesis doesn't know about unittest.SkipTest, so produces failure output, as seen below:

(.venv) coriolinus$ ./manage.py test
Creating test database for alias 'default'...
Installed 39 object(s) from 1 fixture(s)
Installed 21 object(s) from 1 fixture(s)
Installed 9579 object(s) from 1 fixture(s)
Installed 5 object(s) from 1 fixture(s)
Installed 6 object(s) from 1 fixture(s)
.............................s.............................Falsifying example: test_admin_sees_everyone(self=<core.tests.test_user_profile_visibility.ProfileVisibilityTests testMethod=test_admin_sees_everyone>, data=data(...))
Draw 1: <User: alpha>
s..........................................
----------------------------------------------------------------------
Ran 102 tests in 58.985s

OK (skipped=2)
Destroying test database for alias 'default'...
(.venv) coriolinus$ pip list | grep hypothesis
hypothesis (3.6.1)

Expected behavior

Hypothesis does not produce failure output if a test raises unittest.SkipTest.

Observed behavior

Hypothesis produces failure output if a test raises unittest.SkipTest.

[alexwlchan]

Thanks for the report! I had a quick go at trying to write an example – is this sort of like what you were doing?

import unittest

from hypothesis import given
from hypothesis.strategies import integers


class DemoTest(unittest.TestCase):

    @given(integers())
    def test_to_be_skipped(self, x):
        if x == 0:
            raise unittest.SkipTest()
        else:
            assert x == 0


unittest.main()

$ python issue514.py
Falsifying example: test_to_be_skipped(self=<__main__.DemoTest testMethod=test_to_be_skipped>, x=0)
s
----------------------------------------------------------------------
Ran 1 test in 0.021s

OK (skipped=1)

Sounds like we should just recognise unittest.SkipTest as a special exception – I get the same result whether I raise the exception directly or call self.skipTest(reason). Not sure how hard that is off the top of my head.

[coriolinus]

Yes, that's the essence of the issue. Agreed with the proposed course of
action. I feel like I should have submitted a PR to go with this bug
report, but I don't know anything about Hypothesis' internals.

On Thu, Apr 13, 2017 at 3:29 PM, Alex Chan [e-mail] wrote:

Thanks for the report! I had a quick go at trying to write an example – is
this sort of like what you were doing?

import unittest
from hypothesis import givenfrom hypothesis.strategies import integers

class DemoTest(unittest.TestCase):

@given(integers())
def test_to_be_skipped(self, x):
    if x == 0:
        raise unittest.SkipTest()
    else:
        assert x == 0

unittest.main()

$ python issue514.pyFalsifying example: test_to_be_skipped(self=<__main__.DemoTest testMethod=test_to_be_skipped>, x=0)s----------------------------------------------------------------------Ran 1 test in 0.021s
OK (skipped=1)

Sounds like we should just recognise unittest.SkipTest as a special
exception – I get the same result whether I raise the exception directly or
call self.skipTest(reason). Not sure how hard that is off the top of my
head.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/HypothesisWorks/hypothesis-python/issues/514#issuecomment-293895402,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHdeTr5MlcE7IqXHHhCasQKBMK_wfBRDks5rviMxgaJpZM4M8sBy
.

[alexwlchan]

So at a very quick glance, I think we need to make a change somewhere around line 464 of core.py so that if we catch a unittest.SkipTest exception, we abort the entire test run.

I’m not sure what mechanisms we have for marking a test as skipped (as opposed to just silently dropping the error) – I’ll dig in a little further when I have more time.

[DRMacIver]

I’m not sure what mechanisms we have for marking a test as skipped (as opposed to just silently dropping the error)

I think just immediately reraising the error should have the desired effect shouldn't it?

[alexwlchan]

I think just immediately reraising the error should have the desired effect shouldn't it?

Oh yeah. PR incoming. :)

[alexwlchan]

@coriolinus This took longer than I was expecting to finish, but the fix is now released as Hypothesis 3.13.

Thanks for your bug report!

[coriolinus]

I'm very glad to hear it! Thanks for fixing this.

On Thu, Jul 20, 2017 at 5:27 PM, Alex Chan [e-mail] wrote:

@coriolinus https://github.com/coriolinus This took longer than I was
expecting to finish, but the fix is now released as Hypothesis 3.13.

Thanks for your report!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/HypothesisWorks/hypothesis-python/issues/514#issuecomment-316739746,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHdeTt8CtFeHwfw2clHGoqP8cqnjpVQtks5sP3H4gaJpZM4M8sBy
.