Pytest: Introduce match option for xfail

[codekoala]

It would be useful to be able to use the match logic from pytest.raises with pytest.mark.xfail. My current use case involves a lot of tests that use the Amazon boto3 library, which raises a botocore.exceptions.ClientError for pretty much every possible error. The exception has some attribute that can help understand what the error is really about, but the exception message is usually sufficient for my purposes. Currently, I am doing something like this:

def test_a_thing():
    with pytest.raises(ClientError, match="AccessDenied"):
        boto3.client("ec2").do_a_forbidden_thing()

This pattern is littered all throughout the tests. Not only would it be more correct to mark such a test with xfail, but it would help clean up the tests themselves. For example:

raises_access_denied = pytest.mark.xfail(raises=ClientError, match="AccessDenied")

@raises_access_denied
def test_a_thing():
    boto3.client("ec2").do_a_forbidden_thing()

@raises_access_denied
def test_another_thing():
    boto3.client("ec2").do_another_forbidden_thing()

This might be kind of related to the suggestion in https://github.com/pytest-dev/pytest/issues/3362, which could support more interesting matching/checking of the

[RonnyPfannschmidt]

im -1 on this, xfail is for marking tests that are broken, your tests are clearly working as intended

i'd prefer to see something like

raises_access_denied = pytest.mark(pytest.raises(ClientError, match="AccessDenied"))

@raises_access_denied
def test_a_thing():
    boto3.client("ec2").do_a_forbidden_thing()

@raises_access_denied
def test_another_thing():
    boto3.client("ec2").do_another_forbidden_thing()

and resulting in a passed test

[codekoala]

That's fair enough. It certainly is documented that xfail should be used for unfixed bugs, so best not to muddy the waters there. It made sense in my head to use xfail this way until you reminded me of that.

I think it would be useful to support something like what you suggested. I just tried it out and got an error (TypeError: 'MarkGenerator' object is not callable) which I'm guessing you already expected. I noticed there is a plugin on PyPI called pytest-raises that seems to offer this behavior already, but I haven't tried it yet. Would it make sense to support pytest.mark.raises as a built-in option?

Thanks for the quick response!

[RonnyPfannschmidt]

@codekoala thats thinkable im also hoping to introduce generalized marks at some point so we can ge away from the muddy water of string marks and go for more strict matching

but that shouldn't stop the deeper introduction of such a mark

[asottile]

not even sure there needs to be anything from pytest's side here. I think you can just write this decorator yourself:

def raises_access_denied(func):
    @functools.wraps(func)
    def wrapped_test(*args, **kwargs):
        with pytest.raises(ClientError, match='AccessDenied'):
            return func(*args, **kwargs)
    return wrapped_test

[codekoala]

That's very true, but it seems like something that might be common enough to merit including in pytest proper to me. My use cases have probably biased my view on that though.

[nicoddemus]

Hi @codekoala,

While I see your point of view, we need to consider that a built-in decorator is more limited, because it will always apply to the entire function, while the context manager gives you more control to decide which block of code exactly that you expect the exception. Also I suspect other people might have additional requirements to check for (extra attributes to check in the exception for example), so this new decorator might get complex fast.

For those reasons I would say I'm -1 to add a new "raises" decorator.

[codekoala]

Sounds good! Thanks for considering it!

[nicoddemus]

Cool, thanks for your understanding.