Pytest: Question/enhancement: (optionally) report failure if parametrize produces no arguments

[sambarluc]

I noticed a potentially dangerous behaviour of parametrized tests. I have a parametrized test where arguments have to be filled in some "complicated" fashion, let's say via a call to _get_arguments():

import pytest
from mymodule import do_stuff

@pytest.mark.parametrize("argument", _get_arguments())
def test_arg_returns_nonempty(argument):
    assert do_stuff(argument)

It may happen that _get_arguments() returns an empty list. In this case, the test is skipped. However, it may be better sometimes to mark the test as failed or errored in this case. Google suggests this is not currently possible via pytest, is this correct? Clearly, checks can be put elsewhere, but I think it would be useful, and less error prone, to be able to say something like noskip via another mark, or via an option of parametrize.

Thanks for the good work!

[RonnyPfannschmidt]

https://github.com/pytest-dev/pytest/blob/master/src/_pytest/mark/structures.py#L34 implements the marker to use for empty parameterset - currently we only have the options of xfail and skip - one could introduce a marker to always fail based on a condition in order to work around that

[sambarluc]

I'm not sure how one can choose between xfail and skip. Is this documented somewhere? If I understand how this works, I may a try a PR at some point.

[RonnyPfannschmidt]

https://docs.pytest.org/en/latest/changelog.html?highlight=empty_parameter_set_mark#id171
it seems like there are extra docs needed

[sambarluc]

Ok, I'm willing to look more closely at this, as I think it can be important (at least, for my work). However, it won't be before a couple of weeks at least.

[sambarluc]

I looked briefly at the docs and code, it seems to me that:

• indeed the current code does not support failing a test with empty parametrize arguments. I was hoping that I could bend pytest to do what I wanted, but it does not seem possible.
• to implement the desired (by me) functionality, one could: (i) have a command line option which turns xfail into fail, (ii) have a new marker which manually fails/errors a test or (iii) have an option in parametrize, which errors if no arguments are collected.

To me, options (i) and (ii) sound like too much of a hack. Opinions? Suggestions on how to proceed on (iii)?

[RonnyPfannschmidt]

i propose adding a new value to the empty_parameterset_marker option

i would propose a value of fail_at_parameterization which will trigger a direct exception instead of adding a marker

[sambarluc]

Sounds reasonable and not too difficult, I'll try to find some time for that.

[sambarluc]

Can this be closed, after #3970 ?

[nicoddemus]

Definitely, thanks for the ping. 👍