Pytest: Parameter "loop_factory" should be declared explicitly via indirect or in function itself

[dcramer]

As of pytest 5.4.0 there is an issue with async tests, at the very least when using aiohttp. Can confirm that downgrading pytest fixed the issue.

Here's the same details I submitted over there when I thought aiohttp was the culprit:

Parameter "loop_factory" should be declared explicitly via indirect or in function itself

The commonality it turns out were async functions, and it appears to be caused somewhere within aiohttp. I'm not that familiar with asyncio or aiohttp, so I'm happy to debug more with guidance, but I'm hoping someone might have more thoughts here.

Related packages:

aiohttp==3.6.2
pytest==5.4.0
pytest-aiohttp==0.3.0
pytest-cov==2.8.1
pytest-forked==1.1.3
pytest-mock==2.0.0
pytest-responses==0.4.0
pytest-xdist==1.31.0

Some detail around the callstack:

➜  ~/D/zeus (feat/increase-lru-cache) ✗ py.test -x --pdb
=========================================================================================== test session starts ============================================================================================
platform darwin -- Python 3.8.1, pytest-5.4.0, py-1.8.1, pluggy-0.13.1
rootdir: /Users/dcramer/Development/zeus, inifile: setup.cfg
plugins: mock-2.0.0, celery-4.1.1, xdist-1.31.0, aiohttp-0.3.0, forked-1.1.3, responses-0.4.0, cov-2.8.1
collecting 186 items
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
In function "test_health_check":
Parameter "loop_factory" should be declared explicitly via indirect or in function itself
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /Users/dcramer/Development/zeus/.venv/lib/python3.8/site-packages/_pytest/python.py(1144)_validate_explicit_parameters()
-> fail(msg, pytrace=False)
(Pdb) l
1139                    func_name = self.function.__name__
1140                    msg = (
1141                        'In function "{func_name}":\n'
1142                        'Parameter "{arg}" should be declared explicitly via indirect or in function itself'
1143                    ).format(func_name=func_name, arg=arg)
1144 ->                 fail(msg, pytrace=False)
1145
1146
1147    def _find_parametrized_scope(argnames, arg2fixturedefs, indirect):
1148        """Find the most appropriate scope for a parametrized call based on its arguments.
1149
(Pdb) u
> /Users/dcramer/Development/zeus/.venv/lib/python3.8/site-packages/_pytest/python.py(939)parametrize()
-> self._validate_explicit_parameters(argnames, indirect)
(Pdb) l
934
935             self._validate_if_using_arg_names(argnames, indirect)
936
937             arg_values_types = self._resolve_arg_value_types(argnames, indirect)
938
939  ->         self._validate_explicit_parameters(argnames, indirect)
940
941             # Use any already (possibly) generated ids with parametrize Marks.
942             if _param_mark and _param_mark._param_ids_from:
943                 generated_ids = _param_mark._param_ids_from._param_ids_generated
944                 if generated_ids is not None:
(Pdb) u
> /Users/dcramer/Development/zeus/.venv/lib/python3.8/site-packages/aiohttp/pytest_plugin.py(203)pytest_generate_tests()
-> metafunc.parametrize("loop_factory",
(Pdb) l
198                         "Unknown loop '%s', available loops: %s" % (
199                             name, list(factories.keys())))
200                 else:
201                     continue
202             factories[name] = avail_factories[name]
203  ->     metafunc.parametrize("loop_factory",
204                              list(factories.values()),
205                              ids=list(factories.keys()))
206
207
208     @pytest.fixture

https://github.com/aio-libs/aiohttp/issues/4626

[nicoddemus]

Thanks @dcramer,

This is because of https://github.com/pytest-dev/pytest/issues/5712.

Can you provide a reproducible example? Curious that it was being used by aiohttp to parametrize loop_factory.

[dcramer]

I'm not sure how best to isolate it down, but if you take HEAD of getsentry/zeus, and bump pytest to 5.4.0, it will reproduce every time. Zeus isnt a small project, but its somewhat easy to setup since its just python packages (and mostly automated using tooling). There's also a docker build but likely harder to debug from that. Is that useful enough?

[dcramer]

Also possible @snstanton might be able to help on a simpler reproduction?

[nicoddemus]

I have to go to bed now, but if anybody wants to try it, it might be just a matter of passing indirect=True to this parametrized call:

https://github.com/aio-libs/aiohttp/blob/fe647a08d1acb53404b703b46b37409602ab18b4/aiohttp/pytest_plugin.py#L202-L204

[kqf]

@nicoddemus Thanks for the hints. I have a related problem with pytest==5.4.0. There are a lot of tests in my repos that look like this:

import pytest


@pytest.fixture
def config(color):
    return "{}.json".format(color)


@pytest.fixture
def data(config):
    return "/etc/colors/{}".format(config)


@pytest.mark.parametrize("color", ["red", "blue"])
def test_dependent(data):
    print(data)

This yields the same error:

Parameter "color" should be declared explicitly via indirect or in function itself

Now when I use @pytest.mark.parametrize("color", ["red", "blue"], indirect=True) , this gives me another error:

fixture 'color' not found

Finally, I figured out that my test should be modified like this (without indirect=True):

@pytest.mark.parametrize("color", ["red", "blue"])
def test_dependent(data, color):
    print(data)

This works, but I don't know why. Is there a magic configuration that allows to fall back to old behaviour?

[QuLogic]

This is biting Matplotlib tests in the same way as @kqf. Adding indirect=True doesn't help since the fixture doesn't see the parameter, and the whole point of putting it in the fixture was so we _wouldn't_ have to change the functions.

[ishepard]

Same thing for Pydriller, tests are failing with latest version. A small scenario would be this one:

import pytest

@pytest.fixture
def something():
    return None

@pytest.fixture
def using_something(something):
    return 2 + something

@pytest.mark.parametrize('something', [2])
def test_something(using_something):
    assert using_something == 4

This test passes using pytest 5.3.5

> pip install pytest==5.3.5
> pytest tmp.py
=========================== test session starts ===========================
platform darwin -- Python 3.7.6, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: /Users/dspadini/Documents/pydriller, inifile: pytest.ini
collected 1 item

tmp.py::test_something[2] PASSED                                                                                            [100%]

=========================== 1 passed in 0.07s ============================

but it fails with 5.4.0

> pip install pytest==5.4.0
> pytest tmp.py
=========================== test session starts ===========================
platform darwin -- Python 3.7.6, pytest-5.4.0, py-1.8.1, pluggy-0.13.1
rootdir: /Users/dspadini/Documents/pydriller, inifile: pytest.ini
collected 0 items / 1 error

============================== ERRORS ===============================
_________________________________ ERROR collecting tmp.py ___________________________________
In function "test_something":
Parameter "something" should be declared explicitly via indirect or in function itself
===================================================== short test summary info =====================================================
ERROR tmp.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================== 1 error in 0.16s =========================================================

[youtux]

Same problem with dogpile_filesystem

[youtux]

Also pytest_factoryboy is affected by this, and I don't really see an easy way to solve it there, given the nature of the project.

[nicoddemus]

Thanks everyone for the report!

This was intended to remove what we assumed was "surprising" behavior, but as things always goes with projects used by many many people, "surprising" often becomes "used and expected". 😁

@RonnyPfannschmidt I propose we just revert that change, and make it part of the expected behavior of fixtures. What do you think?

[RonnyPfannschmidt]

@nicoddemus we start with reverting due to regressions, but i would like to avoid setting it up as expected api just yet

i think we need a few state graphs to determine the behaviour "yay"

[nicoddemus]

Agreed.

I will prepare the revert and a new hot-fix release.

Thanks everyone here for the quick reports and sorry about the inconvenience! 👍

[nicoddemus]

5.4.1 is out, thanks everyone!

[lburg]

@kqf Just fyi, here's the changes version 5.4.0 would have required in your example:

@pytest.fixture                                                  
def color(request):                                              
    return "{}.json".format(request.param)                       


@pytest.fixture                                                  
def data(color):                                                 
    return "/etc/colors/{}".format(color)                        


@pytest.mark.parametrize("color", ["red", "blue"], indirect=True)
def test_dependent(data):                                        
    print(data)                                                  

The config fixture is renamed color and accepts the request fixture where you can find the parametrized color value, thanks to indirect=True. No need to modify your test function signature.

[kqf]

@lburg Thanks a lot, I missed that one.

[youtux]

Thanks guys for the quick fix!