Pytest: metafunc.parametrize(..., scope="session") fails when mixed with other parametrizations

Created on 6 Jun 2018 · 7Comments · Source: pytest-dev/pytest

Both, python 2.7.13 and 3.5.3, pytest 3.6.1, no additional pytest plugins.

# conftest.py
import pytest

KNOWN_BRANCHES = {
    # The number means how many times given branch has been set-up
    "branch_1": 0,
    "branch_2": 0,
}
def pytest_generate_tests(metafunc):
    if 'tested_branch' in metafunc.fixturenames:
        branches = KNOWN_BRANCHES.keys()
        metafunc.parametrize("tested_branch", branches, scope="session")

@pytest.fixture(scope="session")
def repo_preparation(tested_branch):
    """ If it would have a "session" scopoe then the value should be always 1 for each branch."""
    KNOWN_BRANCHES[tested_branch] += 1
    return KNOWN_BRANCHES[tested_branch]

And the simplest reproduction:

# test_parametrization.py
import pytest

def test_nothing(repo_preparation):
    assert repo_preparation == 1  # that passes

@pytest.mark.parametrize("_", ["param_1", "param_2"])
def test_basic_parametrization(repo_preparation, _):
    # that causes parametrization confisuon
    assert repo_preparation == 1, "Repo prepared more than once."

Gives such an result:

$ pytest -v test_parametrization.py 
================================== test session starts ==================================
platform linux -- Python 3.5.3, pytest-3.6.1, py-1.5.3, pluggy-0.6.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/kamichal/ecws/pyplayground, inifile:
collected 6 items                                                                       

test_parametrization.py::test_nothing[branch_1] PASSED                            [ 16%]
test_parametrization.py::test_basic_parametrization[branch_1-param_1] PASSED      [ 33%]
test_parametrization.py::test_nothing[branch_2] PASSED                            [ 50%]
test_parametrization.py::test_basic_parametrization[branch_1-param_2] PASSED      [ 66%]
test_parametrization.py::test_basic_parametrization[branch_2-param_1] FAILED      [ 83%]
test_parametrization.py::test_basic_parametrization[branch_2-param_2] FAILED      [100%]

======================================= FAILURES ========================================
_____________________ test_basic_parametrization[branch_2-param_1] ______________________

repo_preparation = 2, _ = 'param_1'

    @pytest.mark.parametrize("_", ["param_1", "param_2"])
    def test_basic_parametrization(repo_preparation, _):
        # that causes parametrization confisuon
>       assert repo_preparation == 1, "Repo prepared more than once."
E       AssertionError: Repo prepared more than once.
E       assert 2 == 1

test_parametrization.py:12: AssertionError
_____________________ test_basic_parametrization[branch_2-param_2] ______________________

repo_preparation = 3, _ = 'param_2'

    @pytest.mark.parametrize("_", ["param_1", "param_2"])
    def test_basic_parametrization(repo_preparation, _):
        # that causes parametrization confisuon
>       assert repo_preparation == 1, "Repo prepared more than once."
E       AssertionError: Repo prepared more than once.
E       assert 3 == 1

test_parametrization.py:12: AssertionError
========================== 2 failed, 4 passed in 0.03 seconds ===========================

Setting indirect=True in the metafunc.parametrize call fixes that problem, but there is no information that it's required for larger scopes to work properly. There is only an advice to do so, but...

parametrize bug

Source

kamichal

👍1

Most helpful comment

I came across the same bug, I think...
When using metafunc.parametrize(scope=session) in combination with parametrized tests, all of my session scoped fixtures teared down after each test.

This behavior can be demonstrated very vividly with --setup-show:

SETUP    S session_fixture1 # <-- This fixture is generated with pytest_generate_tests
SETUP    S session_fixture2 (fixtures used: session_fixture1)
    SETUP    M module_fixture (fixtures used: session_fixture2)
        SETUP    F function_fixture (fixtures used: module_fixture)
            test_file.py::TestClass::my_test1()
        TEARDOWN F function_fixture
    TEARDOWN M module_fixture
TEARDOWN S session_fixture2 
TEARDOWN S session_fixture1 # <-- Session scoped fixture gets tear downed
SETUP    S session_fixture1 
SETUP    S session_fixture2 (fixtures used: session_fixture1)
    SETUP    M module_fixture (fixtures used: session_fixture2)
        SETUP    F function_fixture (fixtures used: module_fixture)
            test_file.py::TestClass::my_test2()
        TEARDOWN F function_fixture
    TEARDOWN M module_fixture
TEARDOWN S session_fixture2
TEARDOWN S session_fixture1

Workaround

@kamichal For me, the workaround with indirect=True also works in combination with parameters received from metafunc.config.getoption.

Just define a real fixture with the same name (in the example below my_fixture) and use the request fixture to return the indirect parametrized value:

def pytest_addoption(parser):
    parser.addoption("--my_parser_option")

def pytest_generate_tests(metafunc):
    if "my_fixture" in metafunc.fixturenames:
        metafunc.parametrize("my_fixture", metafunc.config.getoption("my_parser_option"), scope="session", indirect=True)

@pytest.fixture(scope="session")
def my_fixture(request):
    return request.param

def test_fixture(my_fixture):
    assert my_fixture == "world"

Then you can call it with pytest --my_parser_option hello, which will result in "hello" != "world"
I hope this helps someone.

Regards
Winklerrr

winklerrr on 19 Jun 2018

👍2

All 7 comments

GitMate.io thinks possibly related issues are https://github.com/pytest-dev/pytest/issues/634 (metafunc.parametrize overwrites scope), https://github.com/pytest-dev/pytest/issues/519 (fixture scope is ignored when using metafunc.parametrize()), https://github.com/pytest-dev/pytest/issues/570 (indirect=True in parametrize breaks fixture scopes), https://github.com/pytest-dev/pytest/issues/244 (parametrize fails when values are unhashable), and https://github.com/pytest-dev/pytest/issues/1111 (pytest.mark.parametrize fails with lambdas).

pytestbot on 6 Jun 2018

that looks like a bug

note that you can use fun workaround

since you already know your parameters,

# conftest.py
import pytest

KNOWN_BRANCHES = {
    # The number means how many times given branch has been set-up
    "branch_1": 0,
    "branch_2": 0,
}

@pytest.fixture(scope="session", param=KNOWN_BRANCHES.keys())
def repo_preparation(request):
    """ If it would have a "session" scope then the value should be always 1 for each branch."""
    KNOWN_BRANCHES[request.param] += 1
    return KNOWN_BRANCHES[request.param]

might do (untested)

RonnyPfannschmidt on 6 Jun 2018

Thank you Ronny. I know the "workaround" very well.
The point is that I would like to take the tested_branch parameters from parser.addoption.
So, contents of that list is variable in my case, that's why I took pytest_generate_tests hook.

kamichal on 6 Jun 2018

@kamichal i see - in that case - there is a deeper underlying issue that i wont be investigating in near future - i have no idea if any of the other core devs has other plans