Pytest: Provide a way to nicely control fixture execution order

[astraw38]

We sometimes have fixtures that need to execute in a specific order, without having them include one another as an argument. This is to provide flexibility with fixtures, so they aren't always linked to one another.

Quick example:

@pytest.fixture()
def a():
    print "In a"


# Fixture in conftest.py, used in other contexts
@pytest.fixture()
def multi_use():
    # Don't always want to use this fixture with 'a', so isn't included as an argument.
    print "Should always be executed after 'a', if 'a' is included with the test via usefixtures"

I know there's ways to do this w/ using fixture arguments, like having intermediary fixtures that exist purely to order the other fixtures, but it's not very pretty.

[RonnyPfannschmidt]

the idea of declaring indirect dependency orders is tricky to solve,
i'd like to avoid a quick intermediate solution and focus on making the setupstate/di framework more self contained first

[astraw38]

I was thinking of was something similar to what was done w/ plugin order via tryfirst/trylast ... but no idea how quick that would be. Perhaps a sorting system similar to how logging levels work, where fixture order within a level would be indeterminate?

[hpk42]

i guess there could be another parameter to the "pytest.fixture" decorator which influences ordering like "after='a''" but not sure how easy it would be to implement. It would also need to provide proper error messages when ordering can not be established according to fixture configuration. And i somewhat agree to the point Ronny is making that we might need to do some internal refactoring first. That being said, it's always good to have a guiding use case that should be enabled by a refactoring (apart from the need for full backward compat of course). FWIW myself i probably won't try to implement this on unpaid time.

[ktosiek]

For a more concrete use case: I'm trying to implement shared DB state fixtures based on layering subtransactions here: https://github.com/pytest-dev/pytest-django/pull/258, and I need to make sure db (which is function-level) only runs after all users of shared_db_wrapper.

For this case something like _inherited_ before="db" would be needed.

[nipunn1313]

I also have a concrete use case.

A top level conftest.py has a couple of fixtures

def user():
    return new_user

def application(user):
    return new_application(user)

A subdirectory with some special tests needs to create some special state for user before loading the application. Ideally we could do.

@pytest.usefixtures(before='application')
def special_user_setup(user):
    prepare_some_special_stuff_for(user)

We could override the user fixture, but that would be unfortunate if the user fixture was doing a lot of work on its own (we'd have to copy paste).
In this case, application cannot inherit from special_user_setup.

Is there a way to do this today, or will it require such a before= argument?

[The-Compiler]

@nipunn1313 I wonder if you could just do something like:

@pytest.fixture
def user(user):
    pass

Kind of a hack, it might or might not work.

[aurzenligl]

I had the same problem (wanted bar to execute before foo) and used following hack:

@pytest.yield_fixture
def foo():
    ...

@pytest.mark.early
@pytest.yield_fixture
def bar():
    ...

def reorder_early_fixtures(metafunc):
    for fixturedef in metafunc._arg2fixturedefs.values():
        fixturedef = fixturedef[0]
        if getattr(fixturedef.func, 'early', None):
            order = metafunc.fixturenames
            order.insert(0, order.pop(order.index(fixturedef.argname)))

def pytest_generate_tests(metafunc):
    reorder_early_fixtures(metafunc)

def test_baz(foo, bar):

Whereas I suppose marking fixtures - while not intended - is innocuous, I have doubts as to how foolish assumptions of reorder_early_fixtures may be:

1. presence of metafunc._arg2fixturedefs as dictionary with tuples with FixtureDef as first elem,
2. shuffling order of names in metafunc.fixturenames is legal and influences order of execution.

How long would these assumptions remain to be true (I'm using pytest 3.0.3)?
Is there any safer way to achieve this?

[fruch]

@aurzenligl looks like a good idea, question, can we implement the marker with the before param mentioned above, sounds like a great way to enforce the order

[ssalbdivad]

While I do suggest caution in ordering fixtures, I've written up a flexible way to do so when the situation calls for it:

def order_fixtures(metafunc):
    metafunc.fixturenames[:] = []
    orders = {name: getattr(definition[0].func, "order", None)
              for name, definition in metafunc._arg2fixturedefs.items()}
    ordered = {name: getattr(order, "args")[0] for name, order in orders.items() if order}
    unordered = [name for name, order in orders.items() if not order]
    first = {name: order for name, order in ordered.items() if order and order < 0}
    last = {name: order for name, order in ordered.items() if order and order > 0}
    merged = sorted(first, key=first.get) + unordered + sorted(last, key=last.get)
    metafunc.fixturenames.extend(merged)

def pytest_generate_tests(metafunc):
    order_fixtures(metafunc)

Fixtures can then be ordered so that those with a negative order are run before unordered fixtures in ascending order, then unordered fixtures are run, then those with a positive order are run in ascending order, as in the following example:

@pytest.mark.order(-2)
@pytest.fixture
def first():
    print("I go first.")

@pytest.mark.order(-1)
@pytest.fixture
def second():
    print("I go second.")

@pytest.fixture
def third():
    print("I go third.")

@pytest.mark.order(1)
@pytest.fixture
def fourth():
    print("I go fourth.")

@pytest.mark.order(2)
@pytest.fixture
def fifth():
    print("I go fifth.")

def test_something(fifth, fourth, third, second, first):
    print("Running your test.")

"""
Output:
I go first.
I go second.
I go third.
I go fourth.
I go fifth.
Running your test.
"""

[RonnyPfannschmidt]

in general marks are not supported on fixtures, and its not clear how marks of fixtures shouzld map to tests,
so the solution cant use marks

[ssalbdivad]

Hey @RonnyPfannschmidt,
The intent of this solution was not to propose its adoption by pytest but rather to provide a flexible and seemingly reliable way for folks to specify the execution order of their fixtures expressively while a decision is made regarding how best to integrate first-class support for doing so. Based on my testing this seems to be fairly robust, but if there are situations in which relying on fixture marking in the way that I have will cause things to break it'd be really useful to know what those are.
Thanks!

[claweyenuk]

What about using something like z-index from CSS?

@pytest.fixture(teardown_index=-9999)
def first():
    pass

@pytest.fixture(teardown_index=9999)
def last:
    pass

just go in order of teardown index, if 2 items have the same value, no guarantees. The default index can be 0. If teardown should be early, use a negative, if they should be later, use a positive number.

[RonnyPfannschmidt]

that doesn't take scopes into account ^^

[claweyenuk]

Let's say scope takes priority, it doesn't make sense to teardown class fixtures after module fixtures. The kwarg could even by scope_teardown_index, though that is getting long

[RonnyPfannschmidt]

@claweyenuk that still just imposes a random artificial order - which in turn is very likely to bite someone

[romuald]

To complete @aurzenligl solution (that broke with pytest 4.1), here is an updated example:

def reorder_early_fixtures(metafunc):
    """
    Put fixtures with `pytest.mark.early` first during execution

    This allows patch of configurations before the application is initialized

    """
    for fixturedef in metafunc._arg2fixturedefs.values():
        fixturedef = fixturedef[0]
        for mark in getattr(fixturedef.func, 'pytestmark', []):
            if mark.name == 'early':
                order = metafunc.fixturenames
                order.insert(0, order.pop(order.index(fixturedef.argname)))
                break


def pytest_generate_tests(metafunc):
    reorder_early_fixtures(metafunc)


@pytest.fixture
@pytest.mark.early
def early_stuff(self, another_fixture):
    # ...