Pytest: setup_* executed before all fixtures

[pytestbot]

Originally reported by: Haak Saxberg (BitBucket: haaksmash, GitHub: haaksmash)

---

consider the following dummy test file:

#!python

import pytest

@pytest.yield_fixture(scope="session", autouse=True)
def print_session():
    print "session fixture"
    yield
    print "session teardown"


class TestClass(object):

    def setup_method(self, method):
        print "method setup"

    def teardown_method(self, method):
        print "method teardown"

    def test_printer(self):
        print "test method"

pytest --capture=no test_file.py will generate the following output:

#!bash

==================== test session starts ====================
platform darwin -- Python 2.7.5 -- py-1.4.20 -- pytest-2.5.2
collected 1 items

tests/test_ordering.py method setup
session fixture
test method
.method teardown
session teardown


================= 1 passed in 0.01 seconds =================

it looks like all setup_* methods are invoked before any fixtures are processed, but the teardown_* methods are called in the order you'd expect:

• method_fixture post-yield
• method_teardown
• class_fixture post-yield
• class_teardown
• module_fixture post-yield
• module_teardown

etc, etc.

that doesn't seem right, especially since the teardowns are in the correct order. In fact it's pretty bad, because you might (I would think rightly) depend on the session having been set up already in your setup_method.

---

• Bitbucket: https://bitbucket.org/pytest-dev/pytest/issue/517

[pytestbot]

_Original comment by_ Floris Bruynooghe (BitBucket: flub, GitHub: flub):

---

I'm not sure if this is something we should worry about since it is mixing two paradigms. When writing py.test tests one should be using fixtures and the setup and teardown methods is only there for unittest compatibility. Using both together is just asking for corner cases to appear.

All IMHO of course.

[pytestbot]

_Original comment by_ Haak Saxberg (BitBucket: haaksmash, GitHub: haaksmash):

---

yeah, it's a little unorthodox. i'm working with a sorta-legacy system, though, and i needed something that would be guaranteed to run before any other method-scope fixtures. That means setup_method, because as far as I can tell fixtures are applied basically lexicographically and i don't want to rename the fixture a_i_need_this_setup_first or something :P

I'm all ears for other solutions!

[pytestbot]

_Original comment by_ Andreas Pelme (BitBucket: pelme, GitHub: pelme):

---

While tracking down the issue described in https://github.com/pelme/pytest_django/issues/79 I also hit this problem. I wrote a patch for it and went here to open an issue about it to find this, opened 2 days ago. :-)

@flub: I am also not sure which way of running the fixtures before or after setUpClass/setup_method really makes most sense, since it is as you point out, it is a strange mix to begin with.

However, I think it is way more practical to run fixtures first. In the case of pytest-django, I am not sure there is a way to support LiveServerTestCase unless we do some nasty monkey patching on LiveServerTestCase (pretty much removing setUpClass and re-implementing it with autouse fixtures so that we can control the order).

Another way to put it: Are there any reasons to _not_ run fixtures before all unittest setup stuff?

Please have a look at the PR at https://bitbucket.org/hpk42/pytest/pull-request/170/fix-for-issue517/diff, it doesn't break any other test so I guess it cannot be that bad. :-)

[pytestbot]

_Original comment by_ holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

I think the proper way to fix this issue is to internally turn setupClass/Module methods into appropriate pytest autouse fixtures. This way setupClass would execute before any function-level fixtures, setupModule before setupClass etc.

[pytestbot]

_Original comment by_ Andreas Pelme (BitBucket: pelme, GitHub: pelme):

---

If someone else runs into a similar problem, this has been solved in pytest-django with some monkey patching:

https://github.com/pytest-dev/pytest-django/commit/8a9aacbd835a49064ff8999137fa6ede34f3c520

[ghostsquad]

:+1:

[nicoddemus]

For the record, a common pattern I've used when converting tests from xUnit style to pytest style is to transform the xUnit setup methods in autouse fixtures (which coincidentally is the prefferred solution by @hpk42 to fix this internally in pytest):

#!python

import pytest

@pytest.yield_fixture(scope="session", autouse=True)
def print_session():
    print "session fixture"
    yield
    print "session teardown"


class TestClass(object):

    @pytest.yield_fixture(autouse=True)
    def my_method_setup(self, method):
        print "method setup"
        yield
        print "method teardown"

    def test_printer(self):
        print "test method"

This has the advantage of not mixing paradigms and also allows to use the much nicer yield_fixture to collapse both setup and teardown into a single function. :smile:

/cc @haaksmash

[kirill-gr]

There is an another actual case where this problem becomes apparent: a backwards transition from pytest-style fixtures to xunit-style. Our team finds former to be "too much magic" and latter more explicit and (thus) better supported by IDE. But pytest session-scoped fixture is still very usefull for session teardown.

[hpk42]

FTR I once reimplemented xunit fixtures in terms of autouse fixtures and there wasn't much of a problem. However, it would add three extra fixtures to every test by default (for function, class and module scope) and somehow i didn't like that. the three autouse fixtures would introspect and call setup functions. It would be nicer if those autouse fixtures would only appear if there are actual xUnit-style methods. While parsing fixture functions we could probably also check for xunit-style and integrate them. Eventually i think we should go for it.

FWIW xUnit-style fixtures are conceptually autouse fixtures but you access dependencies as attributes through "self" and rely on other xUnit/autouse fixtures to have set the attribute before. Arguably that's also quite some magic but it's true that you do "self.ATTRNAME = value" assignments yourself and don't rely on the invocation of fixture functions by argument names. xUnit, however, makes it harder to keep fixtures modular as you are always tied to inheritance hierarchies and working with Mixins is only so much fun.

[RonnyPfannschmidt]

idea, how about having a hook that tells active fixtures, with implementations in the plugins

the mark based one would go to the mark plugin, the xunit discovery would go to the unittest compat plugin and the python plugin would handle autouse and argument names

[RonnyPfannschmidt]

then the xunit fixtures would not be autouse fixtures, but rather discovered fixtures per item

[kirill-gr]

it harder to keep fixtures modular

May be I'm missing something, but it looks as easy as keeping any other function modular (except when using parametrized fixtures, which I also find confusing in practice)
Also doing "self.ATTRNAME = value" assignment gives typehinting and code navigation in a modern IDE.

[RonnyPfannschmidt]

an inheritance hierachy is per se less composable than a configurable DI engine
i'd prefer Type hint based DI over going back to xunit setup ^^

[kirill-gr]

I meant achieving modularity via encapsulating common setup code in regular functions:

def do_some_setup():
    <setup_steps>

def setup_function(function):
    do_some_setup()

def test_something():
    ...

instead of:

@pytest.fixture
def some_setup_fixture():
    <setup_steps>

def test_something(some_setup_fixture):
    ...

Also it feels easier for people who are not familiar with pytest (we also use python test for testing java parts).

[RonnyPfannschmidt]

it also prevents reconfiguration and dependency controll

[hpk42]

@RonnyPfannschmidt @kromkrom i suggest to not discuss the xunit/fixture matter further here until it relates to better integration of xunit/fixture.

@RonnyPfannschmidt not sure i understand your suggestion. my suggestion targets an internal refactoring but i can imagine we could use a hook there to have plugins discover fixtures themselves. Probably that's what you meant.

@kromkrom you basically want to use a mixture of xUnit style and pytest style fixtures in your project not only now but also in the future, right?

[kirill-gr]

@hpk42 yes, I see cases for both styles in one project

[hpk42]

are you aware that you can integrate xUnit/pytest fixtures nicely e.g. for module level:

@pytest.fixture(autouse=True, scope="module")
def setup_module(session_fix):
      do something on top of the session fixture.

IOW xunit setup methods can typically just be marked (you can have a short-cut marker if you prefer to not have to spell out the above and could have a "@xunit_fixture"). If you use the marker everything will be nicely integrated, no surpises regarding ordering etc.

[hpk42]

and the name doesn't matter. If you want to spell it setupModule that's fine.

[kirill-gr]

@hpk42 Do you mean that explicitly depending xUnit fixture on session-level pytest fixture will solve the ordering issue?

[nicoddemus]

I think @hpk42 meant to just use fixtures everywhere instead of xunit style... often times is just a matter of renaming it to something else and adding the pytest.fixture decorator (see my comment).

[ofayans]

I don't know if these issues are related but I've noticed that setup_method is run not only before each test_* method but also before any arbitrarily named methods within a test class.
Consider this simplified example:

class TestReplicaInstall:
def install(self):
install_master(self.master)

def setup_method(self):
    install_client(self.master, self.client)

def test_replica_installation(self):
    install_replica(self.master, self.client)

Here replica installation requires an installed client. One would expect that setup_method would be called ONLY before 'test_replica_installation', but in reality it is invoked before 'install' as well

[nicoddemus]

@ofayans I could not reproduce your problem:

class TestReplicaInstall:
    def install(self):
        print('install')

    def setup_method(self, m):
        print('setup method')

    def test_replica_installation(self):
        print('test_replica_installation')

Here's my output:

] py.test foo.py -s
============================= test session starts =============================
platform win32 -- Python 2.7.11, pytest-2.8.4, py-1.4.31, pluggy-0.3.1
rootdir: e:\ws\SimBR-pkg\Projects\simbr, inifile:
plugins: catchlog-1.1, cov-1.8.1, localserver-0.3.3, mock-0.7.0, timeout-1.0.0, xdist-1.13.1
collected 1 items

foo.py setup method
test_replica_installation
.

================= 1 passed, 1 pytest-warnings in 0.01 seconds =================

install is never called, as expected. Could you please post a minimal reproducible example? Thanks!

[abhikeny]

Hello,
I have seen this too. For me, the setup_module() (which is defined in the same file/module as the test function) is executed before any fixture that has scope="module", autouse=True (even though the fixture is defined in the conftest.py, which is different than the test module/file). So, the execution sequence is:

setup_module
fixure_with_module_scope
test_1
test_2
...
teadown_module

This seems less intuitive, as I would expect the fixture_with_module_scope to be executed before anything from the test file is executed.

[RonnyPfannschmidt]

xunit setup and fixture setup are distinct

xunit setup is applied before fixture consideration

[nicoddemus]

It has been considered to refactor the xunit support to be implemented in terms of autouse fixtures, but I'm sure there must be test suites depending on the current behavior, so it's not so simple I'm afraid :sad:

[RonnyPfannschmidt]

maybe we should consider deprecating xunit style setup ^^ - its strictly less powerful and more painful than fixtures

[abhikeny]

@nicoddemus and @RonnyPfannschmidt , Thanks for the response.
I would agree that this might not be an easy change, as a lot of people might be using this as it exists.

If there is a known workaround to not have to change any test files (with the setup_module() in it), and still be able to run pseudo code (like the external _module_ level fixture described above) before and/or after the file, I would really appreciate it.

[nicoddemus]

@abhikeny unfortunately nothing comes to mind...

[abhikeny]

Thanks for the response again, @nicoddemus .
I will look into the per-directory conftest.py files with session level fixtures (at least I will get as close to executing the pseudo code before and/or after the set of modules/files in a directory, if not as exactly as what I am looking for).

[ghostsquad]

@nicoddemus major issue updates of pytest should be expected to have potentially backwards breaking changes. What's wrong with rolling out a new version of pytest with this and other wanted features that are hard to implement without breaking everyone?

[nicoddemus]

@nicoddemus major issue updates of pytest should be expected to have potentially backwards breaking changes. What's wrong with rolling out a new version of pytest with this and other wanted features that are hard to implement without breaking everyone?

We avoid as much as possible to introduce backward incompatible changes, because we know how pain that brings for companies and individuals which rely on pytest. Sometimes a backward incompatible change is small and there's a simple way for users which depend on it to workaround it (for example, simply setting an option on pytest.ini to restore the previous behavior). Other times, the change in behavior is too subtle and not easy to workaround, potentially breaking hundreds of test suites and requiring manual and tedious intervention in order to adapt the test suite. This change seems like the latter, that's why we usually are wary of changing it, even for the upcoming 3.0 version.

That being said, I'm not saying we shouldn't change it, just that we usually take great care and try to obtain user's feedback if that backward incompatible change should be made or it would introduce too much breakage... that's why I said it was not so simple. :grin:

[RonnyPfannschmidt]

i can see introducing 2 internal plugins for implementing the old and new behaviour
then a ini option could be used to make things more integratable,

that can be done after 3.0

[rsyring]

FWIW, I'd really like to see xunit & nose style function/methods become implemented in such a way that they are integrated with the more powerful pytest fixture system.

In particular, I find that the methods available on classes are more succinct than having to use the pytest fixture system. My ideal would be the ability to write:

def setup_module(app):
    pass

class TestMe:
    @classmethod
    def setup_class(app):
        pass

    def setup(app):
        pass

Assuming app was a fixture coming from conftest.py

[nicoddemus]

@rsyring thanks for the feedback.

I know what you are saying, but I thought I would mention that your example can easily be tweaked a bit:

@pytest.fixture(scope='module', autouse=True)
def my_setup_module(app):
    pass

class TestMe:
    @classmethod
    @pytest.fixture(scope='class', autouse=True)
    def my_setup_class(app):
        pass

    @pytest.fixture(autouse=True)
    def my_setup(app):
        pass

Of course there's an extra decorator and we can't call the function/methods setup_, but this has the advantage that it works today and is part of the fixture system.

It is even possible to write a plugin which takes these setup_* methods and automatically turns them into autouse-fixtures with the appropriate scope if you are so inclined.

Just thought I would mention the above in case you didn't know.

[rsyring]

@nicoddemus I am aware of those decorators, but appreciate your taking the time to point them out. I realize it's just one additional decorator, but it feels like a real pain when I'm so used to the setup methods.

The plugin is an interesting idea. I might look into that some more.

[nicoddemus]

The plugin is an interesting idea. I might look into that some more.

Great, feel free to ask for guidance. 👍

[tlandschoff-scale]

Of course there's an extra decorator and we can't call the function/methods setup_, but this has the advantage that it works today and is part of the fixture system.

I just tried it (we have tons of tests that are still using xUnit setup/teardown because it would be quite some work to convert them to the fixture based setup) and it appears that a method marked with

class TestMe:
    @pytest.fixture(autouse=True)
    def my_setup(self):
        ...

is actually not called.

[nicoddemus]

Can you provide a reproducible example? This works for me:

import pytest

class TestMe:
    @pytest.fixture(autouse=True)
    def my_setup(self):
        assert 0

    def test_foo(self):
        pass

============================= test session starts =============================
platform win32 -- Python 3.6.0, pytest-3.2.4.dev60+g1a3d9cd.d20171104, py-1.4.34, pluggy-0.4.0
rootdir: C:\pytest\.tmp, inifile: pytest.ini
plugins: xdist-1.20.1, forked-0.2, hypothesis-3.25.0
collected 1 item

.tmp\test_xunit_fixture.py E

=================================== ERRORS ====================================
______________________ ERROR at setup of TestMe.test_foo ______________________

self = <test_xunit_fixture.TestMe object at 0x000002A4FFDA7F60>

    @pytest.fixture(autouse=True)
    def my_setup(self):
>       assert 0
E       assert 0

.tmp\test_xunit_fixture.py:7: AssertionError
=========================== 1 error in 0.07 seconds ===========================

Also we use this trick all the time at work for some time now.

[tlandschoff-scale]

Beats me. I should have started with a trivial example, no idea how I borked it.
It appears to be working now, sorry for the fuss. Unfortunately I did not even stash the modifications so I have no idea how I messed up on this.

Thank for very much for the example which immediately proved me wrong. Worked like a charm and also the part of the test suite I just adapted is working great. Thanks so much!

[nicoddemus]

No worries glad it was not some weird bug and it is actually working for you. 👍

(For the record I imagined you might have another my_setup down the class which was shadowing the one with the decorator, but forgot to mention).

[craigds]

That's a great workaround, except if you need to override it in a subclass and call super(), in which case you have to iterate over it (because it's a generator) :/

I ended up with this awful looking thing

class MySubclassTest(MySuperclassTest):
    @pytest.fixture(autouse=True)
    def my_setup_method(self, a_fixture_to_pass_to_super):
        # Calling super() is kinda gnarly when the super fixture is a generator
        sup = super(MySubclassTest, self).my_setup_method(a_fixture_to_pass_to_super)

        # Enter the super() fixture
        sup.next()

        # Add some stuff
        blah.blah.blah()

        # Run the test
        yield

        # Exit the super() fixture (raises StopIteration, which is handled by pytest.fixture())
        sup.next()

This works, though I suspect a_fixture_to_pass_to_super is setup twice (once for the subclass and once for the superclass). I haven't investigated that though.

[nicoddemus]

Dan Nealschneider e-mail on testing-in-python made me think: perhaps can introduce this change together with a configuration option that allows users to revert to the old behavior if they need to (say use_old_xunit_setup=true). This would alleviate the problem to impacted users and would allow us to move the implementation forward.

[RonnyPfannschmidt]

a starting point would be to move the behaviour into a plugin class, then its a question of implementing a second plugin class and/or adding a switch/boolean

[nicoddemus]

That might be a valid approach, true.

Created #3094 to better discuss this proposal.

[luv]

Yes, I have to say this one is an annoying issue, and unfortunately present for more than 4 years, but really my only issue with pytest :).

I have finally worked around this in my conftest.py by creating a fixture(class scope, autouse) and monekypatching _pytest/unittest.py:UnitTestCase.setup to not call the setup but save it in a dictionary and then I simply call the appropriate setup from the fixture request.node.obj is in the dictionary.

Quick and dirty indeed :).

[luv]

I have also prepared a plugin https://github.com/luv/pytest_setupclass_fix which does the monkey patching (for pytest >=3.2.0).

[nicoddemus]

Indeed it is a bummer that this issue has been around for so long; I think one of the factors that contributed to it is that for a long time people were uncertain on how to fix this. Now that we have a plan at least it should be possible to move forward.

FWIW this is in my personal roadmap for the medium term, after I work on the internal warnings system.

[nicoddemus]

Some bad news folks: I've implemented this, and while almost all tests work as before except for 2, backward compatibility with yield-tests semantics make this impossible to merge right now (see a more detailed explanation in #4091).

This means that while this is perfectly doable, we will need to wait for pytest 4.1 release, which will remove yield-tests for good.

[nicoddemus]

Will be released in 4.2. 👍