Pytest: Pytest should't import conftest if __init__ raises Skipped

[rmorshea]

I have a package that needs to support Python 3 and 2 code. To accommodate this I've created a test directory specifically aimed at testing my Python 3 code. Within that directory I have an __init__.py with the following contents:

import sys
import pytest

if sys.version_info < (3, 0):
    pytest.skip("skipping Python 3 only tests", allow_module_level=True)

If I place a conftest.py file inside the directory containing this __init__.py and running pytest using Python 2 I get the following output:

==================================== test session starts =====================================
platform darwin -- Python 2.7.15, pytest-4.6.9, py-1.8.1, pluggy-0.13.1
plugins: env-info-0.3.0, mock-2.0.0, cov-2.8.1
collected 0 items / 1 skipped                                                                

================================== short test summary info ===================================
SKIPPED [1] xxxxxxxxxxxxxxxxx.py: skipping Python 3 only tests

-- Docs: https://docs.pytest.org/en/latest/warnings.html
=========================== 1 skipped, 1 warnings in 0.12 seconds ============================
ERROR: InvocationError for command xxx/bin/python -m pytest tests (exited with code 5)

I'm not really sure what's gone wrong, but this only occurs if I have a conftest.py within the directory I'm trying to skip, removing the conftest.py allows things to work again. If I use an explicit --ignore test_my_py_3_stuff instead of the programatic skipping, everything seems to work even if I have a conftest.py file.

[SalmonMode]

@rmorshea don't use __init__.py for that. Pytest isn't involved yet when then would run (it would be python itself parsing that package so that pytest can then work with it after), so there's no way for it to know you want to skip those tests.

Use conftest.py instead. Those are basically the __init__.py equivalent for pytest logic.

[rmorshea]

@SalmonMode so I moved the pytest.skip() call to a conftest.py file in the same directory however now pytest crashes with a traceback instead:

  File "xxxxx/conftest.py", line 7, in <module>
    pytest.skip("skipping Python 3 only tests", allow_module_level=True)
  File "xxx/py27/lib/python2.7/site-packages/_pytest/outcomes.py", line 104, in skip
    raise Skipped(msg=msg, allow_module_level=allow_module_level)
  builtins.Skipped: skipping Python 3 only tests

It seems like it's not actually catching Skipped errors inside conftest.py.

[SalmonMode]

@rmorshea you'll want to have it raise it at test runtime. If it's in the global scope, then it will still be happening while python itself is still parsing package, before pytest gets involved (or at least while pytest is importing it, so it'd have to be importing in a try/except, but I'm not sure if it is or isn't). To get it to happen at test runtime, it'll need to be in a fixture, e.g.:

@pytest.fixture(scope="session", autouse=True)
def check_python_version():
    if sys.version_info < (3, 0):
        pytest.skip("skipping Python 3 only tests")

The scope for this is session, so that it only happens once for the whole package, and happens as soon as possible.

It looks like the docs recommend the approach you were taking, so it could be a bug that it didn't initially work for you, but that approach may actually only work for a single module, and the approach in my example is more stable in the long run because it uses more fundamental features of pytest.

[rmorshea]

Ok, I think that makes sense as the flag in skip does mention it being for the "module level".

I think this issue should remain open though as the docs should definitely be updated to explain this.

[SalmonMode]

Definitely. It should also remain open until you can confirm this approach works.

[rmorshea]

@SalmonMode so unfortunately this does not work because pytest still attempts to collect the neighboring test files which are not valid Python 2 syntax so I now get a SyntaxError. The fact that skipping inside __init__.py works if and only if there is not conftest.py file really implies that pytest does in fact have control over whether or not pytest.skip() will or won't cause crash. My guess it that there just needs to be a try/except catch around the import for conftest.

Additionally given this problem with pytest attempting to import modules with Python3-only syntax, it might actually be incorrect to attempt to load the conftest.py file if __init__.py raises Skipped because conftest.py itself might contain Python3-only syntax and thus cause pytest to crash. Although it would be easier to just put a try/except around the conftest import skipping the conftest import if __init__.py skips seems more correct.

[SalmonMode]

I would recommend updating your package to use backwards compatible python 3 code so you don't have this issue.

[rmorshea]

Unfortunately that's not really an option for us at the moment. While I understand if you close this out, it seems intuitive to me that a skip in __init__.py would skip the whole directory regardless of its contents (which is what occurs so long as there is no conftest.py). This behavior of attempting to import conftest.py after __init__.py has been skipped definitely broke my original assumptions about how pytest behaved in this scenario.

[rmorshea]

I guess what I'm trying to get at is that it would be nice if, in some directories, I didn't need to worry about Python2 compatibility as that can add a lot of unnecessary overhead to writing tests especially when the tests in question are targeting features that are only designed to run under Python3.

[SalmonMode]

To be honest, EoL for Python 2 was announced over a decade ago, and we're already passed it. There's been almost 12 years worth of moments to implement backwards compatibility. That should really be a top priority at this point.

[RonnyPfannschmidt]

@rmorshea https://doc.pytest.org/en/latest/skipping.html#skipping-test-functions <- allow_module_level might help, i believe you would still need a conftest with collect_ignore in some way to make sure the files are really skipped

[rmorshea]

Thanks @RonnyPfannschmidt I'll give that a go.

[Zac-HD]

https://docs.pytest.org/en/latest/reference.html#collect-ignore-glob might also be helpful :slightly_smiling_face:

[rmorshea]

Can confirm collect_ignore and collect_ignore_glob work great. Thanks @Zac-HD and @RonnyPfannschmidt

[rmorshea]

Ok, there's still a caveate that all conftest.py files, even those below the one declaring collect_ignore_glob in the directory hierarchy, will get collected, but that's workable.