Pytest: change to config/findpaths.py -> determine_setup in 6.1.0 changes the behavior of pytest find root path

[zzzeek]

the bug starts with the change made at https://github.com/pytest-dev/pytest/commit/70f3ad1c1f31b35d4004f92734b4afd6c8fbdecf .

When running on a Jenkins CI that creates a little bit of a broken directory structure, there's a setup.cfg file in both the working checkout where tests are run from , as well as another one that is one directory upwards. this fools the logic at https://github.com/pytest-dev/pytest/blob/master/src/_pytest/config/findpaths.py#L85 into determining the wrong path as the root path.

to compound the issue, when using pytest-xdist, the logic seems to behave inconsistently, leading some workers to see a different root path, which then causes collection to fail as the test ids are different.

the change in behavior can be illustrated using the following directory layout:

[classic@photon3 foo]$ find .
.
./myproject
./myproject/tests
./myproject/tests/test_foo.py
./myproject/tests/conftest.py
./myproject/setup.cfg
./setup.cfg

[classic@photon3 foo]$ cat myproject/tests/test_foo.py 
def test_foo():
    print("hi")

[classic@photon3 foo]$ cat myproject/tests/conftest.py 
import inspect

def pytest_pycollect_makeitem(collector, name, obj):
    if inspect.isfunction(obj):
        print(collector.config.rootdir)
    return None

[classic@photon3 foo]$ cat setup.cfg 
[tool:pytest]
addopts= --tb native -v -r fxX -p no:warnings -p no:logging --maxfail=25
python_files=tests/test_*.py


[classic@photon3 foo]$ cat myproject/setup.cfg 
[tool:pytest]
addopts= --tb native -v -r fxX -p no:warnings -p no:logging --maxfail=25
python_files=tests/test_*.py

to sum up - the same setup.cfg file in both foo/ and foo/myproject, then the tests in foo/myproject/tests.

now the working directory, where I will normally set --rootdir, which is obviously going to be my workaround here, is foo/myproject. Let's cd there:

[classic@photon3 foo]$ cd myproject/

then run the tests giving the directory (yes, perfect storm of inputs , otherwise locate_config still gets the right answer):

[classic@photon3 myproject]$ pytest -v -s  tests/ 
=========================================================== test session starts ===========================================================
platform linux -- Python 3.8.3, pytest-6.1.0, py-1.9.0, pluggy-0.13.0 -- /home/classic/.venv3/bin/python
cachedir: .pytest_cache
rootdir: /home/classic/tmp/foo, configfile: setup.cfg
plugins: forked-1.1.3, xdist-2.1.0, cov-2.8.1
collecting ... /home/classic/tmp/foo
collected 1 item                                                                                                                          

tests/test_foo.py::test_foo hi
PASSED

============================================================ 1 passed in 0.03s ============================================================
[classic@photon3 myproject]$ 

I just noticed it actually gives us the rootdir anyway, I didn't even have to print it. OK so you see above it's /home/classic/tmp/foo. with pytest 6.0.2, it's one directory inwards which is what it's been for years:

[classic@photon3 myproject]$ pytest -v -s  tests/ 
=========================================================== test session starts ===========================================================
platform linux -- Python 3.8.3, pytest-6.0.2, py-1.9.0, pluggy-0.13.0 -- /home/classic/.venv3/bin/python
cachedir: .pytest_cache
rootdir: /home/classic/tmp/foo/myproject, configfile: setup.cfg
plugins: forked-1.1.3, xdist-2.1.0, cov-2.8.1
collecting ... /home/classic/tmp/foo/myproject
collected 1 item                                                                                                                          

tests/test_foo.py::test_foo hi
PASSED

============================================================ 1 passed in 0.02s ===========================================================

So I can totally get that you're going to say, you have three things confusing it, just set rootdir. Clearly that's what I'm going to do. but this is a behavioral change and most critically it seems to be behaving randomly when I am using pytest-xdist, and in my logs for those runs, it's reporting the correct rootdir but then it does not work the same way in each worker node. An excerpt of this run is below showing the correct rootdir at the top and then inconsistent rootdirs in the workers:

============================= test session starts ==============================
platform linux -- Python 3.7.5, pytest-6.1.0, py-1.9.0, pluggy-0.13.1 -- /home/jenkins/workspace/alembic_gerrit/6a4aefb0/.tox/py37-pyoptimize-sqla13-sqlite-postgresql-mysql-oracle/bin/python
cachedir: .tox/py37-pyoptimize-sqla13-sqlite-postgresql-mysql-oracle/.pytest_cache
rootdir: /home/jenkins/workspace/alembic_gerrit/6a4aefb0, configfile: setup.cfg
plugins: forked-1.3.0, xdist-2.1.0
gw0 I / gw1 I / gw2 I / gw3 I

[gw0] linux Python 3.7.5 cwd: /home/jenkins/workspace/alembic_gerrit/6a4aefb0

[gw1] linux Python 3.7.5 cwd: /home/jenkins/workspace/alembic_gerrit/6a4aefb0

[gw2] linux Python 3.7.5 cwd: /home/jenkins/workspace/alembic_gerrit/6a4aefb0

[gw3] linux Python 3.7.5 cwd: /home/jenkins/workspace/alembic_gerrit/6a4aefb0

[gw0] Python 3.7.5 (default, May 25 2020, 17:49:41)  -- [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]

[gw1] Python 3.7.5 (default, May 25 2020, 17:49:41)  -- [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]

[gw2] Python 3.7.5 (default, May 25 2020, 17:49:41)  -- [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]

[gw3] Python 3.7.5 (default, May 25 2020, 17:49:41)  -- [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
gw0 [49] / gw1 [49] / gw2 [49] / gw3 [49]

scheduling tests via LoadScheduling

==================================== ERRORS ====================================
_____________________________ ERROR collecting gw1 _____________________________
Different tests were collected between gw0 and gw1. The difference is:
--- gw0

+++ gw1

@@ -1,49 +1,49 @@

-6a4aefb0/tests/test_script_consumption.py::ApplyVersionsFunctionalTest::test_steps
-6a4aefb0/tests/test_script_consumption.py::CallbackEnvironmentTest::test_steps
-6a4aefb0/tests/test_script_consumption.py::EncodingTest::test_encode

# ... lots of gw0s

#... then the gw1s come in with *different* root dir:

+tests/test_script_consumption.py::ApplyVersionsFunctionalTest::test_steps
+tests/test_script_consumption.py::CallbackEnvironmentTest::test_steps
+tests/test_script_consumption.py::EncodingTest::test_encode

# ... etc

Workaround is simple, I just set:

BASECOMMAND=python -m pytest --rootdir {toxinidir}

in my tox.ini

[andialbrecht]

I can confirm this issue. In my case I have a sub-project within a project, the directory structure is roughly

top-project/
  - conftest.py
  - pytest.ini
  - some-other-folder/
     - sub-project/
       - conftest.py
       - pytest.ini

When running tests in sub-project with pytest<6.1 pytest picks up conftest.py and pytest.ini from the sub-project. With pytest==6.1 it picks up both files from the top-level project. Adding --rootdir as mentioned above didn't help. I've also had to add --confcutdir. Then pytest picks up conftest.py from sub-project. But even then it picks pytest.ini from the top-level project.

[forzagreen]

I have the same issue with version 6.1.0

Workaround: user parameters --rootdir and -c

pytest --rootdir=path/to/dir -c path/to/pytest.ini

CLI docs here:

  -c file               load configuration from `file` instead of trying to
                        locate one of the implicit configuration files.

  --rootdir=ROOTDIR     Define root directory for tests. Can be relative
                        path: 'root_dir', './root_dir',
                        'root_dir/another_dir/'; absolute path:
                        '/home/user/root_dir'; path with variables:
                        '$HOME/root_dir'.

[bluetech]

Thanks for the report. I am able to reproduce, looking into it.

[bluetech]

Fix is in #7813, will be backported to 6.1.1, which we'll try to release soon.

[cjw296]

Pretty sure I've been hit by this too: https://app.circleci.com/pipelines/github/cjw296/sybil/82/workflows/471524c7-b5b9-4e1f-b3f4-2b7e0ac0f4e2/jobs/460