Pytest: Collection warning emitted for modules which should not being collected

[BeyondEvil]

When setting the testpaths key under the [pytest] section in tox.ini, pytest seems to ignore it.

See the comment thread in this PR for more information.

Result of pip list

appdirs              1.4.3                           
Appium-Python-Client 0.44                            
aspy.yaml            1.3.0                           
atomicwrites         1.3.0                           
attrs                19.1.0                          
black                19.3b0                          
certifi              2019.3.9                        
cfgv                 2.0.0                           
chardet              3.0.4                           
Click                7.0                             
entrypoints          0.3                             
filelock             3.0.12                          
flake8               3.7.7                           
identify             1.4.3                           
idna                 2.8                             
importlib-metadata   0.15                            
mccabe               0.6.1                           
more-itertools       7.0.0                           
nodeenv              1.3.3                           
pip                  19.1.1                          
pluggy               0.12.0                          
pre-commit           1.16.1                          
py                   1.8.0                           
pycodestyle          2.5.0                           
pyflakes             2.1.1                           
pytest               4.5.0                           
pytest-base-url      1.4.1                           
pytest-html          1.20.0                          
pytest-metadata      1.8.0                           
pytest-selenium      1.9.2.dev112+g77fed97.d20190529 /Users/jimbrannlund/dev/pytest-dev/pytest-selenium
pytest-variables     1.7.1                           
PyYAML               5.1                             
requests             2.22.0                          
selenium             3.141.0                         
setuptools           41.0.1                          
six                  1.12.0                          
toml                 0.10.0                          
tox                  3.12.1                          
urllib3              1.25.3                          
virtualenv           16.6.0                          
wcwidth              0.1.7                           
wheel                0.33.4                          
zipp                 0.5.1

python: 3.7.3
macOSX: 10.14.4

I see the same outcome if I run the same command as defined in tox.ini (pytest -n auto -v -r a) w/o tox.

pytest header seems to pickup the value correctly at least:

rootdir: /Users/<user>/dev/pytest-dev/pytest-selenium, inifile: tox.ini, testpaths: testing

@nicoddemus

[nicoddemus]

Thanks @BeyondEvil, I will try to take a look when I have some time. 👍

[blueyed]

pytest header seems to pickup the value correctly at least:

Make sure that you are not passing in args via tox' command then.

[BeyondEvil]

pytest header seems to pickup the value correctly at least:

Make sure that you are not passing in args via tox' command then.

Nope, running tox "pure": tox

@blueyed

[blueyed]

@BeyondEvil
And what is the command it uses?

Looked at https://github.com/pytest-dev/pytest-selenium/pull/220#issuecomment-497017048 - the issue there appears to be that it emits a warning for TestFoo, and not that it does not collect in testpaths only?!

For testpaths check --collect-only.

The warning being emitted might be another/different issue.

[blueyed]

Anyway, I think @nicoddemus will get behind it.. :)

[BeyondEvil]

@blueyed

And what is the command it uses?

I see the same outcome if I run the same command as defined in tox.ini (pytest -n auto -v -r a) w/o tox.

Looked at pytest-dev/pytest-selenium#220 (comment) - the issue there appears to be that it emits a warning for TestFoo, and not that it does not collect in testpaths only?!

Well, those should be mutually exclusive - since Testingbot (link) lives under pytest_selenium > drivers > testingbot.py and not under testing

😊

[blueyed]

@BeyondEvil
Yeah, it looks "just" like the warning is emitted also when not really collecting it, but e.g. maybe when the class is used etc.

[blueyed]

(I've based my comments on the issue title initially ("pytest ignores testpaths in tox context"))

[BeyondEvil]

@blueyed

Yeah, it looks "just" like the warning is emitted also when not really collecting it, but e.g. maybe when the class is used etc.

Yeah, could be.

(I've based my comments on the issue title initially ("pytest ignores testpaths in tox context"))

Yeah, I could've been more accurate in the title. :(

[nicoddemus]

Yeah, I could've been more accurate in the title. :(

Updated the title. 😁

I managed to reproduce this without using tox.

[nicoddemus]

After digging around a bit I found the issue: the problem is that test_testingbot.py imports TestingBot at the top-level, that's why pytest is trying to collect that class as a test-class.

This patches fixes makes the warning go away:

diff --git a/testing/test_testingbot.py b/testing/test_testingbot.py
index 8349993..acfc07a 100644
--- a/testing/test_testingbot.py
+++ b/testing/test_testingbot.py
@@ -6,7 +6,7 @@ import os
 import pytest

 from functools import partial
-from pytest_selenium.drivers.testingbot import TestingBot, HOST, PORT
+from pytest_selenium.drivers.testingbot import HOST, PORT

 pytestmark = [pytest.mark.skip_selenium, pytest.mark.nondestructive]

@@ -95,5 +95,6 @@ def test_invalid_host(failure, monkeypatch, tmpdir):
     ("protocol", "host", "port"), [("http", "localhost", "4445"), ("https", HOST, PORT)]
 )
 def test_executor_url(protocol, host, port):
+    from pytest_selenium.drivers.testingbot import TestingBot
     tb = TestingBot(host, port)
     assert tb.executor == "{}://{}:{}/wd/hub".format(protocol, host, port)

I'm closing this now then. 😁

[BeyondEvil]

@nicoddemus

Thanks! Awesome, I'll apply the patch.

However, is the described behavior (pytest trying to collect an import as a test class) by design? That seems odd. Do you happen to know why that is the expected/wanted behavior?

Thanks again!

[nicoddemus]

@BeyondEvil that's how it works today, pytest will call dir() on the imported module, and look into the objects returned for test classes/functions. It doesn't do any special treatment to see if the objects were declared in that module or were imported. It is surprising in some cases, but is very useful in others: you can import tests from other modules which test other symbols in the importing module.

I opened #5346 to improve the message and make the problem more obvious in the first place. 👍

[BeyondEvil]

Great, thanks @nicoddemus for taking the time to explain it properly. 🙏

[nicoddemus]

Sure thing, my pleasure. 👍