Pytest: doctest plugin runs when unwanted
This file fails when run explicitly with pytest:
$ pytest --doctest-glob *.nope README.rst
...
platform darwin -- Python 3.7.1, pytest-4.3.1, py-1.8.0, pluggy-0.9.0
...
...______ [doctest] README.rst __
093 'banner': Required(str),
...
102 >>> print(open('/etc/my_app/config.yaml').read())
UNEXPECTED EXCEPTION: FileNotFoundError(2, 'No such file or directory')
Things work fine if I just pytest in the root directory. The problem is that the doctest plugin is kicking in when it has not been requested, and indeed even when I try and explicitly stop it from working on this file. I use sybil for testing examples in my documentation as it provides for more flexibility.
Here's my full environment:
$ pip freeze
alabaster==0.7.12
atomicwrites==1.3.0
attrs==19.1.0
Babel==2.6.0
bleach==3.1.0
certifi==2019.3.9
chardet==3.0.4
-e [email protected]:Simplistix/configurator.git@1090b4db182490e523ca9b7415fd87ba784e25ae#egg=configurator
coverage==4.5.3
docutils==0.14
idna==2.8
imagesize==1.1.0
Jinja2==2.10
MarkupSafe==1.1.1
mock==2.0.0
more-itertools==6.0.0
packaging==19.0
pbr==5.1.3
pkginfo==1.5.0.1
pluggy==0.9.0
py==1.8.0
pyfakefs==3.5.8
Pygments==2.3.1
pyparsing==2.3.1
pytest==4.3.1
pytest-cov==2.6.1
pytz==2018.9
PyYAML==5.1
readme-renderer==24.0
requests==2.21.0
requests-toolbelt==0.9.1
setuptools-git==1.2
six==1.12.0
snowballstemmer==1.2.1
Sphinx==1.8.5
sphinxcontrib-websupport==1.1.0
sybil==1.0.9
testfixtures==6.6.1
toml==0.10.0
tqdm==4.31.1
twine==1.13.0
urllib3==1.24.1
voluptuous==0.11.5
webencodings==0.5.1
cjw296
All 17 comments
pytest --doctest-glob *.nope README.rst
Haven't investigate, but make sure that *.nope does not get expanded by your shell already, i.e. quote it.
blueyed
on 20 Mar 2019
Related code: https://github.com/blueyed/pytest/blob/033849a31ec7d2b9f0493fa51c77289eab657af1/src/_pytest/doctest.py#L104-L107.
Added in acd286f8 initially.
The glob pattern is ignored for txt and rst files given on the command line explicitly.
Why do you specify README.rst explicitly?
blueyed
on 20 Mar 2019
I was very sure that *.nope did not match any files ;-)
"The glob pattern is ignored for txt and rst files given on the command line explicitly." feels like a bug to me. Why specify README.rst explicitly? Same reason people normally specify a subset of their tests, I just wanted to run those tests, not the whole unit test suite.
cjw296
on 20 Mar 2019
Yeah.. but if you specify "README.rst" it runs the (doc)tests from it.
It is like withtestpaths but specifying a file explicitly.
blueyed
on 20 Mar 2019
It evidently does, but why should it? it doesn't when I don't otherwise limit discovery.
If I do, pytest some/test_file.py, it doesn't suddenly decide to ignore my plugin configuration and do something I don't want ;-)
To be clear: this is far from a show stopper, but it is a bit of a wart and it would be great to find a way to fix it!
cjw296
on 20 Mar 2019
If I do,
pytest some/test_file.py, it doesn't suddenly decide to ignore my plugin configuration and do something I don't want ;-)
But pytest t-foo.py ignores testpaths also.
blueyed
on 20 Mar 2019
If you pass a file explicitly on the command-line, pytest will run that file as a test, regardless of test file pattern configuration or testpaths. For example if you have tests on foo.py, running just pytest . won't pick up foo.py, but pytest foo.py will. The same happens for doctests as @blueyed pointed out in the code.
I think the rationale is that if you are explicitly passing the file in the command-line, you are telling pytest to run tests from that file.
Not sure what were the requirements that lead to this, but I don't think we should change this without a good reason because this will certainly break some workflows out there.
nicoddemus
on 27 Mar 2019
[fixed this comment up as I realised I was misreading some output]
Sorry, but this doesn't feel right. -vv output gives some more detail: it looks like the file is being processed twice, once with the stuff I've configured in conftest.py and then additionally, erroneously, as a doctest:
$ pytest -vvx README.rst
========================================================================= test session starts =========================================================================
platform darwin -- Python 3.7.1, pytest-4.3.1, py-1.8.0, pluggy-0.9.0 -- /Users/chris/virtualenvs/configurator_37/bin/python3.7
cachedir: .pytest_cache
rootdir: /Users/chris/vcs/git/configurator, inifile:
plugins: cov-2.6.1, pyfakefs-3.5.8
collected 14 items
README.rst::line:26,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED [ 7%]
README.rst::line:45,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED [ 14%]
README.rst::line:66,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED [ 21%]
README.rst::line:92,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED [ 28%]
README.rst::line:107,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED [ 35%]
README.rst::line:108,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED [ 42%]
README.rst::line:112,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED [ 50%]
README.rst::line:114,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED [ 57%]
README.rst::line:116,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED [ 64%]
README.rst::line:122,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED [ 71%]
README.rst::line:124,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED [ 78%]
README.rst::line:126,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED [ 85%]
README.rst::line:128,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED [ 92%]
README.rst::README.rst <- /Users/chris/vcs/git/configurator/README.rst FAILED [100%]
Now, if I specify an item within the file, things work as expected:
$ pytest -vvx README.rst::line:26,column:1
========================================================================= test session starts =========================================================================
platform darwin -- Python 3.7.1, pytest-4.3.1, py-1.8.0, pluggy-0.9.0 -- /Users/chris/virtualenvs/configurator_37/bin/python3.7
cachedir: .pytest_cache
rootdir: /Users/chris/vcs/git/configurator, inifile:
plugins: cov-2.6.1, pyfakefs-3.5.8
collected 1 item
README.rst::line:26,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED [100%]
====================================================================== 1 passed in 0.08 seconds =======================================================================
The problem is that these tests shared state within a module, they're testing the examples within documentation using Sybil, so I really want to be able to run a single file without the annoying and erroneous doctest stuff kicking in.
How do I do that?
cjw296
on 29 Mar 2019
once with the stuff I've configured in conftest.py
What is that?
Do you mean setup.cfg/pytest.ini maybe instead?
I suggest digging into the code maybe yourself (I've left a pointer already).
(btw: with 4.4 it would also display "testpaths: …" in the header, I wonder if that covers this case also?)
blueyed
on 29 Mar 2019
No, I mean conftest.py.
I'm afraid I can't follow how displaying "testpaths: …" in the header is related to this.
cjw296
on 29 Mar 2019
I'm afraid I can't follow how displaying "testpaths: …" in the header is related to this.
It is not really, but I just wondered if files for doctests would show up there (like they should).
blueyed
on 29 Mar 2019
Just to be clear: I'm not using doctests and I don't want to be using doctests. I want pytest to stop erroneously treating my non-doctest .rst files as doctests, which it only does if I specify a particular file on the command line.
My currently workaround is to do:
pytest -k README.rst
That works fine, so there's no massive priority on fixing this, but it feels like you're trying to tell me it's not a bug, in which case I will respectfully disagree ;-)
cjw296
on 29 Mar 2019
Try -p no:doctest then maybe (might require pytest 4.4 / current features branch though).
blueyed
on 29 Mar 2019
I already have a workaround thanks...
cjw296
on 29 Mar 2019
And I agree that the collection logic in the doctest plugin is a bit wonky.
As for your workaround: it will collect all files, and then filter based on a pattern, which is slower than collecting just one file.
blueyed
on 29 Mar 2019
Just to note that this is still an issue in pytest 6, but that putting addopts = -p no:doctest in the config file does stop the pytest doctest plugin from getting in the way.
cjw296
on 2 Dec 2020
Thanks @cjw296,
Not sure how we could change the defaults so they don't break existing workflows though. Any ideas? If so I suggest we open a new "proposal" with a plan to deprecate automatically running doctests in .rst and .txt files, but in a way that won't get users by surprise (for example, someone upgrades pytest to a version which no longer runs those files as doctests, and doesn't notice).
nicoddemus
on 4 Dec 2020
Related issues
RonnyPfannschmidt
·
3Comments
jurisbu
·
3Comments
nschloe
·
3Comments
jtmoon79
·
3Comments
TomNicholas
·
3Comments
Most helpful comment
Just to be clear: I'm not using doctests and I don't want to be using doctests. I want pytest to stop erroneously treating my non-doctest .rst files as doctests, which it only does if I specify a particular file on the command line.
My currently workaround is to do:
That works fine, so there's no massive priority on fixing this, but it feels like you're trying to tell me it's not a bug, in which case I will respectfully disagree ;-)