pytest fails to run with pytest-xdist
base docker image: python:3.8.1-alpine3.11
amqp 2.5.2
apipkg 1.5
apns2 0.7.1
asgiref 3.2.5
attrs 19.3.0
autopep8 1.5
backcall 0.1.0
bandit 1.6.2
billiard 3.6.3.0
celery 4.4.1
certifi 2019.11.28
cffi 1.14.0
chardet 3.0.4
click 7.1.1
coreapi 2.3.3
coreschema 0.0.4
coverage 5.0.3
cryptography 2.8
decorator 4.4.2
Django 3.0.4
django-cors-headers 3.2.1
django-debug-toolbar 2.2
django-environ 0.4.5
django-extensions 2.2.8
django-filter 2.2.0
django-fsm 2.7.0
django-guardian 2.2.0
django-push-notifications 2.0.0
django-redis-cache 2.1.0
django-silk 4.0.1
django-solo 1.1.3
djangorestframework 3.11.0
djangorestframework-simplejwt 4.4.0
dparse 0.4.1
drf-yasg 1.17.1
entrypoints 0.3
eradicate 1.0
execnet 1.7.1
flake8 3.7.9
flake8-broken-line 0.1.1
flake8-bugbear 20.1.4
flake8-builtins 1.4.2
flake8-comprehensions 3.2.2
flake8-eradicate 0.3.0
flake8-executable 2.0.3
gevent 1.4.0
gitdb 4.0.2
GitPython 3.1.0
glob2 0.7
gprof2dot 2019.11.30
greenlet 0.4.15
gunicorn 20.0.4
h2 2.6.2
hiredis 1.0.1
hpack 3.0.0
http-ece 1.1.0
hyper 0.7.0
hyperframe 3.2.0
idna 2.9
inflection 0.3.1
ipython 7.13.0
ipython-genutils 0.2.0
itypes 1.1.0
jedi 0.16.0
Jinja2 2.11.1
kombu 4.6.8
Mako 1.1.2
MarkupSafe 1.1.1
mccabe 0.6.1
more-itertools 8.2.0
mypy 0.770
mypy-extensions 0.4.3
packaging 20.3
parse 1.15.0
parse-type 0.5.2
parso 0.6.2
pbr 5.4.4
pexpect 4.8.0
pickleshare 0.7.5
pip 20.0.2
pluggy 0.13.1
prompt-toolkit 3.0.4
psycopg2-binary 2.8.4
ptyprocess 0.6.0
py 1.8.1
py-vapid 1.7.0
pycodestyle 2.5.0
pycparser 2.20
pyflakes 2.1.1
Pygments 2.6.1
PyJWT 1.7.1
pyparsing 2.4.6
pytest 5.4.1
pytest-bdd 3.2.1
pytest-cov 2.8.1
pytest-django 3.8.0
pytest-flake8 1.0.4
pytest-forked 1.1.3
pytest-xdist 1.31.0
python-dateutil 2.8.1
pytz 2019.3
pywebpush 1.10.1
PyYAML 5.3
redis 3.4.1
requests 2.23.0
responses 0.10.12
ruamel.yaml 0.16.10
ruamel.yaml.clib 0.2.0
safety 1.8.7
sentry-sdk 0.14.2
setuptools 46.0.0
six 1.14.0
smmap 3.0.1
SQLAlchemy 1.3.15
sqlparse 0.3.1
stevedore 1.32.0
traitlets 4.3.3
typed-ast 1.4.1
typing-extensions 3.7.4.1
uritemplate 3.0.1
urllib3 1.25.8
vine 1.3.0
wcwidth 0.1.8
wheel 0.34.2
Tried to update pytest from 5.3.5 to 5.4.1
===================================================================================================================== test session starts =====================================================================================================================
platform linux -- Python 3.8.1, pytest-5.4.1, py-1.8.1, pluggy-0.13.1 -- /usr/local/bin/python
cachedir: .pytest_cache
django: settings: core.settings (from ini)
rootdir: /src, inifile: pytest.ini
plugins: cov-2.8.1, bdd-3.2.1, xdist-1.31.0, django-3.8.0, celery-4.4.1, forked-1.1.3, flake8-1.0.4
[gw0] linux Python 3.8.1 cwd: /src
[gw1] linux Python 3.8.1 cwd: /src
[gw2] linux Python 3.8.1 cwd: /src
[gw3] linux Python 3.8.1 cwd: /src
[gw4] linux Python 3.8.1 cwd: /src
[gw5] linux Python 3.8.1 cwd: /src
[gw6] linux Python 3.8.1 cwd: /src
[gw7] linux Python 3.8.1 cwd: /src
[gw1] Python 3.8.1 (default, Jan 18 2020, 02:42:17) -- [GCC 9.2.0]
[gw2] Python 3.8.1 (default, Jan 18 2020, 02:42:17) -- [GCC 9.2.0]
[gw4] Python 3.8.1 (default, Jan 18 2020, 02:42:17) -- [GCC 9.2.0]
gw0 C / gw1 ok / gw2 ok / gw3 C / gw4 ok / gw5 C / gw6 C / gw7 CINTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/usr/local/lib/python3.8/site-packages/_pytest/main.py", line 191, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
INTERNALERROR> File "/usr/local/lib/python3.8/site-packages/_pytest/main.py", line 247, in _main
INTERNALERROR> config.hook.pytest_runtestloop(session=session)
INTERNALERROR> File "/usr/local/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR> return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR> File "/usr/local/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/usr/local/lib/python3.8/site-packages/pluggy/manager.py", line 84, in
INTERNALERROR> self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR> File "/usr/local/lib/python3.8/site-packages/pluggy/callers.py", line 203, in _multicall
INTERNALERROR> gen.send(outcome)
INTERNALERROR> File "/usr/local/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR> raise ex[1].with_traceback(ex[2])
INTERNALERROR> File "/usr/local/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR> res = hook_impl.function(args)
INTERNALERROR> File "/usr/local/lib/python3.8/site-packages/xdist/dsession.py", line 112, in pytest_runtestloop
INTERNALERROR> self.loop_once()
INTERNALERROR> File "/usr/local/lib/python3.8/site-packages/xdist/dsession.py", line 135, in loop_once
INTERNALERROR> call(*kwargs)
INTERNALERROR> File "/usr/local/lib/python3.8/site-packages/xdist/dsession.py", line 272, in worker_collectreport
INTERNALERROR> self._failed_worker_collectreport(node, rep)
INTERNALERROR> File "/usr/local/lib/python3.8/site-packages/xdist/dsession.py", line 302, in _failed_worker_collectreport
INTERNALERROR> if rep.longrepr not in self._failed_collection_errors:
INTERNALERROR> TypeError: unhashable type: 'ExceptionChainRepr'
Coverage.py warning: No data was collected. (no-data-collected)
Coverage.py warning: No data was collected. (no-data-collected)
Coverage.py warning: No data was collected. (no-data-collected)
Coverage.py warning: No data was collected. (no-data-collected)
Coverage.py warning: No data was collected. (no-data-collected)
Coverage.py warning: No data was collected. (no-data-collected)
Coverage.py warning: No data was collected. (no-data-collected)
Coverage.py warning: No data was collected. (no-data-collected)
MuslimBeibytuly
All 6 comments
Can you provide a simpler reproducing example?
Zac-HD
on 16 Mar 2020
I see the same problem over at astropy/astropy#10038 in the 32-bit job that also serves as a parallel job test. I am even able to reproduce it when SSH into the CircleCI instance. The problem goes away if I downgrade to pytest<=5.3. I tried running it with pytest-faulthandler too but it didn't change anything. FYI.
pllim
on 16 Mar 2020
I got this error when the underlying issue was just an import error.
Error was shown correctly when run serially, but got the INTERNALERROR junk above when run with -n4
lusewell
on 20 Mar 2020
@pllim thanks for the pytest<=5.3 workaround. I can confirm it works (Arch Linux, Python 3.8.2).
I got the same bug in this report with pytest > 5.3 both with Python 3.7 and 3.8
marco-m
on 20 Mar 2020
@pllim it works fine to pytest<=5.3.5
MuslimBeibytuly
on 27 Mar 2020
@Zac-HD, @blueyed
I think that the bug is triggered when an exception is thrown while handling another exception, as can be seen in the examples below. It might also be related to the presence of a conftest.py file.
According to my tests and git bisect, the bug is triggered by commit b11bfa10, which is between 5.3.5 and 5.4.0. Wether the bug is in pytest or in xdist, I cannot say.
My test setup is very simple, what is important is the directory layout: invoking pytest must be done one directory above the pytest repository:
foo/
├── pytest/
├── pytest-xdist/
I then installed pytest in editable mode
> cd foo
> python3.7 -m venv .env
> source .env/bin/activate.fish
> pip install --editable ./pytest
> pip install --editable ./pytest-xdist
I could not reproduce with a small test. Then I noticed the following:
From directory foo, if I run pytest, I get an "acceptable" or unrelated error, with status code 1:
> pytest
platform darwin -- Python 3.7.7, pytest-5.4.1.dev49+g0e4a44db3b, py-1.8.1, pluggy-0.13.1
plugins: forked-1.1.3, xdist-1.31.1.dev1+g6fd5b56491
collected 0 items / 1 error
_____ ERROR collecting test session _____
pytest/src/_pytest/config/__init__.py:495: in _importconftest
return self._conftestpath2mod[key]
E KeyError: PosixPath('/Users/foo/sync/src/foo/pytest/testing/example_scripts/conftest_usageerror/conftest.py')
During handling of the above exception, another exception occurred:
...
pytest/testing/example_scripts/conftest_usageerror/conftest.py:4: in pytest_configure
raise pytest.UsageError("hello")
E _pytest.config.exceptions.UsageError: hello
= short test summary info ==
ERROR - _pytest.config.exceptions.UsageError: hello
!!!!!!!!!!! Interrupted: 1 error during collection
while if I run pytest -n auto (that is, if I enable xdist), I can reproduce the bug, with status code 3:
> pytest -n auto
platform darwin -- Python 3.7.7, pytest-5.4.1.dev49+g0e4a44db3b, py-1.8.1, pluggy-0.13.1
plugins: forked-1.1.3, xdist-1.31.1.dev1+g6fd5b56491
gw0 ok / gw1 ok / gw2 ok / gw3 ok / gw4 ok / gw5 ok / gw6 ok / gw7 okINTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/Users/foo/sync/src/foo/pytest/src/_pytest/main.py", line 191, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
...
INTERNALERROR> File "/Users/foo/sync/src/foo/pytest-xdist/src/xdist/dsession.py", line 302, in _failed_worker_collectreport
INTERNALERROR> if rep.longrepr not in self._failed_collection_errors:
INTERNALERROR> TypeError: unhashable type: 'ExceptionChainRepr'
at this point I had everything I needed for git bisect.
I wrote this script, regression.fish and put in in the root directory of pytest (this is important):
#!/usr/bin/env fish
cd .. # <== important, run pytest one directory above
pytest -n auto
echo pytest returned $status
switch $status
case 0
exit 0 # bisect good
case 1
exit 0 # same error as without xdist, bisect good
case 3
exit 1 # the bug INTERNALERROR, bisect bad
case '*'
exit 128 # what is happening here???, bisect interrupt
end
then:
> git checkout 5.3.0
> git bisect start
> git bisect good
> git checkout master
> git bisect bad
finally:
> git bisect run ./regression.fish
...
b11bfa106cb413773292a52e903e3b3ab4f62b95 is the first bad commit
commit b11bfa106cb413773292a52e903e3b3ab4f62b95
Author: Daniel Hahler <[email protected]>
Date: Tue Mar 3 21:53:28 2020 +0100
Use attrs with all Repr classes (#6739)
Co-authored-by: Ran Benita <[email protected]>
Related issues
wstrm
·
3Comments
nschloe
·
3Comments
pombredanne
·
3Comments
thakkardharmik
·
3Comments
jacquerie
·
3Comments
Most helpful comment
@Zac-HD, @blueyed
I think that the bug is triggered when an exception is thrown while handling another exception, as can be seen in the examples below. It might also be related to the presence of a
conftest.pyfile.According to my tests and git bisect, the bug is triggered by commit b11bfa10, which is between 5.3.5 and 5.4.0. Wether the bug is in pytest or in xdist, I cannot say.
My test setup is very simple, what is important is the directory layout: invoking pytest must be done one directory above the pytest repository:
I then installed pytest in editable mode
I could not reproduce with a small test. Then I noticed the following:
From directory foo, if I run
pytest, I get an "acceptable" or unrelated error, with status code 1:while if I run
pytest -n auto(that is, if I enable xdist), I can reproduce the bug, with status code 3:at this point I had everything I needed for git bisect.
I wrote this script,
regression.fishand put in in the root directory of pytest (this is important):then:
finally: