Pytest: 3.6.0 breaking test: internal error loading conftest.py

[sbouchut]

No error with 3.5.1. Since 3.6.0 I get an unexpected error.

pytest --basetemp=/builds/company/project/.tox/py27/tmp --cov=project --no-cov-on-fail --html reports/test-reports/pytest-py27.html --self-contained-html --junitxml reports/test-reports/pytest-py27.xml tests --dburl postgresql://login:password@postgres:5432/tests

Traceback (most recent call last):
  File "/builds/forcity/ewan/.tox/py27/local/lib/python2.7/site-packages/_pytest/config.py", line 376, in _importconftest
    mod = conftestpath.pyimport()
  File "/builds/forcity/ewan/.tox/py27/local/lib/python2.7/site-packages/py/_path/local.py", line 668, in pyimport
    __import__(modname)
  File "/builds/forcity/ewan/.tox/py27/local/lib/python2.7/site-packages/_pytest/assertion/rewrite.py", line 145, in find_module
    _write_pyc(state, co, source_stat, pyc)
  File "/builds/forcity/ewan/.tox/py27/local/lib/python2.7/site-packages/_pytest/assertion/rewrite.py", line 269, in _write_pyc
    marshal.dump(co, fp.file)
TypeError: marshal.dump() 2nd arg must be file
ERROR: could not load /builds/forcity/ewan/tests/conftest.py

pip freeze

amqp==1.4.9
anyjson==0.3.3
arrow==0.12.1
asn1crypto==0.24.0
atomicwrites==1.1.5
attrs==18.1.0
backports.functools-lru-cache==1.5
bcrypt==3.1.4
bottle==0.12.13
bulb==0.1.4
certifi==2018.4.16
cffi==1.11.5
chardet==3.0.4
cimkit==3.6.1
click==6.7
click-plugins==1.0.3
cligj==0.4.0
configparser==3.5.0
coverage==4.5.1
cryptography==2.2.2
cycler==0.10.0
descartes==1.1.0
enum-compat==0.0.2
enum34==1.1.6
eventlet==0.20.1
project==0.4.1rc1
Fiona==1.7.11.post2
flake8==3.5.0
flake8-html==0.4.0
company-classes==0.20.0rc0
company-modelingtoolkit==1.6.0
companyalchemy==0.10.0
funcsigs==1.0.2
functools32==3.2.3.post2
GeoAlchemy2==0.2.6
geojson==2.4.0
geopandas==0.3.0
greenlet==0.4.13
hypothesis==3.57.0
idna==2.6
invoke==1.0.0
ipaddress==1.0.22
Jinja2==2.10
jsonschema==2.6.0
kiwisolver==1.0.1
kombu==3.0.37
lumis-commons==0.0.1
MarkupSafe==1.0
matplotlib==2.2.2
mccabe==0.6.1
mock==2.0.0
more-itertools==4.2.0
munch==2.3.2
nameko==2.8.5
Nameko-Logger==1.6.1
numexpr==2.6.5
numpy==1.14.3
orca==1.4.0
pandas==0.23.0
paramiko==2.4.1
path.py==11.0.1
patsy==0.5.0
pbr==4.0.3
pluggy==0.6.0
prettytable==0.7.2
psycopg2==2.7.4
py==1.5.3
pyasn1==0.4.3
pycodestyle==2.3.1
pycparser==2.18
pyflakes==1.6.0
Pygments==2.2.0
PyNaCl==1.2.1
pyparsing==2.2.0
pyproj==1.9.5.1
pytest==3.6.0
pytest-cov==2.5.1
pytest-company-db==0.3.1
pytest-html==1.18.0
pytest-metadata==1.7.0
python-dateutil==2.7.3
pytz==2018.4
PyYAML==3.12
requests==2.18.4
retrying==1.3.3
Rtree==0.8.3
scipy==1.1.0
Shapely==1.6.4.post1
simplejson==3.15.0
six==1.11.0
SQLAlchemy==1.2.7
SQLAlchemy-Utils==0.33.3
statsmodels==0.8.0
structlog==18.1.0
subprocess32==3.5.1
tables==3.4.3
tblib==1.3.2
toolz==0.9.0
tox==3.0.0
ujson==1.35
urbansim==3.1.1
urllib3==1.22
virtualenv==16.0.0
Werkzeug==0.14.1
wrapt==1.10.11
yapf==0.22.0
zbox==1.2.0

[pytestbot]

GitMate.io thinks possibly related issues are https://github.com/pytest-dev/pytest/issues/913 (py.test exits with code 0 on errors), https://github.com/pytest-dev/pytest/issues/3181 (3.4.0 logging changes break devpi tests), https://github.com/pytest-dev/pytest/issues/60 (py.test 2.1.0 causes new errors), https://github.com/pytest-dev/pytest/issues/3498 (3.6.0 breaks usefixtures when using unittest/unittest2), and https://github.com/pytest-dev/pytest/issues/1404 (conftest.py loaded in wrong order.).

[sbouchut]

It seems this error may be related to missing __init__.py file in test directory (which was not raising error with previous version?). I will perform some additional checks and close the issue if so.

Edit: Unfortunately it was not enough to solve the issue in my case.

[RonnyPfannschmidt]

@nicoddemus this seems to be a bug in the bytecode writing of the assertion rewriter - based on the diff/blame it correlates with the switch to use atomic-files to avoid race conditions

[nicoddemus]

Hi @sbouchut, thanks for the report.

@RonnyPfannschmidt is right, this is related to #3390: we are now using atomicwrites to write rewritten .pyc files because they might be corrupted on Windows otherwise.

The snippet showing the problem is:

https://github.com/pytest-dev/pytest/blob/48215fdcb90baf9abc1f83661d9d8be3e72a4b22/_pytest/assertion/rewrite.py#L263-L271

I'm surprised fp.file somehow is not a file, given that all our py27 suite works with that code.

@sbouchut could you please run your code with pytest from nicoddemus/temp-check-atomicwrite-file? I added a temporary check and better error message to that branch.

I see you are using tox, so this would be a matter of changing this in your tox.ini:

deps = 
    pytest     

To this:

deps = 
    git+https://github.com/nicoddemus/pytest@temp-check-atomicwrite-file

Could you then post the output? Thanks!

[sbouchut]

Thank you for your help.
The new traceback below.

Traceback (most recent call last):
  File "/home/sbouchut/dev/project/.tox/py27/local/lib/python2.7/site-packages/_pytest/config.py", line 376, in _importconftest
    mod = conftestpath.pyimport()
  File "/home/sbouchut/dev/project/.tox/py27/local/lib/python2.7/site-packages/py/_path/local.py", line 668, in pyimport
    __import__(modname)
  File "/home/sbouchut/dev/project/.tox/py27/local/lib/python2.7/site-packages/_pytest/assertion/rewrite.py", line 145, in find_module
    _write_pyc(state, co, source_stat, pyc)
  File "/home/sbouchut/dev/project/.tox/py27/local/lib/python2.7/site-packages/_pytest/assertion/rewrite.py", line 270, in _write_pyc
    assert isinstance(fp.file, types.FileType), "unexpected file type: {!r} ({!r})".format(fp.file, type(fp.file))
AssertionError: unexpected file type: <open GreenPipe '<fd:10>', mode 'wb' at 0x7f5cc29c4440> (<class 'eventlet.greenio.py2.GreenPipe'>)
ERROR: could not load /home/sbouchut/dev/project/tests/conftest.py

ERROR: InvocationError for command '/home/sbouchut/dev/project/.tox/py27/bin/pytest --basetemp=/home/sbouchut/dev/project/.tox/py27/tmp --cov=project --no-cov-on-fail --html reports/test-reports/pytest-py27.html --self-contained-html --junitxml reports/test-reports/pytest-py27.xml tests --dburl=postgresql://postgres@localhost/test' (exited with code 4)

[RonnyPfannschmidt]

@sbouchut thanks, this looks like gevent/eventlet breaking the world with a monkeypatch

@nicoddemus i'm wondering - would it work to switch to marshal.dumps and manually write the data out (as it would allow to use the file object directly and avoid hitting the marshal check)

[nicoddemus]

@sbouchut thanks, now the problem is clear: like @RonnyPfannschmidt said greenlet monkeypatches things and marshal.dump doesn't know about the greenlet's file object.

@RonnyPfannschmidt: yep, that will work and be simpler even. I'm on it.

[nicoddemus]

PR is up #3509