Coveragepy: coverage regression with Python 3.7 ?
I posted a question on Stackoverflow because the time for my tests running with coverage and Python 3.7 were about 8 times longer than with Python 3.5 and 3.6. Hoefling suggested I run the tests without coverage and times for testing were about the same for the different versions. Hoefling suggested that it might be a coverage regression and that I post an issue.
0LL13
All 30 comments
@0LL13 Can you verify for me the versions of coverage you have running in each Python version? "coverage --version" should do it. In particular, that will tell me whether you are using the C tracer (fast) or the Python tracer (slow).
nedbat
on 12 Jul 2018
(vEnv)$ coverage --version
Coverage.py, version 4.0.3.
0LL13
on 13 Jul 2018
@0LL13 Can you upgrade to the latest version? 4.0.3 is over 2.5 years old. The latest version is 4.5.1. Once you've done that, if there is still a difference in performance, show the coverage version for each Python environment. Thanks.
nedbat
on 13 Jul 2018
(vEnv)$ pip install -U coverage
(vEnv)$ coverage --version
Coverage.py, version 4.5.1 with C extension
#tox.ini
...
[testenv]
basepython =
py35 = python3.5
py36 = python3.6
py37 = python3.7
deps =
pytest
pytest-cov
commands =
py.test --cov=pdf2textbox --duration=0 tests
(vEnv)$ tox
GLOB sdist-make: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/setup.py
py35 recreate: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/.tox/py35
py35 installdeps: check-manifest, pytest, requests, pdfminer2, coverage, pytest-cov
py35 inst: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/.tox/dist/pdf2textbox-0.3.1.zip
py35 installed: atomicwrites==1.1.5,attrs==18.1.0,certifi==2018.4.16,chardet==3.0.4,check-manifest==0.37,coverage==4.5.1,idna==2.7,more-itertools==4.2.0,pdf2textbox==0.3.1,pdfminer2==20151206,pluggy==0.6.0,py==1.5.4,pytest==3.6.3,pytest-cov==2.5.1,requests==2.19.1,six==1.11.0,urllib3==1.23
py35 runtests: PYTHONHASHSEED='2082557123'
py35 runtests: commands[0] | check-manifest
lists of files in version control and sdist match
py35 runtests: commands[1] | python --version
Python 3.5.5
py35 runtests: commands[2] | py.test --cov=pdf2textbox --duration=0 tests
================================= test session starts =================================
platform linux -- Python 3.5.5, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox, inifile:
plugins: cov-2.5.1
collected 15 items
tests/test_cli_slice_option.py ... [ 20%]
tests/test_convert_pdf.py ...... [ 60%]
tests/test_get_pdf.py .... [ 86%]
tests/test_types.py .. [100%]
----------- coverage: platform linux, python 3.5.5-final-0 -----------
Name Stmts Miss Cover
pdf2textbox/pdf2textbox.py 506 201 60%
=============================== slowest test durations ================================
54.32s call tests/test_cli_slice_option.py::test_pdf_to_text_slice
1.26s call tests/test_types.py::test_textbox_structure
1.25s call tests/test_types.py::test_pdf_to_text_all
============================ 15 passed in 59.82 seconds ==============================
py36 recreate: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/.tox/py36
py36 installdeps: check-manifest, pytest, requests, pdfminer2, coverage, pytest-cov
py36 inst: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/.tox/dist/pdf2textbox-0.3.1.zip
py36 installed: atomicwrites==1.1.5,attrs==18.1.0,certifi==2018.4.16,chardet==3.0.4,check-manifest==0.37,coverage==4.5.1,idna==2.7,more-itertools==4.2.0,pdf2textbox==0.3.1,pdfminer2==20151206,pluggy==0.6.0,py==1.5.4,pytest==3.6.3,pytest-cov==2.5.1,requests==2.19.1,six==1.11.0,urllib3==1.23
py36 runtests: PYTHONHASHSEED='2082557123'
py36 runtests: commands[0] | check-manifest
lists of files in version control and sdist match
py36 runtests: commands[1] | python --version
Python 3.6.3
py36 runtests: commands[2] | py.test --cov=pdf2textbox --duration=0 tests
================================= test session starts =================================
platform linux -- Python 3.6.3, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox, inifile:
plugins: cov-2.5.1
collected 15 items
tests/test_cli_slice_option.py ... [ 20%]
tests/test_convert_pdf.py ...... [ 60%]
tests/test_get_pdf.py .... [ 86%]
tests/test_types.py .. [100%]
----------- coverage: platform linux, python 3.6.3-final-0 -----------
Name Stmts Miss Cover
pdf2textbox/pdf2textbox.py 506 201 60%
=============================== slowest test durations ================================
53.42s call tests/test_cli_slice_option.py::test_pdf_to_text_slice
1.23s call tests/test_types.py::test_pdf_to_text_all
1.22s call tests/test_types.py::test_textbox_structure
0.46s call tests/test_convert_pdf.py::test_convert_three_cols_and_header
============================ 15 passed in 58.94 seconds ==============================
py37 recreate: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/.tox/py37
py37 installdeps: check-manifest, pytest, requests, pdfminer2, coverage, pytest-cov
py37 inst: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/.tox/dist/pdf2textbox-0.3.1.zip
py37 installed: atomicwrites==1.1.5,attrs==18.1.0,certifi==2018.4.16,chardet==3.0.4,check-manifest==0.37,coverage==4.5.1,idna==2.7,more-itertools==4.2.0,pdf2textbox==0.3.1,pdfminer2==20151206,pluggy==0.6.0,py==1.5.4,pytest==3.6.3,pytest-cov==2.5.1,requests==2.19.1,six==1.11.0,urllib3==1.23
py37 runtests: PYTHONHASHSEED='2082557123'
py37 runtests: commands[0] | check-manifest
lists of files in version control and sdist match
py37 runtests: commands[1] | python --version
Python 3.7.0
py37 runtests: commands[2] | py.test --cov=pdf2textbox --duration=0 tests
================================= test session starts =================================
platform linux -- Python 3.7.0, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox, inifile:
plugins: cov-2.5.1
collected 15 items
tests/test_cli_slice_option.py ... [ 20%]
tests/test_convert_pdf.py ...... [ 60%]
tests/test_get_pdf.py .... [ 86%]
tests/test_types.py .. [100%]
----------- coverage: platform linux, python 3.7.0-final-0 -----------
Name Stmts Miss Cover
pdf2textbox/pdf2textbox.py 506 201 60%
=============================== slowest test durations ================================
426.08s call tests/test_cli_slice_option.py::test_pdf_to_text_slice
9.55s call tests/test_types.py::test_pdf_to_text_all
9.50s call tests/test_types.py::test_textbox_structure
3.48s call tests/test_convert_pdf.py::test_convert_three_cols_and_header
3.42s call tests/test_convert_pdf.py::test_convert_three_cols
3.15s call tests/test_convert_pdf.py::test_convert_two_cols_and_header
3.09s call tests/test_convert_pdf.py::test_convert_two_cols
2.81s call tests/test_convert_pdf.py::test_convert_text_only
2.74s call tests/test_convert_pdf.py::test_convert_one_col_and_header
0.38s call tests/test_get_pdf.py::test_get_pdf_file_url
...
============================ 15 passed in 465.17 seconds =============================
__________________________________ summary _______________________________________
py35: commands succeeded
py36: commands succeeded
py37: commands succeeded
congratulations :)
Thanks. Now can you add "coverage --version" to the tox.ini [commands] section, so that we can see the coverage version for each of the different Python versions?
nedbat
on 14 Jul 2018
I did that and now as command[2] there is:
py36 runtests: commands[2] | coverage --version
Coverage.py, version 4.5.1 with C extension
Documentation at https://coverage.readthedocs.io
Otherwise no change.
0LL13
on 14 Jul 2018
@0LL13 I don't know how else to say this: I need to see the version of coverage for each Python version. I suspect that Python 3.7 is running with the Python extension, not the C extension.
nedbat
on 14 Jul 2018
Sorry for the delay. You were absolutely right:
ppy35 runtests: commands[0] | check-manifest
lists of files in version control and sdist match
py35 runtests: commands[1] | python --version
Python 3.5.5
py35 runtests: commands[2] | coverage --version
Coverage.py, version 4.5.1 with C extension
Documentation at https://coverage.readthedocs.io
py35 runtests: commands[3] | py.test --cov=pdf2textbox --duration=0 tests
py36 runtests: commands[0] | check-manifest
lists of files in version control and sdist match
py36 runtests: commands[1] | python --version
Python 3.6.3
py36 runtests: commands[2] | coverage --version
Coverage.py, version 4.5.1 with C extension
Documentation at https://coverage.readthedocs.io
py36 runtests: commands[3] | py.test --cov=pdf2textbox --duration=0 tests
py37 runtests: commands[0] | check-manifest
lists of files in version control and sdist match
py37 runtests: commands[1] | python --version
Python 3.7.0
py37 runtests: commands[2] | coverage --version
Coverage.py, version 4.5.1 without C extension
Documentation at https://coverage.readthedocs.io
py37 runtests: commands[3] | py.test --cov=pdf2textbox --duration=0 tests
Otherwise the same output as above.
I will be offline for a couple of days.
0LL13
on 16 Jul 2018
Thanks. 4.5.1 doesn't have a wheel for 3.7, so it tries to install from source. Perhaps you don't have the needed python-dev packages installed? http://coverage.readthedocs.io/en/coverage-4.5.1a/install.html#c-extension
nedbat
on 16 Jul 2018
frodo@theshire:~$ sudo apt-get install python3-dev gcc
[sudo] Passwort für frodo:
Paketlisten werden gelesen... Fertig
Abhängigkeitsbaum wird aufgebaut.
Statusinformationen werden eingelesen.... Fertig
gcc ist schon die neueste Version (4:7.3.0-3ubuntu2).
gcc wurde als manuell installiert festgelegt.
python3-dev ist schon die neueste Version (3.6.5-3ubuntu1).
0 aktualisiert, 0 neu installiert, 0 zu entfernen und 6 nicht aktualisiert.
(vEnv)frodo@theshire:pdf2textbox$ python -m coverage --version
Coverage.py, version 4.5.1 with C extension
Coverage is only within the virtualenv for my project.
0LL13
on 19 Jul 2018
Confirming that pip install coverage comes without C extension under Python 3.7 which I have installed using pyenv. As a result, my pytest --cov took ~20 mins instead of usual ~5mins.
Solution for my case:
pip uninstall coverage
pip install --global-option=build_ext --global-option="$(python-config --includes)" coverage
vst
on 19 Jul 2018
OK, I've uploaded wheels for v4.5.1 for Python 3.7 on mac and manylinux to pypi. Let me know if that solves the problem. Thanks for all the details.
nedbat
on 19 Jul 2018
I believe this change has broken folks using pipenv and a Pipfile/Pipfile.lock as the hash for 4.5.1 has now changed.
```± |master ✓| → pipenv install --dev
Installing dependencies from Pipfile.lock (a03fec)...
An error occurred while installing coverage==4.5.1! Will try again.
🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 25/25 — 00:00:01
Installing initially failed dependencies...
Looking in indexes: https://pypi.python.org/simple00
Collecting coverage==4.5.1
Using cached https://files.pythonhosted.org/packages/c7/d0/337673c08f5b0cc7ada3dfe2a998ae8a97d482722045644be3d79bbcbe05/coverage-4.5.1-cp37-cp37m-macosx_10_13_x86_64.whl
THESE PACKAGES DO NOT MATCH THE HASHES FROM Pipfile.lock!. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.```
cardoe
on 20 Jul 2018
(vEnv)$ pip uninstall coverage
(vEnv)$ pip install --no-cache-dir coverage
Collecting coverage
Downloading https://files.pythonhosted.org/packages/3d/a0/b12090c40e0b8196b973962ec71c1c541a6c04af58ba5ad85683b3de251a/coverage-4.5.1-cp36-cp36m-manylinux1_x86_64.whl (202kB)
100% |████████████████████████████████| 204kB 6.9MB/s
Installing collected packages: coverage
Successfully installed coverage-4.5.1
No change in py.test --cov time for Python 3.7
0LL13
on 20 Jul 2018
@cardoe I don't understand how uploading a new file that never existed before could break an installation. Can you provide more details? What was the old hash computed from, and what is the new hash computed from?
@0LL13 Looks like that wheel you installed was for Python 3.6. Can you double-check that you are installing for 3.7?
nedbat
on 20 Jul 2018
@nedbat It came from running pipenv install --dev coverage a few months back sometime after 4.5.1 was released. pipenv automatically generates a Pipfile.lock which contains all the hashes of that version on pypi. According to pypa folks its a feature. https://docs.pipenv.org/basics/#pipfile-lock-security-features From the rough looks of it they're just using pip hash which appears to have changed for a pip cache containing coverage from about 10 hours ago to a cache that was recently created. Ultimately our tests started failing because pipenv reported that coverage may have been tampered with. I came to the GitHub project and searched and landed on this issue. Our last successful test was 42 minutes before your comment about new wheels being uploaded. Our first failure was 23 minutes after that comment. I very infrequently write Python now days so I'm just trying to go with the tools I've got. 🤷♂️
cardoe
on 20 Jul 2018
@cardoe I guess I understand what is happening. I'm not sure what I can do about it. Uploading new wheels for new Python versions seems like a net positive to me. Perhaps you should re-create your Pipfile.lock file? I'm not sure what the pipenv authors would recommend.
nedbat
on 20 Jul 2018
@nedbat It seems that wheel works fine now for py37. Thanks a lot!
coverage comes with C extension via tox --recreate, too, as expected:
python runtests: commands[0] | coverage --version
Coverage.py, version 4.5.1 with C extension
Documentation at https://coverage.readthedocs.io
python runtests: commands[1] | pytest --cov
...
As for pipenv: I am not using it right now but I guess that the only solution is to recreate Pipfile.lock. Yet, some friends may need to make new releases for packaged applications if they are shipping Pipfile.lock (such as Docker etc...).
vst
on 20 Jul 2018
I asked about pipenv in the pipenv repo: https://github.com/pypa/pipenv/issues/2618
nedbat
on 20 Jul 2018
I can confirm that recreation of Pipfile.lock followed by pipenv install --dev will solve. Thanks @cardoe for noticing this issue. Thank you!
abk-code
on 20 Jul 2018
I guess you need these lines, if not I need advice how to do it:
(vEnv)$ tox
...
py37 inst-nodeps: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/.tox/dist/pdf2textbox-0.3.1.zip
py37 installed: atomicwrites==1.1.5,attrs==18.1.0,certifi==2018.4.16,chardet==3.0.4,check-manifest==0.37,coverage==4.5.1,idna==2.7,more-itertools==4.2.0,pdf2textbox==0.3.1,pdfminer2==20151206,pluggy==0.6.0,py==1.5.4,pytest==3.6.3,pytest-cov==2.5.1,requests==2.19.1,six==1.11.0,urllib3==1.23
py37 runtests: PYTHONHASHSEED='513262920'
py37 runtests: commands[0] | check-manifest
lists of files in version control and sdist match
py37 runtests: commands[1] | python --version
Python 3.7.0
py37 runtests: commands[2] | coverage --version
Coverage.py, version 4.5.1 without C extension
Documentation at https://coverage.readthedocs.io
py37 runtests: commands[3] | py.test --cov=pdf2textbox --duration=0 tests
================================= test session starts =================================
platform linux -- Python 3.7.0, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox, inifile:
plugins: cov-2.5.1
collected 15 items
Times are as before.
0LL13
on 21 Jul 2018
@0LL13 Can you try tox --recreate instead of just tox?
vst
on 21 Jul 2018
Cool.
(vEnv)$ tox --recreate
...
py37 recreate: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/.tox/py37
py37 installdeps: check-manifest, pytest, requests, pdfminer2, coverage, pytest-cov
py37 inst: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/.tox/dist/pdf2textbox-0.3.1.zip
py37 installed: atomicwrites==1.1.5,attrs==18.1.0,certifi==2018.4.16,chardet==3.0.4,check-manifest==0.37,coverage==4.5.1,idna==2.7,more-itertools==4.2.0,pdf2textbox==0.3.1,pdfminer2==20151206,pluggy==0.6.0,py==1.5.4,pytest==3.6.3,pytest-cov==2.5.1,requests==2.19.1,six==1.11.0,urllib3==1.23
py37 runtests: PYTHONHASHSEED='3853648192'
py37 runtests: commands[0] | check-manifest
lists of files in version control and sdist match
py37 runtests: commands[1] | python --version
Python 3.7.0
py37 runtests: commands[2] | coverage --version
Coverage.py, version 4.5.1 with C extension
Documentation at https://coverage.readthedocs.io
py37 runtests: commands[3] | py.test --cov=pdf2textbox --duration=0 tests
================================= test session starts =================================
platform linux -- Python 3.7.0, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox, inifile:
plugins: cov-2.5.1
collected 15 items
tests/test_cli_slice_option.py ... [ 20%]
tests/test_convert_pdf.py ...... [ 60%]
tests/test_get_pdf.py .... [ 86%]
tests/test_types.py .. [100%]
----------- coverage: platform linux, python 3.7.0-final-0 -----------
Name Stmts Miss Cover
.....................................................................................
pdf2textbox/pdf2textbox.py 506 201 60%
=============================== slowest test durations ================================
52.36s call tests/test_cli_slice_option.py::test_pdf_to_text_slice
2.64s call tests/test_get_pdf.py::test_get_pdf_file_url
1.14s call tests/test_types.py::test_textbox_structure
1.14s call tests/test_types.py::test_pdf_to_text_all
...
========================= 15 passed in 59.84 seconds ==============================
__________________________________ summary _______________________________________
py35: commands succeeded
py36: commands succeeded
py37: commands succeeded
congratulations :)
0LL13
on 21 Jul 2018
@0LL13 looks like it's all fixed. :)
nedbat
on 21 Jul 2018
Yep :)
0LL13
on 21 Jul 2018
Hi, sorry for reviving this issue, but are there C extensions for Python 3.7 on windows?
I have:
python --version
Python 3.7.0
coverage --version
Coverage.py, version 4.5.1 without C extension
coffeebenzene
on 17 Oct 2018
@coffeebenzene Right, there are no windows 3.7 files at the moment. I can try to build them, but something is wrong at AppVeyor, and it is difficult to diagnose.
nedbat
on 22 Oct 2018
@coffeebenzene ah, but they are here: https://www.lfd.uci.edu/~gohlke/pythonlibs/#coverage . Perhaps I will upload those.
nedbat
on 22 Oct 2018
@coffeebenzene I uploaded those .whl to pypi. You should be all set.
nedbat
on 22 Oct 2018
Thanks!
I just reinstalled with pip and I got the C extensions now.
coffeebenzene
on 26 Oct 2018
Related issues
cdce8p
·
4Comments
fersarr
·
4Comments
fkromer
·
7Comments
decorator-factory
·
5Comments
pawamoy
·
8Comments
Most helpful comment
@coffeebenzene I uploaded those .whl to pypi. You should be all set.