Pants: The generated coveragerc seems wrong
I've encountered the case where the generated coveragerc looks like this:
[run]
branch = True
timid = True
[report]
exclude_lines =
def __repr__
raise NotImplementedError
[paths]
shared =
shared
.pants.d/pyprep/sources/7612f833acaeb440ee7d4249c5c2c340885c1ec7
/usr/local/google/home/tanin/projects/clusterfuzz-tools/shared
/usr/local/google/home/tanin/projects/clusterfuzz-tools/.pants.d/pyprep/sources/7612f833acaeb440ee7d4249c5c2c340885c1ec7
error =
error
.pants.d/pyprep/sources/7612f833acaeb440ee7d4249c5c2c340885c1ec7
/usr/local/google/home/tanin/projects/clusterfuzz-tools/error
/usr/local/google/home/tanin/projects/clusterfuzz-tools/.pants.d/pyprep/sources/7612f833acaeb440ee7d4249c5c2c340885c1ec7
(The content is generated from this line: https://github.com/pantsbuild/pants/blob/master/src/python/pants/backend/python/tasks2/pytest_run.py#L210)
The run command is: ./pants test.pytest --coverage=auto error:test -- -p no:logging
And here's the BUILD:
python_tests(
name='test',
sources=rglobs('tests/*.py'),
coverage='error',
compatibility=['>=2.7','<3'],
dependencies=[
':src',
'//shared:test_libs',
'//3rdparty/python:pyfakefs',
'//3rdparty/python:mock'
]
)
:src and //shared:test_libs are regular python_librarys. :src has the module error.
This leads to the generated report below:
18:05:50 00:03 [coverage]
Name Stmts Miss Branch BrPart Cover
----------------------------------------------------------------
shared/test_libs/__init__.py 0 0 0 0 100%
shared/test_libs/helpers.py 38 38 8 0 0%
----------------------------------------------------------------
TOTAL 38 38 8 0 0%
You can see that it doesn't include the module error. If I remove //shared:test_libs from the BUILD target, it suddenly shows the coverage from the module error.
After testing, there's something strange around [paths]. I'm not sure.
This is Pants 1.3.0.
tanin47
All 18 comments
More info:
- I am updating from 1.2.1 to 1.3.0 (to avoid
error in cryptography setup command: Invalid environment marker: python_version < '3'). The code coverage works in 1.2.1 but not 1.3.0 - I think
[paths]'s value seems fine from hacking multiple version of coveragercs. But there might be something odd in the current versions of pytest, pytest-cov, and coverage.py.
tanin47
on 12 Jan 2018
Would again recommend trying with 1.3.1rc1. More generally, it's been a very long time since 1.3.0 was released... there have been many changes to pytest in master, but I can't really recommend any particular 1.4.0.devX release as being more likely to fix this issue. If you have time to bisect on 1.4.0.dev releases, that would be neat.
I think that we will be cutting a 1.4.0rc0 tomorrow which you might consider trying out: and in particular, if the coverage issue is not fixed in the rc, we can work to get it fixed before 1.4.0 is released.
stuhood
on 12 Jan 2018
Neither 1.3.1rc1, 1.4.0.dev3, nor 1.4.0.dev27 works. Here is a sample output:
$ ./pants test.pytest --coverage=error error:test -- -p no:logging
WARN] /home/tanin/.cache/pants/setup/bootstrap-Linux-x86_64/1.4.0.dev3/local/lib/python2.7/site-packages/pants/build_graph/build_file_aliases.py:52: DeprecationWarning: DEPRECATED: The `resources=` Python target argument will be removed in version 1.5.0.dev0.
Depend on resources targets instead.
context_aware_object_factory(parse_context, *args, **kwargs)
22:33:44 00:00 [main]
(To run a reporting server: ./pants server)
22:33:44 00:00 [setup]
22:33:45 00:01 [parse]
Executing tasks in goals: bootstrap -> imports -> unpack-jars -> deferred-sources -> gen -> jvm-platform-validate -> resolve -> compile -> pyprep -> resources -> test
22:33:45 00:01 [bootstrap]
22:33:45 00:01 [substitute-aliased-targets]
22:33:45 00:01 [jar-dependency-management]
22:33:45 00:01 [bootstrap-jvm-tools]
22:33:45 00:01 [provide-tools-jar]
22:33:45 00:01 [imports]
22:33:45 00:01 [ivy-imports]
22:33:45 00:01 [unpack-jars]
22:33:45 00:01 [unpack-jars]
22:33:45 00:01 [deferred-sources]
22:33:45 00:01 [deferred-sources]
22:33:45 00:01 [gen]
22:33:45 00:01 [antlr-java]
22:33:45 00:01 [antlr-py]
22:33:45 00:01 [jaxb]
22:33:45 00:01 [protoc]
22:33:45 00:01 [ragel]
22:33:45 00:01 [thrift-java]
22:33:45 00:01 [thrift-py]
22:33:45 00:01 [wire]
22:33:45 00:01 [jvm-platform-validate]
22:33:45 00:01 [jvm-platform-validate]
22:33:45 00:01 [resolve]
22:33:45 00:01 [ivy]
22:33:45 00:01 [cache].
22:33:45 00:01 [bootstrap-nailgun-server]
22:33:45 00:01 [compile]
22:33:45 00:01 [compile-jvm-prep-command]
22:33:45 00:01 [jvm_prep_command]
22:33:45 00:01 [compile-prep-command]
22:33:45 00:01 [compile]
22:33:45 00:01 [zinc]
22:33:45 00:01 [jvm-dep-check]
22:33:45 00:01 [pyprep]
22:33:45 00:01 [interpreter]
22:33:45 00:01 [cache]
No cached artifacts for 4 targets.
Invalidated 4 targets.
22:33:51 00:07 [requirements]
22:33:51 00:07 [cache]
No cached artifacts for 2 targets.
Invalidated 2 targets.
22:33:53 00:09 [sources]
22:33:53 00:09 [cache]
No cached artifacts for 4 targets.
Invalidated 4 targets.
22:33:53 00:09 [resources]
22:33:53 00:09 [prepare]
22:33:53 00:09 [services]
22:33:53 00:09 [test]
22:33:53 00:09 [test-jvm-prep-command]
22:33:53 00:09 [jvm_prep_command]
22:33:53 00:09 [test-prep-command]
22:33:53 00:09 [test]
22:33:53 00:09 [pytest-prep]
22:33:53 00:09 [cache]
No cached artifacts for 6 targets.
Invalidated 6 targets.
22:33:53 00:09 [cache]
No cached artifacts for 1 target.
Invalidated 1 target.
22:33:57 00:13 [pytest]
22:33:57 00:13 [cache]
No cached artifacts for 1 target.
Invalidated 1 target.
22:33:57 00:13 [run]
============== test session starts ===============
platform linux2 -- Python 2.7.12, pytest-3.3.2, py-1.5.2, pluggy-0.6.0
rootdir: /home/tanin/projects/clusterfuzz-tools, inifile:
plugins: timeout-1.2.1, cov-2.4.0, pyfakefs-3.1
collected 1 item
error/tests/error/error_test.py . [100%]
generated xml file: /home/tanin/projects/clusterfuzz-tools/.pants.d/test/pytest/junitxml/TEST-error.test.xml
============ 1 passed in 0.10 seconds ============
22:33:58 00:14 [coverage]
22:33:58 00:14 [coverage]
Name Stmts Miss Branch BrPart Cover
----------------------------------------------------------------
shared/test_libs/__init__.py 0 0 0 0 100%
shared/test_libs/helpers.py 38 38 8 0 0%
----------------------------------------------------------------
TOTAL 38 38 8 0 0%
22:33:58 00:14 [coverage]
22:33:59 00:15 [coverage]
error.test ..... SUCCESS
22:33:59 00:15 [junit]
22:33:59 00:15 [complete]
SUCCESS
The coverage report only shows the coverage for shared/. This only happens when we include 2 different python_librarys. It will choose one of them randomly.
I'll make a PoC over the weekend, and hopefully it's reproducible.
tanin47
on 12 Jan 2018
cc @benjyw , @jsirois : in case you guys see anything obvious.
stuhood
on 12 Jan 2018
Here's the PoC: https://github.com/tanin47/pants-python-coverage-problem
Notice that the coverage for the module abstract isn't generated. It's likely that this problem comes from a newer version of pytest, pytest-cov, or coverage.py
tanin47
on 12 Jan 2018
This seems to be interacting weirdly with test caching, although that may be unrelated, so I'll let @jsirois look into it. :)
benjyw
on 18 Jan 2018
Question: is there a way to control the dependencies' versions in Pants? I want to test with different versions of pytest, pytest-cov, and coverage.py.
tanin47
on 18 Jan 2018
From the command-line:
--pytest-requirements="pytest>=3.0.7,<4.0"
--pytest-cov-requirements="pytest-cov>=2.4,<2.5"
Or in pants.ini:
[pytest]
requirements: "pytest>=3.0.7,<4.0"
cov_requirements: "pytest-cov>=2.4,<2.5"
Note that AFAIK we don't set the version of coverage explicitly - we let
pytest-cov determine the version(s) it's compatible with.
On Thu, Jan 18, 2018 at 1:18 PM, Tanin Na Nakorn notifications@github.com
wrote:
Question: is there a way to control the dependencies' versions in Pants? I
want to test with different versions of pytest, pytest-cov, and coverage.py.—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pantsbuild/pants/issues/5314#issuecomment-358784617,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAfS_JJTg0_bkRFPsGJxZY7rlB5uVLveks5tL7UXgaJpZM4RbuR4
.
benjyw
on 19 Jan 2018
But you may be able to force a different coverage version by sneaking it in
to the requirements via ----unittest2-requirements (I'm assuming you're
using a python version that has a current unittest already), e.g.:
--pytest-cov-requirements="pytest-cov>=2.4,<2.5"
--unittest2-requirements="coverage>=1.0,<2.0"
or:
[pytest]
requirements: "pytest>=3.0.7,<4.0"
unittest2_requirements: "coverage>=1.0,<2.0"
On Thu, Jan 18, 2018 at 4:13 PM, Benjy Weinberger benjyw@gmail.com wrote:
From the command-line:
--pytest-requirements="pytest>=3.0.7,<4.0"
--pytest-cov-requirements="pytest-cov>=2.4,<2.5"Or in pants.ini:
[pytest]
requirements: "pytest>=3.0.7,<4.0"
cov_requirements: "pytest-cov>=2.4,<2.5"Note that AFAIK we don't set the version of coverage explicitly - we let
pytest-cov determine the version(s) it's compatible with.On Thu, Jan 18, 2018 at 1:18 PM, Tanin Na Nakorn <[email protected]
wrote:
Question: is there a way to control the dependencies' versions in Pants?
I want to test with different versions of pytest, pytest-cov, and
coverage.py.—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pantsbuild/pants/issues/5314#issuecomment-358784617,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAfS_JJTg0_bkRFPsGJxZY7rlB5uVLveks5tL7UXgaJpZM4RbuR4
.
benjyw
on 19 Jan 2018
Thanks for all the detail @tanin47! This led to a unit test that reproduces the problem. Still working through a solution, but a solution should be landing by end of week.
jsirois
on 23 Jan 2018
Alright, just a note that the issue in the end is having multiple source roots for python code. This leads to an ambiguity in path conversion in our coverage combine step which leads to an arbitrary source root being picked by coverage. When coverage then looks for files not in that source root, coverage is not found. Still poking about for solutions.
jsirois
on 26 Jan 2018
Tried _alot_ of things here and the only clean way to deal with this is to model the runtime on the source setup, ie: use a separate source pex per source root involved. Giving this a shot.
jsirois
on 27 Jan 2018
Thank you @jsirois for this work. I'd really like to try it out. How can I use it without waiting for a release? Is there an option in pants.ini that can be used for pointing to a specific SHA instead of version?
tanin47
on 29 Jan 2018
If you clone the pants repo, you can follow the instructions here to run
from those sources instead of from a pants release:
https://www.pantsbuild.org/howto_develop.html#running-from-sources
On Sun, Jan 28, 2018 at 4:32 PM, Tanin Na Nakorn notifications@github.com
wrote:
Thank you @jsirois https://github.com/jsirois for this work. I'd really
like to try it out. How can I use it without waiting for a release? Is
there an option in pants.ini that can be used for pointing to a specific
SHA instead of version?—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pantsbuild/pants/issues/5314#issuecomment-361111326,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAfS_Iaun3iFcxE2jHWnyua5Pef42fwKks5tPRG4gaJpZM4RbuR4
.
benjyw
on 29 Jan 2018
As Benjy says. And this ~works against the sample repo you provided:
$ ~/dev/tanin47/pants-python-coverage-problem (master) $ PANTS_VERSION=1.5.0.dev1 ~/dev/pantsbuild/pants/pants test.pytest --coverage=abstract tests:test
...
05:01:17 00:15 [pytest]
05:01:17 00:15 [cache]
No cached artifacts for 1 target.
Invalidated 1 target.
05:01:17 00:15 [run]
============== test session starts ===============
platform linux2 -- Python 2.7.12, pytest-3.3.2, py-1.5.2, pluggy-0.6.0
rootdir: /home/jsirois, inifile:
plugins: cov-2.4.0, timeout-1.2.1
collected 1 item
.pants.d/pyprep/sources/611275cc44eea98e4fa9bc327aec1208efd1f176-DefaultFingerprintStrategy_972b98b13b87/abstract_test.py . [100%]
generated xml file: /home/jsirois/dev/tanin47/pants-python-coverage-problem/.pants.d/test/pytest/tests.test/junitxml/TEST-tests.test.xml
============ 1 passed in 0.07 seconds ============
05:01:17 00:15 [coverage-combine]
05:01:17 00:15 [coverage-report]
Name Stmts Miss Branch BrPart Cover
-----------------------------------------------------------------
abstract/abstract/abstract.py 6 0 0 0 100%
chicken/chicken.py 3 0 0 0 100%
-----------------------------------------------------------------
TOTAL 9 0 0 0 100%
05:01:18 00:16 [coverage-html]
05:01:18 00:16 [coverage-xml]
tests:test ..... SUCCESS
...
N.B: both abstract and chicken are included here. I'm not sure exactly why, but --coverage=abstract.abstract (which selects the abstract.abstract module), --coverage=abstract/abstract (which selects everything under $BUILD_ROOT/abstract/abstract - which is just abstract.py), --coverage=chicken/ and --coverage=chicken all display coverage for exactly what you'd expect and nothing more. Your module and source root structure here are both atypical so the --coverage=abstract case reporting more coverage result than expected may need a follow-up issue if this is a problem for you.
jsirois
on 29 Jan 2018
I've tagged the PR #needs-cherrypick for 1.4.x, so it should probably go in 1.4.0rc2.
stuhood
on 30 Jan 2018
Do you have a timeframe when 1.4.0rc2 will be released? Thank you.
tanin47
on 7 Feb 2018
@tanin47 : Currently it's reverted in the 1.4.x branch and waiting on a fix for #5426... I'll reopen this to signal that.
stuhood
on 7 Feb 2018
Related issues
stuhood
·
5Comments
KLFrost
·
4Comments
pvcnt
·
3Comments
benjyw
·
6Comments
Eric-Arellano
·
7Comments
Most helpful comment
Thanks for all the detail @tanin47! This led to a unit test that reproduces the problem. Still working through a solution, but a solution should be landing by end of week.