Cylc-flow: Report test coverage

[kinow]

Hi,

While toying with the code for #844 , I was trying to find the coverage of tests of what I was altering, but I couldn't locate the report. And with some upcoming changes for the new GUI, Python 3, and new features and bug fixes, probably having the test coverage reported for the current code will help preventing regressions.

I've been learning a bit more about Coverage.py, and have some local results, but nothing that would work on Travis-CI yet. Here's a list of subtasks that I think have to be completed for the report to be available to all here in Github.

• [x] Collect coverage of the same command used for running the tests in Travis-CI (test-battery I believe)
• [x] Aggregate the data for the processes produced with prove and shell script
• [x] Check if with the _chunking_ feature (i.e. matrix use in Travis-CI) we are still able to produce a single report for all chunks
• [x] Produce a report in XML for Codacy and maybe HTML so that we can view it from Travis-CI if necessary (e.g. Codacy is down, or integration is not working as expected)
• [x] Update .travis-ci.yml
• [ ] Update Codacy - if necessary. We can probably add a new badge to the project (and maybe also get the metrics per pull request)
• [ ] Document Coverage.py, Travis-CI, & Codacy setup for Cylc somewhere (maybe CUG, maybe just here in this ticket, possibly comments in the .travis.yml file)

Running the tests with prove and shell script makes it a little more difficult to get the coverage for cylc test-battery, but I have it working (kinda) locally. Still needs a few kicks until it works as expected, but for most small tests I've been doing it reported the correct coverage.

Not sure if the chunks for tests will work with the coverage. What I saw in other projects, was the matrix feature from Travis CI being used for running multiple versions of Python/Cython/etc. And then when generating the report, it would execute for only one value of the matrix.

Our case is somewhat special, as I believe we would need an aggregated report, with the total coverage of the 4 chunks that we have right now.

Also never used Codacy for coverage, only coveralls and sonarqube, but so far it looks good! Will update the ticket later, but if anybody has some more information about coverage, feel free to chime in :-)

[hjoliver]

Thanks @kinow - yes it would be really good to get coverage tests working this year; it sounds a little more difficult than I had imagined!

[kinow]

Indeed it is @hjoliver ! Doable, bit of fun, but Travis CI and the chunks feature are quite a challenge.

I have a local report that I could even upload somewhere. Even though my check-software is saying FOUND for almost all dependencies (missing preprint and texlive only), I am not sure if there is any other configuration that would enable more tests.

My notes for running the tests are in some post-its and a text file on my desktop here. So documenting here how to get coverage for the tests, so that I can run this again in other workstation if necessary - or if anybody would like to take a look.

---

Install coverage.py with pip install coverage.

That should put the executable coverage in your $PATH

Configure coverage.py

Normally you would run something like coverage run nosetests. Where coverage run is simply going to have a chance to prepare and start collecting coverage of your python code. But as running cylc test-battery actually involves running shell scripts and utilities to get the python code running, the best example from coverage.py documentation is the part about subprocesses, with a few tweaks.

First, you will need a .coveragerc in the project root, with the content similar to:

[run]

parallel = True
branch = True
cover_pylib = False

data_file=/home/user/workspace/cylc/.coverage

source=
    ./bin
    ./lib

#debug=
#    callers
#    dataio
#    multiprocess
#    dataio
#    dataop
#    pid
#    plugin
#    process
#    sys
#    trace

timid = False

[report]

exclude_lines =
    pragma: no cover
    def __repr__
    raise NotImplementedError
    if __name__ == .__main__.:
    def parse_args

ignore_errors = False
precision=2
show_missing=True

Read the coverage.py documentation for details about each key in the ini configuration file. The data_file is necessary, as some tests appear to result in the coverage file .coverage in sub directories.

Every time you run coverage.py, it will produce a .coverage file. But as we have the parallel set, then coverage.py will output something like .coverage.computer.12345, which are hostname and process ID. In the end, there will be several .coverage____ files in the root directory.

Running coverage.py

As mentioned in the previous section, one cannot simply run coverage run cylc test-battery. The cylc in that command line is a shell script, and coverage.py will just give up collecting coverage after that, regardless of all the effort you have put in making that configuration file (I spent quite some time until I realized that).

Similarly to Jacoco/Cobertura/Clover+XDebug and so many other tools, you have to "tell coverage.py to instrument the code". There are two ways of doing that, documentation in the subprocess documentation section mentioned here before.

The way that worked for me, was with a .pth file. This file, when put in certain locations, will always be executed before a Python script. The hack, used for coverage.py, is that you can also load modules, and execute them.

So we can instruct all Python processes in our system to always start coverage. That will alert coverage.py that we want it to instrument the code and collect coverage. In my enviroment (a Ubuntu LTS with system Python, not Anaconda), I created the file /usr/lib/python2.7/dist-packages/cylc.pth.

import sys; import coverage; import os; os.environ['COVERAGE_PROCESS_START'] = "/home/kinow/workspace/cylc/.coveragerc"; coverage.process_startup();

_(I entered some garbage in this file, and in different locations, and simply started python in a terminal, until I got a few places where it worked, and I could confirm it was being loaded...)_

Now you can finally run the Cylc tests as you would normally do, and coverage.py will take care to collect the coverage. As you enabled parallel = True, you will have multiple .coverage output files. Well done! よーし!

cylc test-battery

Combining the multiple .coverage files

If you have experience with Java's Emma/Jacoco, this will be a pretty common task, that works exactly the same way.

Given you have multiple coverage output files, you can combine all these files with

coverage combine --append

The --append is a new feature (4.1 or 4.2 I think) that, instead of overwriting values, will sum up and report the totals in the coverage (NB probably useful for the _chunk_ feature in Travis CI).

Reporting the coverage

Finally, you can report the coverage in the terminal with

coverage report

Or get a HTML report with

coverage html

By default the HTML report is created under the folder ./htmlcov.

Notes about the configuration file

Each configuration in the configuration file .coveragerc corresponds more or less to a task in this pipeline. Where run contains information for running the tests, report will be used when combining and reporting the results, and html can also be used for changing style, title, and location of the coverage HTML.

Final notes

This text was quickly written to offload the notes from post-its over my desk, and from a confusing text file from my desktop. My current workstation contains a working set up, but I intend to re-read these notes and re-run step by step, reviewing and correcting it, in the near future on another workstation.

More to come about Travis CI later on :walking_man:

[hjoliver]

@kinow - out of interest, have your experiments resulted in ball-park coverage estimates yet?

(We should bear in mind that the old GUI is a known black spot, and that coverage may also be degraded by batch-system and remote-specific tests that get skipped in the default set-up, but I guess we should still "bite the bullet" and try to figure this out).

[kinow]

@hjoliver I had an old copy of the htmlcov folder, but must have deleted. By chance, I came have lunch at home as I have to take some documents to another building nearby, so left the tests running. Used the latest version from the master branch. Here's the output of my check-software:

cylc check-software
Checking your software...

Individual results:
==========================================================================================
Package (version requirements)                                     Outcome (version found)
==========================================================================================
                                   *REQUIRED SOFTWARE*                                   
Python (2.6+, <3)............................FOUND & min. version MET (2.7.15.candidate.1)

             *OPTIONAL SOFTWARE for the GUI & dependency graph visualisation*             
Python:pygtk (2.0+)......................................FOUND & min. version MET (2.24.0)
graphviz (any)..............................................................FOUND (2.40.1)
Python:pygraphviz (any)......................................................FOUND (1.3.1)

                       *OPTIONAL SOFTWARE for the HTML User Guide*                       
ImageMagick (any)..........................................................FOUND (6.9.7-4)

                  *OPTIONAL SOFTWARE for the HTTPS communications layer*                  
Python:urllib3 (any)..........................................................FOUND (1.23)
Python:OpenSSL (any)........................................................FOUND (18.0.0)
Python:requests (2.4.2+).................................FOUND & min. version MET (2.19.1)

                   *OPTIONAL SOFTWARE for the configuration templating*                   
Python:EmPy (any)............................................................FOUND (3.3.2)

                       *OPTIONAL SOFTWARE for the LaTeX User Guide*                       
TeX:framed (any)...............................................................FOUND (n/a)
TeX (3.0+)...........................................FOUND & min. version MET (3.14159265)
TeX:tocloft (any)..............................................................FOUND (n/a)
TeX:tex4ht (any)...............................................................FOUND (n/a)
TeX:preprint (any)...........................................................NOT FOUND (-)
TeX:texlive (any)............................................................NOT FOUND (-)
==========================================================================================

Summary:
                               ****************************                               
                                  Core requirements: ok                                  
                                Full-functionality: not ok                                
                               **************************** 

Some tests failed, but I believe it must be something that I'm missing:

Test Summary Report
-------------------
tests/authentication/05-full-control.t                                       (Wstat: 0 Tests: 12 Failed: 4)
  Failed tests:  2-5
tests/authentication/02-state-totals.t                                       (Wstat: 0 Tests: 10 Failed: 4)
  Failed tests:  2-5
tests/authentication/06-suite-override.t                                     (Wstat: 0 Tests: 12 Failed: 4)
  Failed tests:  2-5
tests/syntax/00-pep8.t                                                       (Wstat: 0 Tests: 3 Failed: 1)
  Failed test:  3
tests/authentication/03-full-read.t                                          (Wstat: 0 Tests: 11 Failed: 4)
  Failed tests:  2-5
tests/authentication/04-shutdown.t                                           (Wstat: 0 Tests: 12 Failed: 4)
  Failed tests:  2-5
tests/authentication/07-sha-hash.t                                           (Wstat: 0 Tests: 12 Failed: 4)
  Failed tests:  2-5
tests/authentication/01-description.t                                        (Wstat: 0 Tests: 10 Failed: 4)
  Failed tests:  2-5
tests/job-submission/16-timeout.t                                            (Wstat: 0 Tests: 4 Failed: 1)
  Failed test:  3
tests/authentication/00-identity.t                                           (Wstat: 0 Tests: 17 Failed: 8)
  Failed tests:  2-5, 10-13
Files=659, Tests=2437, 2264 wallclock secs ( 2.19 usr  0.77 sys + 1412.94 cusr 274.67 csys = 1690.57 CPU)
Result: FAIL

It created 12116 .coverage.localhost.123.123 coverage output files, which after combine --append result in a single file sized 213K.

_(the settings must need some tweaking, as it apparently included a file from my $PATH which is not related to cylc: Couldn't parse '/home/kinow/bin/a2dp.py' as Python source: 'invalid syntax' at line 118... will investigate later)_

I'm uploading it to a GitHub repository with the pages setting enabled. This way will be easier to review the coverage numbers we have right now. During my tests, I had a simplified script, that covered just the print_tree function. Later, I started running more tests, and checking lib/parsec/validate.py, and lib/cylc/config.py, which I believe are interesting & important files :-)

| File | Coverage |
| ------------- | ------------- |
| lib/cylc/config.py | 34.91% |
| lib/parsec/validate.py | 57.41% |

I am not too crazy about trying to achieve 100% coverage. Instead, normally I try to have enough coverage in the important parts of the system, and later slowly add more as necessary (tests have a cost associated with writing and maintaining them after all).

But I am really looking forward to what's going to happen once we have a proper coverage solution. From past experience, adding coverage to a project that exists for a while, normally has some positive results such as:

• finding code that is not needed (we find it as that code keeps the coverage under a certain level)
• finding code that does not follow the code design (sometimes you may have a SuiteAdapterZorkAbc with 90% coverage, while SuiteAdapterZorba has 10%. Some times it's just because the design of the classes is different, maybe one was added later, and needs an update so that it's easier to test)
• finding - and/or preventing - bugs! Of course. As we add more tests to parts that we consider important, but are not yet covered. This is my favourite, and easiest to achieve.

[hjoliver]

Thanks, this is going to be useful.

[kinow]

Committed here https://github.com/kinow/kinow.github.io, which in theory should be available at https://kinow.github.io/index.html, but I guess it's cached somewhere with the previous test site I had there.

If you see a different page, try again in a few seconds, or clone the repo and open locally, and you should be able to navigate the results. That's the default output of Coverage.py.

[hjoliver]

Wow, this is really useful information!

One thing is, we should exclude bundled external packages (which are only bundled with Cylc for pragrmatic reasons, and should have their own tests): lib/jinja2, lib/cherrypy, lib/markupsafe.

Can we also exclude the current GUI and just acknowledge that this is not tested at all (but is stable and static, and soon to be replaced).

[kinow]

One thing is, we should exclude bundled external packages (which are only bundled with Cylc for pragrmatic reasons, and should have their own tests): lib/jinja2, lib/cherrypy, lib/markupsafe.

Oh, good point. That's fairly easy. Right now I'm simply telling Coverage.py what are the source directories (bin & lib). But I can combine that with omit key. Only limitation is that it's not possible to use the source directories + the include key, as they are exclusive.

Can we also exclude the current GUI and just acknowledge that this is not tested at all (but is stable and static, and soon to be replaced).

+1

[kinow]

Got a branch with a working setup. But here are some important notes:

• Codacy allows partial uploads of coverage (with the --partial argument). However, the Python utility doesn't support it. Instead, you have to use the JAR
• Codacy works with branch/commit. So if you have a Travis-CI build that executes 3 or 4 jobs in parallel, what matters is that they use the same branch and commit
• You also have to tell Codacy that the analysis is over, so it can aggregate the reports, and produce a combined report. That's done with the --final argument.
• In order to have the fork-join kind of tasks in Travis CI, I had to modify the .travis.yml to use stages instead of matrix.
• The first stage to run is test, and the second is report. If you do not specify a stage name, Travis assumes you meant the last available stage, or test if no previous stage.
• Some times Coverage.py gets confused, and fails in coverage xml. To prevent that from stopping the whole coverage analysis of all chunks, we need to use --ignore-warnings with the coverage xml command.
• Travis-CI build scripts are now under .travis
• The .pth file didn't work on Travis-CI very well, but I found a post with a nice set up for the sitecustomize.py. Basically, it consists in running PYTHONPATH=.travis, and having a sitecustomize.py in that folder. This way we can run a script before each Python script, starting up the coverage process.

[hjoliver]

@kinow - #2766 is merged - can we close this now?

[kinow]

Yup, forgot we had an issue too, sorry. Closing.