Platformio-core: Support generation of code coverage data

[r89m]

Hi Ivan,

I searched for this kind of topic and didn't find anything but let me know if this is a duplicate.

I have my Transition library being tested on Travis using my ArduinoFrameworkFake library which has been working well since you gave me directions on how to import the library.

I'd now like to gather some information regarding coverage so that I can identify areas where I am not testing my code sufficiently. I tried using the build_flags section in platformio.ini to pass --coverage but was met with a lot of linker errors that I tried (unsuccessfully) to resolve.
Here's a few of the combinations of options I tried using
-g -O0 --coverage -fPIC -shared -Wl,-fprofile-generate -Wl,-shared -Wl,-fPIC
-fprofile-arcs -ftest-coverage -fprofile-generate -Wl,-lgcov --coverage

I am successfully generating coverage using: https://github.com/r89m/Transition/blob/master/.travis.yml#L56
which is giving me the output I'd like (https://coveralls.io/jobs/22116632) but I'd love to be able to do something along the lines of pio test --coverage or similar and generate the necessary gcno and gcda files.

Any help would be appreciated.

Richard

[tanishqaggarwal]

I know this has probably fallen by the wayside by now, but here's a simple fix I found. Posting this here in case someone, like me, searches the ends of the Earth for how to configure this functionality but can't figure out how:

• Add --coverage as a build option
• Add an extra_script with the following content:

Import("env")

env.Append(
  LINKFLAGS=[
      "--coverage"
  ]
)

Works like a charm.

[tanishqaggarwal]

It's worth noting that I haven't been able to get coveralls to satisfactorily exclude random nonsense files from the coverage report, although this is probably more of a coveralls problem than a PlatformIO one. So far I've been trying:

coveralls -r . -b . -e "./test/output_export.cpp" -e ".pio/libdeps"

This invocation tries to print a test coverage for libdeps within the .pio folder as well. Not sure why.

[tanishqaggarwal]

See https://github.com/pathfinder-for-autonomous-navigation/CommonSoftware/pull/15

[p-vernaeckt]

I'd love to see that feature added to the next release's roadmap ^^

[arduhe]

So do I :-)

[jcw]

With some web searches, I've been able to generate coverage data and visualise it (using lcov + genhtml), but one puzzle still remains: I'm not just running the app (natively, in Linux), I also want to include coverage data from all the test suites I've written. The data from these tests gets generated fine (in .pio/build/native/test/testname/...), but so far I've found it impossible to use it with lcov/gcov, because the test application binaries get overwritten: the final apps build with pio run and pio test all end up being the same executable, i.e. .pio/build/native/program.

Coverage data is really only meaningful when you can take the code though _all_ its paces and _then_ examine the aggregated reports. I suspect that anyone using pio test will run into the same issue.

_In short: how can I generate and combine all coverage data, i.e. several runs of the real app as well as all test apps?_

P.S. For reference, here is a trimmed version of my extra_script.py:

# Import("env") etc ...

def generateCoverageInfo(source, target, env):
    for file in os.listdir("verify"):
        call([".pio/build/native/program", "verify/"+file])
    call(["lcov", "-d", ".pio/build/native/", "-c", "-o", "lcov.info"])
    call(["genhtml", "-o", "cov/", "--demangle-cpp", "lcov.info"])

env.AddPostAction(".pio/build/native/program", generateCoverageInfo)

This works fine for the app, without the test suites, when launched as pio run -t debug.