Tarpaulin: Documentation of how to use in multi package workspace

[bluenote10]

This is most likely just my stupidity, but maybe worth improving the docs.

I'm trying to run tarpaulin in a multi-package workspace. I have a workspace consisting of a package called lib and a package benchmarks. I want to count the coverage only of lib. From the CLI help, my guess was that this can be accomplished by running one following commands in the top-level workspace:

cargo tarpaulin -v -o Html -p lib_package_name
cargo tarpaulin -v -o Html -e benchmarks_package_name
cargo tarpaulin -v -o Html -p lib_package_name -e benchmarks_package_name

All three cases behave the same as not specifying any -e or -p and result in:

I'd like not to count the benchmarks crate though.

Alternatively I tried to cd into the library package and run tarpaulin there. By default running it in a sub-package crashes with:

[ERROR tarpaulin] Failed to create report directory: No such file or directory (os error 2)

Using cargo tarpaulin -v -o Html --root .. helps with the crash, but then the benchmark package gets included in the report again.

A hint in the docs for multi-package workspaces would be great.

[xd009642]

There's always the --exclude-files flag where you could pass benchmarks/* as the argument and it will remove it from the results. Exclude package should just stop tests in that package from being built but the code will still be included in coverage results. If you can send me a repo link I can have a look because I'm curious on why there's some results in the benchmark folder with it being excluded..

[bluenote10]

The --exclude-files option indeed returns the expected coverage. It is just that the -p and -e options seem to have no effect.

Sure, the repository is rust-array-stump. Note that this is my first real Rust package, and it is still unfinished ;). Thanks for looking into it.

[xd009642]

So if I run cargo test --workspace --e and then exclude benchmarks or lib, or use -p to include one of them it behaves exactly the same as tarpaulin does running all the tests. There is an issue where if I cd into the lib folder tarpaulin crashes because the target folder isn't where it's expected to be (unless I specify it explicitly). I'll fix the target issue but I think in terms of the coverage results and what tarpaulin generates there's no issue as it behaves the same as cargo test

[bluenote10]

I've spend another 15 minutes, but I can't get it to work with --workspace / -e / -p. What cargo tarpaulin ... command are you using?

To make it specific on the rust-array-stump example, the top level Cargo.toml contains:

[workspace]
members = [
    "lib",          # contains package called "array_stump"
    "benchmarks",   # contains package called "array_stump_benchmarks"
]

Now all of the following fails to exclude the "benchmarks" coverage (which I conclude from the last printed line 68.34% coverage which would be ~99% if ignoring the benchmarks package):

# pwd is repo root, i.e., the workspace
cargo tarpaulin -v -p array_stump -e array_stump_benchmarks
cargo tarpaulin -v -p array_stump -e array_stump_benchmarks --workspace

# or should I use directory names instead of package names?
cargo tarpaulin -v -p lib -e benchmarks
cargo tarpaulin -v -p lib -e benchmarks --workspace

To me the arguments -p, -e, and --workspace seem to have no effect. Do you get different behavior? The only parameter that has an effect is --exclude-files.

In general I don't understand the semantics of --workspace. What should happen if executing cargo tarpaulin top-level in a workspace, without activating --workspace? Or conversely, what means --workspace when there is no workspace? Shouldn't that be inferred?

[xd009642]

Yeah so the same thing happens with me, the point I was making is that tarpaulin's behaviour matches that of cargo test. Never mind how I use those commands with cargo test it runs the same tests

[danieleades]

i have a similar issue. I have a workspace with multiple packages, for which i need to generate and upload coverage reports separately. is this supported?

[xd009642]

@danieleades I imagine the solution would be to call tarpaulin multiple times with different options and calling a different upload command for the generated report. It should be possible via bash in your CI but will require multiple separate tarpaulin calls

[ia0]

Thanks a lot for the tool. It's very useful.

However I have a similar issue for data-encoding. I am only interested in the coverage of the library code (i.e. what is under lib/src). In particular, I don't want to report coverage for the fuzzing code (e.g. lib/fuzz), the tests (e.g. lib/tests), or other crates code (e.g. bin).

I manage to do it with the following:

cd lib
cargo tarpaulin --ciserver travis-ci --coveralls "$TRAVIS_JOB_ID" \
  --exclude-files '../*' --exclude-files fuzz --exclude-files tests

However if there would be a way to specify positively (in addition to negatively) the list of files for which to report coverage (e.g. --includes-files lib/src) I would be interested. The semantics could be something like:

• If there are any --include-files use their union as a starting point, otherwise use all files.
• Then, if there are any --exclude-files remove their union from the starting point above.

[xd009642]

@ia0 that's a good idea I'll look to work something with that functionality into tarpaulin for the next release :+1:

[xd009642]

Also @azriel91 pointed out and submitted a PR that fixed the fact package arguments weren't being passed into cargo. I'm right now writing some code that passes exclude in because that was missed out as well. I'm also adding a test project for workspaces since the lack of tests for workspace coverage meant these slipped through the net when cargo was removed as a dependency.

I'll aim to have a new release out tomorrow hopefully with the --include-files in as well otherwise I'll leave that for another release later on

[xd009642]

Actually I just found when --exclude was fixed the excluded package was omitted from the results so I'm going to go ahead and push a release out and add --include-files later on

[bluenote10]

Perfect, thanks for the fix!

[ia0]

Thanks! I'll try the new release when this bug is closed. I don't need the --include-files option if --package works.

[xd009642]

@bluenote10 so is your initial issue all sorted?

@danieleades I'll open a new issue for your feature request and tag you in it.

[xd009642]

Gonna take that react as a yes and close this issue :smile: