Tarpaulin: Command line option to skip tests

[hawkw]

It would be nice if tarpaulin had the ability to pass a --skip argument to the test executable (like cargo test -- --skip $TEST). We have a very large set of tests which, when run with coverage, take over an hour to run (resulting in Travis killing the build) and would like to skip those tests when running tarpaulin.

Ideally, it would be great if tarpaulin could accept a -- flag like cargo test to pass any arguments to the test executable...

[xd009642]

Good idea I'll look into this. Currently, until it's implemented you can use the ignore attribute on tests to avoid running exceptionally large tests.

I'll let you know when the feature's implemented

[xd009642]

Oh you can also try the --skip-clean flag to speed up tarpaulin building your application! It's an optional flag as I can't guarantee it won't cause it to behave differently with incremental compilation but I've had no complaints yet.

[hawkw]

Yeah, --skip-clean is nice, but it's the tests that are killing us here. HPACK is really disgustingly slow when instrumented...

[xd009642]

Fair enough, I am working on the feature tonight though just suggesting --skip-clean as a temporary solution in the meantime.

Out of curiosity do you have an open repo just so I can get an idea of the source code volume, test amount etc? Maybe also use it for performance testing on my own machine to get an idea of how I'm improving/regressing performance. From users I've talked to it sounds like you're having the worst performance issues so it could be a good benchmark

[xd009642]

As suggested if you do cargo tarpaulin -- <args>... the following args are forwarded to the test executable. Tested this one some of my projects and it skips tests fine :+1: . Just pushing release 0.4.2 with this new feature

[hawkw]

Thanks, this is excellent! The repo is carllerche/h2, and here is a CI build that was killed for running too long. I believe the HPACK tests are being generated programmatically.

[xd009642]

No problem, happy to help!

Also, I'm going to add a feature to disable counting the hits with line coverage meaning that once an instrumentation point is hit it will be disabled. That should dramatically speed things up for people who don't care about the hits metric. It'll be in the next release, whenever that may be. I've logged it in issue #38 if you want to watch for when it's implemented!