Tarpaulin: Misses in coverage

[xd009642]

Mention any false positives or false negatives in code coverage in this issue with an example where it occurred and it will be added to the list!

• [ ] An else only line https://codecov.io/gh/Smithay/wayland-rs/src/master/wayland-commons/src/socket.rs#L33
• [x] Single logical lines split across multiple ones see #270 for past discussion
• [ ] match statement with enum or *self https://github.com/xd009642/tarpaulin/issues/258
• [ ] macros in function bodies https://codecov.io/gh/simd-lite/simdjson-rs/src/master/src/stage2.rs#L133
• [ ] static values can be uncovered https://codecov.io/gh/simd-lite/simdjson-rs/src/master/src/stage2.rs#L320
• [x] const values can be uncovered https://codecov.io/gh/simd-lite/simdjson-rs/src/master/src/stage2.rs#L320
• [ ] unsafe blocks marked as uncovered https://codecov.io/gh/simd-lite/simdjson-rs/src/master/src/lib.rs#L285
• [x] const initialisations aren't covered.
• [ ] general chained method calls (sometimes can differ in free functions versus methods see https://github.com/xd009642/tarpaulin/issues/238 )
• [ ] PhantomData
• [ ] Inconsistent coverage with macros like assert_eq!
• [ ] Not sure how well inline asm! works
• [ ] Code disabled by feature configs included
• [ ] struct fields in matches with enum structs Foo::X{ref bar} => {} bar can register as uncovered
• [ ] trait's ignore methods without default impl

[jamestwebber]

It looks like my coverage report is counting the lines _in_ tests as part of coverage (including one miss, somehow). See e.g. here if you scroll down to the lines 500+. Not sure whether that's intentional or not but seems a bit odd (and the miss is a false negative I guess)

[xd009642]

There's some weirdness with macros, I need to figure out the best way to handle them. Also if you want to exclude test code you can use --exclude-tests and all lines in anything with the #[test] attribute or #[cfg(test)] will be removed from the results

[pwoolcoc]

Just curious, why is --ignore-tests not the default? Are people really writing tests for their tests? I can't imagine a situation where I'd want my test code to be counted as part of the coverable source code

[xd009642]

Honestly just because as a feature it came later and no ones asked for the default to be switched around, I've got no opposition to switching the default behaviour. I also definitely don't think people are writing tests for their tests (or at least I hope not...)

The main use I can see is for integration tests that involve things like file or network IO some people might want to make sure their tests are actually executing the code as thoroughly as they expect

[mathstuf]

I think this is another instance of the "split logical lines", but the attached report for serde.rs shows some more uncovered chain calls and unexpected (inlined?) not-to-be-covered lines at the very bottom.

tarpaulin-report.html.gz

[Kixiron]

A variable declaration with the left-hand expression on a different line is counted as untested, see this report and this one. Additionally, conditionally compiled code that is not part of the current feature set seems to flag a false negative as well, as shown here

[xd009642]

oh the feature flag is something I've definitely missed! Although I suppose I wonder if that should be filtered out by default, as ideally you should test all your features thoroughly and it is possible to organise different runs in a batch with different feature settings... Maybe I'll add a flag to add/remove them.

Also for the assign expressions thinking of something that maps to logical lines of code for solving that class of issues once and for all

[jtempest]

It looks like the inline, associated functions returning &'static str in this report only have the function signature marked and not the body that returns the string.

Also, thanks for tarpaulin, it's really handy and easy to set up :)

[Recmo]

Are people really writing tests for their tests?

I also definitely don't think people are writing tests for their tests (or at least I hope not...)

I can think of two examples:

• Implementations of Quickcheck / Proptest test case generation. These implementations are sometimes complicated enough to warrant some testing.
• Reference functions. Sometimes I write a slow-but-simple textbook test version of an algorithm (for example an FFT). I then use this to compare with the fast-but-complicated real implementation. But I first add a few tests to make sure the reference is correct.

But I don't care much about coverage in both cases, --ignore-test seems like a good default.

[jonhoo]

Here's another coverage result that seems to have some odd lines marked as not covered:
https://codecov.io/gh/jonhoo/openssh-rs/src/9c82a1b11668033e2361d04f821786a41ce46615/src/lib.rs

The repository is over here: https://github.com/jonhoo/openssh-rs/

[jonhoo]

I believe all of the misses here are erroneous:
https://codecov.io/gh/jonhoo/cliff/src/811bace3d7b1d7c64bdd316ab59d4e355e3d163c/src/lib.rs

[xd009642]

GitHub API: Forbidden

:disappointed:

[jamestwebber]

^ I think that's just GitHub having issues today

[FeuRenard]

I have 2 covered lines containing doc comments using tarpaulin 0.11.1:
https://codecov.io/gh/FeuRenard/mygpoclient-rs/src/c5e4d61f21b548f2efd03429a4d35f62441c42c2/src/episode.rs

[kristiannotari]

False negatives here.

I have setup tarpaulin recently and tested it on my code and uploaded test coverage to Codecov. This is the actual report on Codecov (no custom formatting).

I have found that:

• some write!( ... ) macros arguments, when those are split on multiple lines are flagged as misses
• some assert!( ... ) macros arguments, when those are split on multiple lines are flagged as misses
• some right hand sides of let ... assignments, when on other lines (even if self is on the same line of the let ...) are flagged as misses

Here's my travis-ci build configuration, if needed:

configuration

```
language: rust
sudo: required # required for some configurations

tarpaulin has only been tested on bionic and trusty other distros may have issues

dist: bionic
addons:
apt:
packages:
- libssl-dev
rust:
- stable
- beta
- nightly
jobs:
allow_failures:
- rust: nightly
fast_finish: true
cache: cargo
before_script: |
if [[ "$TRAVIS_RUST_VERSION" == stable ]]; then
cargo install cargo-tarpaulin
fi
after_success: |
if [[ "$TRAVIS_RUST_VERSION" == stable ]]; then
# Get coverage report and upload it for codecov.io.
cargo tarpaulin --out Xml
bash <(curl -s https://codecov.io/bash)
fi
```

[khoek]

I have a whole bunch of match self.blah { <newline> }s where the line containing the match itself is marked not-covered.

[CAD97]

In a fn foo(&mut self, ...) -> &mut Self { ...; self }, the final line returning self is always listed as uncovered.

It seems to be any trailing expression that is just a borrow from self: report

[CAD97]

The item headers for impls in statement position are incorrectly marked as uncovered. Lifting the impl blocks to item position properly does not consider these lines as uncovered. [example]

[mathstuf]

@CAD97 That link is behind a login wall. Not everyone has already given access to codecov.io.

[CAD97]

@mathstuf ah, sorry, I was unaware that that link required a login. (It should be public, the repository in question is public.) (If it was just that the second link was blank, it's because I had forgotten to pin it to a specific commit, which is fixed now.) Here's the linked snippets inline:

+   pub fn builder(&mut self) -> &mut Builder {
-       &mut self.cache
    }

+    pub fn add(&mut self, element: impl Into<NodeOrToken<Arc<Node>, Arc<Token>>) -> &mut Self {
+       self.children.push(element.into());
-       self
    }

```diff

• fn visit_seq(self, mut seq: Seq) -> Result
  where
  Seq: SeqAccess<'de>,
  {
• if seq.size_hint().is_some() {
  struct SeqAccessExactSizeIterator<'a, 'de, Seq: SeqAccess<'de>>(
  &'a mut Builder,
  Seq,
  PhantomData<&'de ()>,
  );
• impl<'de, Seq: SeqAccess<'de>> Iterator for SeqAccessExactSizeIterator<'_, 'de, Seq> {
• type Item = Result, Arc>, Seq::Error>;
• fn next(&mut self) -> Option {
• self.1.next_element_seed(ElementSeed(self.0)).transpose()
  }

[alexschrod]

These that I'm seeing seem almost arbitrary; the function signature itself is marked as missing coverage, even though the function itself clearly is covered[1], and I don't have the same problem for the exact same function implemented elsewhere[2] and which is tested in the exact same manner; a similar issue can be seen with [3] vs [4].

[icefoxen]

Pretty sure I found a false negative, the latest released cargo-tarpaulin (0.13.3) marks two wrapping_mul()'s as unexecuted in the middle of a bunch of other code that is green:

• https://hg.sr.ht/~icefox/oorandom/browse/src/lib.rs?rev=default#L91
• https://hg.sr.ht/~icefox/oorandom/browse/src/lib.rs?rev=default#L214

Report: https://alopex.li/temp/tarpaulin-oorandom.html

Thank you!

[fadeevab]

Simply, what I noticed is that once I get the warning "Instrumentation address clash, ignoring 0x40001234", the lines around of that "address clash" are marked as _uncovered_. So have a simple new method with a straightforward implementation it's marked as uncovered. Therefore the coverage is reported with lower percentage.

[xd009642]

I'll try and solve that instrumentation clash warning better, it's because instrumentation points to lines are a many to many relationship and I worked to simplify it by making it a many to one (many traces to one line).

[xd009642]

Also @fadeevab do you have a public project with the instrumentation clash issue I can test with? Just to save time in hunting down or creating a project where that happens

[fadeevab]

@xd009642 Yeap, https://github.com/fadeevab/cocoon

Thanks!

[xd009642]

Hmm it didn't happen on my machine... I've pushed a branch called permissive_clash_handling if you want to try it out yourself. In the meantime I'll keep looking to recreate the clash issue with a project on my machine

[fadeevab]

Not sure whether it helps, but this is my toolchain:

Default host: x86_64-unknown-linux-gnu

stable-x86_64-unknown-linux-gnu (default)
rustc 1.44.1 (c7087fe00 2020-06-17)

[TheButlah]

Bunch of false negatives in single logical line broken into multiple lines (yellow is uncovered, green is covered):

[scouten]

I filed #496 which appears to be another case of match and function arguments split across multiple lines as described here. rsgit is a public project; feel free to submit a PR (similar to my https://github.com/rust-git/rsgit/pull/56) if you wish to use that for testing.

[xd009642]

Yeah I've got a plan for the chained method calls and some of the match ones. Basically falling back on the logical lines technique, just need to figure out the edge cases in the language etc.

All these issues really come around because there's a lot of noise and instrumentation points that aren't reachable (you need to instrument somewhere else). Would be nice to figure out how to tackle the noise in the DWARF tables and get around it but logical lines based grouping isn't insurmountable to implement

[scouten]

Looks like Rust 1.45.0 added some new cases. :-( Please see https://github.com/rust-git/rsgit/pull/59 for an example.

[stouset]

Several false negatives here:

• unsafe line not considered covered
• straightforward call to mem::size_of()
• lines that are just casts

All of these examples have tests directly testing them at the bottom of the file.

[khvorov45]

String literals are sometimes counted as "not covered". An example here.

I created a small project that identifies 2 cases here:

• A struct method creating a multi-line string variable.
• Async struct method calling an async function that takes a string.

[MarkMcCaskey]

I noticed some strange behavior in the coverage of a diff https://codecov.io/gh/wasmerio/wasmer/pull/1566/diff, specifically:

• constant in if else chain marked as not hit
• unconditional call to clone marked as not hit
• use of type { field, ..type::default() } marked as not hit
• part of if let binding marked as not hit
• creation of variable marked as not hit
• use of that variable being marked as not hit

Perhaps this is a configuration issue on my side; a lot of these issues look like they were caused by optimization.

Thanks!

[jhpratt]

Looks like one of the largest instances of false negatives in the code for time is a parameter or the initialization of a struct (when a variable with that name already exists, typically). Other situations that sometimes aren't covered is when an exprssion is split over multiple lines.

impl Distribution<Duration> for Standard {
    fn sample<R: Rng + ?Sized>(&self, rng: &mut R) -> Duration {
        let seconds = Standard.sample(rng);
        Duration::new(
            seconds, // This line alone isn't covered.
            seconds.signum() as i32 * rng.gen_range(0, 1_000_000_000),
        )
    }
}

pub const fn try_from_hms(
    hour: u8,
    minute: u8,
    second: u8,
) -> Result<Self, error::ComponentRange> {
    ensure_value_in_range!(hour in 0 => 23);
    ensure_value_in_range!(minute in 0 => 59);
    ensure_value_in_range!(second in 0 => 59);
    Ok(Self {
        hour, // this
        minute, // this
        second, // and this line are not covered
        nanosecond: 0,
    })
}

pub(crate) fn from_iso_ywd_unchecked(year: i32, week: u8, weekday: Weekday) -> crate::Date {
    let ordinal = week as u16 * 7 + weekday.iso_weekday_number() as u16
        - (Self::from_yo_unchecked(year, 4)
            .weekday() // this
            .iso_weekday_number() as u16 // this
            + 3); // and this line are uncovered, despite the first chunk of the expression being covered

    if ordinal < 1 {
        return Self::from_yo_unchecked(year - 1, ordinal + days_in_year(year - 1));
    }

    let days_in_cur_year = days_in_year(year);
    if ordinal > days_in_cur_year {
        Self::from_yo_unchecked(year + 1, ordinal - days_in_cur_year)
    } else {
        Self::from_yo_unchecked(year, ordinal)
    }
}

Having no idea how tarpaulin actually works aside from it putting "markers" to detect when certain code is run, I think marking an entire expression as covered is what would be necessary. The struct initialization is almost certainly optimized out, even in debug mode (especially in the example provided), so any marker would be futile.

[Ch00k]

In my case the missed lines are the arguments in a function call.
The test and the missed lines in coverage report.

[jhpratt]

Here's an interesting one! Should a branch of a match arm that consists solely of unreachable!() be considered covered? In my opinion it should be ignored.

[kennetpostigo]

I don't seem to get any coverage on traits even when directly testing the trait methods

[adder46]

Here are my two cents.

[johnoneil]

I'm seeing the issue reported here: https://github.com/xd009642/tarpaulin/issues/270 which redirects to this issue.