Tarpaulin: Missing coverage for logical lines split on multiple lines

[rubik]

System: Linux laptop 5.3.5-arch1-1-ARCH #1 SMP PREEMPT Mon Oct 7 19:03:08 UTC 2019 x86_64 GNU/Linux
Rust version: rustc 1.38.0 (625451e37 2019-09-23)
cargo-tarpaulin version: 0.9.0

It seems that Tarpaulin has the most trouble when logical lines are split across multiple lines.

I have a few examples of this behavior. I maintain rubik/orizuru, and I'm referring to the code at commit https://github.com/rubik/orizuru/tree/6a569c574137549af0d2c9148be694e2254c175c.

It's easier to see Tarpaulin's output on Coveralls, but I'll attach the raw output at the bottom of this post. On Coveralls, we can see that lines 32, 122, 126, 127, 130, 132, 133, 150, 165 of src/consumer.rs are marked as uncovered but they are obviously covered. In src/gc.rs, the same thing is true for line 36, while on the other hand line 37 is wrongly marked as covered, when it should be line 38.

Here is the raw output from cargo tarpaulin -v --ignore-tests:

$ cargo tarpaulin --ignore-tests -v
[INFO tarpaulin] Running Tarpaulin
[INFO tarpaulin] Building project
   Compiling orizuru v0.0.1 (/home/miki/exp/orizuru)
    Finished dev [unoptimized + debuginfo] target(s) in 4.62s
[DEBUG tarpaulin] Processing Target(test: producer)
[INFO tarpaulin] Launching test
[INFO tarpaulin] running /home/miki/exp/orizuru/target/debug/deps/producer-2ef6002cc111b285

running 1 test
test producer_can_enqueue ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

[DEBUG tarpaulin] Processing Target(test: consumer)
[INFO tarpaulin] Launching test
[INFO tarpaulin] running /home/miki/exp/orizuru/target/debug/deps/consumer-fe2596306e0a75d1

running 9 tests
test no_heartbeat ... ok
test register ... ok
test one_heartbeat ... ok
test acked_are_released ... ok
test decodes_job ... ok
test can_be_stopped ... ok
test unacked_to_unack_queue ... ok
test rejected_to_unack_queue ... ok
test multiple_heartbeat ... ok

test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

[DEBUG tarpaulin] Processing Target(test: gc)
[INFO tarpaulin] Launching test
[INFO tarpaulin] running /home/miki/exp/orizuru/target/debug/deps/gc-421ed796d3795885

running 5 tests
test collect_one_runs_with_no_jobs ... ok
test collect_noop_with_no_consumers ... ok
test collect_runs_with_a_consumer_and_no_jobs ... ok
test collect_one_runs_with_some_jobs ... ok
test collect_runs_with_a_consumer_and_some_jobs ... ok

test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

[DEBUG tarpaulin] Processing Target(test: test_utils)
[INFO tarpaulin] Launching test
[INFO tarpaulin] running /home/miki/exp/orizuru/target/debug/deps/test_utils-7a56f348bdb14280

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

[DEBUG tarpaulin] Processing Target(lib)
[INFO tarpaulin] Launching test
[INFO tarpaulin] running /home/miki/exp/orizuru/target/debug/deps/orizuru-7134647b58b7d504

running 4 tests
test message::tests::cant_decode_if_not_string ... ok
test message::tests::cant_decode_if_not_msgpack ... ok
test message::tests::payload_field_is_accessible ... ok
test message::tests::message_field_is_accessible ... ok

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

[INFO tarpaulin] Coverage Results:
|| Uncovered Lines:
|| src/consumer.rs: 32, 119, 122, 126-127, 130, 132-133, 150, 157-159, 165, 167-168
|| src/gc.rs: 36, 38-39
|| src/message.rs: 126-127
|| Tested/Total Lines:
|| src/consumer.rs: 62/77
|| src/gc.rs: 25/28
|| src/message.rs: 28/30
|| src/producer.rs: 8/8
|| 
86.01% coverage, 123/143 lines covered

[jamestwebber]

I've noticed the same issue–I'm trying to keep my code readable and nicely formatted but tarpaulin does not like it.

On the other hand, it doesn't make a ton of sense to count a statement like that as multiple lines for the purposes of coverage. I wonder if a simple-ish (maybe not so simple) solution would be to min-ify the code before measuring coverage? e.g.:

1. collapse all multi-line statements into one-liners, keeping track of the line number transformation
2. run tarpaulin and get coverage info for minified code
3. use the inverse line-number mapping for display purposes

I would maybe use the minified version for the purpose of computing coverage percentage, but opinions might vary there.

[rubik]

@jamestwebber In static analysis there is a concept of "logical line of code", as opposed to the physical source line of code. The latter is simply what is present in the file, whereas the logical one measures the number of executable statements.

You can see some examples on Wikipedia; they are in C but they easily translate to Rust:
https://en.wikipedia.org/wiki/Source_lines_of_code#Measurement_methods

I suppose Tarpaulin could adopt a similar approach. It's very frequently used by static analysis tools.

[haukened]

So no movement on this? The majority of my new lib has multi line statements because:

let local_path = String::from_utf8(
    Command::new("which")
        .arg("keybase")
        .output()
        .expect("which is not installed")
        .stdout,
    )
    .expect("Output not in UTF-8");

Is just so much more readable than the one-line marathon. (Especially when you’re working in vim)

Edited for real world example

[xd009642]

No movement of this, I'm trying to figure out an issue on selecting the right lines for breakpoints that solves this. Though currently I've only managed to do trial and error switching between multiple potential breakpoints and finding there is a correct solution and an incorrect solution. Need to figure out how to do this analytically and change tarpaulin to match that behaviour

[xfix]

This happens for me. The project source code is at https://gitlab.com/pastebinrun/pastebinrun.

Somehow format! is reachable, but concat! is not. CONTENT_SECURITY_POLICY is a constant.

[xd009642]

Hmm I'd have to see how concat is implemented, if it emits no runtime instructions it should probably be filtered out of results..

[xd009642]

Also is concat needed, I think if you remove it and the comma at the end of each arg it's a valid multiline string in rust (though I might be getting confused with C++

[xfix]

It's not needed, but it's going to be a ridiculously long string without it. All what concat! does is concatenate string into a literal, for instance concat!("a", "b") gives "ab". This is a built-in Rust macro. You cannot concatenate strings by typing "a" "b" in Rust either.

I cannot have newlines in content-security-policy either (this is a HTTP header, cannot really have newlines in those).

[orhun]

I think I have the same issue at this report:
https://codecov.io/gh/orhun/kmon/src/master/src/kernel/log.rs#L25

[rye]

FYI, concat! is a compiler built-in, and concatenates literals, yielding a &'static str.

[Razican]

Some other examples:

assert_eq! in its own line:

.split() or .collect() in its own lines:

match self and multiple line match left side:

[jstasiak]

Similar, with a plain multi-line assignment:

The full report with the source code: https://coveralls.io/builds/28315588/source?filename=src/sounddat.rs#L58

[DavidBM]

I have the same problem. Seems that the lines that can be optimized away by the compiler (because they have static values) are not counted. See the image:

[xd009642]

I've popped the issue on the new mega issue for all missing coverage etc so I'll close this one in favour of that one https://github.com/xd009642/tarpaulin/issues/351

[Luro02]

Here is another example https://codecov.io/gh/sile/hls_m3u8/pull/44/src/src/types/byte_range.rs?before=src/types/byte_range.rs