Coverlet: Branch coverage with async calls

[coryflucas]

I have a project using EF Core with async methods that shows a branch at each await call. One of the branches always shows as uncovered. I've created a sample repo that repoduces the problem here. You can see the branch in the coverage report but the referenced line does not have a branch.

Here is the output of dotnet test:

$ dotnet test /p:CollectCoverage=true
Build started, please wait...
Build completed.

Test run for /Users/clucas/Source/clucas/coverlet-test/Test/bin/Debug/netcoreapp2.1/Test.dll(.NETCoreApp,Version=v2.1)
Microsoft (R) Test Execution Command Line Tool Version 15.7.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

Total tests: 1. Passed: 1. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 2.0845 Seconds

Calculating coverage result...
  Generating report '/Users/clucas/Source/clucas/coverlet-test/Test/coverage.json'

+---------+--------+--------+--------+
| Module  | Line   | Branch | Method |
+---------+--------+--------+--------+
| Example | 90.9%  | 50%    | 80%    |
+---------+--------+--------+--------+

I tired another example with await HttpClient.GetAsync(...) but that did not seem to produce the same issue so this doesn't appear to apply to all awaited calls.

[mgw854]

I've noticed the same thing. My hunch (totally a guess here, so take it for what it's worth) is that the branch inserted by the compiler for awaiting code (such that you don't actually wait if the task has already completed) is being interpreted as a branch in your code.

If this is true, I don't think it's a meaningful distinction in 99+% of code. There are only two cases where I can think of that this matters. The first is using .ConfigureAwait, which doesn't do anything if the call was never awaited (such that you can accidentally pass on the captured state to a subsequent call that doesn't use .ConfigureAwait). The other is a timing bug where you expecting the current execution to yield, but it doesn't because the task was completed.

[abe545]

We've noticed the same thing, and I put together a repro where await Task.CompletedTask causes it to report a partial branch: https://github.com/abe545/CoverletAwaitBranchCoverage/tree/master

[tonerdo]

Thanks for reporting this guys. I'm currently working on general improvements to the coverage logic so will add this to the list. Thanks for the test project @abe545

[rmlira]

+1

[sugarjig]

I'm seeing this issue still in version 2.5.1 (that's the version number in Nuget).

[MarcoRossignoli]

@sugarjig can you provide a repro?

[soljarka]

Same with 2.6.0

[MarcoRossignoli]

@soljarka can you provide a repro?

[Rjae]

I don't know if this is old news, but I think I figured out a way through. By adding a simple continuation I was able to get Coverlet to consider all branches hit. For example:

return await _service.SendAsync(command).ContinueWith(x => x.Result);

I hope that helps others.

[MarcoRossignoli]

Thanks @Rjae for the comment...can you provide a repro of the issue (I mean a piece of code without continuation that leads to invalid async branch coverage)?

[Rjae]

Sure thing bro...literally the same line without the ContinueWith. So...

return await _service.SendAsync(command);

I've used same approach in several more lines with same success.

[MarcoRossignoli]

Thanks I'll take a look asap

[Rjae]

Wow...thanks! Additional "discovery": ContinueWith does not appear to solve the coverage reporting for await async Task, only for await async Task<T>.

[Rjae]

BTW, I'm looking for a temporary ignore for these cases. I've tried following attribute-excludes: GeneratedCodeAttribute,CompilerGeneratedAttribute,AsyncStateMachineAttribute but no luck. @MarcoRossignoli if you know a way to ignore these cases from analysis I'd be interested to know.

[MarcoRossignoli]

@Rjae add CompilerGeneratedAttribute should work on next version thank's to https://github.com/tonerdo/coverlet/pull/477 but it was merged on 3 July and last verson was released 1 July.
For the moment you can try to doogfood coverlet using this guide https://github.com/tonerdo/coverlet/blob/master/Documentation/ConsumeNightlyBuild.md
When we'll release next version you can revert.
Sorry but for the moment I don't have better choice...we need to fix issue with async(some investigation but I've already an idea on fix) and do a release 😞 so for the moment you can skip using dogfooding and after using new release.
You could also try to skip using filters https://github.com/tonerdo/coverlet/blob/master/Documentation/MSBuildIntegration.md#filters

[Rjae]

@MarcoRossignoli again, thanks! Will watch for async fix.