Coverlet: Coverage Issue - New Using + Async/Await + ConfigureAwait

[abbotware]

found during #649, this combination of code seems to have incorrect reports

        public async Task<ActionResult<IEnumerable<SystemUserItem>>> SystemUsers(CancellationToken ct)
        {
            using var data = this.factory.CreateReadOnly<SystemUserItem>();

            var list = await data.AllAsync(ct)
                .ConfigureAwait(false);

            return this.Ok(list);
        }

here is the final report output:

here is the diff of the coverage xml:

left side = unit test - code is not executed
right side = integration test - code is there doesnt appear to be any branches - all code execute under a single call

[MarcoRossignoli]

Thank's for reporting this

[moikot]

I've got the same issue. Using version 2.8.0 of coverlet.msbuild. But I don't have ConfigureAwait.

[MarcoRossignoli]

@moikot can provide a "repro"?
I mean a small project affected by the issue?

[moikot]

Please find it here https://github.com/moikot/async-await-coverage

[MarcoRossignoli]

Great thank's I'll take a look asap.

[MarcusOtter]

Having the same problem with my repo. Coverage dropped from 100% to 93% without any changes in source code.

See https://coveralls.io/builds/28220011 (repo: https://github.com/LeMorrow/APOD.Net/tree/develop)
Not sure exactly what information you need from me, let me know how I can help :)

[MarcoRossignoli]

@LeMorrow can you isolate a small repro the @moikot one(where I'm investigating) seem without ConfigureAwait.

[MarcoRossignoli]

Guys I found the issue...I'PR a fix asap.
Are two different issue, one related to netstandard project and the other related on new async machine state code.

[MarcoRossignoli]

Hi devs please can you try the master version using our nightly build and tell me if works as expected? https://github.com/tonerdo/coverlet/blob/master/Documentation/ConsumeNightlyBuild.md tomorrow fix PR will land inside packages!

Thank's a lot for helping coverlet!

[moikot]

Thank you for the fix! I will be able to try in a couple of days.

[moikot]

@MarcoRossignoli I tested my project using master, it seems like the coverage is calculated properly. Thank you for your prompt fix!

[MarcoRossignoli]

Glad to hear thank you very much for the test and for using coverlet!

[MarcusOtter]

Sorry for the late reply. I'm still seeing this issue after trying 2.8.6-g8cb1ebc0da.

This is in the context of a GitHub Actions workflow (specifically, this one)

dotnet add src/ApodTests/ApodTests.csproj package coverlet.msbuild --version 2.8.6-g8cb1ebc0da --source https://www.myget.org/F/coverlet-dev/api/v3/index.json

The coverage remained the same for the commit, as you can see here on coveralls: https://coveralls.io/builds/28326307.

An example of where this error still remains is here: https://coveralls.io/builds/28326307/source?filename=src/Apod/Logic/Errors/ErrorHandler.cs#L71

Again, this codebase was previously at 100% (coveralls report from December 17 - 2019). Not one line of source code changed between now and then.

Edit: Perhaps I'm installing the package wrong? Is the NuGet.Config required even when I install with --version? I'll try adding the config regardless.
Edit 2: Not getting it to work with the NuGet.Config either. See workflow and results here.

[MarcoRossignoli]

@LeMorrow cloned and debugged your sample, I cannot push a sample PR because I don't have permission on worfklow

 ! [remote rejected] coverlettest -> coverlettest (refusing to allow an OAuth App to create or update workflow `.github/workflows/build.yml` without `workflow` scope)
error: failed to push some refs to 'https://github.com/MarcoRossignoli/APOD.Net.git'

BTW this is my version of ApodTests.csproj

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netcoreapp3.0</TargetFramework>

    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <ItemGroup>
    <!-- To substitute int future with stable version this version has got a path to load new version of collectors -->
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.5.0-preview-20200116-01" />
    <PackageReference Include="Moq" Version="4.13.1" />
    <PackageReference Include="xunit" Version="2.4.1" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.4.1">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <!-- To substitute with new version of nightly tomorrow https://www.myget.org/feed/coverlet-dev/package/nuget/coverlet.collector -->
    <PackageReference Include="coverlet.collector" Version="1.2.20-ga2b0373d60">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\Apod\Apod.csproj" />
  </ItemGroup>

</Project>

On root solution add a file named Nuget.Config

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <clear />
    <!-- Coverlet nightly build feed -->
    <add key="coverletNightly" value="https://www.myget.org/F/coverlet-dev/api/v3/index.json" /> 
    <!-- Defaul nuget feed -->
    <add key="nuget" value="https://api.nuget.org/v3/index.json" /> 
    <!-- Add all other needed feed -->
  </packageSources>
</configuration>

Add a runsettings file under ApodTests folder like

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat code coverage">
        <Configuration>
          <Format>lcov</Format>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

Final patch to build.yml

...      
    - name: Run tests
      run: dotnet test src/ApodTests/ApodTests.csproj --collect:"XPlat Code Coverage" --settings ./src/ApodTests/runsettings.xml
...

In this way you're using collectors and not msbuild version, collectors are the best way to consume coverlet because integrated in vstest runtime.

Current result

[MarcusOtter]

@MarcoRossignoli Thank you, I will try using collectors and get back to you within a few days.

[MarcusOtter]

@MarcoRossignoli Hello, the problem for me with using collectors is that I can't get a predicatble path to the coverage.info (unless I'm missing something). They end up in coverage/very-long-hash-number/coverage.info. Example: coverage/d0d865aa-43cb-4b9c-8ca6-bf24c61d84fd/coverage.info.
This is after adding a ResultsDirectory to the .runsettings.

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
  <RunConfiguration>
    <ResultsDirectory>.\coverage\</ResultsDirectory>
  </RunConfiguration>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat code coverage">
        <Configuration>
          <Format>lcov</Format>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

To use https://github.com/coverallsapp/github-action I need a full path-to-lcov which does not support wildcards or glob patterns, as they are using fs.readFileSync() seen here.

So to upload my coverage report with my CI solution, I need the full path to the lcov.

[MarcoRossignoli]

Ok you can use msbuild version but you could hit https://github.com/tonerdo/coverlet/blob/master/Documentation/KnowIssues.md#1-vstest-stops-process-execution-earlydotnet-test
So you could add a simple script that move file to fixed folder before load to coveralls and open an issue on that repo to support glob pattern.

[MarcusOtter]

Adding a script that moves the files was a good call, I got it working and the problems with new using and async/await with ConfigureAwait disappeared :)

For future reference if anyone encounters the problem, take a look at the relevant files below.

build.yml (GitHub Workflow)

- name: Install coverlet
  run: dotnet add src/ApodTests/ApodTests.csproj package coverlet.collector --version 1.2.25-g074a201a91 --source https://www.myget.org/F/coverlet-dev/api/v3/index.json

- name: Run tests
  run: dotnet test src/ApodTests/ApodTests.csproj --collect:"XPlat Code Coverage" --settings ./src/ApodTests/.runsettings

- name: Move test coverage results UNIX
  if: matrix.os == 'ubuntu-latest' || matrix.os == 'macos-latest'
  run: mv src/ApodTests/coverage/**/*.info .

- name: Move test coverage results WINDOWS
  if: matrix.os == 'windows-latest'
  run: move src\ApodTests\coverage\**\*.info .

- name: Coveralls GitHub Action
  uses: coverallsapp/[email protected]
  with: 
    github-token: ${{ secrets.GITHUB_TOKEN }}
    path-to-lcov: coverage.info

ApodTests.csproj

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netcoreapp3.0</TargetFramework>

    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.4.0" />
    <PackageReference Include="Moq" Version="4.13.1" />
    <PackageReference Include="xunit" Version="2.4.1" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.4.1">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="coverlet.collector" Version="1.2.25-g074a201a91">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\Apod\Apod.csproj" />
  </ItemGroup>

</Project>

.runsettings in the root test folder (with ApodTests.csproj)

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
  <RunConfiguration>
    <ResultsDirectory>coverage</ResultsDirectory>
  </RunConfiguration>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat code coverage">
        <Configuration>
          <Format>lcov</Format>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

And finally, the NuGet.Config in the root solution folder

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <clear />
    <!-- Coverlet nightly build feed -->
    <add key="coverletNightly" value="https://www.myget.org/F/coverlet-dev/api/v3/index.json" /> 
    <!-- Default nuget feed -->
    <add key="nuget" value="https://api.nuget.org/v3/index.json" /> 
    <!-- Add all other needed feed -->
  </packageSources>
</configuration>

I'm back to 100% :)
Thank you for the help @MarcoRossignoli!

[MarcoRossignoli]

Glad to hear...thank you for the documentation

[abbotware]

@MarcoRossignoli - when will this be released?

[MarcoRossignoli]

Usually we release every three months so new version is expected at the end of March. We do intermediate release only for serious bug for Instance when fails to run or for security reasons. You can use nightly feed until new release!