Coverlet: dotnet test with coverlet failed when UseSourceLink

[iron9light]

SonarSource/sonar-scanner-msbuild#756
My repo: https://github.com/iron9light/HOCON.Json
It CI with Azure pipelines.
When I add /p:UseSourceLink=true for dotnet test, it will not generate valid coverage file (opencover format) (sonar cannot get coverage info from the output coverage file).
When removed /p:UseSourceLink=true everything works fine.
coverlet.msbuild version: 2.6.3

[MarcoRossignoli]

it will not generate valid coverage file

Can you provide more details?What does "non valid" mean?

[MarcoRossignoli]

I cannot repro

dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   2.2.300
 Commit:    73efd5bd87

 dotnet test Hocon.Json.sln --logger trx --configuration Release /p:CollectCoverage=true "/p:CoverletOutputFormat=\"opencover,cobertura\"" /p:UseSourceLink=true /p:Exclude=[xunit.*]*

Calculating coverage result...
  Generating report 'C:\git\HOCON.Json\src\Hocon.Json\coverage.opencover.xml'
  Generating report 'C:\git\HOCON.Json\src\Hocon.Json\coverage.cobertura.xml'

+--------+------+--------+--------+
| Module | Line | Branch | Method |
+--------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 100% | 100%   | 100%   |
+---------+------+--------+--------+
| Average | 8%   | 8%     | 8%     |
+---------+------+--------+--------+


Calculating coverage result...
  Generating report 'C:\git\HOCON.Json\src\Hocon.Json\coverage.opencover.xml'
  Generating report 'C:\git\HOCON.Json\src\Hocon.Json\coverage.cobertura.xml'

+--------+------+--------+--------+
| Module | Line | Branch | Method |
+--------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 100% | 100%   | 100%   |
+---------+------+--------+--------+
| Average | 8%   | 8%     | 8%     |
+---------+------+--------+--------+

Test run for C:\git\HOCON.Json\src\Hocon.Json.Tests\bin\Release\netcoreapp2.2\Hocon.Json.Tests.dll(.NETCoreApp,Version=v2.2)
Microsoft (R) Test Execution Command Line Tool Version 16.1.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Results File: C:\git\HOCON.Json\src\Hocon.Json.Tests\TestResults\Marco_WS-3_2019-07-03_09_53_36.trx

Test Run Successful.
Total tests: 6
     Passed: 6
 Total time: 1,9367 Seconds

Calculating coverage result...
  Generating report 'C:\git\HOCON.Json\src\Hocon.Json.Tests\coverage.opencover.xml'
  Generating report 'C:\git\HOCON.Json\src\Hocon.Json.Tests\coverage.cobertura.xml'

+------------+--------+--------+--------+
| Module     | Line   | Branch | Method |
+------------+--------+--------+--------+
| Hocon.Json | 78,12% | 73,91% | 83,33% |
+------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 78,12% | 73,91% | 83,33% |
+---------+--------+--------+--------+
| Average | 78,12% | 73,91% | 83,33% |
+---------+--------+--------+--------+

[iron9light]

If you read SonarSource/sonar-scanner-msbuild#756 . You will found that, I got similar output as yours.
The problem is, if I enabled /p:UseSourceLink=true, the sonarcloud will not show the coverage info.
But if I disabled it, I can see the coverage info in sonarcloud.
I will change my azure pipelines and output the content of the coverage file for both cases.

[MarcoRossignoli]

It's a bit different you got strange average https://github.com/SonarSource/sonar-scanner-msbuild/issues/756#issuecomment-507647800 right?

[iron9light]

w/ UseSourceLink (see "Print opencover")
https://iron9light.visualstudio.com/github/_build/results?buildId=106

      <Files>
        <File uid="1" fullPath="https://raw.githubusercontent.com/iron9light/HOCON.Json/654d4ea8ec524f72027e2b2d324aad9acf80b710/src/Hocon.Json/obj/Release/netstandard2.0/Hocon.Json.Version.cs" />
        <File uid="2" fullPath="https://raw.githubusercontent.com/iron9light/HOCON.Json/654d4ea8ec524f72027e2b2d324aad9acf80b710/src/Hocon.Json/HoconJsonExtentions.cs" />
      </Files>

w/o UseSourceLink
https://iron9light.visualstudio.com/github/_build/results?buildId=107

      <Files>
        <File uid="1" fullPath="D:\a\1\s\src\Hocon.Json\obj\Release\netstandard2.0\Hocon.Json.Version.cs" />
        <File uid="2" fullPath="D:\a\1\s\src\Hocon.Json\HoconJsonExtentions.cs" />
      </Files>

@MarcoRossignoli Is the result correct?
@duncanp-sonar I think it's not a issue of coverlet. It just becouse sonar don't support sourcelink file path.

[MarcoRossignoli]

The strange thing I see is that obj/Release/netstandard2.0/Hocon.Json.Version.cs that I don't find on your repo.
The other one is present https://raw.githubusercontent.com/iron9light/HOCON.Json/654d4ea8ec524f72027e2b2d324aad9acf80b710/src/Hocon.Json/HoconJsonExtentions.cs

[MarcoRossignoli]

@AArnott do you have any idea?That file seem nbgv tmp file generated.
Have you never seen something like that?

[iron9light]

The version file is generated by Nerdbank.GitVersioning.

[duncanp-sonar]

@iron9light I've removed the Sonar tasks from my copy of your build .yml and done a couple more tests. The links below are to the Azure DevOps build logs.
1) with UseSourceLink -> fails with

Line 1600
Calculating coverage result...
C:\Users\VssAdministrator\.nuget\packages\coverlet.msbuild\2.6.2\build\coverlet.msbuild.targets(41,5): error : The given key '' was not present in the dictionary. [d:\a\1\s\src\Hocon.Json.Tests\Hocon.Json.Tests.csproj]

2) without UseSourceLink -> succeeds.

[MarcoRossignoli]

@duncanp-sonar that error seem https://github.com/tonerdo/coverlet/issues/418 but I cannot repro on my local after cloning HOCON.Json repo.
Seem that the issue here are 2
1) You get exception that I/we cannot repro
2) Coverlet generates nbgv autogenerated file.

[duncanp-sonar]

Hi @MarcoRossignoli. Yep, looks like there are a couple of different issues.

Re (1): those failing builds were on MS-hosted agents (Hosted Windows 2019 with VS2019).
If it would help, I can make you an admin on the Azure DevOps project I created to do the testing. If that would be useful let me know your Azure DevOps account and I'll send you an invite.

Re: #418: I did point @iron9light at this issue a couple of days ago. However, I saw that the bug was closed and available in your nightly build for 2.6.1, and @iron9light is using 2.6.2. Which version is the bug fix in?

[iron9light]

I'm using Hosted Windows 2019 with VS2019 for my azure pipelines.
I was using 2.6.2 and now 2.6.3, but I do not get an error for dotnet test.

[MarcoRossignoli]

@iron9light is this issue active yet?

[vchirikov]

Yep, I have same situation.

[MarcoRossignoli]

@vchirikov one idea...can you try to exclude autogenerated file with filter?
https://github.com/tonerdo/coverlet/issues/482#issuecomment-508080869
https://github.com/tonerdo/coverlet/blob/master/Documentation/MSBuildIntegration.md#source-files

for instance /p:ExcludeByFile=\"**/*Json.Version.cs\"

[vchirikov]

@MarcoRossignoli yes, with your ExcludeByFile works fine. Maybe it should be in readme about compability with Nerdbank.GitVersioning.

[MarcoRossignoli]

@vchirikov it's not the "only" compatibility issue coverlet has got with other components and environment maybe we could add to "know issue" https://github.com/tonerdo/coverlet/blob/master/Documentation/KnowIssues.md

[MarcoRossignoli]

@iron9light @duncanp-sonar can you tell me if issue is solved?

[MarcoRossignoli]

added to know issue https://github.com/tonerdo/coverlet/blob/master/Documentation/KnowIssues.md#3-nerdbankgitversioning-and-pusesourcelinktrue-option

[MarcoRossignoli]

close for stale conversation, feel free to re-open if needed.