Coverlet: Coverlet crashes VSTest

[clairernovotny]

I'm getting errors when I'm trying to use the vstest integration to coverlet.

This is with the 2.2.300 SDK.

 PS> dotnet test --no-build --no-restore --settings ..\CodeCoverage.runsettings --collect:"XPlat Code Coverage"  -- RunConfiguration.DisableAppDomain=true
Test run for D:\dev\xUnit\visualstudio.xunit\test\test.xunit.runner.visualstudio\bin\Debug\net46\test.xunit.runner.visualstudio.dll(.NETFramework,Version=v4.6)
Microsoft (R) Test Execution Command Line Tool Version 16.1.1
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...


Attachments:
  D:\dev\xUnit\visualstudio.xunit\test\test.xunit.runner.visualstudio\TestResults\730034da-8dc8-4a52-9bdf-db094514cf10\coverage.cobertura.xml
  D:\dev\xUnit\visualstudio.xunit\test\test.xunit.runner.visualstudio\TestResults\730034da-8dc8-4a52-9bdf-db094514cf10\Oren_ATHENA_2019-06-25.16_25_17.coverage

Test Run Aborted.

To repro:

• clone https://github.com/xunit/visualstudio.xunit
• checkout the coverlet branch
• from a VS Command prompt, go to the /test/test.xunit.runner.visualstudio directory
• run msbuild /restore to build the test project (and the project it's testing). You'll need the Windows/UWP 16299 SDK installed to successfully build it.
• From the same directory, run the following command dotnet test --no-build --no-restore --settings ..\CodeCoverage.runsettings --collect:"XPlat Code Coverage" -- RunConfiguration.DisableAppDomain=true

It'll start running the tests and then abort.

[MarcoRossignoli]

I cannot repro I get a different issue seem I don't find collector(in other netcoreapp projects it works) but I also get tests abort.

 dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   2.2.300
 Commit:    73efd5bd87

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.17763
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.2.300\
...
C:\git\visualstudio.xunit\test\test.xunit.runner.visualstudio (coverlet -> origin)
λ dotnet test --no-build --no-restore --settings ..\CodeCoverage.runsettings --collect:"XPlat Code Coverage" -- RunConfiguration.DisableAppDomain=true
Test run for C:\git\visualstudio.xunit\test\test.xunit.runner.visualstudio\bin\Debug\net46\test.xunit.runner.visualstudio.dll(.NETFramework,Version=v4.6)
Microsoft (R) Test Execution Command Line Tool Version 16.1.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Data collection : Could not find data collector 'XPlat code coverage'

Attachments:
  C:\git\visualstudio.xunit\test\test.xunit.runner.visualstudio\TestResults\9e99c4dc-e9c6-4329-a592-749a262b5d6c\Marco_WS-3_2019-06-26.11_47_45.coverage

Test Run Aborted.

Btw it's strange I mean if it cannot find collector should ignore coverage and go on with test...so that output seem maybe unrelated to coverlet
@vagisha-nidhi do you have any idea?

[vagisha-nidhi]

@MarcoRossignoli We set the collector path in coverlet.collector.targets which is in the build folder inside the nuget. Is it possible that since you are setting --no-build, this path is not set and hence collector is not discovered?

[MarcoRossignoli]

Mmm the same without --no-build

C:\git\visualstudio.xunit\test\test.xunit.runner.visualstudio (coverlet -> origin)
λ dotnet test --settings ..\CodeCoverage.runsettings --collect:"XPlat Code Coverage" -- RunConfiguration.DisableAppDomain=true
Test run for C:\git\visualstudio.xunit\test\test.xunit.runner.visualstudio\bin\Debug\net46\test.xunit.runner.visualstudio.dll(.NETFramework,Version=v4.6)
Microsoft (R) Test Execution Command Line Tool Version 16.1.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Data collection : Could not find data collector 'XPlat code coverage'

Attachments:
  C:\git\visualstudio.xunit\test\test.xunit.runner.visualstudio\TestResults\9ab7249b-2bd7-4906-9ae8-9a5645510777\Marco_WS-3_2019-06-26.12_55_32.coverage

Test Run Aborted.

EDIT: I've logged build and I found empty VSTestTestAdapterPath

[vagisha-nidhi]

@MarcoRossignoli This seems like .NETFramework dll
C:\git\visualstudio.xunit\test\test.xunit.runner.visualstudio\bin\Debug\net46\test.xunit.runner.visualstudio.dll(.NETFramework,Version=v4.6)

[MarcoRossignoli]

Yes is a netfx lib...xplat collector doesn't support full .net right?
Is this the issue?

[vagisha-nidhi]

@MarcoRossignoli Yes. At present collector only supports .net core

[MarcoRossignoli]

@MarcoRossignoli Yes. At present collector only supports .net core

cc: @onovotny

[MarcoRossignoli]

@vagisha-nidhi can you confirm that the issue is the same for issue text https://github.com/tonerdo/coverlet/issues/475#issue-460627092?

[clairernovotny]

I'm a bit confused since I was able to get the collector to work with .NET Framework with Ix Async:

https://dev.azure.com/dotnet/Rx.NET/_build/results?buildId=18615&view=logs&jobId=12f1170f-54f2-53f3-20dd-22fc7dff55f9&taskId=63465373-886e-5c09-88f2-acc826db3ddc&lineStart=59

[vagisha-nidhi]

@MarcoRossignoli @onovotny I'll try to repro the issue and get back.

[MarcoRossignoli]

thank's a lot @vagisha-nidhi!

[vagisha-nidhi]

@onovotny
So I have looked at this. In your runsettings you define both data collectors "CodeCoverage" and "Xplat Code coverage"
I tried both, and both of them are aborting the run. I don't have an RCA on the CodeCoverage one but for xplat code coverage, here's my analysis.

In test.xunit.runner.visualstudio, there's a condition

  <ItemGroup Condition=" '$(TargetFramework)' == 'net46' ">
    <PackageReference Include="Microsoft.TestPlatform.ObjectModel" Version="15.9.0" />
  </ItemGroup>

This is causing it to refer to old objectmodel which does not work with coverlet.
Getting this error when I looked at logs using --diag option

Microsoft.VisualStudio.TestPlatform.ObjectModel.DataCollection.SessionStartEventArgs..ctor(System.Collections.Generic.IDictionary`2<System.String,System.Object>)'.
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.EventHandlers.TestCaseEventsHandler.SendSessionStart(IDictionary`2 properties)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.SendSessionStart()
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTests().

Can you try with updating the object model and disabling data collector for CodeCoverage and see if xplat code coverage works?

cc: @MarcoRossignoli

[clairernovotny]

I'm not sure if I can use a later object model? As this is a test adapter for xunit, it has to run in both VS 2017 and VS 2019.

[vagisha-nidhi]

@onovotny The objectmodel should be back compat. You can give this a try. Coverlet integration has a dependency on the newer versions.

[clairernovotny]

Okay, yes, that was the cause. Just wish that vstest output a stacktrace instead of an unhelpful crash :)

[MarcoRossignoli]

Thank's for your time @vagisha-nidhi!