Coverlet: Cannot exclude tests in runsettings when using VSTest integration driver

[civspan]

Hi!
We are using the VSTest integration driver (v1.2.0) to successfully generate a coverage.covertura.xml. Integrated it with MS DevOps, everything looks great. Now we want to exclude some files from our test report and we can't seem to get it to work.

We are running as follows:
dotnet test --collect:"XPlat Code Coverage" --settings .\coverlet.runsettings
and
dotnet test --settings .\coverlet.runsettings.

We've tried many runsettings file configuration, but in its simplest form we think something like this should work:

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="XPlat Code Coverage">
<Configuration>
<Format>cobertura</Format>
<Exclude>[*]*</Exclude>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
</RunSettings>

We have gotten file exclusion to work with the MSBuild integration driver, using the following command:
dotnet test /p:CollectCoverage=true /p:Exclude="[*]*"

[MarcoRossignoli]

I did some test on our test sample https://github.com/tonerdo/coverlet/tree/master/Documentation/Examples/VSTest/HelloWorld

dotnet test --collect:"XPlat Code Coverage" --settings runsettings.xml --diag:log.txt

```xml






lcov,cobertura
[]

TpTrace Verbose: 0 : 29836, 1, 2020/04/03, 17:33:45.846, 712318375319, datacollector.dll, [coverlet]Excluded module filter '[coverlet.]'
TpTrace Verbose: 0 : 29836, 1, 2020/04/03, 17:33:45.846, 712318377069, datacollector.dll, [coverlet]Excluded module filter '[]'
TpTrace Verbose: 0 : 29836, 1, 2020/04/03, 17:33:45.855, 712318463223, datacollector.dll, [coverlet]Excluded module: 'C:gitcoverletDocumentationExamplesVSTestHelloWorldXUnitTestProject1binDebugnetcoreapp3.1ClassLibrary1.dll'
TpTrace Verbose: 0 : 29836, 1, 2020/04/03, 17:33:45.855, 712318465580, datacollector.dll, [coverlet]Excluded module: 'C:gitcoverletDocumentationExamplesVSTestHelloWorldXUnitTestProject1binDebugnetcoreapp3.1Microsoft.TestPlatform.CommunicationUtilities.dll'
TpTrace Verbose: 0 : 29836, 1, 2020/04/03, 17:33:45.856, 712318467118, datacollector.dll, [coverlet]Excluded module: 'C:gitcoverletDocumentationExamplesVSTestHelloWorldXUnitTestProject1binDebugnetcoreapp3.1Microsoft.TestPlatform.CoreUtilities.dll'
TpTrace Verbose: 0 : 29836, 1, 2020/04/03, 17:33:45.856, 712318469231, datacollector.dll, [coverlet]Excluded module: 'C:gitcoverletDocumentationExamplesVSTestHelloWorldXUnitTestProject1binDebugnetcoreapp3.1Microsoft.TestPlatform.CrossPlatEngine.dll'
TpTrace Verbose: 0 : 29836, 1, 2020/04/03, 17:33:45.856, 712318470676, datacollector.dll, [coverlet]Excluded module: 'C:gitcoverletDocumentationExamplesVSTestHelloWorldXUnitTestProject1binDebugnetcoreapp3.1Microsoft.TestPlatform.PlatformAbstractions.dll'
TpTrace Verbose: 0 : 29836, 1, 2020/04/03, 17:33:45.856, 712318472009, datacollector.dll, [coverlet]Excluded module: 'C:gitcoverletDocumentationExamplesVSTestHelloWorldXUnitTestProject1binDebugnetcoreapp3.1Microsoft.TestPlatform.Utilities.dll'
TpTrace Verbose: 0 : 29836, 1, 2020/04/03, 17:33:45.856, 712318473323, datacollector.dll, [coverlet]Excluded module: 'C:gitcoverletDocumentationExamplesVSTestHelloWorldXUnitTestProject1binDebugnetcoreapp3.1Microsoft.VisualStudio.CodeCoverage.Shim.dll'
TpTrace Verbose: 0 : 29836, 1, 2020/04/03, 17:33:45.856, 712318474621, datacollector.dll, [coverlet]Excluded module: 'C:gitcoverletDocumentationExamplesVSTestHelloWorldXUnitTestProject1binDebugnetcoreapp3.1Microsoft.VisualStudio.TestPlatform.Common.dll'
TpTrace Verbose: 0 : 29836, 1, 2020/04/03, 17:33:45.856, 712318475907, datacollector.dll, [coverlet]Excluded module: 'C:gitcoverletDocumentationExamplesVSTestHelloWorldXUnitTestProject1binDebugnetcoreapp3.1Microsoft.VisualStudio.TestPlatform.ObjectModel.dll'
TpTrace Verbose: 0 : 29836, 1, 2020/04/03, 17:33:45.857, 712318477386, datacollector.dll, [coverlet]Excluded module: 'C:gitcoverletDocumentationExamplesVSTestHelloWorldXUnitTestProject1binDebugnetcoreapp3.1Newtonsoft.Json.dll'
TpTrace Verbose: 0 : 29836, 1, 2020/04/03, 17:33:45.857, 712318478746, datacollector.dll, [coverlet]Excluded module: 'C:gitcoverletDocumentationExamplesVSTestHelloWorldXUnitTestProject1binDebugnetcoreapp3.1NuGet.Frameworks.dll'
TpTrace Verbose: 0 : 29836, 1, 2020/04/03, 17:33:45.857, 712318480120, datacollector.dll, [coverlet]Excluded module: 'C:gitcoverletDocumentationExamplesVSTestHelloWorldXUnitTestProject1binDebugnetcoreapp3.1testhost.dll'
TpTrace Verbose: 0 : 29836, 1, 2020/04/03, 17:33:45.857, 712318481424, datacollector.dll, [coverlet]Excluded module: 'C:gitcoverletDocumentationExamplesVSTestHelloWorldXUnitTestProject1binDebugnetcoreapp3.1xunit.abstractions.dll'
TpTrace Verbose: 0 : 29836, 1, 2020/04/03, 17:33:45.857, 712318482707, datacollector.dll, [coverlet]Excluded module: 'C:gitcoverletDocumentationExamplesVSTestHelloWorldXUnitTestProject1binDebugnetcoreapp3.1xunit.assert.dll'
TpTrace Verbose: 0 : 29836, 1, 2020/04/03, 17:33:45.857, 712318483993, datacollector.dll, [coverlet]Excluded module: 'C:gitcoverletDocumentationExamplesVSTestHelloWorldXUnitTestProject1binDebugnetcoreapp3.1xunit.core.dll'
TpTrace Verbose: 0 : 29836, 1, 2020/04/03, 17:33:45.857, 712318485281, datacollector.dll, [coverlet]Excluded module: 'C:gitcoverletDocumentationExamplesVSTestHelloWorldXUnitTestProject1binDebugnetcoreapp3.1xunit.execution.dotnet.dll'
TpTrace Verbose: 0 : 29836, 1, 2020/04/03, 17:33:45.857, 712318486561, datacollector.dll, [coverlet]Excluded module: 'C:gitcoverletDocumentationExamplesVSTestHelloWorldXUnitTestProject1binDebugnetcoreapp3.1xunit.runner.reporters.netcoreapp10.dll'
TpTrace Verbose: 0 : 29836, 1, 2020/04/03, 17:33:45.858, 712318487863, datacollector.dll, [coverlet]Excluded module: 'C:gitcoverletDocumentationExamplesVSTestHelloWorldXUnitTestProject1binDebugnetcoreapp3.1xunit.runner.utility.netcoreapp10.dll'
TpTrace Verbose: 0 : 29836, 1, 2020/04/03, 17:33:45.858, 712318489228, datacollector.dll, [coverlet]Excluded module: 'C:gitcoverletDocumentationExamplesVSTestHelloWorldXUnitTestProject1binDebugnetcoreapp3.1xunit.runner.visualstudio.dotnetcore.testadapter.dll'

Coverage file
```xml
<?xml version="1.0" encoding="utf-8"?>
<coverage line-rate="1" branch-rate="1" version="1.9" timestamp="1585927991" lines-covered="0" lines-valid="0" branches-covered="0" branches-valid="0">
  <sources />
  <packages />
</coverage>

Can you enable logging --diag:log.txt and take a look at log.datacollector.xxxxx.txt file generated in the project folder?Search for [coverlet] trace and check if module are excluded or not.

[civspan]

Thank you for your prompt response Marco.
We added the logging flag and the exclusion started working as expected right away. Very strange, but most welcome! I'm sure we did something wrong before but I have no idea what. I'll close this for now and if we detect the issue again I'll post it here.

Thanks again!

[MarcoRossignoli]

Glad to hear!

[mrcoymeeks]

Apologize for commenting on a closed issue, but had a related question that might be appropriate to ask here.

I'm able to exclude by namespace using the above runsettings example as a template but I'm only able to get the first exclude to work. I'd like to exclude multiple namespaces, like this:

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat code coverage">
        <Configuration>
          <Format>lcov,cobertura</Format>
          <Exclude>[*]Fake.Namespace1.*</Exclude>
          <Exclude>[*]Fake.Namespace2.*</Exclude>
          <Exclude>[*]Fake.Namespace3.*</Exclude>
          <Exclude>[*]Fake.Namespace4.*</Exclude>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

The above results in only Fake.Namespace1 being excluded from the report. Syntactically, is the above correct or is there a way to apply multiple in a single line?

Thanks!

[MarcoRossignoli]

Hi @mrcoymeeks you should add namespaces with a comma , like in the documentation sample https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md

...
 <Exclude>[coverlet.*.tests?]*,[*]Coverlet.Core*</Exclude> <!-- [Assembly-Filter]Type-Filter -->
 <Include>[coverlet.*]*,[*]Coverlet.Core*</Include> <!-- [Assembly-Filter]Type-Filter -->
...

[mrcoymeeks]

@MarcoRossignoli just what I was looking for, thank you!