Vstest: Is there a good example of excluding modules from code coverage using vstest.console.exe only?

Created on 30 May 2019  ·  7Comments  ·  Source: microsoft/vstest

I have seen examples of using a .runsettings file to exclude modules, but when I try to use a runsettings file with my tests and the vstest.console from the microsoft.testplatform nuget package it gives me an error about not being able to launch codecoverage.exe.
I see that vstest.console.exe accepts Runsettings parameters; is there a way to list modules to exclude in that command line?

question
Source
picture shueybubbles
👍1

Most helpful comment

ok. I figured out that the settings file parser is not resilient to improperly formed regular expression on excludes

IE this leads to the failure:

<ModulePath>*moq.*</ModulePath>

Note the missing leading character class.

 .*moq.* works fine.

So it's a user error but the tool should be more resilient.

picture shueybubbles on 3 Jun 2019
👍2

All 7 comments

@shueybubbles
Sharing example of using .runsettings to exclude modules: https://docs.microsoft.com/en-gb/visualstudio/test/customizing-code-coverage-analysis?view=vs-2019#sample and this is how you configure unit tests with .runsettings file.

Also, how are you trying to collect code coverage? We recommend folks to always use vstest.console to gather coverage. Can you also share logs? https://github.com/microsoft/vstest-docs/blob/master/docs/diagnose.md contains how to collect logs for test paltform.

vagisha-nidhi picture vagisha-nidhi on 31 May 2019

Here's a sample command line using a .runsettings file. Notice I'm trying to use the Microsoft.TestPlatform nuget package so I don't have to install VS on build agents.
```
"D:gitSqlManagementObjectspackagesMicrosoft.TestPlatform.16.1.0toolsnet451Common7IDEExtensionsTestPlatformvstest.console.exe" "D:gitSqlManagementObjectstargetdistribDebugnet461ConnectionInfoUnitTests.dll"
/TestCaseFilter:"TestCategory=Unit" /Settings:D:gitSqlManagementObjectssrcUnitTestunittest.runsettings /logger:console /logger:trx;LogFileName=Testrun_ConnectionInfoUnitTests.trx /ResultsDirectory:"D:gitSqlManagement
ObjectstargetdistribDebugnet461UnitTestResults"



the test immediately pops up an error dialog
testhost.x86.exe - Fatal error
CLR error: 80004005.
 The program will now terminate.
OK   

with this output

Microsoft (R) Test Execution Command Line Tool Version 16.1.0
Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait...
M i c r o s o f t ( R ) C o v e r a g e C o l l e c t i o n T o o l V e r s i o n 1 6 . 0 . 3 0 3 1 9 . 0

C o p y r i g h t ( c ) M i c r o s o f t C o r p o r a t i o n . A l l r i g h t s r e s e r v e d .

2.7439
Data collector 'Code Coverage' message: Failed to launch CodeCoverage.exe. No code coverage will be collected..
vstest.console process failed to connect to testhost process after 90 seconds. This may occur due to machine slowness, please set environment variable VSTEST_CONNECTION_TIMEOUT to increase timeout.

Test Run Aborted.
Results File: D:gitSqlManagementObjectstargetdistribDebugnet461UnitTestResultsTestrun_ConnectionInfoUnitTests.trx


Alternately, if I remove the .runsettings file and use --collect, no problem:

"D:gitSqlManagementObjectspackagesMicrosoft.TestPlatform.16.1.0toolsnet451Common7IDEExtensionsTestPlatformvstest.console.exe" "D:gitSqlManagementObjectstargetdistribDebugnet461ConnectionInfoUnitTests.dll"
/TestCaseFilter:"TestCategory=Unit" --collect:"Code Coverage" /logger:console /logger:trx;LogFileName=Testrun_ConnectionInfoUnitTests.trx /ResultsDirectory:"D:gitSqlManagementObjectstargetdistribDebugnet461UnitTestRe
sults"
Microsoft (R) Test Execution Command Line Tool Version 16.1.0
Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait...
M i c r o s o f t ( R ) C o v e r a g e C o l l e c t i o n T o o l V e r s i o n 1 6 . 0 . 3 0 3 1 9 . 0

C o p y r i g h t ( c ) M i c r o s o f t C o r p o r a t i o n . A l l r i g h t s r e s e r v e d .

2.8319
√ VerifyInteractiveModeConnectionSettings [436ms]
√ SqlConnectionInfo_supports_ApplicationIntent [34ms]
√ TypeConverter_DatabaseEngineEdition_ConvertsEnumToStringAndBack [25ms]
√ TypeConverter_DatabaseEngineType_ConvertsEnumToStringAndBack [< 1ms]
√ When_AccessToken_is_null_SqlConnectionObject_has_null_AccessToken [36ms]
√ When_AccessToken_is_not_null_SqlConnectionObject_has_valid_AccessToken [218ms]
√ When_AccessToken_is_in_config_SqlConnectionObject_has_valid_AccessToken [2ms]
√ ServerConnection_copies_AccessToken_from_source_ServerConnection [4ms]
√ ServerConnection_constructor_throws_if_AccessToken_is_null_and_UseIntegratedSecurity_is_false [< 1ms]
√ When_AccessToken_changes_SqlConnectionObject_has_new_AccessToken_value [2ms]
M i c r o s o f t ( R ) C o v e r a g e C o l l e c t i o n T o o l V e r s i o n 1 6 . 0 . 3 0 3 1 9 . 0

C o p y r i g h t ( c ) M i c r o s o f t C o r p o r a t i o n . A l l r i g h t s r e s e r v e d .

 √ When_constructed_from_SqlConnection_with_sql_auth_LoginSecure_is_false [1ms]
√ When_SqlConnection_uses_SqlCredential_ServerConnection_copies_it [5ms]
√ When_assigned_ConnectionString_after_construction_Clone_is_correct [1ms]
√ GetServerInformation_executes_one_query_and_accepts_three_result_tables [199ms]
√ When_ServerVersion_is_less_than_14_HostPlatform_is_hard_coded_Windows [3ms]

Attachments:
D:gitSqlManagementObjectstargetdistribDebugnet461UnitTestResults19631726-cdda-4591-af67-432b4bbdbf68davidshi_DAVIDSHI-WIN10 2019-05-31 11_24_02.coverage

Test Run Successful.
Total tests: 15
Passed: 15
Total time: 3.8732 Seconds
Results File: D:gitSqlManagementObjectstargetdistribDebugnet461UnitTestResultsTestrun_ConnectionInfoUnitTests.trx
```
The only issue I have with this usage is the .coverage file has data for DLLs I don't care about like moq and NUnit which I want to exclude.

shueybubbles picture shueybubbles on 31 May 2019

@shueybubbles
This is what I have tried, and code coverage seems to be working fine along with the runsettings provided.

PS E:\agent\_work\_tool\VsTest\16.1.1\x64\tools\net451\Common7\IDE\Extensions\TestPlatform> .\vstest.console.exe "C:\Users\vanidhi\source\repos\UnitTestProject26\UnitTestProject26\bin\Debug\netcoreapp2.0\UnitTestProject26.dll" --settings:"E:\codecoverage.runsettings"
Microsoft (R) Test Execution Command Line Tool Version 16.1.1
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
  √ TestMethod1 [9ms]

Attachments:
  E:\agent\_work\_tool\VsTest\16.1.1\x64\tools\net451\Common7\IDE\Extensions\TestPlatform\TestResults\efbf9ae2-fb85-4e28-826e-9070341b5508\vanidhi_VANIDHI-DESK 2019-06-03 15_31_17.coverage

This is the exact runsettings I provided: https://docs.microsoft.com/en-gb/visualstudio/test/customizing-code-coverage-analysis?view=vs-2019#sample-runsettings-file
I am getting the same result with Test Platform version 16.1.0.

Can you please share your runsettings file as well as diag logs with us?
You can collect the diag logs by adding --diag:"E:\log.txt" to the command line invocation.

vagisha-nidhi picture vagisha-nidhi on 3 Jun 2019

It works for me with the same file, but as soon as I remove a few of the excludes sections (my binaries use the same public key listed , for example) it fails with this error in the log.host

TpTrace Information: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183983555, testhost.dll, AssemblyResolver.OnResolve: Microsoft.VisualStudio.ArchitectureTools.PEReader: Resolving assembly.
TpTrace Information: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183985018, testhost.dll, AssemblyResolver.OnResolve: Microsoft.VisualStudio.ArchitectureTools.PEReader: Resolved from cache.
TpTrace Verbose: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183986209, testhost.dll, CurrentDomain_AssemblyResolve: Resolving assembly 'Microsoft.VisualStudio.ArchitectureTools.PEReader'.
TpTrace Information: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183987640, testhost.dll, AssemblyResolver.OnResolve: Microsoft.VisualStudio.ArchitectureTools.PEReader: Resolving assembly.
TpTrace Information: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183988858, testhost.dll, AssemblyResolver.OnResolve: Microsoft.VisualStudio.ArchitectureTools.PEReader: Resolved from cache.
TpTrace Verbose: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183989996, testhost.dll, CurrentDomain_AssemblyResolve: Resolving assembly 'Microsoft.VisualStudio.ArchitectureTools.PEReader'.
TpTrace Verbose: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183991080, testhost.dll, CurrentDomainAssemblyResolve: Failed to resolve assembly 'Microsoft.VisualStudio.ArchitectureTools.PEReader'.
TpTrace Verbose: 0 : 3464, 4, 2019/06/03, 10:36:45.065, 11942183995574, testhost.dll, CurrentDomainAssemblyResolve: Failed to resolve assembly 'Microsoft.VisualStudio.ArchitectureTools.PEReader'.
TpTrace Warning: 0 : 3464, 4, 2019/06/03, 10:36:45.065, 11942184001821, testhost.dll, TestPluginDiscoverer: Failed to get types from assembly 'Microsoft.VisualStudio.TraceDataCollector, Version=16.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.  Skipping test extension scan for this assembly.  Error: System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types.
Could not load file or assembly 'Microsoft.VisualStudio.ArchitectureTools.PEReader, Version=16.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
   at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at System.Reflection.RuntimeModule.GetTypes()
   at System.Reflection.Assembly.GetTypes()
   at System.Reflection.AssemblyExtensions.GetTypes(Assembly assembly)
   at Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.TestPluginDiscoverer.GetTestExtensionsFromAssembly[TPluginInfo,TExtension](Assembly assembly, Dictionary`2 pluginInfos)
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.VisualStudio.ArchitectureTools.PEReader, Version=16.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
File name: 'Microsoft.VisualStudio.ArchitectureTools.PEReader, Version=16.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'

TpTrace Verbose: 0 : 19196, 4, 2019/06/03, 10:15:33.531, 11929468585820, testhost.dll, TestExecutionRecorder.RecordStart: Starting test: Microsoft.SqlServer.ConnectionInfoUnitTests.ConnectionSettingsTests.SqlConnectionInfo_supports_ApplicationIntent.
TpTrace Error: 0 : 19196, 4, 2019/06/03, 10:15:33.583, 11929469110382, testhost.dll, TestCaseEventsHandler.RaiseTestCaseStart: Exception occurred while calling handler of type Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.ProxyOutOfProcDataCollectionManager for TestCaseStartEventArgs: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.SocketCommunicationManager.ReceiveRawMessage()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.SocketCommunicationManager.ReceiveMessage()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.DataCollectionTestCaseEventSender.SendTestCaseStart(TestCaseStartEventArgs e)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Delegate.DynamicInvokeImpl(Object[] args)
   at Microsoft.VisualStudio.TestPlatform.Utilities.MulticastDelegateUtilities.SafeInvoke(Delegate delegates, Object sender, EventArgs args, String traceDisplayName)
TpTrace Information: 0 : 19196, 4, 2019/06/03, 10:15:33.749, 11929470768379, testhost.dll, AssemblyResolver.OnResolve: Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.resources: Resolving assembly.
TpTrace Information: 0 : 19196, 4, 2019/06/03, 10:15:33.749, 11929470771554, testhost.dll, AssemblyResolver.OnResolve: Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.resources: Failed to load assembly.
shueybubbles picture shueybubbles on 3 Jun 2019

ok. I figured out that the settings file parser is not resilient to improperly formed regular expression on excludes

IE this leads to the failure:

<ModulePath>*moq.*</ModulePath>

Note the missing leading character class.

 .*moq.* works fine.

So it's a user error but the tool should be more resilient.

shueybubbles picture shueybubbles on 3 Jun 2019
👍2

It works for me with the same file, but as soon as I remove a few of the excludes sections (my binaries use the same public key listed , for example) it fails with this error in the log.host

TpTrace Information: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183983555, testhost.dll, AssemblyResolver.OnResolve: Microsoft.VisualStudio.ArchitectureTools.PEReader: Resolving assembly.
TpTrace Information: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183985018, testhost.dll, AssemblyResolver.OnResolve: Microsoft.VisualStudio.ArchitectureTools.PEReader: Resolved from cache.
TpTrace Verbose: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183986209, testhost.dll, CurrentDomain_AssemblyResolve: Resolving assembly 'Microsoft.VisualStudio.ArchitectureTools.PEReader'.
TpTrace Information: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183987640, testhost.dll, AssemblyResolver.OnResolve: Microsoft.VisualStudio.ArchitectureTools.PEReader: Resolving assembly.
TpTrace Information: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183988858, testhost.dll, AssemblyResolver.OnResolve: Microsoft.VisualStudio.ArchitectureTools.PEReader: Resolved from cache.
TpTrace Verbose: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183989996, testhost.dll, CurrentDomain_AssemblyResolve: Resolving assembly 'Microsoft.VisualStudio.ArchitectureTools.PEReader'.
TpTrace Verbose: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183991080, testhost.dll, CurrentDomainAssemblyResolve: Failed to resolve assembly 'Microsoft.VisualStudio.ArchitectureTools.PEReader'.
TpTrace Verbose: 0 : 3464, 4, 2019/06/03, 10:36:45.065, 11942183995574, testhost.dll, CurrentDomainAssemblyResolve: Failed to resolve assembly 'Microsoft.VisualStudio.ArchitectureTools.PEReader'.
TpTrace Warning: 0 : 3464, 4, 2019/06/03, 10:36:45.065, 11942184001821, testhost.dll, TestPluginDiscoverer: Failed to get types from assembly 'Microsoft.VisualStudio.TraceDataCollector, Version=16.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.  Skipping test extension scan for this assembly.  Error: System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types.
Could not load file or assembly 'Microsoft.VisualStudio.ArchitectureTools.PEReader, Version=16.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
   at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at System.Reflection.RuntimeModule.GetTypes()
   at System.Reflection.Assembly.GetTypes()
   at System.Reflection.AssemblyExtensions.GetTypes(Assembly assembly)
   at Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.TestPluginDiscoverer.GetTestExtensionsFromAssembly[TPluginInfo,TExtension](Assembly assembly, Dictionary`2 pluginInfos)
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.VisualStudio.ArchitectureTools.PEReader, Version=16.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
File name: 'Microsoft.VisualStudio.ArchitectureTools.PEReader, Version=16.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'

TpTrace Verbose: 0 : 19196, 4, 2019/06/03, 10:15:33.531, 11929468585820, testhost.dll, TestExecutionRecorder.RecordStart: Starting test: Microsoft.SqlServer.ConnectionInfoUnitTests.ConnectionSettingsTests.SqlConnectionInfo_supports_ApplicationIntent.
TpTrace Error: 0 : 19196, 4, 2019/06/03, 10:15:33.583, 11929469110382, testhost.dll, TestCaseEventsHandler.RaiseTestCaseStart: Exception occurred while calling handler of type Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.ProxyOutOfProcDataCollectionManager for TestCaseStartEventArgs: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.SocketCommunicationManager.ReceiveRawMessage()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.SocketCommunicationManager.ReceiveMessage()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.DataCollectionTestCaseEventSender.SendTestCaseStart(TestCaseStartEventArgs e)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Delegate.DynamicInvokeImpl(Object[] args)
   at Microsoft.VisualStudio.TestPlatform.Utilities.MulticastDelegateUtilities.SafeInvoke(Delegate delegates, Object sender, EventArgs args, String traceDisplayName)
TpTrace Information: 0 : 19196, 4, 2019/06/03, 10:15:33.749, 11929470768379, testhost.dll, AssemblyResolver.OnResolve: Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.resources: Resolving assembly.
TpTrace Information: 0 : 19196, 4, 2019/06/03, 10:15:33.749, 11929470771554, testhost.dll, AssemblyResolver.OnResolve: Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.resources: Failed to load assembly.

This issue is a different issue. https://github.com/microsoft/vstest/issues/2008. We will be giving fix for this soon.

ok. I figured out that the settings file parser is not resilient to improperly formed regular expression on excludes

IE this leads to the failure:

<ModulePath>*moq.*</ModulePath>

Note the missing leading character class.

 .*moq.* works fine.

So it's a user error but the tool should be more resilient.

Seems like you are able to exclude modules in code coverage now.
Closing the thread. Feel free to open another issue if needed.

vagisha-nidhi picture vagisha-nidhi on 3 Jun 2019

I'm pulling my hair out trying to get this to work. I am using the sample .runsettings file from the previously linked path.

I am running:
vstest.console.exe "*bin**Test.dll" --settings:c:sourceCodeCoverage.runsettings --testcasefilter:Backend!=Database --diag:"c:sourcelog.txt"

(do I need --enablecodecoverage? Seems to be the same with or without it)

Then:
$TestResultsPath = resolve-path "c:sourceTestResults***.coverage" | Select -ExpandProperty Path
CodeCoverage.exe analyze /output:DynamicCodeCoverage.coveragexml "$TestResultsPath"

The XML contains DLLs with Microsoft in the name. (which to me looks like they should be excluded).

I have added XUnit to exclude and still get XUnit listed in the XML and the report generated from it with report generator.



..dll$
..exe$


.CPPUnitTestFramework.
.XUnit.

XUnit dlls are included in the reports as well.

I have looked at the log and it shows my runsettings with the added XUnit exclusion.

log.txt

BTW: I am running this in the mcr.microsoft.com/dotnet/framework/sdk:4.8 windows container.

Any help appreciated. I would expect not to see Microsoft.SignalR.xx or XUnit.* dlls at all.

PilotBob picture PilotBob on 9 Mar 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

alastairtree picture alastairtree  ·  4Comments
ejcoyle88 picture ejcoyle88  ·  3Comments
GreenDavidA picture GreenDavidA  ·  5Comments
dlemstra picture dlemstra  ·  5Comments
aumanjoa picture aumanjoa  ·  3Comments