Vstest: Launching Code Coverage Collector from TestCaseStart Handler and Stopping in TestCaseEnd

[t-carob]

Description

I am trying to create a tool that selects which tests to run based on the code coverage of the tests. To do so, I need to collect code coverage information for each test, while the current code coverage DataCollector collects coverage information for the union of all tests.

I am working on modifying your code coverage DataCollector to launch a new CodeCoverage.exe task for each test case rather than one for the entire session. To make things simple, I am filtering to only one run test for now.

I added event handlers for TestCaseStart and TestCaseEnd, and tried moving the call to this.StartVanguard(context) and this.StopVanguard(context) to these handlers. When I do this, no coverage information is collected, whether I use the test context or session context. However, if I leave the calls to StartVanguard and StopVanguard in SessionStart and SessionEnd (as you have them), coverage information is collected as expected.

I know that all of my handlers are being called in the correct order, so I’m confused as to why this minor change breaks the program.

Steps to reproduce

I am invoking vstest.console.exe from the developer console as follows:
vstest.console.exe D:\src\Intune\Svc\DeviceConfig\out\retail-amd64-unittest\StatelessDeviceConfigurationFEServiceUnitTests\StatelessDeviceConfigurationFEServiceUnitTests.dll /TestCaseFilter:"(Name=CustomAllListsNonEmpty)" --collect:"Per Test Code Coverage" /testadapterpath:"D:\src\vstest\src\DataCollectors\TraceDataCollector\bin\Debug\netstandard2.0" /collect:"Event Log"

My handlers look something like this once modified:

    public void SessionStart(object sender, SessionStartEventArgs e)
    {
        // this.StartVanguard(e.Context);
        this.context = e.Context;
        this.logger.LogWarning(e.Context, "Starting session");
    }


    public void SessionEnd(object sender, SessionEndEventArgs e)
    {
        // this.StopVanguard(e.Context);
        this.logger.LogWarning(e.Context, "Ending session");
    }

    public void TestCaseStart(object sender, TestCaseStartEventArgs e)
    {
        this.logger.LogWarning(e.Context, "Starting Vanguard from TestCaseStart");
        this.StartVanguard(this.context);
        this.logger.LogWarning(e.Context, "Starting test case: " + e.TestCaseName);
    }

    public void TestCaseEnd(object sender, TestCaseEndEventArgs e)
    {
        this.StopVanguard(this.context);
        this.logger.LogWarning(e.Context, "Ending test case: " + e.TestCaseName);
    }

The test dll that it points to is built with Framework 45.

Expected behavior

A .coverage file is output that shows that the test case that was run is covered.
Note: this works as expected when StartVanguard and StopVanguard are launched from the SessionStart and SessionEnd handlers, but not from the TestCaseStart and TestCaseEnd handlers.

Actual behavior

A .coverage file is output that shows that no lines of code are covered.
Note: all of the WarningLogs I added for debugging purposes are logged as expected

Edit:
The Event Log Data Collector outputs the following error in the case when StartVanguard is launched from TestCaseStart:
<Table1>
<Type>Error</Type>
<DateTime>2018-07-19T14:14:49-07:00</DateTime>
<Source>Dynamic Code Coverage</Source>
<Category>(0)</Category>
<EventID>0</EventID>
<Description>The description for Event ID '0' in Source 'Dynamic Code Coverage' cannot be found. The` local computer may not have the necessary registry information or message DLL files to display the message, or you may not have permission to access them. The following information is part of the event:'engine::notify_process_attach failed with exception: Session "1a8998c0-e845-4597-a09d-d15c55306c90" does not exist.'</Description>
<Computer>DESKTOP-3RB7J8R.northamerica.corp.microsoft.com</Computer>
</Table1>

No errors are output when StartVanguard is launched from SessionStart, as usual.

Environment

Using Test Execution Command Line Tool Version 15.7.2

@smadala

[smadala]

@t-carob Can you share your code changes on github as well?

I think you are trying to create tool similar to TestImpact. See below blogs for more details and code for TestImpact is here

• https://blogs.msdn.microsoft.com/devops/2017/03/02/accelerated-continuous-testing-with-test-impact-analysis-part-1/
• https://blogs.msdn.microsoft.com/devops/2017/05/16/accelerated-continuous-testing-with-test-impact-analysis-part-2/
• https://blogs.msdn.microsoft.com/devops/2017/06/13/accelerated-continuous-testing-with-test-impact-analysis-part-3/
• https://blogs.msdn.microsoft.com/devops/2017/08/04/accelerated-continuous-testing-with-test-impact-analysis-part-4/

[mayankbansal018]

@t-carob , Additional thing to note, CodeCoverage currently does not subscribe to TestCase level event. We add handling of events here. Can you try subscribing to TestCase level events at this location, & see if you get those events during CodeCoverage

[t-carob]

@smadala you're correct, I am essentially recreating the functionality of test impact because unfortunately there doesn't seem to be a way to use that feature with our CDP pipeline.

@mayankbansal018 yes, I added the event handling as you described & was able to receive those events.

After doing some digging, I found some writing on vsperfmon from 2007... it isn't exactly the codecoverage.exe tool, but it seems like the original version:

https://blogs.msdn.microsoft.com/phuene/2007/05/14/code-coverage-collection/

If you read the second paragraph, it says "When VSCover90.dll is loaded and initialized, it will attempt a connection to the monitor. If the monitor is not running, the connection will silently fail and coverage collection will be disabled for the lifetime of the process." My current thinking is that vstest still uses a similar system, and this is why the codecoverage.exe monitor needs to be launched at SessionStart rather than TestCaseStart: the monitor already needs to be running when the instrumented dlls are first loaded.

If this is the case, it may not be possible to collect code coverage on a test by test basis using this method, since I'm guessing the instrumented test dlls are loaded sometime after session start but before TestCaseStart, and the coverage monitor already needs to be running before these dlls are loaded.

This is mostly speculation since I haven't been able to see how these tools are written, but it seems to explain the behavior I'm seeing.

[smadala]

I am essentially recreating the functionality of test impact because unfortunately there doesn't seem to be a way to use that feature with our CDP pipeline.

@t-carob Let's not reinvent wheel again. We would like to address missing feature in CDP.

[t-carob]

@smadala I agree that using TIA with CDP would be the ideal solution. What do you suggest for next steps?

[smadala]

@t-carob Just sent you email internally, We will follow up on it.

[mayankbansal018]

@t-carob , @smadala can you close this issue

[smadala]

Done.