Vstest: Add support for merging coverage files from multiple test projects using dotnet test --collect:"Code Coverage"

[lomholdt]

Description

When collecting code coverage using dotnet test --collect:"Code Coverage" I would like to be able to merge multiple coverage files into one if I have multiple test projects in a solution.

See #981 for reference

Expected behavior

A single coverage file that contains coverage for all test projects in a solution.

Actual behavior

Multiple coverage files are created in each test project

AB#1272814

[kennethkryger]

+1 🤞

[KLuuKer]

Any decent size application (heck even small ones) will have many projects each with there own tests, so this is a must!

[mayankbansal018]

@cltshivash , @PBoraMSFT can you check where it can be fit in our backlog

[michael-colclough]

This is very important to our team. Without it, the code coverage results are meaningless. Please could this be implemented as soon as possible

[cheenamalhotra]

We are also waiting for this support to be added, it's a basic requirement in order to gather cumulative code coverage. Is there an ETA to this request?

[ManishJayaswal]

bumping the priority and also assigning to @jakubch1 as he is working on code coverage. Let's see if we can get to this in 16.6 release.

[nohwnd]

For what it's worth, the types that can merge the code coverage files already ship with the test platform. You can parse the coverage files out from the output and merge them yourself. The only hiccup here is that the interop library is built for specific architecture, so you need the correct bitness. Both 32-bit and 64-bit versions are shipped in TP.

Here is an example that works for solution that mixes netcoreapp2.1 and 3.1 and produces two coverage files as a result.

dotnet build
dotnet test --collect:"Code Coverage" | Tee-Object -Variable output

$coverage = if ($null -ne $output) { 
    $output | 
    Select-String \.coverage | 
    ForEach-Object { $_.Line.Trim() }
}

if ($null -ne $coverage){ 
    if (1 -eq $coverage.Count) {
        $coverage
    }
    else {
        $architecture = if (64 -eq [System.IntPtr]::Size * 8) { "Amd64" } else { "X86" }

        $libs = Get-ChildItem  "~\.nuget\packages\microsoft.testplatform\*\tools\net451\Common7\IDE\Extensions\TestPlatform\Microsoft.VisualStudio.Coverage.Interop.dll" -Recurse | 
            Sort-Object -Property CreationTime -Descending

        $lib = foreach ($l in $libs) { 
            $an = [Reflection.Assemblyname]::GetAssemblyName($l)
            if ($architecture -eq $an.ProcessorArchitecture) {
                $l
                break
            }
        }

        if (-not $lib) { 
            throw "Could not find Code Coverage interop."
        }

        Add-Type -Path $lib
        $cc = [Microsoft.VisualStudio.Coverage.Interop.CoverageData]

        $firstFile = $coverage[0]
        $mergedFile = [IO.Path]::Combine([IO.Path]::GetDirectoryName($firstFile), [IO.Path]::GetFileNameWithoutExtension($firstFile) + "_merged" + [IO.Path]::GetExtension($firstFile))
        $mergeWith = $firstFile
        foreach ($file in $coverage | Select-Object -Skip 1)
        {
            $cc::MergeCoverageFiles($file, $mergeWith, $mergedFile)
            $mergeWith = $mergedFile
        }

        $mergeWith
    }

    if ($mergeWith -and $dte) { 
        # try opening open the file if we are in VS 
        # package manager console
        $dte.itemoperations.OpenFile($mergeWith)
    }
}

[nohwnd]

@jakubch1 let's syncup on this tomorrow, maybe this will be easy to fix in test console. ^^^

[nohwnd]

I've been looking into this and the main issue here is that we are just invoking msbuild which is in turn running VSTest task which invokes vsconsole. This only outputs into console but does not report back to dotnet test which invoked the build and could possibly merge the results before returning. This requires quite a big change in how the projects are invoked. Possibly simplifying the vstest task to just collect the data produced by msbuild (most importantly the target path which is the dll that we will invoke against), and then communicate that back to the test task. The test task would then spawn processes and register callbacks.

Or do it similarly but instead of starting the vstest console that would write into console we would invoke it in the client mode like in VS and collect the results.

I will try the second approach because that has smaller impact, hopefully that won't be too slow.

[nohwnd]

@jakubch1 let's sync on this this week, we need to start pushing to get it to 16.6, and I have a lot of other prio1 issues :)

[MarcoRossignoli]

@nohwnd this could resolve also coverlet users complaints on report merge.

[pavelhorak]

AB#1018529