Azure-pipelines-tasks: VSTest task saves test results to wrong TestResults directory

[harmpauw]

Hi all,

I've noticed that since a couple of weeks, Unit Tests results and Code Coverage files are no longer picked up by the SonarQube Scanner when running a build on VSTS using Hosted Agents. When I looked in the log, I saw that SonarQube can't find any results files:

2016-10-03T09:58:06.3589980Z SonarQube Scanner for MSBuild End Step 2.1 2016-10-03T09:58:06.3589980Z 09:58:03.798 Fetching code coverage report information from TFS... 2016-10-03T09:58:06.3589980Z 09:58:03.799 Attempting to locate a test results (.trx) file... 2016-10-03T09:58:06.3599982Z 09:58:03.799 No test results files found

I found out that SonarQube is looking in the $(Common.TestResultsDirectory) for files (e.g. c:\a\1\TestResults), while the results and coverage files are written to $(build.SourcesDirectory)\TestResults (e.g. c:\a\1\s\TestResults). I expected them to be written to the Common.TestResultsDirectory.

It seems like the change in the VSTest task responsible for this was introduced by this commit. Line 159, which reads Common.TestResultsDirectory was deleted and replaced by this line.

    $testResultsDirectory = Get-TaskVariable -Context $distributedTaskContext -Name "Common.TestResultsDirectory"

    if([string]::IsNullOrEmpty($testResultsDirectory))
    {
        $testResultsDirectory = $workingDirectory + [System.IO.Path]::DirectorySeparatorChar + "TestResults"
    }`

Could you please look into this? Thanks!

[allendm-msft]

@harmpauw I would simply pick the test run and download the trx instead of scanning it locally since the task isnt promising to retain it in place either. If you still want to do it locally you can either a) pick the directory from code (fragile) or b) do a file scan (non optimal)

In general I would recommend against taking dependencies on the internal implementations of the task.

[harmpauw]

@allendm-msft Thanks for you reply. After some more digging. I found the problem: The SonarQube Scanner for MSBuild had a hard dependency to the location of the TestResults directory (see this pull request). They've fixed it in master a couple of days ago but it's not released yet.

For others that have the same problems and read this thread: What I've done for now a a temporary hack is I added a Copy Files task to copy the test results to the directory that the SonarQube Scanner for MSBuild expects by copying from $(build.SourcesDirectory)/TestResults to $(Agent.BuildDirectory)/TestResults (for now the same as $(Common.TestResultsDirectory)). I inserted this task between the Test Assemblies task and the Finish analysis task of SonarQube.

That leaves me with one question: Shouldn't the VSTest task (or another one responsible for this) set the $(Common.TestResultsDirectory) variable to the right location and if not, for what purpose should we use it since this is a publicly documented variable?

[allendm-msft]

@bryanmacfarlane any recommendation on this variable? Should the test task be setting this?
We didnt want to update it with the path we use, assuming that there would be someone taking a dependency on the existing value already.

[allendm-msft]

This is a variable set by the Agent currently and not being updated by the test task. We don't intend to change this to avoid breaking others who have taken dependencies on it as-is. If we introduce a similar variable it will be posted in the docs.

[mack0196]

Why doesn't vstest create the test results in $(Common.TestResultsDirectory)? I don't understand why the same breaking consideration wasn't considered when changing the TestResults directory.

[allendm-msft]

@mack0196 let me try to explain. The change to the location where the testresults are placed by the vstest task is internal to the task. This location was not published as a output/common variable to be consumed by external tasks. Since another task made the assumption that this location wont change they broke.
This is pretty much similar to using non documented apis.

[bobylevm]

@allendm-msft $(Common.TestResultsDirectory) is published as output/common variable at visualstudio.com: https://www.visualstudio.com/en-us/docs/build/define/variables

[jerryjoseph-net]

@allendm-msft since VSTest is a VSTS task it has to respect the VSTS variable $(Common.TestResultsDirectory) IMO

No need to update it with the value you use. You just need to use that value instead of ignoring it. This was the behavior until the recent change.