Azure-pipelines-tasks: vstest output path changed recently

[baywet]

Environment

• Server - VSTS or TFS on-premises?

  • If using TFS on-premises, provide the version:

  • If using VSTS, provide the account name, team project name, build definition name/build number: 2tolead, , build definition id 32, build id 3998 (working), 4191 (non working)

• Agent - Hosted or Private: Hosted VS2017

Issue Description

Recently the output path of the vstest coverage file change from D:\a\1\s\TestResults\63a93925-fc5e-4a09-9725-0a1850687297\VssAdministrator_factoryvm-az361 2018-07-18 13_20_02.coverage to D:\a\_temp\TR_47834234-9496-460e-b9c3-e66d33e8c426\8d2bd737-fbf2-4323-b34c-26dd8f45cba4\VssAdministrator_factoryvm-az389 2018-08-08 16_01_27.coverage.
Not only this is not located within the build directory, it also broke subsequent scripts in our pipeline (calling codecoverage.exe to convert to xml and later import to sonarqube).
To my knowledge there's no way to specify the output path other than setting the RunSettings.RunConfiguration.ResultsDirectory node in the runsettings file. However setting that value would require extra scripts and tokenization.
My ask: please add an output path configuration for the task relative to the build folder.

Error logs

not relevant, it's our subsequent PowerShell task failing because it doesn't find the file at the right spot.

[prawalagarwal]

@baywet
The output location of the vstest task is not guranteed and does not follow a fixed contract. Taking a direct dependency on this is not a good idea. The recent changes we made to generate output in agent temp directory were to ensure proper cleaning of files generated by us and ensuring the right artifacts get uploaded to the build.

That said, you can control where your coverage file gets generated by specifying the results directory in run settings file.https://blogs.msdn.microsoft.com/aseemb/2013/09/14/how-to-specify-results-directory-with-vstest-console/

[bryanmacfarlane]

+1 to what @prawalagarwal said. that's the location layout of our code.

[baywet]

Thanks for the answer. If I'm not mistaking the runsettings file only accepts full paths and not relatives correct? (and the build working path is dynamic by nature)
With that in mind it means we'd have to replace tokens before running the tests. I'm not saying it's impossible, I'm only saying this is a huge overhead than simply having a parameter for that in the task that'd be relative to the build working path.

[bryanmacfarlane]

You can use variables to supply a full path that's not hardcoded.

https://docs.microsoft.com/en-us/vsts/pipelines/build/variables?view=vsts

The agent replaces those for you. For example $(System.DefaultWorkingDirectory)

[baywet]

thanks for that answer @bryanmacfarlane
Again, I understand it's feasible with a runsettings file and token replacement (and the build variables). My feedback is: it'd be much simpler if the test task provided a setting for the output location.

[acesiddhu]

Tagging @PBoraMSFT for the feedback on providing it as a task option. Prachi whats your take on this?

[PBoraMSFT]

@acesiddhu @baywet - Trying to add a setting in the task for every single config will simply bloat it. We can consider enhancing the override test run parameters as an easier means to override values in runsettings - but that would be an enhancement and not a bug fix. Marking this issue as such.

[baywet]

thanks for the answer @PBoraMSFT ! Yes I think there might be other settings that are not currently exposed that people might want to provide to the task/process. A generic parameters override section would probably be better.

[Daniel-Svensson]

Any progress on this?

@prawalagarwal you say that the there is no documented , but the documentation clearly states that the variable "_Common.TestResultsDirectory_" should point to the "The local path on the agent where the test results are created." and the trx and coverage files are the result of the test task.

I ve been trying to get the sonarcloud extension working again for the last week after you changed the behaviour without finding a solution I am satisfied with.

i've been trying to look for both to look in the temp directory as well as copy the output back to the correct directory, but in the latter case the .trx file is not found.

copy step can be found here but as you can se in the build log it does not find the trx file, do you delete it after publish?

Can you provide an example of how to access the trx file?

As long as you don't follow the expected outcome and place the test results in the test results directory we really need an easy way to get back the expected behavior.

The best solution would clearly be to put the test results under the test result directory (even if you need separate subfolders). If you really cannot do that you should provide an easy to use setting to store test results in the test results directory ("redirect test result to agent temp folder" or similar).

[kaadhina]

Hi,
The output location of the vstest task is not guaranteed to be the same. And we do not recommend using files from there. The task did not and does not drop results file in Common.TestResultsDirectory. Making the results file available from a drop location such as this is not supported by the task and we clean up the files once the task is done uploading them.
However you can access the uploaded file from the run attachments. There are APIs that will allow you to do the same:

1. You can find the run associated with the build using https://docs.microsoft.com/en-us/rest/api/azure/devops/test/runs/list?view=azure-devops-rest-5.0
2. Then get the run attachments using https://docs.microsoft.com/en-us/rest/api/azure/devops/test/attachments/get%20test%20run%20attachments?view=azure-devops-rest-5.0

[kaadhina]

Closing this now. Feel free to reopen if needed or open another one in case you hit into other issues.

[cocowalla]

@bryanmacfarlane the agent doesn't seem to be replacing variables. I tried this in my runsettings file:

<RunConfiguration>
    <ResultsDirectory>$(System.DefaultWorkingDirectory)\TestResults</ResultsDirectory>
</RunConfiguration>

But in the logs I see:

##[debug]CommonSettingsManager : TestResultsDirectory intialized to d:\a\_temp\$(System.DefaultWorkingDirectory)\TestResults\TestResults

and this from the updated runsettings file:

<ResultsDirectory>d:\a\_temp\$(System.DefaultWorkingDirectory)\TestResults\TestResults</ResultsDirectory>

Is there some way to access the generated trx files? At present it seems like it's impossible to access them in a Publish Test Results task, because the trx files are always dumped in a temp folder that's cleared after every task!

@kaadhina pointed out that there are APIs to access the attachments, but that's going to be a major PITA :frowning_face:

[ShreyasRmsft]

@cocowalla may I know why you need access to the test result files when the vstest task already uploads the test results for you?

because the trx files are always dumped in a temp folder that's cleared after every task!

This is not true by the way. The agent clears the temp folder _at the end of the pipeline and not after each task._

[cocowalla]

@ShreyasRmsft for 2 different reasons:

1. I want to use the ReportGenerator task to convert the code coverage reports into another format
2. I want to upload the test result and code coverage reports to Sonar Cloud (could just as well be any other SAST service tho)

[ShreyasRmsft]

Fair enough. You can find the trx and the coverage files in the $(Agent.TempDirectory)/TestResults

The reason we started putting them in temp directory is two fold:

• $(System.DefaultWorkingDirectory) is the place where the source code is present, the task generating the tests results there was basically altering the customer's source structure. Conceptually this was wrong and was a bug in the legacy task code. The distributed flow in vstest v2 task always used to use the temp directory.

• The second reason is cleanup, if the task crashed/hung/aborted there were cases where the results were left as is in the sources directory and never cleaned up. The agent today takes care of cleaning up the temp dir after the build/release (not after each task i re-iterate).

This left us a choice between using Common.TestResultsDirectory or Agent.TempDirectory. The distributed flow and re-run flows of the task (non legacy code) have always been using the temp directory and it made sense for the task experience to be uniform across all these flows.

Another reason we leaned towards Agent.TempDirectory is because these files we generate are intermediate and not a well documented contract (We did stop the task code from cleaning up the results when he heard that folks have taken a dependency on it though, the results should be present in the temp directory till the build is complete).

The recommended way is to get these metrics via the well documented apis.

This experience can simplified if the SonarQube task (and others if any) could get these attachments from the well-documented apis for you, thus ensuring this is a contract between the tasks that will be honored and customers will not have to bother about the internal implementation of where the task stores it's intermediate result files)

[ShreyasRmsft]

We are in the process of initiating conversations with the SonarQube folks to see what we can do towards this end.

[cocowalla]

@ShreyasRmsft

Fair enough. You can find the trx and the coverage files in the $(Agent.TempDirectory)/TestResults
For some reason this didn't work for me - I could see from the logs that the build task wrote the files here, but in the next task they were gone.

$(System.DefaultWorkingDirectory) is the place where the source code is present, the task generating the tests results there was basically altering the customer's source structure... The second reason is cleanup...

Few alternative options I see:

1. Place the files in Common.TestResultsDirectory (seems the obvious place for them, IMO!)
2. Place the files in a subfolder of $(Build.ArtifactStagingDirectory)
3. Better yet, make the path _configurable_ in the task

The recommended way is to get these metrics via the well documented apis

Yes, but that is a major PITA - it's a lot of effort to just access some files that you'd expect to be easily accessible.

This experience can simplified if the SonarQube task ... We are in the process of initiating conversations with the SonarQube folks

I'd politely request that you don't get carried away because I specifically mentioned SonarQube - this is a more general requirement, not specific to a single SAST tool or usage; for example, I also mentioned the ReportGenerator task. Ultimately, the best option for customers is to simply make the files easily accessible in the file system.

[ShreyasRmsft]

Option 3 ie. exposing this input in the task is something we are considering and I should be able to get back to you on this very soon.

As for not finding the files in the temp folder after the task:

May I know what version of the VsTest task you are running? Also let me know the version of the azure-pipelines agent you are running the build on.

Providing the full zip logs (obtained by clicking on download all logs in the build summary page) with system.debug = true should help me get to the bottom of this very quickly.