Vstest: dotnet test does not respect LogFileName -- adds date

[livarcocc]

_From @hpbieker on March 11, 2019 9:59_

Steps to reproduce

• Install .NET Core 3.0 preview3
• Create a test project
• Run dotnet test --logger 'trx;LogFileName=somename.trx'

Expected behavior

A file named somename.trx should be created. In .NET Core 2.2 / Test Execution Command Line Tool Version 15.9.0 I get this behaviour.

Actual behavior

A file named somename_2019-03-11_10-38-36-764.trx was created.

$ dotnet test --logger 'trx;LogFileName=somename.trx'
Test run for C:\Users\nohabie\AppData\Local\Temp\test\bin\Debug\netcoreapp3.0\test.dll(.NETCoreApp,Version=v3.0)
Microsoft (R) Test Execution Command Line Tool Version 16.0.0-preview-20190124-02
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Results File: C:\Users\XXX\AppData\Local\Temp\test\TestResults\somename_2019-03-11_10-38-36-764.trx

Total tests: 1. Passed: 1. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 1,1557 Seconds

Environment data

$ dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   3.0.100-preview3-010431
 Commit:    d72abce213

Runtime Environment:
 OS Name:     Windows
 OS Version:  6.1.7601
 OS Platform: Windows
 RID:         win7-x64
 Base Path:   C:\Program Files\dotnet\sdk\3.0.100-preview3-010431\

Host (useful for support):
  Version: 3.0.0-preview3-27503-5
  Commit:  3844df9537

.NET Core SDKs installed:
  2.1.600 [C:\Program Files\dotnet\sdk]
  2.2.200 [C:\Program Files\dotnet\sdk]
  3.0.100-preview3-010431 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0-preview3-19153-02 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0-preview3-27503-5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.0.0-preview3-27504-2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

_Copied from original issue: dotnet/cli#10962_

[ShreyasRmsft]

@livarcocc as of today this is by design.

We do this to create unique files. Consider the option as more of a logFilePrefix rather than logFileName.

And changing the param name is contract breaking change so we won't be able to rename it.

[hpbieker]

But it did not behave this way in In .NET Core 2.2 / Test Execution Command Line Tool Version 15.9.0 I get this behaviour? This change broke my build scripts because the trx file was not found anymore.

[ShreyasRmsft]

You are right on the behavior changing from 15.9.0 to 16.0.0. Let me double check the exact reason we made this change and get back to you.

[ShreyasRmsft]

@hpbieker this was done when dotnet test support at solution level was added i believe. The scenario was that when you ran it at a solution level multiple csprojs are present and multiple trx files are generated. To ensure these don't override each other this change was made.

I'm sorry this broke your build scripts. But I don't see a way to go back to honoring the exact file name without adding a post fix since it will break the multi csproj scenario.

You can look for trx files with a regex that checks for the name you have as the prefix. (This is what we do in the AzureDevops pipelines, to publish the trx-es). I hope the explanation was satisfactory?

[brittonbeckham]

BTW: this happens even when running tests against a single .csproj file. We have the same problem where it broke our CI pipes.

[ShreyasRmsft]

@brittonbeckham can you please create a new issue and provide the entire set of logs and screenshots (where applicable) for this?

[tremblaysimon]

@ShreyasRmsft This change broke our pipelines too.

Is it normal that a patch version (.NET Core SDK 2.1.602) brings a breaking change like that?

Workaround: change your build script or use SDK version 2.1.505.

Note that we used the microsoft/dotnet:2.1-sdk docker image available on Docker Hub. Since that image point to 2.1.602 instead of 2.1.505 it broke our pipelines.

[jonjomckay]

This also broke all of our build pipelines too. Nothing exotic is happening in this section of our pipelines - it's merely pulling the microsoft/dotnet:2.2-sdk Docker image and running dotnet vstest /logger:"trx;LogFileName=report.unit.xml", so it looks like this was a breaking change made in a non-major version update...?

[Kemyke]

Same here, broke our build pipelines too.

[Marusyk]

@ShreyasRmsft This change broke our pipelines too. What's going on?

[brittonbeckham]

I think MS wants us to open another issue for this, but seems like it shouldn't have been closed. A patch like this to a container image shouldn't have been a breaking change without, at minimum, a minor version number change. We found a work around for the meantime, but this is definitely frustrating.

On the flip side, I am happy and excited about where all this .NET Core stuff is going and have been using it heavily. Having used .NET since v1, i can say I have never been more excited to code in .NET Core. Cheers to the entire team!

[cltshivash]

@brittonbeckham @Marusyk @Kemyke Apologies for the break here and do acknowledge that we should have called this out broadly and with an equivalent version update. The cause of the change was to address the scenario of multi targeted test project where the log file provided was being overwritten and ending up with data loss. We will revisit the change and see if we could retain the earlier behavior and at the same time address the issue with multi target test projects.

[mayankbansal018]

Adding reference to https://github.com/Microsoft/vstest/issues/1978 , we plan to publish an RFC on how trx file name would be generated, will update once that is out.

[andyturner]

+1, broke our build chain too. I'm surprised such a change was implemented without any backwards compatibility. Was it not more sensible to implement a "UniqueLogFileName" parameter for those people with the multi-target issue to use, and to leave the behaviour of "LogFileName" untouched/unbroken?

[subprime]

+1, broke our build chain too.

[crhairr]

This broke our build scripts that execute inside of Jenkins. The test result collector at the end of our build pipeline stages can no longer find the files. We have many, many dozens of these build scripts, all run inside Docker images. Fixing this is a huge issue for us, and a reason not to adopt the newer .NET Core versions.

We also have many multi-test solutions in place. Our pattern is to run each test project separately and collect the result files at the end of the stage. Your "multi test" solution is for a problem that many developers have already worked around.

The better solution would have been to allow developers the options to change the suffix if we had needed to, rather than forcing it on us and breaking our builds without warning.

Your solution to a problem that didn't exist is bad.

[SeppPenner]

+1, broke our build chain too. I'm surprised such a change was implemented without any backwards compatibility. Was it not more sensible to implement a "UniqueLogFileName" parameter for those people with the multi-target issue to use, and to leave the behaviour of "LogFileName" untouched/unbroken?

Broke our scripts as well. I totally agree with @andyturner and @crhairr. It should be backwards compatible and simply provide the possibility to add prefixes/postfixes if needed.

[vany0114]

+1 broke my build too

[mikrj]

+1 It broke our builds too, it would be better with possibility for prefixes/postfixes.

Also the description in dotnet test -h is now misleading:
Log in trx format using the specified file name: --logger "trx;LogFileName=<TestResults.trx>"

[ecoffman]

+1 Our scripts are broken now, too.

I fixed it like so:

sh 'dotnet test src/hub-backend.sln --verbosity m --logger "trx;LogFileName=TestResult.xml"'
script {
    TRX_FILE = sh (script: "ls -t src/<hidden>/TestResults/TestResult*.xml | head -1", returnStdout: true).trim()
}
step([$class: 'MSTestPublisher', testResultsFile:"$TRX_FILE", failOnError: true, keepLongStdio: true])

[Brunni]

+1
It broke our builds, too!
Our command: dotnet test --logger "trx;LogFileName=TestResults.trx"

[alirezava]

+1 broke our build too.

Microsofts cowboy builder at it again...

Any updates on this?

[mayankbansal018]

It was fixed via https://github.com/microsoft/vstest/pull/1996
Release: https://github.com/microsoft/vstest/releases/tag/v16.0.2-preview-20190502-01
Notes: https://github.com/Microsoft/vstest-docs/blob/master/docs/releases.md#1602-preview-20190502-01

[hrai]

@hpbieker this was done when dotnet test support at solution level was added i believe. The scenario was that when you ran it at a solution level multiple csprojs are present and multiple trx files are generated. To ensure these don't override each other this change was made.

I'm sorry this broke your build scripts. But I don't see a way to go back to honoring the exact file name without adding a post fix since it will break the multi csproj scenario.

You can look for trx files with a regex that checks for the name you have as the prefix. (This is what we do in the AzureDevops pipelines, to publish the trx-es). I hope the explanation was satisfactory?

Using the trx regex like TestResult_*.trx in my build script fixed it temporarily until we get the sdk up to date.

[hunyhabib]

Now My tests also are breaking with Microsoft.Net.Test.SDK V16.3.0 for same reason