Azure-pipelines-tasks: dotnet test "No test result files were found"

Created on 30 Jun 2018 · 11Comments · Source: microsoft/azure-pipelines-tasks

Environment

Server - VSTS

Account Name: msdata
Team Project: CosmosDB
Build Definition: CosmosSDK-Gated-clone
Build Number: 1116757

Agent - Hosted VS 2017

Issue Description

When running dotnet test the test run, succeed, but even though Publish test results is checked, we get a No test result files were found.

We can see that the command is added the --logger trx argument in the call.
2018-06-30T20:47:10.5674902Z [command]"C:\Program Files\dotnet\dotnet.exe" test D:\a\1\s\Product\SDK.net\Microsoft.Azure.Documents.Client.Test\Microsoft.Azure.Cosmos.SDK.NetFramework.Tests.csproj --no-build --filter TestCategory!=Quarantine --configuration Debug /p:TargetPlatform=x64 -- RunConfiguration.TargetPlatform=x64 --logger trx --results-directory D:\a_temp
`

Error logs

    2018-06-30T20:47:10.5674902Z [command]"C:\Program Files\dotnet\dotnet.exe" test D:\a\1\s\Product\SDK\.net\Microsoft.Azure.Documents.Client.Test\Microsoft.Azure.Cosmos.SDK.NetFramework.Tests.csproj --no-build --filter TestCategory!=Quarantine --configuration Debug /p:TargetPlatform=x64 -- RunConfiguration.TargetPlatform=x64 --logger trx --results-directory D:\a\_temp
    2018-06-30T20:47:13.6658343Z Test run for D:\a\1\s\Product\SDK\.net\Microsoft.Azure.Documents.Client.Test\bin\Debug\net461\Microsoft.Azure.Cosmos.SDK.NetFramework.Tests.dll(.NETFramework,Version=v4.6.1)
    2018-06-30T20:47:14.1842026Z Microsoft (R) Test Execution Command Line Tool Version 15.7.0
    2018-06-30T20:47:14.1842252Z Copyright (c) Microsoft Corporation.  All rights reserved.
    2018-06-30T20:47:14.1842344Z 
    2018-06-30T20:47:15.4323958Z Starting test execution, please wait...
    2018-06-30T20:47:54.4002303Z 
    2018-06-30T20:47:54.4011105Z Total tests: 176. Passed: 176. Failed: 0. Skipped: 0.
    2018-06-30T20:47:54.4011573Z Test Run Successful.
    2018-06-30T20:47:54.4017649Z Test execution time: 38.6527 Seconds
    2018-06-30T20:47:54.4497955Z ##[debug]rc:0
    2018-06-30T20:47:54.4499172Z ##[debug]success:true
    2018-06-30T20:47:54.4504087Z ##[debug]BuildConfiguration=Debug
    2018-06-30T20:47:54.4505110Z ##[debug]BuildPlatform=AnyCPU
    2018-06-30T20:47:54.4505416Z ##[debug]defaultRoot: 'D:\a\_temp'
    2018-06-30T20:47:54.4505722Z ##[debug]findOptions.followSpecifiedSymbolicLink: 'true'
    2018-06-30T20:47:54.4505998Z ##[debug]findOptions.followSymbolicLinks: 'true'
    2018-06-30T20:47:54.4506261Z ##[debug]matchOptions.debug: 'false'
    2018-06-30T20:47:54.4506542Z ##[debug]matchOptions.nobrace: 'true'
    2018-06-30T20:47:54.4507142Z ##[debug]matchOptions.noglobstar: 'false'
    2018-06-30T20:47:54.4507811Z ##[debug]matchOptions.dot: 'true'
    2018-06-30T20:47:54.4508084Z ##[debug]matchOptions.noext: 'false'
    2018-06-30T20:47:54.4508356Z ##[debug]matchOptions.nocase: 'true'
    2018-06-30T20:47:54.4508610Z ##[debug]matchOptions.nonull: 'false'
    2018-06-30T20:47:54.4508883Z ##[debug]matchOptions.matchBase: 'false'
    2018-06-30T20:47:54.4509156Z ##[debug]matchOptions.nocomment: 'false'
    2018-06-30T20:47:54.4513852Z ##[debug]matchOptions.nonegate: 'false'
    2018-06-30T20:47:54.4514172Z ##[debug]matchOptions.flipNegate: 'false'
    2018-06-30T20:47:54.4514601Z ##[debug]pattern: '**/*.trx'
    2018-06-30T20:47:54.4514907Z ##[debug]findPath: 'D:\a\_temp'
    2018-06-30T20:47:54.4515159Z ##[debug]statOnly: 'false'
    2018-06-30T20:47:54.4515427Z ##[debug]findPath: 'D:\a\_temp'
    2018-06-30T20:47:54.4515689Z ##[debug]findOptions.followSpecifiedSymbolicLink: 'true'
    2018-06-30T20:47:54.4515987Z ##[debug]findOptions.followSymbolicLinks: 'true'
    2018-06-30T20:47:54.4516242Z ##[debug]  D:\a\_temp (directory)
    2018-06-30T20:47:54.4516574Z ##[debug]1 results
    2018-06-30T20:47:54.4517273Z ##[debug]found 1 paths
    2018-06-30T20:47:54.4517628Z ##[debug]applying include pattern
    2018-06-30T20:47:54.4517897Z ##[debug]adjustedPattern: 'D:\a\_temp\**/*.trx'
    2018-06-30T20:47:54.4518205Z ##[debug]0 matches
    2018-06-30T20:47:54.4518478Z ##[debug]0 final results

Test bug

Source

ealsur

All 11 comments

@ealsur We are able to reproduce the issue. We are working on a fix for this.

abhishkk on 5 Jul 2018

👍1

@abhishkk I was able to detect the issue. It seems the order of parameters is affecting the behavior.

If the --logger parameter is after the /p:TargetPlatform switch, then no file gets generated (this happens all the time when using the Publish test results checkbox since it appends it). If the --logger parameter is before then it works. Not sure if it's an issue with dotnet CLI though.

Instead of using the dotnet test task, I went with a dotnet custom task and manually added the --logger parameter before the /p:TargetPlatform switch.

2018-07-05T22:03:46.4696014Z ##[section]Starting: Run .NET Framework Tests
2018-07-05T22:03:46.4718250Z ==============================================================================
2018-07-05T22:03:46.4718407Z Task         : .NET Core
2018-07-05T22:03:46.4718562Z Description  : Build, test, package, or publish a dotnet application, or run a custom dotnet command. For package commands, supports NuGet.org and authenticated feeds like Package Management and MyGet.
2018-07-05T22:03:46.4718710Z Version      : 2.135.0
2018-07-05T22:03:46.4718822Z Author       : Microsoft Corporation
2018-07-05T22:03:46.4718933Z Help         : [More Information](https://go.microsoft.com/fwlink/?linkid=832194)
2018-07-05T22:03:46.4719052Z ==============================================================================
2018-07-05T22:03:47.2448367Z [command]C:\Windows\system32\chcp.com 65001
2018-07-05T22:03:47.2534777Z Active code page: 65001
2018-07-05T22:03:51.5858206Z [command]"C:\Program Files\dotnet\dotnet.exe" test D:\a\1\s\Product\SDK\.net\Microsoft.Azure.Documents.Client.Test\Microsoft.Azure.Cosmos.SDK.NetFramework.Tests.csproj --no-build --filter TestCategory!=Quarantine --configuration Debug --logger trx --results-directory D:\a\_temp /p:TargetPlatform=x64 -- RunConfiguration.TargetPlatform=x64
2018-07-05T22:03:54.4983312Z Test run for D:\a\1\s\Product\SDK\.net\Microsoft.Azure.Documents.Client.Test\bin\Debug\net461\Microsoft.Azure.Cosmos.SDK.NetFramework.Tests.dll(.NETFramework,Version=v4.6.1)
2018-07-05T22:03:55.0182688Z Microsoft (R) Test Execution Command Line Tool Version 15.7.0
2018-07-05T22:03:55.0182933Z Copyright (c) Microsoft Corporation.  All rights reserved.
2018-07-05T22:03:55.0183052Z 
2018-07-05T22:03:56.4967542Z Starting test execution, please wait...
2018-07-05T22:05:24.5065651Z Results File: D:\a\_temp\VssAdministrator_factoryvm-az217_2018-07-05_22_04_04.trx
2018-07-05T22:05:24.5065895Z 
2018-07-05T22:05:24.5066001Z Total tests: 178. Passed: 178. Failed: 0. Skipped: 0.
2018-07-05T22:05:24.5066104Z Test Run Successful.
2018-07-05T22:05:24.5068525Z Test execution time: 1.4597 Minutes
2018-07-05T22:05:24.5746035Z ##[section]Finishing: Run .NET Framework Tests

ealsur on 6 Jul 2018

Closing since this sounds rather an issue in dotnet CLI. But just in case someone else runs into this, it's good info.

ealsur on 12 Jul 2018

@ealsur
Firstly correcting one minor mistake. /p:TargetPlatform=x64 -- RunConfiguration.TargetPlatform=x64. Both these parameters are not required. Only -- RunConfiguration.TargetPlatform=x64 is sufficient.

Secondly, your above interpretation is partially correct. Actually in dotnet commandline args, parameters updating runsettings should always come in last.

Check below lines in https://github.com/Microsoft/vstest-docs/blob/master/docs/RunSettingsArguments.md

All the arguments after -- will be treated as RunSettings arguments, means RunSettings arguments should be at the end of the command line.

For example,
"C:\Program Files\dotnet\dotnet.exe" test D:\a\1\s\Product\SDK\.net\Microsoft.Azure.Documents.Client.Test\Microsoft.Azure.Cosmos.SDK.NetFramework.Tests.csproj --no-build --filter TestCategory!=Quarantine --configuration Debug /p:TargetPlatform=x64 -- RunConfiguration.TargetPlatform=x64 --logger trx --results-directory D:\a\_temp

Here --logger trx and --results-directory D:\a\_temp are coming after --. Thus trx file is not getting generated.

In another example,
"C:\Program Files\dotnet\dotnet.exe" test D:\a\1\s\Product\SDK\.net\Microsoft.Azure.Documents.Client.Test\Microsoft.Azure.Cosmos.SDK.NetFramework.Tests.csproj --no-build --filter TestCategory!=Quarantine --configuration Debug --logger trx --results-directory D:\a\_temp /p:TargetPlatform=x64 -- RunConfiguration.TargetPlatform=x64

Here --logger trx and --results-directory D:\a\_temp are coming before -- and thus trx file is getting created.

Now the problem which happens from dotnet task is that whenever Publish test results option is checked, we are adding --logger trx at the end. Thus whenever runsettings parameters are added by user in args, trx file does not generates.

So, re-opening the issue as we need to fix this and add --logger trx before user provided args.

abhishkk on 12 Jul 2018

Thanks for the detailed explanation @abhishkk ! It makes total sense 😄

ealsur on 13 Jul 2018

Thus whenever runsettings parameters are added by user in args, trx file does not generates.

I wonder if my problem is related?

When I run my tests with the 'Rerun failed tests' option the .trx file is deleted or not generated.

2018-07-17T05:14:46.2800684Z Results File: d:\VSO2\116\s\TestResults\STS-TFS-BUILD2$_STS-TFS-BUILD2_2018-07-17_07_11_01.trx
2018-07-17T05:14:46.3753721Z **************** Completed test execution *********************
2018-07-17T05:14:46.3761559Z Test results files: d:\VSO2\116\s\TestResults\STS-TFS-BUILD2$_STS-TFS-BUILD2_2018-07-17_07_11_01.trx
2018-07-17T05:14:46.9294958Z Created test run: 30410
2018-07-17T05:14:46.9299099Z Publishing test results: 12
2018-07-17T05:14:46.9325479Z Publishing test results to test run '30410'.
2018-07-17T05:14:46.9967503Z TestResults To Publish 12, Test run id:30410
2018-07-17T05:14:47.0012539Z Test results publishing 12 ( HierarchicalCount 7), remaining: 0. Test run id: 30410
2018-07-17T05:14:48.2934002Z Published test results: 12
2018-07-17T05:14:48.2935151Z Publishing Attachments: 1
2018-07-17T05:14:48.6038842Z Failed tests: 7; Total tests: 12;
2018-07-17T05:14:48.6039085Z ************* Rerunning failed tests for Test run 30410 is disabled ************
2018-07-17T05:14:48.6705568Z Completed TestExecution Model...
2018-07-17T05:14:48.7072560Z ##[warning]Vstest failed with error. Check logs for failures. There might be failed tests.
2018-07-17T05:14:48.7407342Z ##[error]Error: d:\VSO2\_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\2.136.10\Modules\DTAExecutionHost.exe failed with return code: 1
2018-07-17T05:14:48.8314265Z ##[error]VsTest task failed.
2018-07-17T05:14:48.8609409Z ##[section]Finishing: Specflow tests
2018-07-17T05:14:48.8633587Z ##[section]Starting: Rename testresults file
2018-07-17T05:14:48.8762025Z ==============================================================================
2018-07-17T05:14:48.8762159Z Task         : Command Line
2018-07-17T05:14:48.8762284Z Description  : Run a command line with arguments
2018-07-17T05:14:48.8762393Z Version      : 1.1.3
2018-07-17T05:14:48.8762495Z Author       : Microsoft Corporation
2018-07-17T05:14:48.8762668Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkID=613735)
2018-07-17T05:14:48.8762800Z ==============================================================================
2018-07-17T05:14:48.8845352Z ##[command]Rename "d:\VSO2\116\s\TestResults\*.trx" TestResults.trx
2018-07-17T05:14:48.9467115Z The system cannot find the file specified.
2018-07-17T05:14:49.0278866Z ##[error]Process completed with exit code 1.

Also notice the line

************* Rerunning failed tests for Test run 30410 is disabled ************

weird?

anniekvandijk on 17 Jul 2018

@anniekvandijk
Issues seems different. Can you please create a separate issue for this so as to track it properly. In addition, can you share debug logs with us?

abhishkk on 17 Jul 2018

@anniekvandijk is your issue resolved? If you are still facing it please open a new issue and provide a snapshot of your build definition and provide the entire build logs with system.debug=true

ShreyasRmsft on 3 Sep 2018

Fixed

abhishkk on 3 Jan 2019

🎉1

@abhishkk I just ran into the same issue with Azure Pipelines with .NET Core SDK 2.2.301. Here is how my test task looks like

- task: DotNetCoreCLI@2
      displayName: 'Test'
      inputs: 
        command: 'test'
        projects: $(testProjects)
        arguments: '--configuration $(buildConfiguration) --no-build'

Here is the output

##[section]Starting: Test
==============================================================================
Task         : .NET Core
Description  : Build, test, package, or publish a dotnet application, or run a custom dotnet command
Version      : 2.154.6
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/dotnet-core-cli
==============================================================================
[command]/vsts/agent/_work/_tool/dncs/2.2.301/x64/dotnet test /vsts/agent/_work/2/s/test/XXX.Tests/XXX.Tests.csproj --logger trx --results-directory /vsts/agent/_work/_temp --configuration Release --no-build
##[warning]No test result files were found.
##[section]Finishing: Test

syedhassaanahmed on 19 Jul 2019

@syedhassaanahmed,

Make sure your test projects contain <IsTestProject>true<IsTestProject>
Make sure you're doing a dotnet build with the same configuration before dotnet test
Can you try this without --no-build to check whether that flag is the culprit

Since this is a very old thread, I'll be closing this. If you still face issues, feel free to create a new one with references to my handle and this issue.