Azure-devops-docs: CTest support

Created on 6 Apr 2019 · 11Comments · Source: MicrosoftDocs/azure-devops-docs

Does CTest supported in Azure DevOps Server 2019 (On-Premise)? Using CTest as testResultsFormat cause Unknown Test Runner error.

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ba91e46a-de53-c0c1-f7f7-c02e797d527c
Version Independent ID: 6f478148-461f-694a-ed82-deb4d731854e
Content: Publish Test Results task - Azure Pipelines
Content Source: docs/pipelines/tasks/test/publish-test-results.md
Product: devops
Technology: devops-cicd
GitHub Login: @alexhomer1
Microsoft Alias: ahomer

Pri1 devops-cictech devopprod doc-bug

Source

evakili

Most helpful comment

I managed to get it working despite the lack of documentation. You need to execute ctest with parameter -T Test. This will generate a Testing/<generated tag>/Test.xml in the expected format. Then use something like

  - task: PublishTestResults@2
    inputs:
      testResultsFormat: cTest
      testResultsFiles: build/Testing/*/Test.xml
    ...

(build is my cmake build directory)

Kha on 28 Jun 2019

👍5

All 11 comments

The docs suggest that cTest (with a lowercase c) is how that's specified. Did you try that?

rjmholt on 26 Apr 2019

@rjmholt
Yes. I try both CTest and cTest.

evakili on 27 Apr 2019

👍1

I am also unable to use cTest as the results format. When I try to publish results from ctest I receive the following from the publish test results task:

##[debug]Evaluating condition for step: 'Publish Test Results **/*.xml'
##[debug]Evaluating: succeededOrFailed()
##[debug]Evaluating succeededOrFailed:
##[debug]=> True
##[debug]Result: True
##[section]Starting: Publish Test Results **/*.xml
==============================================================================
Task         : Publish Test Results
Description  : Publish test results to Azure Pipelines
Version      : 2.152.3
Author       : Microsoft Corporation
Help         : [More Information](https://go.microsoft.com/fwlink/?LinkID=613742)
==============================================================================
##[debug]agent.TempDirectory=REDACTED
##[debug]loading inputs and endpoints
##[debug]loading INPUT_TESTRUNNER
##[debug]loading INPUT_TESTRESULTSFILES
##[debug]loading INPUT_SEARCHFOLDER
##[debug]loading INPUT_MERGETESTRESULTS
##[debug]loading INPUT_FAILTASKONFAILEDTESTS
##[debug]loading INPUT_TESTRUNTITLE
##[debug]loading INPUT_PLATFORM
##[debug]loading INPUT_CONFIGURATION
##[debug]loading INPUT_PUBLISHRUNATTACHMENTS
##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
##[debug]loaded 12
##[debug]Agent.ProxyUrl=undefined
##[debug]Agent.CAInfo=undefined
##[debug]Agent.ClientCert=undefined
##[debug]Agent.SkipCertValidation=undefined
##[debug]check path : REDACTED/_work/_tasks/PublishTestResults_0b0f01ed-7dde-43ff-9cbb-e48954daf9b1/2.152.3/task.json
##[debug]adding resource file: REDACTED/_work/_tasks/PublishTestResults_0b0f01ed-7dde-43ff-9cbb-e48954daf9b1/2.152.3/task.json
##[debug]system.culture=en-US
##[debug]testRunner=CTest
##[debug]testResultsFiles=**/*.xml
##[debug]mergeTestResults=false
##[debug]platform=buildPlatform
##[debug]configuration=buildConfiguration
##[debug]testRunTitle=Test results
##[debug]publishRunAttachments=true
##[debug]failTaskOnFailedTests=false
##[debug]searchFolder=REDACTED/_work/28/a
##[debug]testRunner: CTest
##[debug]testResultsFiles: **/*.xml
##[debug]mergeResults: false
##[debug]platform: buildPlatform
##[debug]config: buildConfiguration
##[debug]testRunTitle: Test results
##[debug]publishRunAttachments: true
##[debug]failTaskOnFailedTests: false
##[debug]defaultRoot: 'REDACTED/_work/28/a'
##[debug]findOptions.allowBrokenSymbolicLinks: 'true'
##[debug]findOptions.followSpecifiedSymbolicLink: 'true'
##[debug]findOptions.followSymbolicLinks: 'true'
##[debug]matchOptions.debug: 'false'
##[debug]matchOptions.nobrace: 'true'
##[debug]matchOptions.noglobstar: 'false'
##[debug]matchOptions.dot: 'true'
##[debug]matchOptions.noext: 'false'
##[debug]matchOptions.nocase: 'false'
##[debug]matchOptions.nonull: 'false'
##[debug]matchOptions.matchBase: 'false'
##[debug]matchOptions.nocomment: 'false'
##[debug]matchOptions.nonegate: 'false'
##[debug]matchOptions.flipNegate: 'false'
##[debug]pattern: '**/*.xml'
##[debug]findPath: 'REDACTED/_work/28/a'
##[debug]statOnly: 'false'
##[debug]findPath: 'REDACTED/_work/28/a'
##[debug]findOptions.allowBrokenSymbolicLinks: 'true'
##[debug]findOptions.followSpecifiedSymbolicLink: 'true'
##[debug]findOptions.followSymbolicLinks: 'true'
##[debug]  REDACTED/_work/28/a (directory)
##[debug]  REDACTED/_work/28/a/artifacts (directory)
[REDACTED]
##[debug]125 results
##[debug]found 125 paths
##[debug]applying include pattern
##[debug]adjustedPattern: 'REDACTED/_work/28/a/**/*.xml'
##[debug]97 matches
##[debug]97 final results
##[debug]Detected 97 test result files
##[debug]PublishTestResults.OverrideExeFlow=undefined
##[debug]OS type: Linux
##[debug]Reading test results from file 'REDACTED'
##[debug]Reading test results from file 'REDACTED'
##[debug]Reading test results from file 'REDACTED'
##[debug]Reading test results from file 'REDACTED'
##[debug]Reading test results from file 'REDACTED'
##[debug]Reading test results from file 'REDACTED'
##[debug]Reading test results from file 'REDACTED'
##[debug]Reading test results from file 'REDACTED'
##[debug]Reading test results from file 'REDACTED'
##[warning]Failed to publish test results: Object reference not set to an instance of an object.
##[debug]Processed: ##vso[results.publish type=CTest;mergeResults=false;platform=buildPlatform;config=buildConfiguration;runTitle=Test results;publishRunAttachments=true;resultFiles=REDACTED;failTaskOnFailedTests=false;testRunSystem=VSTS - PTR;]
##[debug]task result: Succeeded
##[debug]Processed: ##vso[task.complete result=Succeeded;]
##[debug]Release.ReleaseUri=undefined
##[debug]Release.ReleaseId=undefined
##[debug]Build.BuildUri=vstfs:///Build/Build/REDACTED
##[debug]Build.Buildid=REDACTED
##[debug]Processed: ##vso[telemetry.publish area=TestExecution;feature=PublishTestResultsTask]{"builduri":"vstfs:///Build/Build/REDACTED","buildid":"REDACTED","osType":"Linux","testRunner":"CTest","failTaskOnFailedTests":"false","mergeResultsUserPreference":"false","config":"buildConfiguration","platform":"buildPlatform","testResultsFilesCount":97,"subFeature":"publishTestResultsTaskConsolidatedCiEvent"}
##[section]Async Command Start: Publish test results
##[section]Async Command End: Publish test results
##[section]Finishing: Publish Test Results **/*.xml

ablomber on 13 Jun 2019

  - task: PublishTestResults@2
    inputs:
      testResultsFormat: cTest
      testResultsFiles: build/Testing/*/Test.xml
    ...

(build is my cmake build directory)

Kha on 28 Jun 2019

👍5

I am having same issue Unknown Test Runner . I tried generating Tag as mentioned below but it didnt work out

bindunalam on 24 Jul 2019

@bindunalam The runner is working perfectly for us, see https://github.com/leanprover/lean4/blob/4187a099c2cba410252a031f9d402e6f9a81e225/azure-pipelines.yml#L28. Could you post your configuration?

Kha on 24 Jul 2019

👍2

Here is the output

##[section]Starting: Publish UnitTests

Task : Publish Test Results
Description : Publish test results to Azure Pipelines
Version : 2.154.0
Author : Microsoft Corporation

Help : Learn more about this task

[error]Unable to process command '##vso[results.publish type=CTest;mergeResults=false;runTitle=ctest;publishRunAttachments=true;resultFiles=**/Test.xml;failTaskOnFailedTests=false;testRunSystem=VSTS - PTR;]' successfully. Please reference documentation (http://go.microsoft.com/fwlink/?LinkId=817296)

[error]Unknown Test Runner.

[section]Finishing: Publish UnitTests

bindunalam on 24 Jul 2019

It looks like you're using CTest instead of cTest.

Kha on 24 Jul 2019

Thankyou for pointing that out . I used a regular publish test task and above is the YAML code which i got from PublishTest task . So i didnt choose CTest . so i went ahead and used a yaml pipeline just to publish tests and it worked .This is a bug testrunner format in predefined task is taking as CTest

bindunalam on 24 Jul 2019

I have the same problem.
I use:

  - task: PublishTestResults@2
    condition: succeededOrFailed()
    inputs:
      testRunner: cTest
      testResultsFiles: '**/Testing/*/Test.xml'
      testRunTitle: 'test_win'

It's on Azure DevOps server 17.143.28511.3 with agent version 2.141.1.
There is restriction on Agent version ?