Azure-pipelines-tasks: PublishTestResults@2 version 2.170.0 fails to upload tasks due to missing PTR_TEST_RUNSUMMARY.json file

[colindembovsky]

Question, Bug, or Feature?
Bug

Enter Task Name: PublishTestResults@2

Environment

• Server - Azure Pipelines

  • If using Azure Pipelines, provide the account name, team project name, build definition name/build number:
    Account: MQAutomate
    Team Project Name: MOSAIQLab
    Build Definition Name: Prereq Test
    Build number - all

• Agent - Private (VM Environment) on Windows Servers:

Issue Description

Task version 2.170.0 fails to upload tasks, even though it completes successfully.

"Normal" log:

C:\azagent\A1\_work\_tasks\PublishTestResults_0b0f01ed-7dde-43ff-9cbb-e48954daf9b1\2.170.0\modules\TestResultsPublisher.exe @C:\azagent\A1\_work\_temp\40166ab0-a773-11ea-b9d3-91ff84b586bc.txt 
Finishing: Upload test reports 

With diagnostic logs on, we see the following:

##[debug]Exit code 2148734720 received from tool 'C:\azagent\A1\_work\_tasks\PublishTestResults_0b0f01ed-7dde-43ff-9cbb-e48954daf9b1\2.170.0\modules\TestResultsPublisher.exe' 
##[debug]STDIO streams have closed for tool 'C:\azagent\A1\_work\_tasks\PublishTestResults_0b0f01ed-7dde-43ff-9cbb-e48954daf9b1\2.170.0\modules\TestResultsPublisher.exe' 
##[debug]Exit code of TestResultsPublisher: 2148734720 
##[debug]Agent.Version=2.169.1 
##[debug]Agent.TempDirectory=C:\azagent\A1\_work\_temp 
##[debug]Unable to publish the test run summary to evidencestore, error details:Error: ENOENT: no such file or directory, open 'C:\azagent\A1\_work\_temp\PTR_TEST_RUNSUMMARY.json' 
##[debug]task result: Succeeded 

It appears that the code is looking for the PTR_TEST_RUNSUMMARY.json but this file does not exists. I took a quick scan through the repo and can only find this instance of _reading_ the file - but not where the file is _produced_.

Task logs

Inline above.

Troubleshooting

Checkout how to troubleshoot failures and collect debug logs: https://docs.microsoft.com/en-us/vsts/build-release/actions/troubleshooting

Error logs

No error logs.

[navin22]

@colindembovsky The debug statement is actually misleading. Since there are no tests found for the task, so there's no summary may be we should change those lines to avoid confusion.

To root cause the original issue.
What are you trying to publish .trx, .xml (vstest, nunit, xunit or junit) tests ? If possible can you share the pipeline details or the test reports to help you further ?

[colindembovsky]

Thanks @navin22. The pipeline is publishing junit test result files that are generated by Pester. We have this pipeline in 2 orgs: one is a "master" org and the other is for testing.

When we run the pipeline in the master org, the 2.169.0 version of PublishTestResults@2 runs and we get tests in the summary as expected. When we run the same pipeline in the test org, version 2.170.0 of the PublishTestResults@2 runs and we get the error above in DEBUG and there are not test results.

When there have been problems with the test results file (or this file is not found) there is usually a warning saying that no test results were found. The 2.170.0 task is finding test results (I can see this in the log) but it is erroring as above.

A run of the pipeline that works (in master org with test results as expected, using 2.169.0) is here.

Unfortunately, we've deleted the team project with an example of the pipeline that does not work but I will add a link to that when we have one. By the way, we've deleted and created the Team Project and pipeline multiple times and every time PublishTestResults@2 runs on version 2.170.0 we do not get any test results.

[navin22]

@colindembovsky can we get to see the entire debug logs ?
and are you using the self hosted docker container agent ?

[colindembovsky]

@navin22 I don't have a log for a non-working pipeline. In our original org the version that runs is 2.169.0 which works. New orgs seem to get 2.170.0. As soon as I have a log for a failure again I will scrub and post. However. I'm sure just uploading a junit XML results file will allow you to repro?

This job is running in a VM environment (yml deployment job) so it's a private agent. No docker involved at all.

[navin22]

@colindembovsky Okay.

I see the TestResultsPublisher.exe tool is exiting with different exit code than usual, that we saw only docker hosted agent. Can you try passing PublishTestResults.OverrideExeFlow = true to your pipeline and run to see if exe is causing the issue.?

[colindembovsky]

Hi @navin22 - looks like that switch helped!

Here's a run without the switch: link
Here's a run with diagnostics enabled and the switch: link

Here's a snippet of the log from the second pipeline (that works as expected):

##[debug]PublishTestResults.OverrideExeFlow=true
##[debug]OS type: Windows_NT
##[debug]Reading test results from file 'C:\azagent\A4\_work\1\s\tests\SQL-Prereqs.results.xml'
Timestamp is not available for one or more testsuites. Total run duration is being calculated as the sum of time durations of detected testsuites
##[debug]Processed: ##vso[results.publish type=JUnit;mergeResults=false;runTitle=SQL Prereqs;publishRunAttachments=true;resultFiles=C:\azagent\A4\_work\1\s\tests\SQL-Prereqs.results.xml;failTaskOnFailedTests=false;testRunSystem=VSTS - PTR;]
##[debug]task result: Succeeded
##[debug]Processed: ##vso[task.complete result=Succeeded;]
##[debug]Release.ReleaseUri=undefined
##[debug]Release.ReleaseId=undefined
##[debug]Build.BuildUri=vstfs:///Build/Build/13
##[debug]Build.Buildid=13
##[debug]Processed: ##vso[telemetry.publish area=TestExecution;feature=PublishTestResultsTask]{"builduri":"vstfs:///Build/Build/13","buildid":"13","osType":"Windows_NT","testRunner":"JUnit","failTaskOnFailedTests":"false","mergeResultsUserPreference":"false","config":null,"platform":null,"testResultsFilesCount":1,"dotnetVersion":"","subFeature":"publishTestResultsTaskConsolidatedCiEvent"}
Async Command Start: Publish test results
Publishing test results to test run '2'
Test results remaining: 35. Test run id: 2
##[debug]Uploading test run attachements individually
Published Test Run : https://tfsprodcus5.visualstudio.com/Aab78a5bf-b4fe-4d61-94c4-3a08dc88f8f2/MOSAIQ%20Training%20Lab/_TestManagement/Runs?runId=2&_a=runCharts
TestRunPublisher.IsTestRunFailed: Getting test run summary with run id: 2 using stats api
TestRunPublisher.IsTestRunFailed: checking if test run is failed using run summary input
Number of failed tests which are non-flaky is: 12
Failed Results are published
##[debug]RESOURCE_URIS:
##[debug]Setting task variable METADATA_8e005faf-00a8-404a-8d4c-ab0cbb50243e: {"name":"4a70e4e3-6377-4168-8999-a34982821314","resourceUris":[],"metadata":{"description":"","relatedUrls":[{"url":"https://dev.azure.com/MQAutomate/MOSAIQ Training Lab/_build/results?buildId=13","label":"pipeline-url"}],"humanReadableName":"Test Results from Publish Test Results utility","serializedPayload":"{\"testId\":\"PublishTestResults\",\"testTool\":\"JUnit\",\"testResultAttestation\":{\"total\":35,\"failed\":12,\"passed\":20,\"skipped\":0},\"testDurationSeconds\":0.0,\"testPassPercentage\":\"57.14285714285714\",\"relatedUrls\":[{\"url\":\"

[navin22]

@colindembovsky good that you have workaround now to unblock yourself.

Meanwhile I will work on the root cause. I don't have acess to the link you have provided. Can you please download the entire logs and share here for the one without switch which is causing the problem.

[colindembovsky]

@navin22 Can you give me an email address to share the logs?

[navin22]

@colindembovsky it's [email protected]

[colindembovsky]

@navin22 just emailed logs to you

[MuckT]

@colindembovsky seeing a similar issue on 2.170.0

After running the tests I can see the results

pwd
Directory: D:\a\1\s\WebApp\ClientApp\test-results
ls
Contents of directory:
Mode                LastWriteTime         Length Name                                                                  
----                -------------         ------ ----                                                                  
-a----        6/17/2020   3:23 AM            120 junit.xml 

When attempting to publish the results:
No Result Found to Publish 'D:\a\1\s\WebApp\ClientApp\test-results\junit.xml'.

[colindembovsky]

@MuckT I think that's a different issue. If it cannot fine the XML file, you'll get that error message. My issue was that the file was found but there was some sort of error when uploading it.

You should check the path that you're publishing the test results to or that your searching to upload.

[navin22]

@colindembovsky This exit code(2148734720 ) is related to .net Framework. (I feel it's related to https://github.com/microsoft/azure-pipelines-tasks/issues/12224)

What's the .net framework on VM? Can you try upgrading to 4.7?

[Arwinder4]

@colindembovsky can you please confirm if you have tried upgrading the .net framework as mentioned above?

[ncourtiller]

I have the same issue with Publish Test Results version 2.171.0 and .Net 4.0. For me this issue looks like this one : #12300.

[upadhyayap]

@colindembovsky upgrading to .net framework 4.7.2 should fix the problem. We will update the docs and now closing this issue.