Vstest: Test results missing in trx file

[urbankovak]

Dear Sir/Madam, we have a problem with publishing test results. Some of them are missing in trx file. We use Visual Studio Test task for running UI tests in the TFS Release Pipeline.

I asked Azure DevOps team for help, but they recommended me to ask you - @vstest.console.exe owners. According to their findings the issue is in the trx where for the missing tests is a warning which is result it not being present in the trx. Please see the link to the Azure DevOps team answer below.

https://developercommunity.visualstudio.com/comments/1146832/view.html

Would you be so kind to explore the issue and let me know where the problem is?
If you need any other information, please let me know.

Thank you in advance
Katerina Urbankova

[Sanan07]

Hello @urbankovak.
Did you recently update Test Platform in Azure DevOps, so after update you get that issue? I mean when did you start facing that behavior? And do you get the same problem when running test on your machine in Debug and Release mode inside Visual Studio?

[urbankovak]

Hello @Sanan07 and thank you for your reply.

The problem occurs for some time. Definitely before updating to Azure DevOps too. When I run tests on my machine, I don't get this problem. All run test results are published in VS.

We made an experiment and run the same VS Test task with the same list of tests once in TFS build definition and once in TFS release definition. In the release definition occured the problem described above. But in the build definition everything was OK. The trx file contained all the run tests and all of them were published in TFS test summary table.

So the problem occurs only when the VS Test task is included in release pipeline. Hope this information helps you to solve the problem.

[Sanan07]

@urbankovak These 6 test methods are failing in Release pipeline while vstest trying to publish them. The reason for that vstest trying to combine that name with path and getting PathTooLongException.
You can see it in your Release logs :

3_d0586030-c201-11ea-b3fd-d5545a8ca027.txt

This happening while combining attachment url with trx logger url. Maybe in Release mode you get different path for trx and while publishing the results, it gets very long name.

So try to reduce the name of your test methods and try again. Lets us know about the result.

ZalozEditujStornujKlientskyFwdAmortizacni
ZalozHromadnouPlatbuVerifikaceHromadna
ZalozKlientskouOpciNakupCallEvropskaAutomaticke
ZalozKlientskouPlatbuPresunEditujZaplat
ZalozProdluzAmSwapemKlientskyFwdAmortizacni
ZalozVerifikujSuvZaplatPrevodNostroD0

[nohwnd]

It is recommended to make the path to the agent as short as possible this: C:\TFSAgents\TFS-BUILD-16_Agent1_SBS_Interactive\, should ideally be C:\a1\, you might also use the subst utility to map that directory to a different "disk", e.g. subst x: C:\TFSAgents\TFS-BUILD-16_Agent1_SBS_Interactive.

[urbankovak]

Thank you for your comments.
The full path in the screenshot you attached is C:TFSAgentsTFS-BUILD-16_Agent1_SBS_Interactive_workr1aINDSW-APP11 - Deploy with testsUITests_CSharp_AlfabinDebugZalozEditujStornujKlientskyFwdAmortizacni.png, which is 167 characters long. It doesn't exceed 248 or 260 mentioned in the log file. So we are confused what exactly is wrong.
We will try to shorten the paths, but it must do another team in our company, so it will take some time.
Would you be so kind ale let us know what is the real limit for test full path / full name?

[nohwnd]

@urbankovak I don't think the path that you see in the log is the one that directly causes the issue, instead the TestResultHandler, calls LegacyNormalizePath and GetFullPathName which throws.

@Sanan07 Could you please try making a project locally with the long path as shown above to see what is happing in the TRX resolve? We should be able to reproduce this locally.

[Sanan07]

@urbankovak Issue is that, you have attachment of failed tests and trx logger is creating additional folder to put that attachments.

This is attachment name : ZalozEditujStornujKlientskyFwdAmortizacni.png.

The limit is for fully qualified names is set by .NET : The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters - this is from exception in log.

It is difficult to say what exact path was created during your test run, by I can assume, by looking at code, it could by something like that :

C:\TFSAgents\TFS-BUILD-16_Agent1_SBS_Interactive\_work\r1\a\INDSW-APP11 - Deploy with tests\UITests_CSharp_Alfa\bin\Debug\TestResults\Deploy_TFS-BUILD-16_Agent1_SBS_Interactive 2020-08-21 12_02_27\In\TFS-BUILD-16_Agent1_SBS_Interactive\ZalozEditujStornujKlientskyFwdAmortizacni.png

Pay attention to \In\ directory, it should be created in the same path line, where your trx files stored. Try to find it inside your agent file system. It is created for attachments after test run. When you find that directory, see if you get any attachments there.

This is line where that trx fails : https://github.com/microsoft/vstest/blob/30d13b39111ea0a26126abbfbd070b7f3b9da815/src/Microsoft.TestPlatform.Extensions.TrxLogger/Utility/TrxFileHelper.cs#L201

path - is path to that attachment file inside \In\ folder.

For quick test, you can change your test name to something short, like Test1 and run in your agent and see if that test successfully got in trx file.

Hope that helps. Keep us updated, if this helps.

[urbankovak]

Thank you for you comment. We already got to the same idea too, that the problem is connected to the creating a screenshot file. Because when the test fails from some other reason, where the png file is not created, the trx file is OK and the test result is published in TFS correctly.

We are going to shorten paths on our agents through the next week. I will let you know if it helps.

Thank you once more for your kind help.

[urbankovak]

Hi @Sanan07 and @nohwnd, it helped! We changed the directory structure on our agents so the final path for failed tests screenshots doesn't exceed the limits and the failed tests which weren't published correctly before are correctly published in TFS now. Thank you once more for your kind help.
I'm going to close this issue, because it looks OK now.
Wish you all a lovely day.