Azure-pipelines-tasks: Publish multiple code coverage files results in wrong code coverage summary sometimes

[nigurr]

https://github.com/Microsoft/vstest/issues/981#issuecomment-433896495

[acesiddhu]

Latest on this one is that we are re architecture the coverage merge. Most of the code changes are done and there are few changes that are currently pending before we can role out that broadly. I will update this thread once done

[RamiSh]

Hey, any updates on the roll out?

[steinmr]

Also, in what version of DevOps/TFS Server will this be available?

[RamiSh]

Hi -- any updates on the release?

[acesiddhu]

Hi all
the update is in progress. In the meantime can share the response of this rest api
https://.visualstudio.com/CCI/_apis/test/CodeCoverage?buildId=&flags=1

[RamiSh]

https://.visualstudio.com/CCI/_apis/test/CodeCoverage?buildId=&flags=1

Fyi -- Page not found. Also, it has been almost two months on this issue, when should the community expect a release ? :)

[acesiddhu]

Rami

Can u add account name before .visualstudio.com

And try

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: Rami Shareef [e-mail]
Sent: Monday, July 1, 2019 7:34:53 PM
To: microsoft/azure-pipelines-tasks
Cc: Siddhartha Pandey; Comment
Subject: Re: [microsoft/azure-pipelines-tasks] Publish multiple code coverage files results in wrong code coverage summary sometimes (#9581)

https://.visualstudio.com/CCI/_apis/test/CodeCoverage?buildId=&flags=1

Fyi -- Page not found

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fazure-pipelines-tasks%2Fissues%2F9581%3Femail_source%3Dnotifications%26email_token%3DAAI5LN54VF34EHTRKQKLMP3P5IFILA5CNFSM4GXMZBG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY6HL2I%23issuecomment-507278825&data=02%7C01%7Csiddhartha.pandey%40microsoft.com%7Cae94824e67464769b16d08d6fe2d173f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636975866948679141&sdata=0ZMRoiHLV7CbjDVvacI1OPGYVTpnxbUFgWW7q2ZlHyw%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAI5LNYHTU6IG52ZGWGORS3P5IFILANCNFSM4GXMZBGQ&data=02%7C01%7Csiddhartha.pandey%40microsoft.com%7Cae94824e67464769b16d08d6fe2d173f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636975866948679141&sdata=WgCwRr1k2ndpGISUuElR0hgzeLpxQ5gKTpb95%2B8W%2FYg%3D&reserved=0.

[acesiddhu]

All,

Thanks for the patience. we actually rolled out the new arch job to few scale units and found some issues because of which this took time. the latest on this is that we have rolled out the new merge to more than half of the scale units with all the fixes (including top three scale units which use the code coverage merge job the most). The rollout to remaining scale units should complete mostly by this week

If folks are interested to know if this has reached their account please do mail me the the account name at this address "devops_tools AT microsoft DOT com" and i will confirm over email.

The reason why i asked about rest apis output is because we also fixed a small issue recently which we wanted to confirm is the latest most of the folks are hitting.

Also do let me know if any of you have stopped seeing this issue since sometime last week (or even today). That would help us validate the changes.

[macwier]

Will those fixes be available to on premise TFS Server?

[acesiddhu]

@MaciejWierzchowski these changes wont make to TFS 2019 Update 1 given that we need baked time with these big changes in AzureDevops first. Once this is enabled everywhere and we have gotten sufficient baked time with the changes we will push this to next release of on prem. Can you reach out to me "devops_tools AT microsoft DOT com" I have few workarounds which we can try even now.

[acesiddhu]

Hi folks
the new merge is enabled on all scale units. do let us know if the issue is fixed now. if the issue still persist please drop me a mail (devops_tools AT microsoft DOT com) with account name, build id where it occurred

[steinmr]

With the help of @acesiddhu, I have create a task group that works around the problem (see attached file)

In short, first publish code coverage for a fake/irrelevant test (using this task group) and then do your normal publishing using "dotnet test" on your normal tests. Please let me know if there are any feedback on this task group.

Build_ Publish mock test to workaround bug in TFS 2018.zip

Note: This requires that your tests are in the same flavor/platform as the fake tests. If needed, you can modify the task to run several flavors/platforms to match your test sets (given that you have more than one test project within each variant)

[acesiddhu]

Hi all
I am resolving the issue now. let me know if you are still hitting the issue. you can reach us at devops_tools AT microsoft DOT com with account name and build id where this issue reproed.

[gremlm]

Will this fix work under _vstest.console.exe /Enablecodecoverage_ ?
We need to create .coverage file for solution with multiple projects and later translate it to xml.
Execution of the _vstest.console.exe /Enablecodecoverage_ creates a new _.coverage_ file in random directory, so this _.coverage_ file is impossible to specify as argument for _CodeCoverage analyze_.
We have tried to start _vstest.console.exe_ under CodeCoverage, like the following
_CodeCoverage collect /output:.\TestOut\VisualStudio4.coverage /vstest.console.exe test1.dll test2.dll /ResultsDirectory:.\TestOut_.
This create VisualStudio4.coverage file in directory .\TestOut, but execution of the
_CodeCoverage analyze /output:.\TestOut\VisualStudio4.coverage.xml_ generate an empty output like

\
\
\

Some interesting observation: size of the file generated through _vstest.console.exe /Enablecodecoverage_ _.coverage_ file is much bigger than than _.coverage_ file generated through _CodeCoverage collect ..._ (3,111Kb vs 54Kb)