Azure-pipelines-tasks: Code Coverage Charts not shown on VSTS

[lucas-natraj]

Environment

• Azure Pipelines
  
  
  
  • Account Name: slb1-swt
  
  
  • Project: planck
  
  
  • Build Definition Name: Delete-Me
  
  
  • Build Number: 912449
  
  
• Agent - Private:
  
  
  
  • OS: Windows Server 2012
  
  

Issue Description

I've been able to have my .NET Solution build successfully, using the Visual Studio Build Task, and
have run the Visual Studio Test Task to execute Unit Tests with code coverage enabled.

Everything seems to build and test fine with a .coverage file being generated.

However, the Code Coverage tab on the VSTS Build page only gives me the option to Download code coverage results. That downloaded results (.coverage) file does open in Visual Studio showing me the correct coverage information.

However, I was hoping that it would show charts directly in the Code Coverage Tab. But I don't see anything.
Is there something that needs to be enabled for that to work?
Is this supported in .NET 4.6.1 projects?

The following is by build definition yaml

queue:
  name: ...
  demands: 
  - msbuild
  - visualstudio

steps:
- task: VSBuild@1
  displayName: 'Build Dummy.sln'
  inputs:
    solution: 'Dummy.sln'
    configuration: Release
    restoreNugetPackages: true

- task: VisualStudioTestPlatformInstaller@1
  displayName: 'Visual Studio Test Platform Installer'

- task: VSTest@2
  displayName: 'Test'
  inputs:
    testAssemblyVer2: |
     **\*.UnitTest.dll
     !**\*TestAdapter.dll
     !**\obj\**

    vsTestVersion: toolsInstaller
    overrideTestrunParameters: '-Platform x64'
    codeCoverageEnabled: true

code: .NET Framework 4.6.1
agent version: 2.141.1
visual studio test version: 2.142.11

[kaadhina]

Hi, thanks for you feedback!
This is a known limitation. We are in the process of improving this experience. Thanks!

[lucas-natraj]

Groovy. Thanks for the quick response.

Is this documented anywhere?

I would imagine other people have had the same problem, but I was unable to find any indication that this doesn't yet work or even that people have experienced this.

The closest that I've found is a couple of issues on SO saying that publishing .coverage or coveragexml files doesn't work. But even those suggest that using the Visual Studio Build / Test tasks do work.

Any rough timeline on this support?
Or perhaps guidance on how to use .coverage files (perhaps some convertors to alternate formats that are supported?

[kaadhina]

@PBoraMSFT to comment on the timelines.

[PBoraMSFT]

@lucas-natraj - there are two aspects here:

1. Charts (summary of lines covered / not covered) - this is shown in the build summary page already.
2. Detailed reporting and Drill down - this is on the backlog and pretty high on the priority list. We are still in the design/storyboard phase for this, so I don't have an ETA to share just yet.

[xenalite]

@PBoraMSFT We found the summary page to be very lacking.

I run code coverage for TypeScript, Python and C# code projects. The above picture is from our build - the top bar is for Python, bottom for C#.

I have 2 publish code coverage tasks with Cobertura for TS and Python.

• Only the last publish task's results are displayed. The first task's summary results are overwritten.
• There is no way to tag the coverage result with the language name. There is no way to tell the top bar is for Python. At least in VSTest it tags it with Configuration and Platform.
• Only total line coverage is shown. No block coverage is displayed.
• Line / block coverage for individual bars is not displayed. It is unknown what individual Python / TypeScript / C# coverage is.
• There is no way to run .NET Core and .NET Framework assemblies in a single VSTest task with code coverage enabled. I have to run 2 separate VSTest tasks with assembly filters on.
• If HTML reports are produced for code coverage, their files are mashed together and overwritten, making them useless to display.
• Even without above point, HTML CSS and JavaScript is not displayed in the browser for security reasons, making HTML report viewing a useless feature. Most HTML reports rely on CSS and JavaScript for interactivity.
• No way of specifying a generic code coverage threshold to fail the build if below. I have to resort to custom PowerShell.

If memory serves me right, this page is a regression from TFS 2015, which had much more comprehensive code coverage page.

[samsmithnz]

I agree with everything xenalite has here..

I'm not sure why we can't show blocks on the summary page - given it only shows blocks in Visual Studio.

[sreeraj-rajendran]

Hello @PBoraMSFT thanks for sharing the updates.

Can you please let me know if we have any tentative dates for this feature request. We have a requirement to show the detailed Code Coverage report on the summary page, thought of checking here before trying to make something custom.

Thank you.

[KumarS-Naveen]

Hello @PBoraMSFT ,

Could you please let us know when will the chart feature for code coverage can be available

[ganesp]

@IAm-Naveen , @sreeraj-rajendran
Coverage reporting improvements are on the backlog and pretty high on the priority list.

Some of the improvements are in planning phase.
Would you like to meet us for reviewing storyboards and provide feedback?

//cc: @PBoraMSFT

[sreeraj-rajendran]

@ganesp thanks for the update 👍. We would like to be part of the meetings, please let us know.
Thanks for asking.

[AlexChongMicrosoft]

Hi, I have met this same issue. I couldn't see the code coverage tab in my VSTS. I wonder whether there is something wrong or my misunderstanding. Is there any update of this topic?

[ShreyasRmsft]

@AlexChongMicrosoft may I know what your build definition looks like (yaml/screenshot would help) and what your account name is? Also logs with system.debug = true would also help.

[sreeraj-rajendran]

@ganesp can you please let us know if there are any new updates? Please let us know if you need any feedback from our side. Thanks.

[mithsapurva]

@kaadhina @PBoraMSFT Is there any update on this? Code coverage tab is blank stating "Code coverage report cannot be rendered as report HTML was not found. Please verify that "Report Directory" containing an HTML report was specified when publishing code coverage."
Please note that the HTML reports are present in the specified location and we are using this for .NET 4.6.1 framework (Not .NetCore).
Please suggest if this is supported by the task or not.

[karanjitsingh]

@mithsapurva, in your build artifacts do you see an artifact with the name CoverageReport? If yes can you open it up to see whether it contains an index.html or not.

Additionally could you also post the build logs.

[AlexChongMicrosoft]

Hi Karan,

Thanks for your attention.

I invite you to my DevOps project. You could check the build pipeline directly.
https://v-chucho-microsoft.visualstudio.com/testCodeCoverage/_build/results?buildId=140

If there is any information needed, please let me know.

Best Regards,

Alex Chong

[image001]
Support Engineer
Microsoft APAC Developer Support Team
Customer Service & Support (CSS)
Email: [email protected][e-mail]
Office: [phone]
Time zone: (UTC+08:00) Beijing, Chongqing, Hong Kong, Urumqi
Working time: 9:00am-6:00pm, Mon-Tue-Wed-Thu-Fri

From: Karan Jit Singh [e-mail]
Sent: Friday, July 26, 2019 2:31 PM
To: microsoft/azure-pipelines-tasks [e-mail]
Cc: Alex Chong (International Supplier) [e-mail]; Mention [e-mail]
Subject: Re: [microsoft/azure-pipelines-tasks] Code Coverage Charts not shown on VSTS (#8749)

@mithsapurvahttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmithsapurva&data=02%7C01%7Cv-chucho%40microsoft.com%7C22b95b562fd84192fca408d71192cfb9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636997194556456423&sdata=xfwah%2FJNxWBXYXQuEO3a8ghbtev38LRR817kS1%2BKn5U%3D&reserved=0, in your build artifacts do you see an artifact with the name CoverageReport? If yes can you open it up to see whether it contains an index.html or not.

Additionally could you also post the build logs.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fazure-pipelines-tasks%2Fissues%2F8749%3Femail_source%3Dnotifications%26email_token%3DAK5BZNVHOIWWPACB5PABYWTQBKKZ5A5CNFSM4GAV76FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD23UMZI%23issuecomment-515327589&data=02%7C01%7Cv-chucho%40microsoft.com%7C22b95b562fd84192fca408d71192cfb9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636997194556466418&sdata=8aIRUUn5DJ%2FexoMs3KH2RTL209PRhl8SwTnJ%2FX3Rdnc%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAK5BZNR5RPCXY2CEUVQ3XITQBKKZ5ANCNFSM4GAV76FA&data=02%7C01%7Cv-chucho%40microsoft.com%7C22b95b562fd84192fca408d71192cfb9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636997194556466418&sdata=DKAA2FgyF8BrgwXduOKGKLQu224B7PvVnWWMT0gNVBg%3D&reserved=0.

[karanjitsingh]

@AlexChongMicrosoft, if you see your build, there was no test run. You need to fix that first to get coverage up and running.

[AlexChongMicrosoft]

Hi Karan,

From the log file of test task, I find .coverage data is generated. Could you tell me how to run test here?
[cid:[email protected]]

I also attach another build log, there is test result, but there is no Code Coverage Tab in DevOps. Could you give me some suggestion?

Best Regards,

Alex Chong

[image001]
Support Engineer
Microsoft APAC Developer Support Team
Customer Service & Support (CSS)
Email: [email protected][e-mail]
Office: [phone]
Time zone: (UTC+08:00) Beijing, Chongqing, Hong Kong, Urumqi
Working time: 9:00am-6:00pm, Mon-Tue-Wed-Thu-Fri

From: Karan Jit Singh [e-mail]
Sent: Friday, July 26, 2019 3:06 PM
To: microsoft/azure-pipelines-tasks [e-mail]
Cc: Alex Chong (International Supplier) [e-mail]; Mention [e-mail]
Subject: Re: [microsoft/azure-pipelines-tasks] Code Coverage Charts not shown on VSTS (#8749)

@AlexChongMicrosofthttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAlexChongMicrosoft&data=02%7C01%7Cv-chucho%40microsoft.com%7C81a9de96d6a04b7998e308d71197b378%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636997215554871652&sdata=stY77K6%2BGohtWsBn7Rx4N6Zhu3QojiY1iuSax6Di4nI%3D&reserved=0, if you see your build, there was no test run. You need to fix that first to get up and running.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fazure-pipelines-tasks%2Fissues%2F8749%3Femail_source%3Dnotifications%26email_token%3DAK5BZNVPKQNSDPADDOMWH53QBKO5DA5CNFSM4GAV76FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD23WOYI%23issuecomment-515336033&data=02%7C01%7Cv-chucho%40microsoft.com%7C81a9de96d6a04b7998e308d71197b378%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636997215554881646&sdata=rQe4Glapd1uY1lX4JW7vLR4DLUVws%2FsdMJtQUodM1Jg%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAK5BZNRFANQRR2NV3ESPOALQBKO5DANCNFSM4GAV76FA&data=02%7C01%7Cv-chucho%40microsoft.com%7C81a9de96d6a04b7998e308d71197b378%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636997215554891641&sdata=9XjMSHMgOx2QP%2BoTf55e02mFYzEY11L3lEcWLCnsOVc%3D&reserved=0.

[karanjitsingh]

@AlexChongMicrosoft, you were missing the IsTestProject property in your test projects, I've updated your definition. It should be working now.

[karanjitsingh]

@AlexChongMicrosoft, @mithsapurva. This thread is pretty old and has bloated out of context. If you're still facing problems please create a new issue and feel free to tag me directly (for better traction).