Azure-pipelines-tasks: Inconsistent behavior: .NET Core pipelines task does not surface warnings/errors in Azure Pipelines (whereas the MSBuild task does)

[NenoLoje]

Type: Bug
Task Name: VSBuild

Environment

Azure Pipelines, any org
Hosted Agent (Hosted Ubuntu 1604).
aspnetcore 2.2 app

Issue Description

Error description:
Consider a project using the Microsoft.CodeAnalysis.FxCopAnalyzers package and running code analysis rules as part of the build.

While warnings are visible in the log output, the VSBuild task itself does not show any warnings, and thus the build report does not surface them.

Is this a known issue?

Screenshots

Figure #1: Warnings are shown in the log file:

Figure #2: No warnings in build result report:

[NenoLoje]

Any news here?

[moswald]

Sorry @NenoLoje, this was assigned to Austin but he is no longer with the company. Can you include a build log with diagnostics turned on?

[NenoLoje]

@moswald Sure thing, see attached.

log_20_11361.zip

Also, if you prefer, I can also give you access to the AzDevOps org.

Thanks for looking into this!

[moswald]

Well, I guess I just didn't know this, but build warnings don't get bubbled up to the UI like that. Any warnings you see are from the pipeline itself. If this is an important feature for you, then I recommend creating a feature request on the developer community site. I'm sorry I can't give you a more positive response.

[NenoLoje]

@moswald, Thanks for looking into this! However, I'm struggling if creating a feature request on the developer community site is the right way to handle this issue.

The current behavior is really not what users would expect, and it's inconsistent among the different experiences:

• Code analysis errors/warnings have been an integral part of TFS since version 2005. Not supporting this means removing functionality from the product which users are relying on.
• Visual Studio does show Code Analysis warnings in the error window.
• XAML Builds show Code Analysis warnings in the build errors/warnings.
• So, why would the (new) build pipelines not show them?

Any chance to reconsider this? Thanks! :)

[moswald]

Code analysis errors/warnings have been an integral part of TFS since version 2005. Not supporting this means removing functionality from the product which users are relying on.
XAML Builds show Code Analysis warnings in the build errors/warnings.

If XAML used to do it it's probably an oversight in the original design of the msbuild-based tasks, but the new build platform has been the standard for several years now and has (apparently) never promoted build warnings to be pipeline warnings. I agree it's not what users expect since it surprised me. Regardless, the best way to get traction on something like this is through the developer community site.

[NenoLoje]

@moswald Matt, just to check if there's maybe a larger issue here: the preffered way to use Code Analysis these days seems to be to use the NuGet package, which is using Roslyn analyzers. Wouldn't the limitations (can't fail the build, errors/warnings do not show up in the build result) apply to all kinds of Roslyn analyzers then?

[NenoLoje]

I created an issue on DeveloperCommunity to track this further as it's now closed here (and I can't reopen it):
https://developercommunity.visualstudio.com/content/problem/616986/pipelines-code-analysis-analyzers-warningserrors-d.html

[moswald]

Thanks @NenoLoje. I'll make sure the appropriate person sees it.

If the code analysis tool causes msbuild to fail because of an error, the task should also fail. Depending on the task used, I think the errors will also surface as pipeline errors. It's only the warnings that we don't listen for.

[NenoLoje]

@moswald, I could-checked. Indeed, if a code analysis rule is set to fail the build, indeed msbuild (or dotnet.exe) fail and thus the build breaks.

However, in both cases - you need to search the logs to find the actual warning and error messages. It's worse for warnings, because you might not have noticed them at all. It's just such an unpleasing experience right now.

I'm not sure where would be the best place to fix this... on Azure Pipelines? In the pipeline task? Or maybe in msbuild/dotnet.exe?

[moswald]

The msbuild and vsbuild tasks would have to be modified to check for warnings in the logs and then surface them in the UI. Your Developer Community post is the best way to get traction on that.

[NenoLoje]

@moswald The issue was closed at VS Developer Community with the comment to add an issue here in the GitHub repo. Not sure what to do now.

[moswald]

🙄 I'll fix it.

[NenoLoje]

Thanks. :)

[moswald]

Forgive us, there are a lot of chefs in our kitchen. 😁

[NenoLoje]

Well, then I guess, I was just too hungry then. ;)

[yaananth]

It's a task issue so you know :)
but yeah dev community helps in tracking :), agreed.
If this was mentioned there that it was just for tracking, we are good!

[ArieHein]

in a lot of tools the return value will usually be success even if the content shows warning or even security issues, for example npm when it does its validate step will still consider the step a success even if it finds high severity issues. The tool itself is successful in executing npm, its successful in executing the msbuild command, In some cases I had to go for a cmd task or batch task and call the execution from there so I can control the %ERRORLEVEL% and decide if the task return error is good or bad to fail the build based on it.

[NenoLoje]

It seems that the title is actually misleading now. Changing it to reflect the current state of research.