Msbuild: MSBuild Generally Inconsistent/Buggy Behavior

[jnevins-gcm]

We are trying to leverage MSBuild 15 for our build system and have just hit bug after bug due to inconsistencies in the overall MSBuild implementation. It’s always been the case that it’s tricky dealing with differences between how devenv and command line MSBuilds work. In MSBuild 15 and with dotnet core MSBuild, it’s gotten so bad that we have ended up needing to write unit tests just to verify all the different build variations actually work. There are builds via

• devenv
• MSBuild.exe (VS Tools version)
• dotnet build
• dotnet pack (with implicit build)
• dotnet msbuild
• dotnet restore && dotnet build

Each one of the above behaves differently...in some cases, massively differently.

Then the above list gets multiplied by 2 because sln file builds behave totally different from (cs)proj file builds.

What I’d like to know is - which of the above build scenarios/models should we be designing for to get the best level of support and stability going forward? It’s borderline impossible to support them all.

Our basic desire is to have a build script that runs

dotnet pack my.sln -o artifacts

A test script that runs

dotnet test my.sln -a ...

And have Visual Studio builds work as well

Calling msbuild.exe would be equally okay vs. dotnet.exe if it actually worked.

It sounds easy but it’s so far from it....

With a simple solution consisting of:

• classic csproj
• sdk csproj targeting multiple frameworks (with GeneratePackageOnBuild set to true)
• a net461 console app that needs a PackageReference to the nuget package produced by the aforementioned csproj (must be a PackageReference because the package has custom msbuild targets)
• a pssproj with Pester tests
• a sqlproj

Basically, what we’ve found

• solution restores behave vastly differently between msbuild.exe /t:Restore, dotnet restore and Visual Studio restore. We can hack around it but calling nested MSBuilds in custom targets, but that will make one scenario work and break another... There are tons of issues filed around this behavior but it’s never been addressed. Here is one
  https://github.com/NuGet/Home/issues/4627

Then there is the issue with calling pack on the sln when a csproj in it has GeneratePackageOnBuild set to true.

Then there is the issue that vstest via dotnet test doesn’t find PowerShell tests (only vstest.exe does)...but calling msbuild.exe /t:vstest doesn’t seem to be supported.

What’s a good strategy here?

[jp2masa]

Then there is the issue that vstest via dotnet test doesn’t find PowerShell tests (only vstest.exe does)...but calling msbuild.exe /t:vstest doesn’t seem to be supported.

Did you try using dotnet vstest?

Also, I've been using custom targets with msbuild for some time and I've never hit big inconsistencies between VS, the MSBuild CLI and the .NET CLI. Do you have some concrete examples for those differences?

[jnevins-gcm]

Thanks for the reply. To your first question - no, no tests are found even when doing dotnet vstest - I assume it might be because the PowerShell Tools TestAdapter is compiled for the NETFramework version of MSBuild (task assemblies are not cross compatible)....but I can’t be sure.

Regarding inconsistencies - the biggest is around what targets get called for sln package restores

• Visual Studio integrated restore doesn’t seem to call any targets I can hook into at all (CollectPackageReferences, etc). Further, package level imports don’t seem to get loaded at all when performing a solution restore from Visual Studio (e.g. a Directory.Build.props in the solution path will get imported but not one in the project folder). It’s different than all other scenarios because of the generated meta project.

• dotnet cli performs an implicit solution restore too (it calls a sln level restore across all projects). Some targets I can hook into do actually get called for this one.

• the common targets imported by dotnet msbuild are of course completely different than the ones used by visual studio

Essentially I have two goals here:

• be able to use the CLI (msbuild desktop, dotnet, any is fine as long as I can run it on a build machine) and Visual Studio to consistently get the same behavior when restoring, building, rebuilding, packing, cleaning a solution with the above makeup. If you create a really simple solution with the makeup I mentioned (projects totally empty), you’ll see it’s broken out of the box in so many different ways.

• be able to have Project A in an sln refer to Project B in the same sln as a PackageReference (so the custom build props/targets in Project B’s nupkg get imported when A is built

[jp2masa]

AFAIK, the common targets are the same for all environments. The main difference is that on VS it loads design time targets, which include some targets used at design time to resolve references, as well as project capability includes used by the project system.

Restore is the target which creates the assets file I think, I'm not sure if VS calls that target, as it calls restore using some NuGet VS interop APIs (https://github.com/dotnet/project-system/blob/master/src/Microsoft.VisualStudio.ProjectSystem.Managed.VS/ProjectSystem/VS/NuGet/PackageRestoreInitiator.PackageRestoreInitiatorInstance.cs).

[jnevins-gcm]

That link is actually great - the interop APIs explain why Restore isn’t called.

But the problem is that you literally can’t call the Restore target yourself if an sln Restore is performed by VS or the dotnet CLI (AFAIK)...or at least you can’t without a ton of hacking around stuff (which introduces even more variability between CLI vs VS

https://github.com/Microsoft/msbuild/issues/2455

[jnevins-gcm]

Also - dotnet vstest doesn't work at all ...seems like another known bug that's marked as resolved...but not actually

I'm running
.NET Command Line Tools (2.1.201)

https://github.com/Microsoft/vstest/issues/326

I get

$ dotnet vstest Build.Tests.ps1
Microsoft (R) Test Execution Command Line Tool Version 15.7.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
System.IO.FileNotFoundException: Unable to find tests for D:\....\Build.Tests.ps1. Make sure test project has a nuget reference of package "Microsoft.NET.Test.Sdk" and framework version settings are appropriate. Rerun with /diag option to diagnose further.
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DotnetTestHostManager.GetTestHostProcessStartInfo(IEnumerable`1 sources, IDictionary`2 environmentVariables, TestRunnerConnectionInfo connectionInfo)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyOperationManager.SetupChannel(IEnumerable`1 sources, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyExecutionManager.StartTestRun(TestRunCriteria testRunCriteria, ITestRunEventsHandler eventHandler)

Test Run Aborted.

I am, of course, referencing that package

[jnevins-gcm]

And msbuild.exe /t:vstest also fails:

$ "C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\Bin\MSBuild.exe" IntegrationTests.pssproj /t:vstest
D:\Dev\...\IntegrationTests.pssproj : error MSB4057: The target "vstest" does not exist in the project.
And yet another scenario behaves differently:
dotnet msbuild AzureFunctions.PowerShellIntegrationTests.pssproj /t:vstest

doesn't error out but still doesn't find my tests

Test run for D:\....\Build.Tests.ps1(.NETFramework,Version=v4.0)
Microsoft (R) Test Execution Command Line Tool Version 15.7.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Multiple versions of same extension found. Selecting the highest version.
  Microsoft.VisualStudio.TestPlatform.Extensions.GenericTestAdapter : 15.0.27703.2009
  Microsoft.VisualStudio.TestPlatform.Extensions.OrderedTestAdapter : 15.0.27703.2009

VS Test Explorer also fails to load the tests...but with yet another different error

This is what I mean by inconsistent/buggy - different behavior (all non-functional) depending on where you're running from. It's truly horrific.

[MeikTranel]

I can only agree and add:
We have the same kind of inconsistencies with dotnet pack, nuget pack & msbuild /t:Pack.
Depending on which are used packages are generated differently. Some even outright fail with features documented as polyglot of all three.

I understand why these issues exist & i'm not trying to pile on, It just begs the question as to where the responsibilities actually reside? Why does the nuget cli have a /build flag? Why is VSTest built into dotnet to achieve dotnet test but not built into msbuild the same as dotnet restore is built into the dotnet cli but also msbuild /t:Restore ? Where is the single source of truth? Why is dotnet adhering to a "global.json" file for SDK nuget resolution, while nuget is about to introduce a repo-wide packages.json? I am sorry, but this whole thing is starting to get out of hand. I am maintaining about 1000 csprojs at work desperately trying to distribute this responsibility, but i just keep being thrown back by stuff like this.