Msbuild: Spec Proposal: MSBuild Extension support for .NET Core

[AArnott]

NuGet and MSBuild should work more closely together for MSBuild extensions.

Creation of an MSBuild extension

I should be able to dotnet new buildextension to get a new project that:

1. Includes an MSBuild Task that compiles to net45 and netcoreapp1.x
2. Includes an MSBuild .props and .targets file in source code that invoke the Task from a target, choosing
3. Packs to a NuGet package, that
   
   
   
   1. expresses package dependencies for package dependencies referenced in the csproj that compiled the MSBuild Task. No need to embed dependencies in the MSBuild extension package itself.
   
   
   2. Sets DevelopmentDependency=true automatically in the nuspec file.
   
   

Consumption of this MSBuild extension:

1. Happens by adding a BuildExtensionReference item to the receiving project. This isn't PackageReference because its dependency graph does not blend into the project's dependency graph. But it isn't DotNetCliToolReference because those items do not contribute MSBuild .props and .targets files to the project. This BuildExtensionReference item may propagate across P2P references if desired (so that a single 'root' project in a solution may add the extension to all other projects that reference it). Or perhaps we can unify this new item with DotNetCliToolReference by calling it ProjectExtension (similar to MSBuild's special element type, but this one would appear inside an ItemGroup element.
2. Upon NuGet Restore of the project, the MSBuild extension is downloaded to the machine package cache along with all its dependencies in their own respective packages.
3. The project automatically imports the .props and .targets from the package.
4. The project can build and invoke the Task both on MSBuild full and MSBuild Core, and dependencies the Task has can resolve from the package cache.
5. The MSBuild Task controls its own dependencies independently of other Tasks that may run in that project because each one runs in its own AssemblyLoadContext in MSBuild Core and AppDomain in MSBuild Desktop.

Optional dotnet CLI tool as well:

An MSBuild extension may also want to provide convenient dotnet CLI invocation as well. Currently for an MSBuild extension to both modify the build but also make tools accessible by dotnet CLI the user must add both a PackageReference and a DotNetCliToolReference item to their project. This is cumbersome, especially when such an extension applies to all projects in a solution. So dotnet CLI should allow one package to offer both an MSBuild extension and a dotnet CLI tool.

An example is Nerdbank.GitVersioning which both modifies the build with special version semantics, as well as offers a couple of CLI tools to translate a commit to a version and vice versa.

As discussed with @nguerrera and @tmat on another issue.

[rainersigwald]

Do you have an objection to splitting this into two issues: one for having a template, and one for the second half?

[AArnott]

Thanks for looking, @rainersigwald.

There are a bunch of "issues" to resolve to deliver on this. I see this issue more of a spec for folks to agree to the vision of, then to link it to all the various issues (including across repos) that will deliver on it. Having a template won't be very compelling on its own if the rest of the story isn't solid.

But if you'd really prefer two top level specs, that may reference each other, I can break it up.

[dasMulli]

What's probably needed is a proper acquisition story for SDKs as has been discussed in https://github.com/Microsoft/msbuild/issues/1493 and https://github.com/Microsoft/msbuild/issues/1436, as well as tooling(/templates/defaults) to build & publish SDKs.

I've come across this recently when trying to patch together a few build utilities. While you can tell NuGet to pack the resulting dlls in a folder other than lib through <BuildOutputTargetFolder>build</BuildOutputTargetFolder> and mess with PrivateAssets="All" on the PackageReference item, you soon run out of luck trying to reference & include 3rd party nugets (unless you would emit items with a custom pack path directly out of the nuget packages folder which is also accessible through an msbulid property).

Another issue: At some point, you'll want to explicitly depend on a specific SDK being present.
E.g., if I consume SDK properties, I want to make sure that Microsoft.NET.Sdk is present in the build. If I do fancy F# code generation, I'd want to depend on the F# SDK.
Or maybe integrate with web-specific targets from the web SDK..

What currently works fine is to build packages that make consuming projects include props and targets files via convention (e.g. PackageName.targets):

<None Update="build\**\*" Pack="true" PackagePath="\build" />

Combined with <IncludeBuildOutput>false</…> This also helps replace a lot of nuspec use cases since dotnet pack doesn't directly support packing nuspec files anymore.

[natemcmaster]

I'm interested in this idea too. ASP.NET's build system attempts to extend MSBuild with NuGet packages containing tasks/targets. PackageReference is close to a good solution, but imports happen too late for things like loggers, .NET Framework reference assemblies, and custom task assemblies.

I'm hoping the SDK acquisition experience helps.

And +1 for creating a csproj template for MSBuild tasks projects. Currently this requires knowing how to manipulate NuGet's internal pack targets to get assemblies and files into the right place.

[dasMulli]

Another approach could be to (ab)use NuGet's package type string and introduce a new kind of reference item. Along with <PackageReference> and <DotNetCliToolReference> there could be sth like a <BuildToolReference> that - like the cli tool reference - does not affect the consuming project's dependency tree and does not flow into parent projects (so you can avoid the PrivateAssets="All" now typically used for build-only dependencies) but would just add props and targets files imports.

This would be a pure NuGet feature but would more closely align how distributing and consuming works with nuget packages than "SDK packages" (for which you probably want only one version across the solution and maybe not ship it with NuGet but register a custom SDK resolver etc.).

[stazz]

Hi,

I agree with @AArnott that there should be better support for NuGet package -based MSBuild tasks.
Overall this issue has some good points, and I hope that we will get support for BuildExtensionReference soon.

However, while the current state of MSBuild Extension support via NuGet packages is not optimal, I have created a custom MSBuild Task Factory, which will execute other MSBuild Tasks, which are NuGet package-based.
I've tested this against MSBuild 15.1 in .NET 4.6, and MSBuild 15.3-Preview in .NET Core (since task factories are not supported in MSBuild 15.1 for .NET Core), and it seems that things work out nicely in both scenarios.

There are no special requirements for developing tasks which are useable by this task factory, other than their target framework is suitable (you can just target .netstandard 1.3, and the task will work in both .NET Desktop and .NET Core MSBuild).
The task may reference other third-party NuGet packages freely - the task factory will take care of loading dependent assemblies on-the-fly.

More information available at UtilPack.NuGet.MSBuild, I hope this will help other people who develop complex NuGet package-based MSBuild tasks!

[AArnott]

That sounds very interesting, @stazz. Thanks for sharing. I hope to check it out when I get some time.

[stalek71]

Very good idea @stazz :)
Thanks for sharing it with us.
I found something like this also:
https://blog.nuget.org/20170316/NuGet-now-fully-integrated-into-MSBuild.html

[AArnott]

@stazz said:

you can just target .netstandard 1.3, and the task will work in both .NET Desktop and .NET Core MSBuild

I'm afraid this is likely not true. Unless you've taken special care to load portable assemblies on desktop Framework. The problem is an MSBuild task that compiles against .netstandard1.3 will require facade assemblies at runtime when on .NETFramework -- assemblies that MSBuild (full) doesn't have, leading to at least some MSBuild Tasks failing at runtime.

That's why the MSBuild team's position is you have to dual compile tasks, targeting each of .NET Core and .NET Framework for it to work reliably on each platform.

[stazz]

@stalek71 said:

Thanks for sharing it with us.

Glad to hear! :)

@AArnott said:

Unless you've taken special care to load portable assemblies on desktop Framework.

That's exactly what I had to do. :)

The problem is an MSBuild task that compiles against .netstandard1.3 will require facade assemblies at runtime when on .NETFramework

You're right - that's what happens on .NET Desktop. I encountered this long time ago, and had to do appropriate modifications to my code. But those modifications do work: for example, the CBAM.SQL.MSBuild task is compiled only against .NET Standard 1.3, but it runs successfully on both .NET Desktop and .NET Core. It is not a trivial task either, since it uses disk IO to read database configuration and SQL files, and network IO to communicate with the database. So the dual-compiling is not really mandatory.

[stazz]

Oh, one more thing. One special problem specific only to this issue is that when executing .NET Core MSBuild, the NuGet assemblies are part of trusted assembly set, and thus if e.g. task factory uses NuGet stuff, it won't see the NuGet libraries it was compiled against, but the NuGet libraries loaded by .NET Core SDK. Furthermore, even the minor version updates in NuGet libraries introduce binary-incompatible changes (the ILogger changes in 4.2.0 -> 4.3.0 update, and then the introduction of LocalNuspecCache and usage in RestoreCommandProviders.Create method in 4.3.0 -> 4.4.0 update).

This is why, for UtilPack.NuGet.MSBuild version 2.0.0, I had to introduce facade task factory for .NET Core assembly, which examines the version of NuGet library loaded by SDK, and uses appropriate actual task factory assembly (currently two: for NuGet version 4.3.0, and for NuGet version 4.4.0).

That is something I need to create a issue about once I get time. Not sure if anything can be done about that tho.

[Solokiller]

Is there a good/simple/clean workaround to get this to work until there is NuGet/MSBuild support for task assembly dependencies?

I've tried to work around it by embedding the assembly my task assembly depends on but that assembly also has dependencies and i don't know of any way to handle this recursively, and i fully expect a brute force inclusion of all dependencies to cause conflicts between the local and system versions of certain assemblies (e.g. mscorlib).