Msbuild: Create NuGet packages for all MSBuild assemblies

[niklaslundberg]

I want to reference MSBuild from my projects to be able to programmatically parse files like .csproj. There seems to be no way of doing this for .NET Core projects.

[daveaglick]

I was actually looking for this exact same thing this morning and was a little surprised to find that there isn't already an official package. I'd love to know if one is planned, even if it won't be available for a while.

I did find a Roslyn.Microsoft.Build package that contains the MSBuild assemblies, but I don't know how "official" it is or if it'll get updates:
https://www.nuget.org/packages/Roslyn.Microsoft.Build/

[rainersigwald]

We're currently publishing packages to a private feed: https://dotnet.myget.org/gallery/dotnet-buildtools. But they may not be what you want, since they're (currently) only consumed by dotnet-buildtools to build ourselves and CoreFX.

Having a usable set of packages is on our radar, but not solved yet. The main thing is an interesting complication: in addition to our DLLs, you must have the right set of targets files in order to evaluate an MSBuild project. It'd be frustrating to set up MSBuild package references and then just get "invalid project: can't import Microsoft.CSharp.targets"!

Since this same problem is encountered by OmniSharp, NuGet, and dotnet CLI, I expect we'll have something together in the dev15/dotnet tooling RTM timeframe (but I'm not promising anything here :innocent:).

[daveaglick]

@rainersigwald Thanks a bunch for the feedback. I totally understand that this is a moving target, it's helpful to know where you're at. Personally, my main interest is in evaluation of MSBuild files and not compilation (I.e., the Microsoft.Build.Evaluation namespace in the Microsoft.Build.dll assembly). Looking through the packages in the MyGet feed, I don't actually see that assembly in any of them so I'll continue to sit tight until the packaging story irons out.

[rainersigwald]

@daveaglick There should be a Microsoft.Build package in there that should do the trick (note that due to #528 you may have to explicitly reference the right versions of package dependencies that will be automatic Real Soon Now).

However, even to just evaluate a project, you need to be able to locate files that are <Import>ed.

I forgot to note that while we're working toward this nice packaged world, you can currently use the MSBuild assemblies that are installed when you have Visual Studio (or Build Tools) installed to evaluate or build projects, and they should be able to find the appropriate targets without configuration.

[daveaglick]

There should be a Microsoft.Build package in there that should do the trick

Thanks - totally missed that when it didn't show up on the first page in the NuGet Package Explorer. I'll check it out.

[wjk]

I just wanted to throw in my 2¢ regarding this issue. My main use-case regarding the lack of MSBuild NuGet packages is that I want to add new tasks to an MSBuild-based build, when using the xplat branch’s version. While the portable task demonstrates that this is possible, it still isn’t easy because the portable task approach requires the usage of a *.csproj-based build system. For the short term, I use .NET CLI because of the better design-time experience it gives me (i.e. IntelliSense).

Without the NuGet packages, there is no way (that I know of) that I can get .NET CLI to link against any version of the MSBuild assemblies (much less the ones loaded in the current MSBuild process). I agree with @rainersigwald regarding the *.targets files.

However, given the pending switch to MSBuild that Microsoft announced, this whole issue may become moot soon enough.

[cdmihai]

@wjk

The PortableTask sample references the csproj MSBuild projects out of convenience. You can actually remove those references and replace them with a project.json pointing to the nuget msbuild packages on https://dotnet.myget.org/gallery/dotnet-buildtools, and it should still work.

Ideally, we could update our build so the portable task consumes local nuget packages produced from the current build, from a local nuget feed.

Here's some samples I did by hand:
PortableTests.zip

Let us know if this addresses your problem.

[eerhardt]

This has been open for a few weeks and is not assigned to anyone. Can we get an update on this? The .NET Core SDK team is blocked on moving forward until we can get official MSBuild NuGet packages. We have currently hacked together something for prototypes, but until we get what the official packages look like, and can log issues against them, we are unable to do real production work.

Here's the current outstanding issues that I'm aware of:

• The MSBuild packages are on the "build-tools" feed, which is not an official package feed. Our official product can't be dependent on a feed that is only meant for internal tooling.
• The MSBuild packages don't declare their dependencies correctly.
• The MSBuild packages don't have their assets packaged in the correct folder structure (ex. the .targets files are currently under "runtime/any/native").
• The files use inconsistent casing in some cases - some .targets, some .Targets. This is an issue on non-Windows platforms that have case sensitive file systems.
• The current packages on https://dotnet.myget.org/feed/dotnet-buildtools are not updated for every new build of MSBuild. As I'm writing this (19 July 2016) there hasn't been a new package since 27 Jun 2016.

@rainersigwald @NikolaiTillmann @AndyGerlicher @cdmihai - Any thoughts on when we can expect the NuGet package work to be completed?

/cc @piotrpMSFT @brthor

[cdmihai]

@eerhardt
What about full framework msbuild? Will it need to have its own nuget packages, separate from the .net core ones? Do you have a scenario for this combination?

[eerhardt]

What about full framework msbuild?

If it is expected that the CLI will run full framework msbuild, then we will need to get those assets through NuGet package(s).

Will it need to have its own nuget packages, separate from the .net core ones?

I don't think the CLI has a requirement that they are together or separate. So whichever makes more sense from your end.

Do you have a scenario for this combination?

From what I've been hearing from folks is that when on Windows, the CLI is supposed to use the full framework MSBuild. But there are problems with that approach - namely if the full framework isn't installed on the machine. For example, on Windows Nano you cannot install the full .NET Framework.

[cdmihai]

If it is expected that the CLI will run full framework msbuild, then we will need to get those assets through NuGet package(s).

Can you please loop in the people who'll make the decision to this issue? it would be good to know whether to plan for this or not.

[eerhardt]

Can you please loop in the people who'll make the decision to this issue?

I assume it was the MSBuild and project system teams that were driving this requirement.

/cc @AndyGerlicher @rainersigwald @davkean

[rainersigwald]

Yes, it should be strictly better to run full-framework MSBuild if possible. See #711 for discussion about why.

I think the same-NuGet versus not will come down to how hard it is to build and package both flavors together. If we can do it without extreme hoop-jumping, one package seems better.

[jeffkl]

Design

Ref assemblies for net46 and netstandard1.3

1. The source for these will be generated at build time by the origination assembly, git status will show if anything changed
2. The source files will then be checked in under /ref/[tfm]/[assembly]
3. Developers will have to remember to build both desktop and netcore when making public API changes
4. Eventually when MSBuild is built targeting netstandard1.6, we won't need two ref assemblies

Lib assemblies for netstandard1.3 and net46

1. Enable XML doc comment files, ensure they are in the packages

Create a new BuildPackages.proj to create packages

1. Will be under /setup/
2. First iteration of this will assume you already ran cibuild --target Desktop and cibuild --target CoreCLR
3. This will also be simplified once MSBuild is built against only 1 framework

First iteration of these packages will not have symbols, this work is deferred to #969

*_Should fix #942 #682, #556 *_

I will also look at fixing #800 by implementing some sanity checks on the packages to ensure quality. There are already some NuGet package validators out there.

[eerhardt]

Why check in the ref assemblies?

[jeffkl]

Sorry I meant ref assembly source files. I'll update my comment.

[anurse]

Why do we need ref assemblies at all? ref assemblies should only be necessary when we are shipping the lib assemblies in separate packages (i.e RID-split packages). Having a lib/netstandard1.3 and a lib/net46 assembly should be sufficient, no?

There are already some NuGet package validators out there

We [ASP.NET] have a pretty good one based on some original NuGet code (https://github.com/aspnet/DnxTools/tree/dev/src/NuGetPackageVerifier) . @Eilon can help, as can I, offline to help you use it.

[jeffkl]

@anurse This was the guidance we got from @ericstj

@ericstj is it true we don't need any ref assemblies if we have a lib\net46 and a lib\netstandard1.3?

[ericstj]

If you have lib\netstandard1.3 that is correct. The reason I reccomended reference assemblies is because you had a .NETCore implementation where you wanted to use netstandard1.5 API, but you wanted to expose your own API to all platforms that supported netstandard1.3.

[jeffkl]

Thanks @ericstj. Does that mean we should have:

• lib\net45
• lib\netstandard1.5
• ref\netstandard1.3

[ericstj]

If you want to support net45 as well then ideally your ref would be netstandard1.1. That may be difficult since I believe your API surface for MSBuild depends on some things like ProcessStartInfo that were only introduced in netstandard1.3, so your suggested layout will work.

I want to be clear about what this is "saying" about support.

ref/netstandard1.3 means that you have surface area that depends on netstandard1.3 and you run on one or more frameworks that support netstandard1.3.

lib/net45 means you have an implementation in the package that runs net45 and later.
lib/netstandard1.5 means you have an implementation in the package that runs on any framework supporting netstandard1.5 or later.

Note that there is one framework that support netstandard1.3, but not netstandard1.5, and is not net45 or greater; UWP apps. You're _intentionally_ not supporting using MSBuild from within those apps with this layout. You might want to have that as a disclaimer in your package description since the error surfaced by NuGet will be a bit obscure ("package provides a ref but no lib...").

[anurse]

So the idea here is to allow people to create _libraries_ that target netstandard1.3 and use MSBuild APIs? If so, @ericstj 's approach is probably your best bet.

[ericstj]

Yep: enable for folks to create tasks and whatnot that is portable between desktop (min net46 per Roslyn) and .NETCore.

[jeffkl]

I've committed this and plan on iterating on it if there are any issues. I'm also working on a separate change for the "targets" package which allows users to do project evaluation and in-proc builds