Msbuild: Question: How to write an MSBuild extension that is extensible

[nguerrera]

Moved from https://github.com/dotnet/sdk/issues/898 on behalf of @AArnott

I'm working to port my CodeGeneration.Roslyn MSBuild extension to .netstandard so that it works under dotnet build. The design is that folks can build on my SDK NuGet package to write their Roslyn-based code generator, and then the end user chains in my build-time NuGet package which "discovers" these code generation extensions and loads the code generator assemblies to do the work on my scaffolding.

In the conversion, I hit two major roadblocks from .netstandard limitations:

1. Assembly.LoadFile is missing. I understand this is coming in .netstandard 2.0 but is there any supported alternative in the meantime for an MSBuild Task to load some assembly based on its path? I mean, MSBuild Core itself can do it, so how is that done so I can emulate it?
2. No AppDomain support. This is super useful for two reasons: I can control the version of Roslyn assemblies that are loaded to the ones I was compiled against, thus avoiding assembly load failures. It also lets me unload these extension assemblies when I'm done with them so they don't lock files on disk when the build is over (including VS in proc design-time builds). If MSBuild Core similarly keeps msbuild.exe processes running after a build, we have a significant problem, but if it doesn't, I might omit AppDomain support from just the .netstandard build of this MSBuild Task, then cross my fingers and hope for the best about the Roslyn assembly versions.

Thoughts?

[nguerrera]

I mean, MSBuild Core itself can do it, so how is that done so I can emulate it?

See CoreCLRAssemblyLoader.cs

cc @tmeschter

[AArnott]

Thanks, @nguerrera. I'll target my MSBuild Task to net452 and netcore instead of netstandard then. That works as MSBuild is only either of those and I can already cross target that. The meat of my code is in a netstandard library and can remain that way so that extensions to my extension can remain .netstandard based.

That leaves the AppDomain support question and whether Roslyn assembly versions and "TSR" style OOP build nodes will be a problem. I may find that out myself soon.

[rainersigwald]

Currently, .NET Core MSBuild does not support node reuse (TSR OOP build worker nodes), so that should be fine. We may want to bring it back at some point, but it's not currently on our radar.

[AArnott]

Thanks. I can't close this transferred issue, so feel free to close it anyone who can.

[AArnott]

I notice that when I compile my MSBuild Task project for net452, the dependency assemblies are copied into the outdir as well. But when I compile for netcoreapp1.0, no PackageReference dependencies get copied in (though ProjectReferences do). Why is that? I'm investigating diagnostic logs now to find a workaround in the meantime because I need all non-platform dependencies regardless of target platform.

[AArnott]

I haven't been able to figure this out. I've tried forcing all references to copy local and it doesn't work:

  <Target Name="CopyLocalPackageReferences"
          AfterTargets="ResolveReferences"
          Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
    <ItemGroup>
      <Reference>
        <Private>true</Private>
        <CopyLocal>true</CopyLocal>
      </Reference>
      <ReferencePath>
        <Private>true</Private>
        <CopyLocal>true</CopyLocal>
      </ReferencePath>
    </ItemGroup>
  </Target>

I wonder though if this is even the right approach. I'd settle for it, to be sure, if it worked. But CoreCLR patterns are that dependencies can come from the package cache (at least in some cases). And sometimes this can be quite important because suppose I depended on an assembly that varies its implementation based on the operating system it runs on. NuGet can express that, but I can't simply "copy local" all the DLLs into my MSBuild Core task's bin directory and have it run everywhere that msbuild core can.

Is there a supported path for MSBuild Core tasks to have dependencies that are as rich as a CoreCLR app's dependencies can be?

[AArnott]

In particular, my MSBuild Tasks dll, when built for netcoreapp1.0, produces a *.deps.json file that it seems (ideally) MSBuild Core or CoreCLR itself could pick up and use to find all the runtime dependencies. Of course somehow these packages would need to have been restored -- without being part of the dependency graph of the app being built.

[AArnott]

Ya, the lack of AppDomains on .NET Core _is_ breaking me. MSBuild Core ships with _some_ Roslyn DLLs but not all that I need, so I have to ship some, but unless the versions I ship are exactly the version that MSBuild ships, I'm broken on .NET Core. :(

[AArnott]

So after updating the Roslyn my MSBuild task compiles against to match MSBuild Core's (which is just a temporary workaround), I hit a similar issue with Validation.dll. I compile against 2.4.0.0, but another MSBuild extension this test project uses itself depends on 2.3.0.0. And despite CoreCLR's policy (AFAIK) of just loading the DLL it finds and disregarding the version, CoreCLR is only willing to read the 2.3.0.0 version (because that came first) and then throws because it can't load the 2.4.0.0 version, even though it's right there and could be loaded. So CoreCLR both discriminates on assembly version and isn't willing to load both. Now I don't know what to do.

[tmat]

@AArnott I am not familiar with the details of your problem, but it seems to me that .NET Core version of MSBuild _could_ use a separate AssemblyLoadContext for each build task it loads and load its dependencies to that context to isolate the task from other task dependencies. Is it not doing so currently?

[AArnott]

Thanks for your thoughts, @tmat.
Evidently not. The Nerdbank.gitversioning task's Validation.dll is polluting the task I'm working on.
Even if msbuild created separate AssemblyLoadContexts for each Task, would that protect each Task from pulling in the Roslyn assemblies that ship with msbuild?

Is a separate AssemblyLoadContext something my task can create, or is that reserved for just the host?

[nguerrera]

To copy package dependencies to output, you can use CopyLocalLockFileAssemblies=true. This is defaulted to false for netcoreapp because .NET Core has the capability to resolve from nuget cache.

And despite CoreCLR's policy (AFAIK) of just loading the DLL it finds and disregarding the version

The CoreCLR policy is to bind to DLL with >= version, not to disregard version.

it seems to me that .NET Core version of MSBuild could use a separate AssemblyLoadContext for each build task it loads and load its dependencies to that context to isolate the task from other task dependencies. Is it not doing so currently?

It seems it did at one time and there were problems so the default context was used for everything: https://github.com/Microsoft/msbuild/commit/8099cb33cffcd822341edf15167c7f5212b83ac0

EDIT: I think it previously used one extra non-default load context and not one for each task. Regardless, it seems the problems that blocked that would also block doing one per task.

Is a separate AssemblyLoadContext something my task can create, or is that reserved for just the host?

AFAIK, you can create them yourself. cc @gkhanna on that and for any other clarification that might help resolve this.

In general, the whole story around deploying an msbuild task via nuget needs significant work. I should be able to package only my assemblies and manifest my dependencies in the package somehow.

[AArnott]

you can use CopyLocalLockFileAssemblies=true

Thanks. That worked great. EDIT: except that it copies a bunch of framework assemblies as well that I'm sure msbuild core already ensures are available elsewhere, but I'll live with that for now.

In general, the whole story around deploying an msbuild task via nuget needs significant work. I should be able to package only my assemblies and manifest my dependencies in the package somehow.

That's what I was starting to realize and hope.

The CoreCLR policy is to bind to DLL with >= version, not to disregard version.

That sounds OK. Except when someone needs version 2.3.0.0 and finds it, and loads it. Then later when another assembly needs 2.4.0.0 and can find it, but isn't allowed to load it because one is already loaded.

[tmat]

It seems it did at one time and there were problems so the default context was used for everything: 8099cb3
EDIT: I think it previously used one extra non-default load context and not one for each task. Regardless, it seems the problems that blocked that would also block doing one per task.

It seems to me that the use of ALC per task wouldn't be blocked anymore since the CoreCLR bug has been fixed.

Except when someone needs version 2.3.0.0 and finds it, and loads it. Then later when another assembly needs 2.4.0.0 and can find it, but isn't allowed to load it because one is already loaded.

You can't replace an assembly with another assembly in-place, other than doing EnC on the difference :). I think in a plugin system like msbuild the plugins need to be isolated in different ALCs and only msbuild assemblies that implement types used to communicate between the plugins be loaded in a default ALC. In such a model each ALC would only load assemblies that are known at build time of the plugin.

[AArnott]

Is a separate AssemblyLoadContext something my task can create, or is that reserved for just the host?

AFAIK, you can create them yourself.

Yes, it turns out you can. It's almost trivially easy. I think I might like 'em better than AppDomains. :) One just has to derive from AssemblyLoadContext and then can set whatever assembly resolving policy they want within it. I proved that I can load two versions of the Validation assembly in the same dotnet.exe console app using what I would fully expect to work in an msbuild task.

I'm not sure this is a free ticket to solve my msbuild task problems though, since msbuild itself and all other tasks run in AssemblyLoadContext.Default, it might be that I can keep my own dependencies pure -- except when they are also found in the default context (and they will be, with the wrong versions). And my understanding is that my load context doesn't get asked if the assembly is already discoverable in the default one.

[tmeschter]

Implementing an AssemblyLoadContext per plugin in MSBuild was definitely something I thought about while fixing up the MSBuild assembly loading for .NET Core. At the time, however, there were bugs in assembly loading that would have broken scenarios we had to support, and the primary goal was parity with the MSBuilds assembly loading behavior on Desktop.

I think the idea still makes sense once those bugs are worked out (though we would probably want an ALC per _folder_ rather than per _task_ since you can probably assume all the assemblies in a given directory are meant to work together, and you won't have multiple versions of an assembly in the same folder anyway, but that's an implementation detail).

Being able to query a *.deps.json to find the location of a desired assembly would certainly be useful. That functionality should be exposed by .NET Core rather than MSBuild itself, however--you don't really want every app with an extensibility model to reinvent that logic.

[AArnott]

Yes, these are fine points. Thanks, @tmeschter.

Being able to query a *.deps.json to find the location of a desired assembly would certainly be useful. That functionality should be exposed by .NET Core rather than MSBuild itself

How do you feel about driving CoreCLR to add such a feature?

[tmeschter]

@AArnott

I'm not sure this is a free ticket to solve my msbuild task problems though, since msbuild itself and all other tasks run in AssemblyLoadContext.Default, it might be that I can keep my own dependencies pure -- except when they are also found in the default context (and they will be, with the wrong versions). And my understanding is that my load context doesn't get asked if the assembly is already discoverable in the default one.

Your AssemblyLoadContext will only be asked to resolve an assembly when it is requested by an assembly already loaded in that context.

I think what you want to do is separate things out so the task assembly contains very little logic--just enough to create a new AssemblyLoadContext and then use that ALC to load the assembly with the real "meat" of your extension. From that point on your ALC will be called on to resolve everything needed by your extension. I expect it would handle some known assemblies itself, and fall back on the default ALC for everything else.

Does that make sense? If MSBuild itself created per-extension ALCs this wouldn't be necessary, but I think it's workable.

[tmeschter]

@AArnott

How do you feel about driving CoreCLR to add such a feature?

I don't think I'm the right person for that--I'm back on Roslyn after a brief stint on MSBuild, so I no longer have all the context.

[nguerrera]

EDIT: except that it copies a bunch of framework assemblies as well that I'm sure msbuild core already ensures are available elsewhere, but I'll live with that for now.

I'm bumping in to this myself: https://github.com/dotnet/sdk/issues/933