Msbuild: Support reference assemblies in build flows

[rainersigwald]

The C# and VB compilers are building support for emitting a _reference assembly_ while compiling. The reference assembly contains the accessible interface of the assembly, but _no implementation details_. That means it changes less often than the full assembly--many common development activities don't change the interface, only the implementation. That means that incremental builds can be much faster--if you invert the sense of an if in a common library, today you must rebuild every assembly that references it, but with reference assemblies, you would only rebuild the library and copy it around to a few new output directories.

Roslyn spec for reference assemblies. Initial implementation: https://github.com/dotnet/roslyn/pull/17558

These assemblies should be supported in MSBuild scenarios.

Design criteria

• [ ] Create reference assemblies by default when using deterministic C#/VB compilation
• [ ] Use reference assemblies (when available) instead of implementation assemblies when invoking
  
  
  
  • [ ] the compiler
  
  
  • [ ] other tools? (resgen?)
  
  
• [ ] Targets that use ref assemblies are marked up to date and skipped when a reference's implementation assembly has changed but its ref assembly has not
• [ ] Final output folders must still contain implementation assemblies
• [ ] Project-to-project (p2p) references must be resolvable into both implementation and reference assemblies
• [ ] Design-time builds in Visual Studio build against the same assembly as the command line
• [ ] Build augmentations that are unaware of reference assemblies
  
  
  
  • [ ] continue to work (against implementation assemblies)
  
  
  • [ ] are able to learn about implementation assemblies if they want the incrementality benefits
  
  
• [ ] _Possible_ Consider how this interacts with NuGet reference assemblies. Can the concepts be unified?

[rainersigwald]

Implementation option: find ref assembly by filename transformation convention.

For example: after RAR and before CoreCompile, sub @(ReferencePath->'%(RootDir)%(Directory)\ref\%(Filename)%(Extension)') into the ReferencePath item if it exists.

• ➕ minimal MSBuild dataflow changes
• ➖ convention based (the compiler option allows arbitrary naming of the ref asm)

[rainersigwald]

Implementation option: pass ref assembly path as metadata on the primary compiler output

For example: after RAR and before CoreCompile, sub @(ReferencePath->'%(RootDir)%(Directory)\ref\%(Filename)%(Extension)') into the ReferencePath item if it exists.

• ➕ allows user flexibility in path conventions
• ➖ requires passing data though the convoluted p2p + RAR process
• ➖ Design time p2p resolution doesn't return an item but just the output path (currently)

[rainersigwald]

Ok, I've gotten a proof-of-concept put together based on the just-look-for-a-filename-pattern strategy, and it looks fairly reasonable.

Very little of this is production-ready, but it does work as hoped for: a whitespace change in src\Compilers\Core\Portable\AssemblyUtilities.cs causes recompilation for CodeAnalysis.csproj but doesn't propagate the ref assembly:

15:04:27.747     7>CopyFilesToOutputDirectory: (TargetId:67)
                     Reference assembly "S:\roslyn\Binaries\Debug\Dlls\CodeAnalysis\Microsoft.CodeAnalysis.interface.dll" already has latest information. Leaving it untouched. (TaskId:33)

diff --git a/MSBuild/15.0/Bin/MSBuild.exe.config b/MSBuild/15.0/Bin/MSBuild.exe.config
index eafd444..bb0f31a 100644
--- a/MSBuild/15.0/Bin/MSBuild.exe.config
+++ b/MSBuild/15.0/Bin/MSBuild.exe.config
@@ -46,6 +46,15 @@
           <assemblyIdentity name="XamlBuildTask" culture="neutral" publicKeyToken="31bf3856ad364e35" />
           <bindingRedirect oldVersion="4.0.0.0" newVersion="15.0.0.0" />
         </dependentAssembly>
+        <dependentAssembly>
+          <assemblyIdentity name="System.Collections.Immutable" publicKeyToken="b03f5f7f11d50a3a" culture="neutral"/>
+          <bindingRedirect oldVersion="0.0.0.0-1.2.1.0" newVersion="1.2.1.0" />
+        </dependentAssembly>
+        <dependentAssembly>
+          <assemblyIdentity name="System.IO.FileSystem" publicKeyToken="b03f5f7f11d50a3a" culture="neutral"/>
+          <bindingRedirect oldVersion="0.0.0.0-4.0.2.0" newVersion="4.0.2.0" />
+        </dependentAssembly>
+

         <!-- Workaround for crash in C++ CodeAnalysis scenarios due to https://github.com/Microsoft/msbuild/issues/1675 -->
         <dependentAssembly>
diff --git a/MSBuild/15.0/Bin/Microsoft.Common.CurrentVersion.targets b/MSBuild/15.0/Bin/Microsoft.Common.CurrentVersion.targets
index 8aa426d..6f58d49 100644
--- a/MSBuild/15.0/Bin/Microsoft.Common.CurrentVersion.targets
+++ b/MSBuild/15.0/Bin/Microsoft.Common.CurrentVersion.targets
@@ -347,6 +347,7 @@ Copyright (C) Microsoft Corporation. All rights reserved.
   </PropertyGroup>
   <ItemGroup>
     <IntermediateAssembly Include="$(IntermediateOutputPath)$(TargetName)$(TargetExt)"/>
+    <IntermediateRefAssembly Include="$(IntermediateOutputPath)$(TargetName).interface$(TargetExt)"/>
     <FinalDocFile Include="@(DocFileItem->'$(OutDir)%(Filename)%(Extension)')"/>
   </ItemGroup>

@@ -4003,6 +4004,17 @@ Copyright (C) Microsoft Corporation. All rights reserved.

     </Copy>

+    <!-- Copy the build product (.dll or .exe). -->
+    <CopyRefAssembly
+        SourcePath="@(IntermediateRefAssembly)"
+        DestinationPath="$(OutDir)@(IntermediateRefAssembly->'%(Filename)%(Extension)')"
+        Condition="'$(CopyBuildOutputToOutputDirectory)' == 'true' and '$(SkipCopyBuildProduct)' != 'true' and Exists('@(IntermediateRefAssembly)')"
+            >
+
+      <Output TaskParameter="DestinationPath" ItemName="ReferenceAssembly"/>
+
+    </CopyRefAssembly>
+
     <Message Importance="High" Text="$(MSBuildProjectName) -&gt; @(MainAssembly->'%(FullPath)')" Condition="'$(CopyBuildOutputToOutputDirectory)' == 'true' and '$(SkipCopyBuildProduct)'!='true'" />

     <!-- Copy the additional modules. -->
diff --git a/MSBuild/15.0/Bin/Microsoft.Common.tasks b/MSBuild/15.0/Bin/Microsoft.Common.tasks
index d284df7..4c7fbee 100644
--- a/MSBuild/15.0/Bin/Microsoft.Common.tasks
+++ b/MSBuild/15.0/Bin/Microsoft.Common.tasks
@@ -172,5 +172,6 @@
     <!-- Roslyn tasks are now in an assembly owned and shipped by Roslyn -->
     <UsingTask TaskName="Microsoft.CodeAnalysis.BuildTasks.Csc"                       AssemblyFile="$(RoslynTargetsPath)\Microsoft.Build.Tasks.CodeAnalysis.dll" Condition="'$(MSBuildAssemblyVersion)' != ''" />
     <UsingTask TaskName="Microsoft.CodeAnalysis.BuildTasks.Vbc"                       AssemblyFile="$(RoslynTargetsPath)\Microsoft.Build.Tasks.CodeAnalysis.dll" Condition="'$(MSBuildAssemblyVersion)' != ''" />
+    <UsingTask TaskName="Microsoft.CodeAnalysis.BuildTasks.CopyRefAssembly"           AssemblyFile="$(RoslynTargetsPath)\Microsoft.Build.Tasks.CodeAnalysis.dll" Condition="'$(MSBuildAssemblyVersion)' != ''" />
 </Project>

diff --git a/MSBuild/15.0/Bin/Roslyn/Microsoft.CSharp.Core.targets b/MSBuild/15.0/Bin/Roslyn/Microsoft.CSharp.Core.targets
index 5612c75..020adbb 100644
--- a/MSBuild/15.0/Bin/Roslyn/Microsoft.CSharp.Core.targets
+++ b/MSBuild/15.0/Bin/Roslyn/Microsoft.CSharp.Core.targets
@@ -1,13 +1,20 @@
 <?xml version="1.0" encoding="utf-8"?>
 <!-- Copyright (c)  Microsoft.  All Rights Reserved.  Licensed under the Apache License, Version 2.0.  See License.txt in the project root for license information. -->
 <Project ToolsVersion="14.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+  <Target Name="MungeRefs" BeforeTargets="CoreCompile">
+    <!--<Warning Text="RP: @(ReferencePath) %(RootDir)%(Directory)%(Filename).interface%(Extension)" />-->
+    <ItemGroup>
+      <TrueReferences Include="@(ReferencePath)" Condition="!Exists('%(RootDir)%(Directory)%(Filename).interface%(Extension)')" />
+      <TrueReferences Include="@(ReferencePath->'%(RootDir)%(Directory)%(Filename).interface%(Extension)')" Condition="Exists('%(RootDir)%(Directory)%(Filename).interface%(Extension)')" />
+    </ItemGroup>
+  </Target>
   <Target Name="CoreCompile"
           Inputs="$(MSBuildAllProjects);
                   @(Compile);
                   @(_CoreCompileResourceInputs);
                   $(ApplicationIcon);
                   $(AssemblyOriginatorKeyFile);
-                  @(ReferencePath);
+                  @(TrueReferences);
                   @(CompiledLicenseFile);
                   @(LinkResource);
                   @(EmbeddedDocumentation);
@@ -117,7 +124,7 @@
          Prefer32Bit="$(Prefer32Bit)"
          PreferredUILang="$(PreferredUILang)"
          ProvideCommandLineArgs="$(ProvideCommandLineArgs)"
-         References="@(ReferencePath)"
+         References="@(TrueReferences)"
          ReportAnalyzer="$(ReportAnalyzer)"
          Resources="@(_CoreCompileResourceInputs);@(CompiledLicenseFile)"
          ResponseFiles="$(CompilerResponseFile)"

Explanations:

• Binding redirects: the Roslyn task assembly now has (transitive) references to these two assemblies that don't match the versions delivered in MSBuild or Roslyn. The plan is to move the task assembly to its own folder with a possibly-different closure. MSBuild might still want to set these binding redirects to match VS as part of #1542.
• Ref file pattern: I got lazy and did this instead of mkdir ref/. It's likely a requirement to keep the filename the same.
• Extra copy task does the ref magic incremental copy. Initially I was thinking of telling the compiler to emit to a temporary location and doing the magic copy from there to obj, but now I think it's sufficient to always update obj and only update bin when necessary, because ProjectReferences use the output from bin.
• The target here probably belongs in common rather than C# targets. I did notice it not working for VB in my testing.
• TrueReferences and MungeRefs are terrible, no-good, very-bad names.

[rainersigwald]

Design points to be settled:

• We talked about making this "default when Deterministic is turned on". Where should that be expressed? Currently common targets don't know about Deterministic. Set a property in CSharp.Core.targets that's observed in common, or just bake it into common?
  
  
  
  • 💬 I propose just baking it into Common. It's already a mess and this doesn't make it appreciably worse.
  
  
• What's the convention for paths? Based on conversation with @jaredpar, it's likely useful to make sure that the reference assembly's filename matches the implementation assembly's.
  
  
  
  • 💬 I propose what we'd talked about initially $(OutDir)\ref\$(TargetFileName).
  
  
• Should the reference assembly be passed along explicitly (via metadata) or inferred by probing for an assembly at the ref-assembly-relative-path convention?
  
  
  
  • 💬 I propose passing it along via metadata. GetTargetPath already returned a metadata-decorated item, and it seems low-risk to do the same from Build, giving us the freedom to have arbitrary paths to the reference assembly (if user-desired) and "push" the knowledge of the ref asm from the producing project instead of inferring it from the consuming side.
  
  
• Should the Csc task consume metadata on the existing @(ReferencePath), a permanently changed ReferencePath, a temporarily changed ReferencePath, or a different item?
  
  
  
  • Tasks consume metadata:
  
  
  • ➕ Scoped impact
  
  
  • ➖ Incrementality on CoreCompile becomes hard: the "real" inputs are either the path in Identity or the path in ReferenceAssembly of the references.
  
  
  • Permanently changed ReferencePath:
  
  
  • ➖ Unclear what targets besides CoreCompile consume this (including 3rd-party extensions).
  
  
  • Temporarily changed ReferencePath:
  
  
  • ➕ Scoped impact
  
  
  • ➖ No other target could opt into using the ref assemblies
  
  
  • ➖ Two extra targets would be run to replace the list and to restore it.
  
  
  • New item:
  
  
  • ➕ Scoped impact
  
  
  • ➕ Other targets could opt in to ref-assembly-only list
  
  
  • ➖ Targets would _have_ to opt in
  
  
  • ➖ Microsoft.CSharp.Core.targets would need a compat shim to be able to run with older common targets.
  
  
  • 💬 I propose using a new item, with a compat shim in C# core.
  
  
• Is there a better name than ReferenceAssembly for the MSBuildification of these concepts? ReferenceAssembly is already used to find full-framework reference assemblies, for example in GetReferenceAssemblyPaths.
  
  
  
  • 💬 It's confusing but the concepts are very similar. I think I prefer just using ReferenceAssembly.
  
  

@jaredpar @jcouv @Microsoft/msbuild-maintainers: does that sound good? Anything I've forgotten?

current implementation

diff --git a/MSBuild/15.0/Bin/MSBuild.exe.config b/MSBuild/15.0/Bin/MSBuild.exe.config
index eafd444..bb0f31a 100644
--- a/MSBuild/15.0/Bin/MSBuild.exe.config
+++ b/MSBuild/15.0/Bin/MSBuild.exe.config
@@ -46,6 +46,15 @@
           <assemblyIdentity name="XamlBuildTask" culture="neutral" publicKeyToken="31bf3856ad364e35" />
           <bindingRedirect oldVersion="4.0.0.0" newVersion="15.0.0.0" />
         </dependentAssembly>
+        <dependentAssembly>
+          <assemblyIdentity name="System.Collections.Immutable" publicKeyToken="b03f5f7f11d50a3a" culture="neutral"/>
+          <bindingRedirect oldVersion="0.0.0.0-1.2.1.0" newVersion="1.2.1.0" />
+        </dependentAssembly>
+        <dependentAssembly>
+          <assemblyIdentity name="System.IO.FileSystem" publicKeyToken="b03f5f7f11d50a3a" culture="neutral"/>
+          <bindingRedirect oldVersion="0.0.0.0-4.0.2.0" newVersion="4.0.2.0" />
+        </dependentAssembly>
+

         <!-- Workaround for crash in C++ CodeAnalysis scenarios due to https://github.com/Microsoft/msbuild/issues/1675 -->
         <dependentAssembly>
diff --git a/MSBuild/15.0/Bin/Microsoft.Common.CurrentVersion.targets b/MSBuild/15.0/Bin/Microsoft.Common.CurrentVersion.targets
index 8aa426d..4881519 100644
--- a/MSBuild/15.0/Bin/Microsoft.Common.CurrentVersion.targets
+++ b/MSBuild/15.0/Bin/Microsoft.Common.CurrentVersion.targets
@@ -281,6 +281,8 @@ Copyright (C) Microsoft Corporation. All rights reserved.
     <!-- Example, c:\MyProjects\MyProject\bin\debug\MyAssembly.dll -->
     <TargetPath Condition=" '$(TargetPath)' == '' ">$(TargetDir)$(TargetFileName)</TargetPath>

+    <TargetRefPath Condition=" '$(TargetRefPath)' == '' and '$(Deterministic)' == 'true' ">$([System.IO.Path]::Combine(`$([System.IO.Path]::GetDirectoryName($([System.IO.Path]::GetFullPath(`$(TargetPath)`))))`, 'ref', `$(TargetFileName)`))</TargetRefPath>
+
     <!-- Example, c:\MyProjects\MyProject\ -->
     <ProjectDir Condition=" '$(ProjectDir)' == '' ">$(MSBuildProjectDirectory)\</ProjectDir>

@@ -350,6 +352,13 @@ Copyright (C) Microsoft Corporation. All rights reserved.
     <FinalDocFile Include="@(DocFileItem->'$(OutDir)%(Filename)%(Extension)')"/>
   </ItemGroup>

+  <ItemGroup Condition="'$(Deterministic)' == 'true'">
+    <!-- TODO: should this be configurable? Default path obeys conventions. -->
+    <IntermediateRefAssembly Include="$(IntermediateOutputPath)ref\$(TargetName)$(TargetExt)" Condition="'@(IntermediateRefAssembly)' == ''" />
+    <CreateDirectory Include="@(IntermediateRefAssembly->'%(RootDir)%(Directory)')" />
+    <CreateDirectory Include="$(OutDir)ref" />
+  </ItemGroup>
+
   <ItemGroup Condition="'$(_DebugSymbolsProduced)' == 'true'">
     <_DebugSymbolsIntermediatePath Include="$(IntermediateOutputPath)$(TargetName).compile.pdb" Condition="'$(OutputType)' == 'winmdobj' and '@(_DebugSymbolsIntermediatePath)' == ''"/>
     <_DebugSymbolsIntermediatePath Include="$(IntermediateOutputPath)$(TargetName).pdb" Condition="'$(OutputType)' != 'winmdobj' and '@(_DebugSymbolsIntermediatePath)' == ''"/>
@@ -771,7 +780,7 @@ Copyright (C) Microsoft Corporation. All rights reserved.
       Name="Build"
       Condition=" '$(_InvalidConfigurationWarning)' != 'true' "
       DependsOnTargets="$(BuildDependsOn)"
-      Returns="$(TargetPath)" />
+      Returns="@(TargetPathWithTargetPlatformMoniker)" />

   <!--
     ============================================================
@@ -1797,6 +1806,7 @@ Copyright (C) Microsoft Corporation. All rights reserved.
       <TargetPathWithTargetPlatformMoniker Include="$(TargetPath)">
         <TargetPlatformMoniker>$(TargetPlatformMoniker)</TargetPlatformMoniker>
         <TargetPlatformIdentifier>$(TargetPlatformIdentifier)</TargetPlatformIdentifier>
+        <ReferenceAssembly Condition="'$(TargetRefPath)' != ''">$(TargetRefPath)</ReferenceAssembly>
       </TargetPathWithTargetPlatformMoniker>
     </ItemGroup>
   </Target>
@@ -2015,8 +2025,18 @@ Copyright (C) Microsoft Corporation. All rights reserved.
       <Output TaskParameter="FilesWritten" ItemName="FileWrites"/>
       <Output TaskParameter="DependsOnSystemRuntime" PropertyName="DependsOnSystemRuntime"/>
     </ResolveAssemblyReference>
+
+    <ItemGroup>
+      <ReferencePathWithInterfaceOnlyAssemblies Include="@(ReferencePath->'%(ReferenceAssembly)')" />
+    </ItemGroup>
   </Target>

+  <ItemDefinitionGroup>
+    <ReferencePath>
+      <ReferenceAssembly>%(FullPath)</ReferenceAssembly>
+    </ReferencePath>
+  </ItemDefinitionGroup>
+
   <!--
     ====================================================================================================

@@ -4003,6 +4023,17 @@ Copyright (C) Microsoft Corporation. All rights reserved.

     </Copy>

+    <!-- Copy the build product (.dll or .exe). -->
+    <CopyRefAssembly
+        SourcePath="@(IntermediateRefAssembly)"
+        DestinationPath="$(TargetRefPath)"
+        Condition="'$(CopyBuildOutputToOutputDirectory)' == 'true' and '$(SkipCopyBuildProduct)' != 'true' and Exists('@(IntermediateRefAssembly)')"
+            >
+
+      <Output TaskParameter="DestinationPath" ItemName="ReferenceAssembly"/>
+
+    </CopyRefAssembly>
+
     <Message Importance="High" Text="$(MSBuildProjectName) -&gt; @(MainAssembly->'%(FullPath)')" Condition="'$(CopyBuildOutputToOutputDirectory)' == 'true' and '$(SkipCopyBuildProduct)'!='true'" />

     <!-- Copy the additional modules. -->
diff --git a/MSBuild/15.0/Bin/Microsoft.Common.tasks b/MSBuild/15.0/Bin/Microsoft.Common.tasks
index d284df7..4c7fbee 100644
--- a/MSBuild/15.0/Bin/Microsoft.Common.tasks
+++ b/MSBuild/15.0/Bin/Microsoft.Common.tasks
@@ -172,5 +172,6 @@
     <!-- Roslyn tasks are now in an assembly owned and shipped by Roslyn -->
     <UsingTask TaskName="Microsoft.CodeAnalysis.BuildTasks.Csc"                       AssemblyFile="$(RoslynTargetsPath)\Microsoft.Build.Tasks.CodeAnalysis.dll" Condition="'$(MSBuildAssemblyVersion)' != ''" />
     <UsingTask TaskName="Microsoft.CodeAnalysis.BuildTasks.Vbc"                       AssemblyFile="$(RoslynTargetsPath)\Microsoft.Build.Tasks.CodeAnalysis.dll" Condition="'$(MSBuildAssemblyVersion)' != ''" />
+    <UsingTask TaskName="Microsoft.CodeAnalysis.BuildTasks.CopyRefAssembly"           AssemblyFile="$(RoslynTargetsPath)\Microsoft.Build.Tasks.CodeAnalysis.dll" Condition="'$(MSBuildAssemblyVersion)' != ''" />
 </Project>

diff --git a/MSBuild/15.0/Bin/Roslyn/Microsoft.CSharp.Core.targets b/MSBuild/15.0/Bin/Roslyn/Microsoft.CSharp.Core.targets
index 5612c75..2aee7ca 100644
--- a/MSBuild/15.0/Bin/Roslyn/Microsoft.CSharp.Core.targets
+++ b/MSBuild/15.0/Bin/Roslyn/Microsoft.CSharp.Core.targets
@@ -7,7 +7,7 @@
                   @(_CoreCompileResourceInputs);
                   $(ApplicationIcon);
                   $(AssemblyOriginatorKeyFile);
-                  @(ReferencePath);
+                  @(TrueReferences);
                   @(CompiledLicenseFile);
                   @(LinkResource);
                   @(EmbeddedDocumentation);
@@ -117,7 +117,7 @@
          Prefer32Bit="$(Prefer32Bit)"
          PreferredUILang="$(PreferredUILang)"
          ProvideCommandLineArgs="$(ProvideCommandLineArgs)"
-         References="@(ReferencePath)"
+         References="@(TrueReferences)"
          ReportAnalyzer="$(ReportAnalyzer)"
          Resources="@(_CoreCompileResourceInputs);@(CompiledLicenseFile)"
          ResponseFiles="$(CompilerResponseFile)"

[jaredpar]

These all look good to me.

[jcouv]

Looks good. I especially like the leveraging of existing /obj and /bin distinction. At least we don't have to create a third copy of the ref assembly just to manage timestamps :-)

[rainersigwald]

Had a design review with @AndyGerlicher and @jeffkl today. No changes to the plan, but a couple of options were discussed and dismissed:

• Idea: extracting the MVIDs of references to compute incremental up-to-dateness. Rejected because while we now (post-#1327) have an easy way to _cause_ CoreCompile to run even if up-to-date-checks would otherwise skip it, there's no good way to do the opposite.
• Idea: leave the reference assembly in obj, never copying it to bin. Rejected because we think the ref asm is a real project output, and because this would involve telling the compiler to generate it to an intermediate location and copying it around--so there's little gain.

Also discussed was https://github.com/dotnet/roslyn/issues/19103.

I'm going to go forward with PRs now based on this design.

[borgdylan]

Will this be property based? I.e. will be be a property based switch that turns reference assembly making on/off? Please allow compilers other than roslyn to implement the feature.

[jaredpar]

There isn't anything specific to Roslyn that I'm aware of here. Most of the work is around making the target graphs able to benefit from reference assemblies in general.

[rainersigwald]

@borgdylan I'm planning to implement that property next up in PR #2039.

This is absolutely available to compilers other than Roslyn--the core compilation target just needs to consume the new item rather than @(ReferencePath) to opt into using reference assemblies, and needs to define the ReferenceAssembly metadata on its output if it _produces_ ref assemblies.

[rainersigwald]

@davkean @jviau I'm proposing to change the item that gets passed to Csc in CoreCompile to a new one that has interface-only reference assemblies substituted in for implementation assemblies where available. @jasonmalinowski is worried that this might break something in the project system. Do you know if the project system or CPS directly cares about @(ReferencePath)? If it does, what's the nature of that relationship?

To summarize: I'm concerned about swapping in the ref assembly _everywhere_ because I don't know what targets (including third-party extension things) have a dependency on @(ReferencePath) containing the full output assembly of references. But we _do_ want to use the reference assemblies for things like design-time builds.

[borgdylan]

Thanks for clarifying. I will try to implement this for my compiler since it would be for me to tell thr compiler to forgo generation of method bodies. That would allow me to implement the MSBuild specific bits that interface with your PR. Would MSBuild suggest a path where the reference assembly should be placed?

[rainersigwald]

@borgdylan Yes, in my PR the common targets suggest a default--though you'll want to opt in to the feature once I've made that possible. Csc consumes it fairly simply: https://github.com/rainersigwald/roslyn/blob/2164877d0099c291f2aee22773058ba01fa78266/src/Compilers/Core/MSBuildTask/Microsoft.CSharp.Core.targets#L126, and I would expect that you could do the same.

Also possibly relevant: https://github.com/dotnet/roslyn/pull/19133 adds some requirements to the shape of the ref assembly to keep it compatible with the new incremental CopyRefAssembly task.

[borgdylan]

Thanks for heads up. I did not know of @(IntermediateAssembly) and $(PdbFile). I was concatenating the components myself.

[rainersigwald]

@borgdylan I have a meeting set up with a doc writer to figure out how to start to document some of the soup that is common targets. That kind of detail will likely be low on the priority list since it's primarily useful for compiler writers, but maybe someday. . .

[borgdylan]

In the mean time I will look at the CSharp targets for inspiration once this makes its way into the mono fork.

[davkean]

@rainersigwald I had a quick look, I don't know of anything that would break by using a different item to pass to the compiler instead @(ReferencePath).

I do suspect, however, that the legacy project system (and our yet to be written up-to-date check) would need to be taught about these concepts, preferable via MSBuild items.

[rainersigwald]

The initial implementation of this is complete, but opt-in.

[borgdylan]

@rainersigwald how does msbuild signal that it wants a reference assembly?

[rainersigwald]

@borgdylan

The design we landed on was:

• The _producing_ project respects $(ProduceReferenceAssembly) to determine whether or not to produce the ref assembly. If it does:
  
  
  
  • The return value of GetTargetPath and Build should be an item that has the ReferenceAssembly and CopyUpToDateMarker metadata set to appropriate values.
  
  
• The _referencing_ project respects $(CompileUsingReferenceAssemblies) (as an emergency escape hatch), and if it's not set to false, passes @(ReferencePath->'%(ReferenceAssembly)') to the compiler instead of just @(ReferencePath).

So there's no real way to indicate from a referencing project that you want ref assemblies, it's up to the producing project to provide them or not.

Does that answer the question?

[borgdylan]

On the receiving end I am just using ReferencePathWithRefAssemblies since I know that I will be having v15.6 installed. I want to compile against reference assemblies whenever they are available.

Thanks for answering the part on the producing end. For now I just want to have my targets set up to handle the reference assembly stuff prior to my compiler actually emitting them. In the mean time I will be tricking msbuild by giving a full assembly regardless of what it asks for. However I plan to be writing the metadata and passing a flag to the compiler in the reference assembly production case.

[borgdylan]

What I did not understand: How do I change the values for the mentioned targets from within CoreCompile?

[borgdylan]

Also, what causes $(ProduceReferenceAssembly) to be set to true? It is always false in my builds. As I see it something must be in place to automatically trigger a ref assembly for all projects that are dependencies of the one being built.

[rainersigwald]

@borgdylan Right now ref assemblies are an opt-in feature, so a project must explicitly set <ProduceReferenceAssemblies>true</ProduceReferenceAssemblies> (or get it through an import--Directory.Build.props makes that easy).

I don't think I understand this:

As I see it something must be in place to automatically trigger a ref assembly for all projects that are dependencies of the one being built.

Why? It's not a great situation, but it's perfectly valid to have a non-ref-assembly-aware project depend on a ref assembly but not produce one in turn. You'd still get incremental build benefits since it would only rebuild when its own implementation or the reference's interface changes.

[borgdylan]

What I meant is that a dependency project should know that it is currently treated as a dependency and is not the focus, so it would be enough for it to produce a reference assembly only. Making both a normal and a reference assembly is too wasteful in the same build and defeats the use of reference assemblies to get a faster build.

[rainersigwald]

Ah, I see. That's not a scenario in the current design.

The Roslyn compiler simultaneously emits the reference and implementation assemblies, so it's not significantly slower than emitting only the reference assembly (AFAIK).

Even if that wasn't the case, I don't think I agree with your premise. If you didn't compile the implementation assembly when it was out of date but the build was triggered from a reference, you would potentially be using stale assemblies downstream.

Consider this situation (apologies if I'm butchering the Dylan):

define method adder  (a, b :: <integer>)
    a - b
end;

in one assembly adder.dylproj, and a unit test that verifies that adder adds in test.dylproj. Build test.dylproj, producing adder.dll, ref/adder.dll, and test.dll and observe that the test fails because the implementation of the method is wrong. Fix the error, then build the test project again. What you want is adder.dll', with internals, but an unmodified ref/adder.dll. Under your proposal, if only the ref assembly is regenerated, the test would be rerun against the stale copy of the assembly.

[borgdylan]

Apologies accepted. As things stand, I will be holding off my implementation as I will not have any advantages if I do. If a file in a project changes, the build would still trigger making a new reference assembly anyways. I may do a partial implementation to make distributable reference assemblies be possible.

Side note: dylan.NET is a misnamed language as it does not use the original "Dylan" syntax. An "adder" method would be written as

method public integer adder(var a as integer, var b as integer)
    return a + b
end method

or inspired by C# 6 arrow methods

method public integer adder(var a as integer, var b as integer) =>  a + b