Azure-sdk-for-net: Microsoft.WindowsAzure.targets in combination with PackageReference resolves ref target

[danielmarbach]

When Cloud Service Solution with Microsoft.WindowsAzure.targets is used in combination with PackageReference (old csproj format with package reference interop mode) in combination with a reference assembly like System.ValueTuple the reference assembly is copied into the output path instead of the implementation assembly for the target framework.

Prerequisites

Azure SDK 2.9
VS2017 15.3.3 or higher

Steps to reproduce

• Create a new cloud services solution
• Add a new worker role, target framework < 4.7 (preferably 4.6.2 or smaller)
• Reference System.ValueTuple from nuget via packages.config
• Add

        private static (int foo, bool bar) Read()
        {
            return (42, true);
        }

• Use method in Run like Trace.TraceInformation(Read().ToString());
• Debug locally with F5
• Works see how csx\Debug\roles\WorkerRole1\approot contains 78 kb size ValueTuple implemenation assembly
  
• Add another worker role, target framework < 4.7 (preferably 4.6.2 or smaller)
• Remove packages.config
• Reference System.ValueTuple via package references

  <ItemGroup>
    <PackageReference Include="Microsoft.Azure.KeyVault.Core" Version="2.0.4" />
    <PackageReference Include="Microsoft.Data.Edm" Version="5.*" />
    <PackageReference Include="Microsoft.Data.OData" Version="5.*" />
    <PackageReference Include="Microsoft.Data.Services.Client" Version="5.*" />
    <PackageReference Include="Microsoft.WindowsAzure.ConfigurationManager" Version="3.*" />
    <PackageReference Include="Newtonsoft.Json" Version="8.*" />
    <PackageReference Include="System.Spatial" Version="5.*" />
    <PackageReference Include="System.ValueTuple" Version="4.*" />
    <Reference Include="Microsoft.WindowsAzure.Diagnostics, Version=2.8.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35">
      <Private>True</Private>
    </Reference>
    <Reference Include="Microsoft.WindowsAzure.ServiceRuntime, Version=2.7.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35">
      <Private>False</Private>
    </Reference>
    <PackageReference Include="WindowsAzure.Storage" Version="8.4.0" />
    <Reference Include="Microsoft.CSharp" />
    <Reference Include="System.Data" />
    <Reference Include="System.Xml" />
  </ItemGroup>

• Add

        private static (int foo, bool bar) Read()
        {
            return (42, true);
        }

• Use method in Run like Trace.TraceInformation(Read().ToString());
• Debug locally with F5
• Observe BadImageFormatException

Could not load file or assembly 'System.ValueTuple, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. Reference assemblies should not be loaded for execution.  They can only be loaded in the Reflection-only loader context. (Exception from HRESULT: 0x80131058)

   at WorkerRole2.WorkerRole.Run()
   at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.StartRoleInternal()
   at Microsoft.WindowsAzure.ServiceRuntime.Implementation.Loader.RoleRuntimeBridge.<StartRole>b__2()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

• Works see how csx\Debug\roles\WorkerRole2\approot contains 40 kb size ValueTuple ref assembly without implemenations
  

Repro Solution https://github.com/danielmarbach/PackageRef.CloudServicesRepro

Expected behavior

Target copies implementation assembly instead of ref assembly to csx output path.

[shahabhijeet]

@danielmarbach I am not sure for what product you are reporting this bug.
This repo is for azure service SDK (management plane and data plane .NET SDK for azure resource providers)
Can you clarify.

[danielmarbach]

I think the issue is related to Microsoft.WindowsAzure.targets and since this seems to be part of the SDK I raised the issue here. If you can tell me where the issue should be raised instead I'd be more than happy to move it over.

https://github.com/Azure/azure-sdk-for-net/tree/psSdkJson6/tools/legacy/Microsoft.WindowsAzure.Build.Tasks

[danielmarbach]

I closed this issue since it seems there is no interest in fixing it.

[jfshark]

Hello!
@danielmarbach is there a workaround? I have exactly the same issue, and no idea ATM how to fix it in most natural way

[danielmarbach]

Not sure because the library I was trying to use did drop the reference to the value tuple in the end.

[yirkha]

Hi all,

Analysis

In case of a NuGet package that provides both kinds of assemblies, the build-only reference assemblies from ref/ end up in a MSBuild property @(ReferencePath) and the full runtime assemblies from lib/ end up in @(ReferenceCopyLocalPaths).

A target CollectWorkerRoleFiles in Microsoft.WindowsAzure.targets later queries all binaries to copy and package from the individual role projects by asking the dependent project to "build" a target BuiltProjectOutputGroup from Microsoft.Common.CurrentVersion.targets:

<Target
    Name="BuiltProjectOutputGroupDependencies"
    DependsOnTargets="$(BuiltProjectOutputGroupDependenciesDependsOn)"
    Returns="@(BuiltProjectOutputGroupDependency)">
  <ItemGroup>
    <BuiltProjectOutputGroupDependency
                Include="@(ReferencePath->'%(FullPath)');
                         @(ReferenceDependencyPaths->'%(FullPath)');
                         @(NativeReferenceFile->'%(FullPath)');
                         @(_DeploymentLooseManifestFile->'%(FullPath)');
                         @(ResolvedIsolatedComModules->'%(FullPath)')"/>
  </ItemGroup>
</Target>

As you can see, this includes @(ReferencePath) in the result, not @(ReferenceCopyLocalPaths). Therefore it will provide the reference assemblies only and result in a broken final package with incorrect dependencies.

Solution

Apparently BuiltProjectOutputGroupDependencies should not be used in these cases and ReferenceCopyLocalPathsOutputGroup should be used instead. See https://github.com/microsoft/msbuild/issues/3069

So the fix should happen in Microsoft.WindowsAzure.targets, which is unfortunately not a part of this open source Azure SDK for .NET project here on GitHub. It comes from what's called "Azure Cloud Services build tools" component of Visual Studio and developed internally at Microsoft (DevDiv/AT-OneSdk).

I have no idea how to raise a bug against that, developercommunity.visualstudio.com maybe?

Workaround

Adding this at the end of the inner .csproj files (for the individual worker roles) fixed the issue for me. A side effect is that it resulted in including some auxiliary .xml files too, maybe it could be made less dumb.

  <!-- When the final Azure Cloud Service project is retrieving all binaries to include in the deployment package,
       it mistakenly queries the BuiltProjectOutputGroupDependencies target in this project, which doesn't contain
       the full runtime assemblies from NuGet packages that have both the lib/ and ref/ directories. So override
       the target here to include ReferenceCopyLocalPaths as a workaround.
       See also https://github.com/Azure/azure-sdk-for-net/issues/3699 -->
  <Target
      Name="BuiltProjectOutputGroupDependencies"
      DependsOnTargets="$(BuiltProjectOutputGroupDependenciesDependsOn)"
      Returns="@(BuiltProjectOutputGroupDependency)">
    <ItemGroup>
      <BuiltProjectOutputGroupDependency Include="@(ReferenceCopyLocalPaths->'%(FullPath)');
                                                  @(ReferenceDependencyPaths->'%(FullPath)');
                                                  @(NativeReferenceFile->'%(FullPath)');
                                                  @(_DeploymentLooseManifestFile->'%(FullPath)');
                                                  @(ResolvedIsolatedComModules->'%(FullPath)')"/>
    </ItemGroup>
  </Target

[dejanberic]

@yirkha Great solution! I have one more problem. Do you know how to copy pdb files of all referenced projects so those can be included in Cloud Service deployment?
Right now only one pdb file (the worker role project) is included.
This is preventing me from getting accurate stack trace information when exceptions happen.

[yirkha]

@dejanberic Not really, sorry. Maybe there is some other predefined property to use for that. But all I learned about how this works was from running MS Build with verbose logs, reading those, inspecting the default build target definitions bundled with VS etc. – and I'll gladly leave that to someone who needs it. Good luck!