Msbuild: Help understanding some batching behavior

[nguerrera]

Debugging https://github.com/dotnet/sdk/issues/3257 led me to this curiosity, which I think is showing a gap in my understanding of some msbuild semantics.

Steps to reproduce

test.proj

<Project>
  <ItemGroup>
     <RuntimePackAsset
        Include="c:\a\fr\foo.resources.dll"
        DestinationSubDirectory="fr\"
        DestinationSubPath="fr\fr.resources.dll" 
        />

    <RuntimePackAsset 
        Include="c:\a\bar.dll"
        DestinationSubPath="bar.dll" 
        />

    <ReferenceCopyLocalPaths Include="@(RuntimePackAsset)" />
  </ItemGroup>

  <Target Name="Repro">
    <ItemGroup>
      <_ResolvedCopyLocalPublishAssets Include="@(ReferenceCopyLocalPaths)"
                                       Exclude="@(RuntimePackAsset)"
                                       Condition="'$(PublishReferencesDocumentationFiles)' == 'true' or '%(Extension)' != '.xml'">
        <DestinationSubPath>%(ReferenceCopyLocalPaths.DestinationSubDirectory)%(Filename)%(Extension)</DestinationSubPath>
      </_ResolvedCopyLocalPublishAssets>
    </ItemGroup>

    <Message Text="@(_ResolvedCopyLocalPublishAssets)" Importance="High" />
  </Target>
</Project>

Directory contents:

/
- test.proj
````

#### Command line

msbuild test.proj /m /v:m /nologo

### Expected  behavior

I'm guessing this isn't a bug, but my naive expectation was that since Include and Exclude have the same items, nothing is printed.

### Actual behavior

msbuild test.proj /m /v:m /nologo
c:a\fr\foo.resources.dll

This patch fixes it, but I don't yet grasp *why*:

``` diff
     <ItemGroup>
       <_ResolvedCopyLocalPublishAssets Include="@(ReferenceCopyLocalPaths)"
                                        Exclude="@(RuntimePackAsset)"
-                                       Condition="'$(PublishReferencesDocumentationFiles)' == 'true' or '%(Extension)' != '.xml'">
-        <DestinationSubPath>%(ReferenceCopyLocalPaths.DestinationSubDirectory)%(Filename)%(Extension)</DestinationSubPath>
+                                       Condition="'$(PublishReferencesDocumentationFiles)' == 'true' or '%(ReferenceCopyLocalPaths.Extension)' != '.xml'">
+        <DestinationSubPath>%(ReferenceCopyLocalPaths.DestinationSubDirectory)%(ReferenceCopyLocalPaths.Filename)%(ReferenceCopyLocalPaths.Extension)</DestinationSubPath>
       </_ResolvedCopyLocalPublishAssets>
     </ItemGroup>

Environment data

Microsoft (R) Build Engine version 16.2.0-preview-19274-03+103f944e0 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

16.200.19.27403

[nguerrera]

@rainersigwald

[rainersigwald]

This is partially a result of the _intensely_ confusing behavior that the MSPress MSBuild book calls "multi-batching". This is somewhat documented under no particularly clear name at https://docs.microsoft.com/en-us/visualstudio/msbuild/item-metadata-in-task-batching?view=vs-2019#divide-several-item-lists-into-batches.

Basically, if you have a single batch-eligible thing (here let's just say task invocation; pretty sure this works for target batching too) with multiple item lists, a bare metadata reference like %(Filename) applies to _all lists simultaneously_.

In this case, the engine decided to bucket on:

1. %(Extension)
2. %(ReferenceCopyLocalPaths.DestinationSubDirectory)
3. %(Filename)

All the extensions match, so that doesn't produce new buckets. So the buckets are for each _unique combination_ of %(ReferenceCopyLocalPaths.DestinationSubDirectory) + %(Filename):

1. fr\ + foo.resources (from the foo in ReferenceCopyLocalPaths)
2. empty + bar (from both ReferenceCopyLocalPaths and RuntimePackAsset since neither has DestinationSubDirectory and both have a bar item)
3. empty + foo.resources (from the RuntimePackAsset item: there's no match for %(ReferenceCopyLocalPaths.DestinationSubDirectory), so it MSBuild-ily expands to the empty string)

Adding this line to your example project may help:

    <Message Text="A batch. ReferenceCopyLocalPaths = @(ReferenceCopyLocalPaths), RuntimePackAsset = @(RuntimePackAsset), ReferenceCopyLocalPaths.DestinationSubDirectory = %(ReferenceCopyLocalPaths.DestinationSubDirectory) Filename = %(Filename) Extension = %(Extension)" Importance="High" />

Microsoft (R) Build Engine version 16.2.0-preview-19274-03+103f944e0 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

  A batch. ReferenceCopyLocalPaths = c:\a\fr\foo.resources.dll, RuntimePackAsset = , ReferenceCopyLocalPaths.DestinationSubDirectory = fr\ Filename = foo.resources Extension = .dll
  A batch. ReferenceCopyLocalPaths = c:\a\bar.dll, RuntimePackAsset = c:\a\bar.dll, ReferenceCopyLocalPaths.DestinationSubDirectory =  Filename = bar Extension = .dll
  A batch. ReferenceCopyLocalPaths = , RuntimePackAsset = c:\a\fr\foo.resources.dll, ReferenceCopyLocalPaths.DestinationSubDirectory =  Filename = foo.resources Extension = .dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:00.65

Gory details in and around
https://github.com/microsoft/msbuild/blob/28ca8b4eaac0862aa08ccad8f0608af6c1957068/src/Build/BackEnd/Components/RequestBuilder/BatchingEngine.cs#L74-L79

[danmoseley]

https://github.com/microsoft/msbuild/blob/28ca8b4eaac0862aa08ccad8f0608af6c1957068/src/Build/BackEnd/Components/RequestBuilder/BatchingEngine.cs#L20-L68

[jzabroski]

This is the best MSBuild thread ever. Thank you, @rainersigwald .

[danmoseley]

I had to reverse engineer the code to write that comment above, my goal was that nobody else would have to 😄

[jzabroski]

@danmosemsft I did appreciate your comments, too!

However, the broader problem is there are very few idioms in MSBuild, and searching StackOverflow was not very fruitful. I spent a lot of time the past week improving our build scripts across all of our organization, and its surprising to me how difficult it is to do some basic boolean algebra set computations in MSBuild. About ~10 years ago I read the MSPress MSBuild book, but frankly completely forgot all these idiosyncratic details.

The best tutorial on MSBuild I found online was this random github repository code (apologize that the repo contains a curse word - hope a bot doesn't auto-ban me): https://github.com/Enzogord/fucking_workable_monodevelop/blob/c17606619baf24d0777c0436de13982447e5fc1d/main/tests/test-projects/msbuild-tests/transforms.csproj - based on these bizarre examples, I'm not sure MSBuild has a future in my build process.

[danmoseley]

Hi @jzabroski -- note there were two books - the MSPress book and the Trickery book? The latter one had all the crazy examples.

I encourage you to use whatever build tool best fits your purposes 😃

[jzabroski]

@danmosemsft I did not realize you wrote the forward to an MSBuild book. I was unaware of the Trickery book. I suppose native speakers of MSBuild may have this book and speak some of these tricks as idioms, but, if I can raise the bar on my original comment, they're not "widely known idioms". May be since you have a relationship with the author, you could suggest open sourcing the book on GitHub? Thank you.

[danmoseley]

@jzabroski I have not spoken to him in 10 years unfortunately. You could try reaching out here possibly? (found with Bing..) https://stackoverflow.com/users/610674/brian-kretzler