Msbuild: Need a way to indicate dependencies on other projects

[KirillOsenkov]

Currently MSBuild uses <ProjectReference> items to indicate dependencies between projects and there's logic in the common targets to ensure that the build order is correct and ProjectReferences are respected.

However it is my current understanding that this is specific to typical C# projects, and there's no MSBuild generic way to indicate a project's dependency on other projects. There also doesn't seem to be a way to hint to the solution build in which order to build the projects, apart from the ProjectReference mechanism.

If so, we need to design a generic way to indicate dependency across projects, to minimize work and initially building the projects in the right order vs. imperatively triggering a build for dependencies once we encounter the dependency.

[KirillOsenkov]

Example investigation:
https://github.com/Microsoft/Git-Credential-Manager-for-Windows/pull/529

Example fix 1:
https://github.com/Microsoft/Git-Credential-Manager-for-Windows/pull/529/commits/83f2e7aefe23c83091eab0a5d6a41b9de7ad4e1e
Here we imperatively trigger the build for dependencies instead of declaratively specifying them.

Example fix 2:
https://github.com/Microsoft/Git-Credential-Manager-for-Windows/pull/529/commits/bfdf2b549063880c0a1f94712bedd14c9d79f949
Here we need to make sure we enumerate the directory produced by our dependencies AFTER the dependencies have been built imperatively. If we enumerate the directory normally (during evaluation), it will run before the dependencies have been built. If we had a declarative mechanism to specify project dependencies the evaluation of the dependent project would start after the dependencies have been built, and so everything would work as expected, no surprises.

[rainersigwald]

Solutions allow specifying build order dependencies, which MSBuild respects. And ProjectReference works for vcxproj too, at least . . . what have you seen to make you think otherwise?

[whoisj]

Well, we've seen that solutions only work if the VS toolchain is relied upon. If a dev makes a project from scratch, there's very little rolling to assist with dependencies and easy way to debug / understand the problem.

Additionally, most code bases are too large for solutions

[chrispat]

@rainersigwald we regularly get complaints of project build order in solutions not being respected by msbuild.

[rainersigwald]

@chrisrpatterson Send them to us, please. Any such problem is a bug in MSBuild.

[whoisj]

@rainersigwald Hi, I was sent here by @KirillOsenkov and @chrisrpatterson 😏

So the net issue was that I had a custom script that was being replaced with MSBuild. The script did the following:

1. Use Pandoc to convert .md into .html for help files.
2. Collect the output of a bunch of related projects.
3. Combine the outputs into an installer using InnoSetup.
4. Generate a ZIP xcopy-able alternative to the installer.

The primary motivations for moving away from the script were:

1. The script was fairly opaque.
2. Binary signing wasn't something we could do using existing tool set from a script.
3. Custom scripts aren't supposed to be easier/better/more reliable than build systems.

Since I was replacing a .cmd BATCH script, I started with an empty .proj file and built it up from there. I believed that should be completely doable, and it mostly worked. Dependencies and therefore build ordering didn't.

I'd attempted to establish dependencies via stack of ProjectReference nodes like the one below

    <ProjectReference Include="..\Cli-Askpass\Cli-Askpass.csproj">
      <Project>{19770407-B33A-4EBE-92B8-04C93F43CAE0}</Project>
      <Name>Cli-Askpass</Name>
    </ProjectReference>

But they were seemingly ignored, and every build was failing. The solution (found by @KirillOsenkov) was <MSBuild Projects="@(ProjectReference)" Targets="Build" />.

TL:DR:

The outcome was that MSBuild only cared about project dependencies when invoked from Visual Studio. Why? When building from the "Developer Command Prompt for VS 2017" and/or from AppVeyor, MSBuild seemed to completely ignore the dependencies, with absolutely no indication as to why.

[rainersigwald]

The outcome was that MSBuild only cared about project dependencies when invoked from Visual Studio. Why?

Because the solution lies to Visual Studio and claims that this new project is a normal .csproj solution line: https://github.com/Microsoft/Git-Credential-Manager-for-Windows/blob/b77f58639f18a6525f07967ef2006ff1a4b9f9bb/GitCredentialManager.sln#L49

That confused VS into thinking that the ProjectReferences would do something, because it has a parallel-but-inaccurate understanding of the nature of references. MSBuild sees that nothing consumes @(ProjectReference) and does nothing with it.

This is a bug in VS, though probably not one worth fixing.

I'd love to get your feedback on how we can make the documentation on how to implement ProjectReferences more obvious!

[whoisj]

I'd love to get your feedback on how we can make the documentation on how to implement ProjectReferences more obvious!

Firstly, I would have love to have know that docs on GitHub even existed. I trolled Bing and Google and came up mostly empty. In fact, the only reason I used ProjectReferences and I "lied" to Visual Studio is that I was attempting to make anything work using one part first-principles and one part hack-it.

Secondly, does msbuild even have a help command? I mean a deep diving or, at least, redirecting to official documentation command?

[rainersigwald]

What kinds of search terms were you using? Maybe we can SEO a bit.

Secondly, does msbuild even have a help command? I mean a deep diving or, at least, redirecting to official documentation command?

No, and we should. Filed #2892 as the minimum.

[whoisj]

What kinds of search terms were you using? Maybe we can SEO a bit.

"msbuild project ordering", "msbuld project dependencies", "msbuild choosing last project built"

There were lots of stackoverflow.com threads of people with questions about the same topic in the results, none of which had sufficient answers. :unamused:

There were lots of examples of how to order Targets, and even conflicting reports of how to use AfterTargets and BeforeTargets, and let's be honest the "wording" of those names can be looked at two different ways (read: very confusing). Still, nothing to unblock me - except on blog about adding ProjectReferences and using that .sln hack which mostly unblocked me.

Remember, I was adamant that the final solution did not require Visual Studio to be installed or even available. I work on Visual Studio for a living, and I love Visual Studio, but I am/was not willing to accept that I needed an IDE to build source code. 😏

[japj]

If I may add to this discussion, we have a software product that consists out of 30+ solutions (containing a mix of c++/c#/java/etc) and we created our own msbuild project files that orchestrate building these “subsystems” solutions in the right order (using a similar dependency mechanism as project references does in c#).
It would be really appreciated it if there is a better/official/documented way of managing and building dependencies of (also non c#) projects.

[breyed]

@rainersigwald When considering whether MSBuild should lean on solution-based dependencies, it's worth keeping in mind that MSBuild behavior differs from that of Visual Studio when working with dependencies specified in a solution. In particular, if a solution specifies a dependency with ProjectSection(ProjectDependencies) = postProject and the project being depended on has an item with <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>, MSBuild will include that item in the dependent project's output, whereas Visual Studio will not.

This behavior caused surprise results on the automated build for us when working with a project that was a dependency because it needed to build so that MSBuild would use it as a custom task. To avoid the unwanted files in the dependent project's output, we ended up building the dependency using a MSBuild task inside our custom target. In this way, MSBuild does a nice job of supporting arbitrary dependencies.