Runtime: Build errors: install_force has not been created.

[sandreenko]

The full error:

C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(231,5): error MSB8065: Custom build for item "F:\workspace\_work\1\s\artifacts\obj\native\netcoreapp5.0-Windows_NT-Release-x86\CMakeFiles\01d9c98fded5d1e41907661a010b2e75\INSTALL_force.rule" succeeded, but specified output "f:\workspace\_work\1\s\artifacts\obj\native\netcoreapp5.0-windows_nt-release-x86\cmakefiles\install_force" has not been created. This may cause incremental build to work incorrectly. [F:\workspace\_work\1\s\artifacts\obj\native\netcoreapp5.0-Windows_NT-Release-x86\install.vcxproj] [F:\workspace\_work\1\s\src\libraries\Native\build-native.proj]
##[error]C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(231,5): error MSB8065: (NETCORE_ENGINEERING_TELEMETRY=Build) Custom build for item "F:\workspace\_work\1\s\artifacts\obj\native\netcoreapp5.0-Windows_NT-Release-x86\CMakeFiles\01d9c98fded5d1e41907661a010b2e75\INSTALL_force.rule" succeeded, but specified output "f:\workspace\_work\1\s\artifacts\obj\native\netcoreapp5.0-windows_nt-release-x86\cmakefiles\install_force" has not been created. This may cause incremental build to work incorrectly. [F:\workspace\_work\1\s\artifacts\obj\native\netcoreapp5.0-Windows_NT-Release-x86\install.vcxproj]

Saw this error in a several runs today:
https://dev.azure.com/dnceng/public/_build/results?buildId=537120
https://dev.azure.com/dnceng/public/_build/results?buildId=537213

The symptoms match this old issue #32483, could cmake be downgraded on these machines?

PTAL @dotnet/runtime-infrastructure

[jaredpar]

Changed label to blocking-official-build. Generally the blocking-clean-ci is meant for issues that block the core PR / CI loop and this isn't affecting that. It is blocking our optional CI runs though. We should probably add a new label for that.

The reason for having a clean separatation of labels here is about prioritization and tracking.

[jaredpar]

Added blocking-clean-ci-optional to track this type of issue

[jaredpar]

Actually this is happening in our CI jobs right now. Consider this build https://dev.azure.com/dnceng/public/_build/results?buildId=536905

      warning Code C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(231,5): warning MSB8065: (NETCORE_ENGINEERING_TELEMETRY=Build) Custom build for item "F:\workspace\_work\1\s\artifacts\obj\native\netcoreapp5.0-Windows_NT-Release-x86\CMakeFiles\01d9c98fded5d1e41907661a010b2e75\INSTALL_force.rule" succeeded, but specified output "f:\workspace\_work\1\s\artifacts\obj\native\netcoreapp5.0-windows_nt-release-x86\cmakefiles\install_force" has not been created. This may cause incremental build to work incorrectly. [F:\workspace\_work\1\s\artifacts\obj\native\netcoreapp5.0-Windows_NT-Release-x86\install.vcxproj]

That seems to be the root of the problem here. We've let this be a warning in PR have it as an error elsewhere. @ViktorHofer @jashook why do we do this? Seems like a significant hole.

[jkoritzinsky]

@safern looks like you might have missed another hole in your recent change.

[jaredpar]

@jkoritzinsky at the moment @safern is OOF. Can you go about flipping this back to error here? This is blocking a lot of changes at the moment.

[jkoritzinsky]

We can't flip this back to error without blocking all CI. The current workaround is to mirror #32853 for all of the optional CI runs. (#32853 fixes this for the official build). I can go do that, but I want to wait for #32853 to get merged first so we can decide how we want to handle this specifically since there's some discussion about what fix we want to do.

[jaredpar]

@jkoritzinsky

We can't flip this back to error without blocking all CI.

Why can't we flip it in a PR and resolve the breaks?

[jkoritzinsky]

The errors are caused by the new version of VS causing the warning but the CMake that ships with VS not having the fix for the VS warning (pre-CMake 3.15.5). Since we have warnAsError set to true, this causes the failures. The VS preview pool has a VS that ships with a new enough CMake that this isn't a problem.

[jaredpar]

@jkoritzinsky

I find your comment confusing. Can you clarify? We own all the infra here so what pieces need to change to make this happen.

I want to make sure we have the appropriate priority here. Our official builds are 100% blocked right now. The M2's have commented that official builds must be our highest priority item here. We need to resolve this.

[jkoritzinsky]

Merging https://github.com/dotnet/runtime/pull/32853 will fix official builds.

The Windows VS machines don't actually have the correct version of CMake on them. The warning we're seeing was introduced during the VS 16.x version cycle. CMake fixed the warning in the CMake generated scripts in 3.15.5. The version of VS on our build machines is > "version which introduced detecting and reporting the warning" but bundles a CMake < "version which fixes the new VS warning".

We initially fixed this by disabling warnAsError on PR, CI, and official builds in #31909 and #31890 since that was easier than getting a new enough version of CMake globally installed on the build pool.

The VS Preview build pool now has VS versions which bundle a CMake >= "version which fixes the new VS warning". So, the libraries build moved to the preview pool in #32626 for PR and CI, but accidentally didn't move to the new pool for the official build. PR #32853 moves the libraries build in the official build to the preview pool.

To fix the optional ci legs, they should also move to the preview pool.

The reason I didn't want to make the change for the optional ci legs immediately is that there's been some discussion in #32853 about if we want to use different build pools for different legs.

[jaredpar]

@jkoritzinsky thanks for the explanation.

So how do we control the VS that is used on our build pool machines? Given the sensitivity to the native tooling here it seems that we need to have fairly fine grained control over what version of the tools are installed. Or rather I would expect that when the build pool used by dotnet/runtime is updated to a new version of VS there is a test run done first to validate that our builds remain good.

[jkoritzinsky]

As far as I know, we don't directly control it. Someone on the core-eng team does. @MattGal any idea how we can get more validation before build machines are updated on us across the board?

[safern]

I’ve just updated my PR https://github.com/dotnet/runtime/pull/32853 to disable the warning in the native build which help us not depend on a specific version of cmake for the build per @jkotas suggestion.

[MattGal]

As far as I know, we don't directly control it. Someone on the core-eng team does. @MattGal any idea how we can get more validation before build machines are updated on us across the board?

Simply by inserting "-Int" into the pool provider name (i.e. "`NetCoreInternal-Int-Pool" or "NetCorePublic-Int-Pool", accordingly) in the agent pool name in your yaml will build using machines in the preproduction environment.

The problem here will be that your pipelines use dozens of machines at once and we can't afford to run 4-core VMs at that scale in our staging environment (nor do they have such a quota) so a build pipeline like this would be very, very slow to complete. Further, given the runtime team is not the only team, if we start doing such a test pass for one specific team it's hard to imagine there won't be others asking for the same; it does not scale.

This sort of problem is absolutely not constrained to the .NET Core engineering team's build machines either; users get zero warning when the VS 2019 setup on a hosted build agent changes and never will. We do strive to make our rollouts as painless as possible, but the problem in this case (an interaction between VS 2019 16.4 and the CMake version that a clean install of it indirectly installs) is really best solved by bootstrapping in the version of CMake used. Even if the copy of cmake you randomly find from VS on a machine works, it's more likely to have a subtle breaking change than a locked-in, bootstrapped-with-the-build version.

This particular scenario was made more complex because we also have other customers who absolutely needed 16.4 (ask folks on https://github.com/dotnet/aspnetcore/issues/18019 for details) and maintaining our machines with the latest public GA of VS 2019 has been the policy since the beginning; otherwise rolling back to the older 16.3 image would have been a potential workaround.

[jkotas]

really best solved by bootstrapping in the version of CMake used

I disagree that bootstrapping is going to solve these problems. When the version of the tools that you bootstrap is incompatible with the silent compiler update, there is still going to be a sudden build break and it is going to take some time for folks to get to the bottom of it and fix it.

[markwilkie]

As @jkotas has said many times before, managing our underlying toolsets in a sane way is important, and not necessarily straightforward to do. Unfortunately the long term work is not currently funded due to other "10's", HOWEVER....

@mmitche and I spoke with @shawnro and we agreed that we (the infra team) should do more to proactively test by building key repos when updating toolsets (including Arcade). @jaredpar has been a big proponent of this, and we're getting that effort going right away.