Sdk: Multi-proc build of a .sln file is 'The process cannot access the file because it is being used by another process.'

[eerhardt]

In the CLI repo, we have a Test.sln file which we dotnet build. However, we are seeing random failures that look like the following:

05:25:35.797     1>CSC : error CS2012: Cannot open 'D:\j\workspace\debug_windows---54dcb9ec\test\Msbuild.Tests.Utilities\obj\Debug\netcoreapp1.0\Msbuild.Tests.Utilities.dll' for writing -- 'The process cannot access the file 'D:\j\workspace\debug_windows---54dcb9ec\test\Msbuild.Tests.Utilities\obj\Debug\netcoreapp1.0\Msbuild.Tests.Utilities.dll' because it is being used by another process.' [D:\j\workspace\debug_windows---54dcb9ec\test\Msbuild.Tests.Utilities\Msbuild.Tests.Utilities.csproj] [D:\j\workspace\debug_windows---54dcb9ec\build.proj]

See https://ci.dot.net/job/dotnet_cli/job/rel_1.0.0/job/debug_windows_nt_x86_prtest/810/console

To repro:

1. On Windows x64 machine, clone the CLI repo and checkout https://github.com/dotnet/cli/pull/5465/commits/f6b0196b0f8a7342d4b5fe5709568af9e3ede665.
2. Then remove the line https://github.com/dotnet/cli/pull/5465/commits/f6b0196b0f8a7342d4b5fe5709568af9e3ede665#diff-70eb062c7648e65aa7aff25e924d014bR84 which is working around this problem by turning off multi-proc build.
3. build.cmd

An easier repro once you have built one time is:

1. F:\cli>git clean test -xdf
2. F:\cli>.dotnet_stage0\x64\dotnet.exe restore test\Microsoft.DotNet.Cli.Tests.sln
3. F:\cli>.dotnet_stage0\x64\dotnet.exe build test\Microsoft.DotNet.Cli.Tests.sln

F:\cli>.dotnet_stage0\x64\dotnet.exe build test\Microsoft.DotNet.Cli.Tests.sln
Microsoft (R) Build Engine version 15.1.523.56541
Copyright (C) Microsoft Corporation. All rights reserved.

F:\cli\.dotnet_stage0\x64\sdk\2.0.0-alpha-004616\Sdks\Microsoft.NET.Sdk\build\Microsoft.NET.Sdk.DefaultItems.targets(101,5): warning : A PackageReference for 'NETStandard.Library' was included in your project. This package is implicitly referenced by the .NET SDK and you do not typically need to reference it from your project. For more information, see https://aka.ms/sdkimplicitrefs [F:\cli\test\Microsoft.DotNet.Tools.Tests.Utilities\Microsoft.DotNet.Tools.Tests.Utilities.csproj]
F:\cli\.dotnet_stage0\x64\sdk\2.0.0-alpha-004616\Sdks\Microsoft.NET.Sdk\build\Microsoft.NET.Sdk.DefaultItems.targets(101,5): warning : A PackageReference for 'NETStandard.Library' was included in your project. This package is implicitly referenced by the .NET SDK and you do not typically need to reference it from your project. For more information, see https://aka.ms/sdkimplicitrefs [F:\cli\test\Microsoft.DotNet.Tools.Tests.Utilities\Microsoft.DotNet.Tools.Tests.Utilities.csproj]
  ArgumentsReflector -> F:\cli\test\ArgumentsReflector\bin\Debug\netcoreapp2.0\ArgumentsReflector.dll
  Microsoft.DotNet.Cli.Utils -> F:\cli\src\Microsoft.DotNet.Cli.Utils\bin\Debug\netstandard1.5\Microsoft.DotNet.Cli.Utils.dll
  Microsoft.DotNet.TestFramework -> F:\cli\src\Microsoft.DotNet.TestFramework\bin\Debug\netstandard1.5\Microsoft.DotNet.TestFramework.dll
  Microsoft.DotNet.InternalAbstractions -> F:\cli\src\Microsoft.DotNet.InternalAbstractions\bin\Debug\netstandard1.3\Microsoft.DotNet.InternalAbstractions.dll
CSC : error CS2012: Cannot open 'F:\cli\test\Microsoft.DotNet.Tools.Tests.Utilities\obj\Debug\netstandard1.5\Microsoft.DotNet.Tools.Tests.Utilities.dll' for writing -- 'The process cannot access the file 'F:\cli\test\Microsoft.DotNet.Tools.Tests.Utilities\obj\Debug\netstandard1.5\Microsoft.DotNet.Tools.Tests.Utilities.dll' because it is being used by another process.' [F:\cli\test\Microsoft.DotNet.Tools.Tests.Utilities\Microsoft.DotNet.Tools.Tests.Utilities.csproj]

[eerhardt]

@srivatsn @dsplaisted @nguerrera

This seems like an issue we'd want to fix for "1.0".

[natidea]

Looks like a reoccurrence of https://github.com/dotnet/sdk/issues/475 . I'll see if I can get a repro

[eerhardt]

We see this in our CLI Jenkins runs. Here's another:
https://ci.dot.net/job/dotnet_cli/job/rel_1.0.0/job/debug_windows_nt_x86_prtest/865/

@natidea - let me know if you need help repro'ing it. On my machine it happens pretty consistently.

[TheRealPiotrP]

CLI Jenkins CI is now hitting this issues multiple times on each PR. Likely running dotnet build test/test.sln in a loop will show a repro pretty quickly, with a 'git clean -xfd' of that folder on each iteration.

[TheRealPiotrP]

FYI, we're trying to disable multi-proc builds of the affected SLN in CLI until the base issue gets resolved. Once that merge goes through, you will want to delete this line when attempting to repro:
https://github.com/dotnet/cli/pull/5490/commits/540c2b009115e911f21f4486eab361ac090755b3#diff-70eb062c7648e65aa7aff25e924d014bR86

[natidea]

I have a repro that I am looking at this weekend. Also /cc @rainersigwald @nguerrera

[nguerrera]

Do we have any insight on what source changes were made when this started happening? Was it an upgrade of stage0 sdk? From what build to what build?

[eerhardt]

@piotrpMSFT just changed our build script this week to build the test sln file instead of globbing *.csproj under the test directory and building each project separately.

https://github.com/piotrpMSFT/cli/commit/0facfa29440192940cc125e54ecd8bc9881f7dea

The error doesn't happen consistently, so it made it through our CI, but then started cropping up.

[nguerrera]

I didn't see how the globs were processed before in that change. Were they batched serially or passed to msbuild all at once? Trying to understand if true parallelism was really only first introduced with that change or there really is a difference in sln build case

[eerhardt]

Here's how the test projects were found:

https://github.com/dotnet/cli/blob/1dfee9ead8fcc63b68baba801a88fd2f9b7d251b/build/test/TestProjects.targets#L7-L31

      <PreTestProjects Include="test$(PathSeparator)*$(PathSeparator)*.csproj;" />

      <TestProjects Include="%(PreTestProjects.Fullpath)"
                    Exclude="@(TestProjectsToExclude)" />

And then here's how the tests were built:

https://github.com/dotnet/cli/pull/5403/files#diff-70eb062c7648e65aa7aff25e924d014bL77

  <Target Name="BuildTests"
          DependsOnTargets="RestoreTests;
                             SetupTestProjectData;"
           Inputs="%(TestProjects.BuildInputs)"
           Outputs="%(TestProjects.BuildOutput)">
     <DotNetBuild  Configuration="$(Configuration)"
                   Framework="%(TestProjects.Framework)"
                   ToolPath="$(Stage0Directory)"
                   ProjectPath="%(TestProjects.ProjectPath)" />
   </Target>

[nguerrera]

Ok they were serialized before

[TheRealPiotrP]

Yes, they were serialized. We originally tried parallel build but hit issues due to shared dependencies. Tried sln build as a dogfooding and code cleanup exercise. Reducing build nodes to 1 seems to have mitigated but it should not be required...

[eerhardt]

I have a /v:diag trace of the error. For any MSFT employee: \\scratch2\scratch\eerhardt\multi-proc-build.log. For anyone without access, but wants to look at it, I can upload it publicly if you give me a place that takes 250 MB files.

The interesting things to me are these two entries:

10:20:51.097  4:16>CoreCompile: (TargetId:274)
                     Task Parameter:NoStandardLib=True (TaskId:145)
                     Task Parameter:Optimize=False (TaskId:145)
                     Task Parameter:Deterministic=True (TaskId:145)
                     Task Parameter:OutputAssembly=obj\Debug\netstandard1.5\Microsoft.DotNet.Tools.Tests.Utilities.dll (TaskId:145)
                     Task Parameter:Prefer32Bit=False (TaskId:145)

and

10:20:51.132  4:22>Target "CoreCompile: (TargetId:131)" in file "F:\cli\.dotnet_stage0\x64\sdk\2.0.0-alpha-004622\Roslyn\Microsoft.CSharp.Core.targets" from project "F:\cli\test\Microsoft.DotNet.Tools.Tests.Utilities\Microsoft.DotNet.Tools.Tests.Utilities.csproj" (target "Compile" depends on it):
                   Building target "CoreCompile" completely.
                   Output file "obj\Debug\netstandard1.5\Microsoft.DotNet.Tools.Tests.Utilities.dll" does not exist.
                   Set Property: NoWarn=1701;1702;1705;1701;1702
                   Set Property: UseSharedCompilation=true
                   Using "Csc" task from assembly "F:\cli\.dotnet_stage0\x64\sdk\2.0.0-alpha-004622\Roslyn\Microsoft.Build.Tasks.CodeAnalysis.dll".
                   Task "Csc" (TaskId:70)
                     Task Parameter:AllowUnsafeBlocks=False (TaskId:70)
                     Task Parameter:DebugType=portable (TaskId:70)
                     Task Parameter:DefineConstants=TRACE;NETSTANDARD1_5;DEBUG (TaskId:70)
                     Task Parameter:DisabledWarnings=1701;1702;1705;1701;1702 (TaskId:70)
                     Task Parameter:EmitDebugInformation=True (TaskId:70)
                     Task Parameter:ErrorReport=prompt (TaskId:70)
                     Task Parameter:FileAlignment=512 (TaskId:70)
                     Task Parameter:KeyFile=../../tools/Key.snk (TaskId:70)
                     Task Parameter:NoConfig=True (TaskId:70)
                     Task Parameter:NoLogo=True (TaskId:70)
                     Task Parameter:NoStandardLib=True (TaskId:70)
                     Task Parameter:Optimize=False (TaskId:70)
                     Task Parameter:Deterministic=True (TaskId:70)
                     Task Parameter:OutputAssembly=obj\Debug\netstandard1.5\Microsoft.DotNet.Tools.Tests.Utilities.dll (TaskId:70)

and then that last task is failing:

                     F:\cli\.dotnet_stage0\x64\sdk\2.0.0-alpha-004622\Roslyn\RunCsc.cmd /noconfig /unsafe- /nowarn:1701,1702,1705,1701,1702 /nostdlib+ /errorreport:prompt /warn:4 /define:TRACE;NETSTANDARD1_5;DEBUG /reference:C:\Users\eerhardt\.nuget\packages\fluentassertions\4.0.0\lib\dotnet\FluentAssertions.Core.dll /reference:C:\Users\eerhardt\.nuget\packages\fluentassertions\4.0.0\lib\dotnet\FluentAssertions.dll /reference:C:\Users\eerhardt\.nuget\packages\microsoft.build\15.1.0-preview-000523-01\lib\netstandard1.5\Microsoft.Build.dll /reference:C:\Users\eerhardt\.nuget\packages\microsoft.build.framework\15.1.0-preview-000523-01\lib\netstandard1.3\Microsoft.Build.Framework.dll /reference:C:\Users\eerhardt\.nuget\packages\microsoft.build.utilities.core\15.1.0-preview-000523-01\lib\netstandard1.3\Microsoft.Build.Utilities.Core.dll /reference:C:\Users\eerhardt\.nuget\packages\microsoft.csharp\4.0.1\ref\netstandard1.0\Microsoft.CSharp.dll /reference:F:\cli\src\Microsoft.DotNet.Cli.Utils\bin\Debug\netstandard1.5\Microsoft.DotNet.Cli.Utils.dll /reference:F:\cli\src\Microsoft.DotNet.InternalAbstractions\bin\Debug\netstandard1.3\Microsoft.DotNet.InternalAbstractions.dll /reference:C:\Users\eerhardt\.nuget\packages\microsoft.dotnet.platformabstractions\1.0.1-beta-000933\lib\netstandard1.3\Microsoft.DotNet.PlatformAbstractions.dll /reference:F:\cli\src\Microsoft.DotNet.TestFramework\bin\Debug\netstandard1.5\Microsoft.DotNet.TestFramework.dll /reference:C:\Users\eerhardt\.nuget\packages\microsoft.extensions.dependencymodel\1.0.1-beta-000933\lib\netstandard1.3\Microsoft.Extensions.DependencyModel.dll /reference:C:\Users\eerhardt\.nuget\packages\microsoft.win32.primitives\4.0.1\ref\netstandard1.3\Microsoft.Win32.Primitives.dll /reference:C:\Users\eerhardt\.nuget\packages\newtonsoft.json\9.0.1\lib\netstandard1.0\Newtonsoft.Json.dll /reference:C:\Users\eerhardt\.nuget\packages\nuget.common\4.0.0-rc3-2193\lib\netstandard1.3\NuGet.Common.dll /reference:C:\Users\eerhardt\.nuget\packages\nuget.configuration\4.0.0-rc3-2193\lib\netstandard1.3\NuGet.Configuration.dll /reference:C:\Users\eerhardt\.nuget\packages\nuget.dependencyresolver.core\4.0.0-rc3-2193\lib\netstandard1.3\NuGet.DependencyResolver.Core.dll /reference:C:\Users\eerhardt\.nuget\packages\nuget.frameworks\4.0.0-rc3-2193\lib\netstandard1.3\NuGet.Frameworks.dll /reference:C:\Users\eerhardt\.nuget\packages\nuget.librarymodel\4.0.0-rc3-2193\lib\netstandard1.3\NuGet.LibraryModel.dll /reference:C:\Users\eerhardt\.nuget\packages\nuget.packaging.core\4.0.0-rc3-2193\lib\netstandard1.3\NuGet.Packaging.Core.dll /reference:C:\Users\eerhardt\.nuget\packages\nuget.packaging.core.types\4.0.0-rc3-2193\lib\netstandard1.3\NuGet.Packaging.Core.Types.dll /reference:C:\Users\eerhardt\.nuget\packages\nuget.packaging\4.0.0-rc3-2193\lib\netstandard1.3\NuGet.Packaging.dll /reference:C:\Users\eerhardt\.nuget\packages\nuget.projectmodel\4.0.0-rc3-2193\lib\netstandard1.3\NuGet.ProjectModel.dll /reference:C:\Users\eerhardt\.nuget\packages\nuget.protocol.core.types\4.0.0-rc3-2193\lib\netstandard1.3\NuGet.Protocol.Core.Types.dll /reference:C:\Users\eerhardt\.nuget\packages\nuget.protocol.core.v3\4.0.0-rc3-2193\lib\netstandard1.3\NuGet.Protocol.Core.v3.dll /reference:C:\Users\eerhardt\.nuget\packages\nuget.repositories\4.0.0-rc3-2193\lib\netstandard1.3\NuGet.Repositories.dll /reference:C:\Users\eerhardt\.nuget\packages\nuget.runtimemodel\4.0.0-rc3-2193\lib\netstandard1.3\NuGet.RuntimeModel.dll /reference:C:\Users\eerhardt\.nuget\packages\nuget.versioning\4.0.0-rc3-2193\lib\netstandard1.0\NuGet.Versioning.dll /reference:C:\Users\eerhardt\.nuget\packages\system.appcontext\4.1.0\ref\netstandard1.3\System.AppContext.dll /reference:C:\Users\eerhardt\.nuget\packages\system.collections.concurrent\4.0.12\ref\netstandard1.3\System.Collections.Concurrent.dll /reference:C:\Users\eerhardt\.nuget\packages\system.collections\4.0.11\ref\netstandard1.3\System.Collections.dll /reference:C:\Users\eerhardt\.nuget\packages\system.collections.immutable\1.2.0\lib\netstandard1.0\System.Collections.Immutable.dll /reference:C:\Users\eerhardt\.nuget\packages\system.collections.nongeneric\4.0.1\ref\netstandard1.3\System.Collections.NonGeneric.dll /reference:C:\Users\eerhardt\.nuget\packages\system.console\4.0.0\ref\netstandard1.3\System.Console.dll /reference:C:\Users\eerhardt\.nuget\packages\system.diagnostics.contracts\4.0.1\ref\netstandard1.0\System.Diagnostics.Contracts.dll /reference:C:\Users\eerhardt\.nuget\packages\system.diagnostics.debug\4.0.11\ref\netstandard1.3\System.Diagnostics.Debug.dll /reference:C:\Users\eerhardt\.nuget\packages\system.diagnostics.fileversioninfo\4.0.0\ref\netstandard1.3\System.Diagnostics.FileVersionInfo.dll /reference:C:\Users\eerhardt\.nuget\packages\system.diagnostics.process\4.1.0\ref\netstandard1.4\System.Diagnostics.Process.dll /reference:C:\Users\eerhardt\.nuget\packages\system.diagnostics.tools\4.0.1\ref\netstandard1.0\System.Diagnostics.Tools.dll /reference:C:\Users\eerhardt\.nuget\packages\system.diagnostics.tracesource\4.0.0\ref\netstandard1.3\System.Diagnostics.TraceSource.dll /reference:C:\Users\eerhardt\.nuget\packages\system.diagnostics.tracing\4.1.0\ref\netstandard1.5\System.Diagnostics.Tracing.dll /reference:C:\Users\eerhardt\.nuget\packages\system.dynamic.runtime\4.0.11\ref\netstandard1.3\System.Dynamic.Runtime.dll /reference:C:\Users\eerhardt\.nuget\packages\system.globalization.calendars\4.0.1\ref\netstandard1.3\System.Globalization.Calendars.dll /reference:C:\Users\eerhardt\.nuget\packages\system.globalization\4.0.11\ref\netstandard1.3\System.Globalization.dll /reference:C:\Users\eerhardt\.nuget\packages\system.io.compression\4.1.0\ref\netstandard1.3\System.IO.Compression.dll /reference:C:\Users\eerhardt\.nuget\packages\system.io.compression.zipfile\4.0.1\ref\netstandard1.3\System.IO.Compression.ZipFile.dll /reference:C:\Users\eerhardt\.nuget\packages\system.io\4.1.0\ref\netstandard1.5\System.IO.dll /reference:C:\Users\eerhardt\.nuget\packages\system.io.filesystem\4.0.1\ref\netstandard1.3\System.IO.FileSystem.dll /reference:C:\Users\eerhardt\.nuget\packages\system.io.filesystem.primitives\4.0.1\ref\netstandard1.3\System.IO.FileSystem.Primitives.dll /reference:C:\Users\eerhardt\.nuget\packages\system.io.pipes\4.0.0\ref\netstandard1.3\System.IO.Pipes.dll /reference:C:\Users\eerhardt\.nuget\packages\system.linq\4.1.0\ref\netstandard1.0\System.Linq.dll /reference:C:\Users\eerhardt\.nuget\packages\system.linq.expressions\4.1.0\ref\netstandard1.3\System.Linq.Expressions.dll /reference:C:\Users\eerhardt\.nuget\packages\system.linq.parallel\4.0.1\ref\netstandard1.1\System.Linq.Parallel.dll /reference:C:\Users\eerhardt\.nuget\packages\system.net.http\4.1.0\ref\netstandard1.3\System.Net.Http.dll /reference:C:\Users\eerhardt\.nuget\packages\system.net.primitives\4.0.11\ref\netstandard1.3\System.Net.Primitives.dll /reference:C:\Users\eerhardt\.nuget\packages\system.net.sockets\4.1.0\ref\netstandard1.3\System.Net.Sockets.dll /reference:C:\Users\eerhardt\.nuget\packages\system.objectmodel\4.0.12\ref\netstandard1.3\System.ObjectModel.dll /reference:C:\Users\eerhardt\.nuget\packages\system.reflection\4.1.0\ref\netstandard1.5\System.Reflection.dll /reference:C:\Users\eerhardt\.nuget\packages\system.reflection.extensions\4.0.1\ref\netstandard1.0\System.Reflection.Extensions.dll /reference:C:\Users\eerhardt\.nuget\packages\system.reflection.metadata\1.3.0\lib\netstandard1.1\System.Reflection.Metadata.dll /reference:C:\Users\eerhardt\.nuget\packages\system.reflection.primitives\4.0.1\ref\netstandard1.0\System.Reflection.Primitives.dll /reference:C:\Users\eerhardt\.nuget\packages\system.reflection.typeextensions\4.1.0\ref\netstandard1.5\System.Reflection.TypeExtensions.dll /reference:C:\Users\eerhardt\.nuget\packages\system.resources.reader\4.0.0\lib\netstandard1.0\System.Resources.Reader.dll /reference:C:\Users\eerhardt\.nuget\packages\system.resources.resourcemanager\4.0.1\ref\netstandard1.0\System.Resources.ResourceManager.dll /reference:C:\Users\eerhardt\.nuget\packages\system.runtime\4.1.0\ref\netstandard1.5\System.Runtime.dll /reference:C:\Users\eerhardt\.nuget\packages\system.runtime.extensions\4.1.0\ref\netstandard1.5\System.Runtime.Extensions.dll /reference:C:\Users\eerhardt\.nuget\packages\system.runtime.handles\4.0.1\ref\netstandard1.3\System.Runtime.Handles.dll /reference:C:\Users\eerhardt\.nuget\packages\system.runtime.interopservices\4.1.0\ref\netstandard1.5\System.Runtime.InteropServices.dll /reference:C:\Users\eerhardt\.nuget\packages\system.runtime.interopservices.runtimeinformation\4.0.0\ref\netstandard1.1\System.Runtime.InteropServices.RuntimeInformation.dll /reference:C:\Users\eerhardt\.nuget\packages\system.runtime.loader\4.0.0\ref\netstandard1.5\System.Runtime.Loader.dll /reference:C:\Users\eerhardt\.nuget\packages\system.runtime.numerics\4.0.1\ref\netstandard1.1\System.Runtime.Numerics.dll /reference:C:\Users\eerhardt\.nuget\packages\system.runtime.serialization.primitives\4.1.1\ref\netstandard1.3\System.Runtime.Serialization.Primitives.dll /reference:C:\Users\eerhardt\.nuget\packages\system.runtime.serialization.xml\4.1.1\ref\netstandard1.3\System.Runtime.Serialization.Xml.dll /reference:C:\Users\eerhardt\.nuget\packages\system.security.cryptography.algorithms\4.2.0\ref\netstandard1.4\System.Security.Cryptography.Algorithms.dll /reference:C:\Users\eerhardt\.nuget\packages\system.security.cryptography.encoding\4.0.0\ref\netstandard1.3\System.Security.Cryptography.Encoding.dll /reference:C:\Users\eerhardt\.nuget\packages\system.security.cryptography.primitives\4.0.0\ref\netstandard1.3\System.Security.Cryptography.Primitives.dll /reference:C:\Users\eerhardt\.nuget\packages\system.security.cryptography.protecteddata\4.0.0\ref\netstandard1.3\System.Security.Cryptography.ProtectedData.dll /reference:C:\Users\eerhardt\.nuget\packages\system.security.cryptography.x509certificates\4.1.0\ref\netstandard1.4\System.Security.Cryptography.X509Certificates.dll /reference:C:\Users\eerhardt\.nuget\packages\system.security.principal\4.0.1\ref\netstandard1.0\System.Security.Principal.dll /reference:C:\Users\eerhardt\.nuget\packages\system.text.encoding\4.0.11\ref\netstandard1.3\System.Text.Encoding.dll /reference:C:\Users\eerhardt\.nuget\packages\system.text.encoding.extensions\4.0.11\ref\netstandard1.3\System.Text.Encoding.Extensions.dll /reference:C:\Users\eerhardt\.nuget\packages\system.text.regularexpressions\4.1.0\ref\netstandard1.3\System.Text.RegularExpressions.dll /reference:C:\Users\eerhardt\.nuget\packages\system.threading\4.0.11\ref\netstandard1.3\System.Threading.dll /reference:C:\Users\eerhardt\.nuget\packages\system.threading.tasks.dataflow\4.6.0\lib\netstandard1.1\System.Threading.Tasks.Dataflow.dll /reference:C:\Users\eerhardt\.nuget\packages\system.threading.tasks\4.0.11\ref\netstandard1.3\System.Threading.Tasks.dll /reference:C:\Users\eerhardt\.nuget\packages\system.threading.thread\4.0.0\ref\netstandard1.3\System.Threading.Thread.dll /reference:C:\Users\eerhardt\.nuget\packages\system.threading.threadpool\4.0.10\ref\netstandard1.3\System.Threading.ThreadPool.dll /reference:C:\Users\eerhardt\.nuget\packages\system.threading.timer\4.0.1\ref\netstandard1.2\System.Threading.Timer.dll /reference:C:\Users\eerhardt\.nuget\packages\system.xml.readerwriter\4.0.11\ref\netstandard1.3\System.Xml.ReaderWriter.dll /reference:C:\Users\eerhardt\.nuget\packages\system.xml.xdocument\4.0.11\ref\netstandard1.3\System.Xml.XDocument.dll /reference:C:\Users\eerhardt\.nuget\packages\system.xml.xmldocument\4.0.1\ref\netstandard1.3\System.Xml.XmlDocument.dll /reference:C:\Users\eerhardt\.nuget\packages\system.xml.xpath\4.0.1\ref\netstandard1.3\System.Xml.XPath.dll /reference:C:\Users\eerhardt\.nuget\packages\system.xml.xpath.xmldocument\4.0.1\ref\netstandard1.3\System.Xml.XPath.XmlDocument.dll /reference:C:\Users\eerhardt\.nuget\packages\xunit.abstractions\2.0.1\lib\netstandard1.0\xunit.abstractions.dll /reference:C:\Users\eerhardt\.nuget\packages\xunit.assert\2.2.0-beta4-build3444\lib\netstandard1.0\xunit.assert.dll /reference:C:\Users\eerhardt\.nuget\packages\xunit.extensibility.core\2.2.0-beta4-build3444\lib\netstandard1.0\xunit.core.dll /reference:C:\Users\eerhardt\.nuget\packages\xunit.extensibility.execution\2.2.0-beta4-build3444\lib\netstandard1.0\xunit.execution.dotnet.dll /debug+ /debug:portable /filealign:512 /keyfile:../../tools/Key.snk /nologo /optimize- /out:obj\Debug\netstandard1.5\Microsoft.DotNet.Tools.Tests.Utilities.dll /target:library /warnaserror- /utf8output /deterministic+ Assertions\CommandResultAssertions.cs Assertions\CommandResultExtensions.cs Assertions\DirectoryInfoAssertions.cs Assertions\DirectoryInfoExtensions.cs Assertions\FileInfoAssertions.cs Assertions\FileInfoExtensions.cs Assertions\FileInfoExtensions.FileInfoLock.cs Assertions\FileInfoExtensions.FileInfoNuGetLock.cs Assertions\StringAssertionsExtensions.cs Commands\AddReferenceCommand.cs Commands\BuildCommand.cs Commands\BuildPJCommand.cs Commands\CleanCommand.cs Commands\DependencyToolInvokerCommand.cs Commands\DotnetCommand.cs Commands\DotnetTestCommand.cs Commands\HelpCommand.cs Commands\ListReferenceCommand.cs Commands\MigrateCommand.cs Commands\MSBuildCommand.cs Commands\NewCommand.cs Commands\PackCommand.cs Commands\PublishCommand.cs Commands\RemoveReferenceCommand.cs Commands\RestoreCommand.cs Commands\RestoreProjectJsonCommand.cs Commands\RunCommand.cs Commands\TestCommand.cs Commands\VSTestCommand.cs DotnetLegacyRuntimeIdentifiers.cs Extensions\IDisposableExtensions.cs Extensions\ProcessExtensions.cs Extensions\TestCommandExtensions.cs Mock\EnvironmentMockBuilder.cs Mock\FileSystemMockBuilder.cs Mock\ITemporaryDirectoryMock.cs NetworkUtils\NetworkHelper.cs NetworkUtils\PortManager.cs NuGetConfig.cs PeReaderUtils.cs ProjectUtils.cs Properties\Properties.cs RepoDirectoriesProvider.cs TempFileSystem\DisposableDirectory.cs TempFileSystem\DisposableFile.cs TempFileSystem\FileNameUtilities.cs TempFileSystem\ImmutableArrayTestExtensions.cs TempFileSystem\PathKind.cs TempFileSystem\PathUtilities.cs TempFileSystem\TempDirectory.cs TempFileSystem\TempFile.cs TempFileSystem\TempRoot.cs TestBase.cs WindowsOnlyFactAttribute.cs "C:\Users\eerhardt\AppData\Local\Temp\.NETStandard,Version=v1.5.AssemblyAttributes.cs" obj\Debug\netstandard1.5\Microsoft.DotNet.Tools.Tests.Utilities.AssemblyInfo.cs (TaskId:70)
10:20:52.906  4:22>CSC : error CS2012: Cannot open 'F:\cli\test\Microsoft.DotNet.Tools.Tests.Utilities\obj\Debug\netstandard1.5\Microsoft.DotNet.Tools.Tests.Utilities.dll' for writing -- 'The process cannot access the file 'F:\cli\test\Microsoft.DotNet.Tools.Tests.Utilities\obj\Debug\netstandard1.5\Microsoft.DotNet.Tools.Tests.Utilities.dll' because it is being used by another process.' [F:\cli\test\Microsoft.DotNet.Tools.Tests.Utilities\Microsoft.DotNet.Tools.Tests.Utilities.csproj]
                     Output Item(s): CscCommandLineArgs= (TaskId:70)
                   Done executing task "Csc" -- FAILED. (TaskId:70)

As you can see, node 16 and node 22 are trying to compile the same project to the same output file roughly 35ms apart.

@rainersigwald @jeffkl @cdmihai - doesn't MSBuild have a locking mechanism so 2 nodes don't try building the same project at the same time?

[rainersigwald]

No, the rule is that the engine builds only once for each tuple of:

• Project
• Target
• Set of global properties

It's definitely concerning that something in the build is building project 4 22 times.

I'm going to apply some of my magic internal tracing tools to the Sdk repo to help chase this down.

[AlgorithmsAreCool]

I am running into this reliably if I target the my .sln file. Targeting the .csproj build correctly. I can provide logs if it will help.

[rainersigwald]

@AlgorithmsAreCool in the CLI repo or another public repo? Unfortunately, standard logging is hard to decipher for this problem. I have a still-locked-inside-the-firewall tool that can help, but to use it I need to be able to build using full framework MSBuild. If you have a public repo that hits this I could take a look at it.

[AlgorithmsAreCool]

I'm afraid the code is private, I can try to put together a minimal repro and upload it on here. I should be able to get to that early tomorrow.

[rainersigwald]

@AlgorithmsAreCool I think the relevant information should just be: what projects do you have, what is the project-to-project reference structure, and what are the project's TFMs?

[rainersigwald]

Ok, my "omg 22" business above may have been a bit premature. I'm throwing together a logger to help with debugging this sort of issue: https://github.com/rainersigwald/ParallelBuildDebuggingLogger

It turns out that many/most of the requests will be because of the determine-tfm-from-requesting-tfm dance. For example, on a .sln with 2 projects each dual targeted, I see:

Project {2: "C:\src\sdk-multiproc\SdkMultiprocTestbed\SdkMultiprocTestbed.sln" + <>} built by project -1 -- targets 'Build'
Project {3: "C:\src\sdk-multiproc\SdkMultiprocTestbed\ClassLibrary1\ClassLibrary1.csproj" + <SolutionDir = C:\src\sdk-multiproc\SdkMultiprocTestbed\; Configuration = Debug; SolutionFileName = SdkMultiprocTestbed.sln; SolutionName = SdkMultiprocTestbed; CurrentSolutionConfigurationContents = <SolutionConfiguration>
  <ProjectConfiguration Project="{AB0FAE36-5A34-49C4-B8EC-B29628F7CF48}" AbsolutePath="C:\src\sdk-multiproc\SdkMultiprocTestbed\ClassLibrary1\ClassLibrary1.csproj" BuildProjectInSolution="True">Debug|AnyCPU</ProjectConfiguration>
  <ProjectConfiguration Project="{70B64473-D0EE-47DE-8A3A-9E7DBAB82456}" AbsolutePath="C:\src\sdk-multiproc\SdkMultiprocTestbed\ClassLibrary2\ClassLibrary2.csproj" BuildProjectInSolution="True">Debug|AnyCPU</ProjectConfiguration>
</SolutionConfiguration>; Platform = AnyCPU; SolutionExt = .sln; BuildingSolutionFile = true; SolutionPath = C:\src\sdk-multiproc\SdkMultiprocTestbed\SdkMultiprocTestbed.sln>} built by project 2 -- targets ''
Project {5: "C:\src\sdk-multiproc\SdkMultiprocTestbed\ClassLibrary1\ClassLibrary1.csproj" + <TargetFramework = netstandard1.4>} built by project 3 -- targets 'Build'
Building target 'CoreCompile' in {5: "C:\src\sdk-multiproc\SdkMultiprocTestbed\ClassLibrary1\ClassLibrary1.csproj" + <TargetFramework = netstandard1.4>}
  ClassLibrary1 -> C:\src\sdk-multiproc\SdkMultiprocTestbed\ClassLibrary1\bin\Debug\netstandard1.4\ClassLibrary1.dll
Project {6: "C:\src\sdk-multiproc\SdkMultiprocTestbed\ClassLibrary1\ClassLibrary1.csproj" + <TargetFramework = net46>} built by project 3 -- targets 'Build'
Building target 'CoreCompile' in {6: "C:\src\sdk-multiproc\SdkMultiprocTestbed\ClassLibrary1\ClassLibrary1.csproj" + <TargetFramework = net46>}
  ClassLibrary1 -> C:\src\sdk-multiproc\SdkMultiprocTestbed\ClassLibrary1\bin\Debug\net46\ClassLibrary1.dll
Project {4: "C:\src\sdk-multiproc\SdkMultiprocTestbed\ClassLibrary2\ClassLibrary2.csproj" + <SolutionDir = C:\src\sdk-multiproc\SdkMultiprocTestbed\; Configuration = Debug; SolutionFileName = SdkMultiprocTestbed.sln; SolutionName = SdkMultiprocTestbed; CurrentSolutionConfigurationContents = <SolutionConfiguration>
  <ProjectConfiguration Project="{AB0FAE36-5A34-49C4-B8EC-B29628F7CF48}" AbsolutePath="C:\src\sdk-multiproc\SdkMultiprocTestbed\ClassLibrary1\ClassLibrary1.csproj" BuildProjectInSolution="True">Debug|AnyCPU</ProjectConfiguration>
  <ProjectConfiguration Project="{70B64473-D0EE-47DE-8A3A-9E7DBAB82456}" AbsolutePath="C:\src\sdk-multiproc\SdkMultiprocTestbed\ClassLibrary2\ClassLibrary2.csproj" BuildProjectInSolution="True">Debug|AnyCPU</ProjectConfiguration>
</SolutionConfiguration>; Platform = AnyCPU; SolutionExt = .sln; BuildingSolutionFile = true; SolutionPath = C:\src\sdk-multiproc\SdkMultiprocTestbed\SdkMultiprocTestbed.sln>} built by project 2 -- targets ''
Project {7: "C:\src\sdk-multiproc\SdkMultiprocTestbed\ClassLibrary2\ClassLibrary2.csproj" + <TargetFramework = netstandard1.4>} built by project 4 -- targets 'Build'
Project {8: "C:\src\sdk-multiproc\SdkMultiprocTestbed\ClassLibrary1\ClassLibrary1.csproj" + <ReferringTargetFramework = .NETStandard,Version=v1.4>} built by project 7 -- targets 'GetTargetFrameworkProperties'
Reentering project {5: "C:\src\sdk-multiproc\SdkMultiprocTestbed\ClassLibrary1\ClassLibrary1.csproj" + <>} to build targets 
Reentering project {5: "C:\src\sdk-multiproc\SdkMultiprocTestbed\ClassLibrary1\ClassLibrary1.csproj" + <>} to build targets GetNativeManifest
Building target 'CoreCompile' in {7: "C:\src\sdk-multiproc\SdkMultiprocTestbed\ClassLibrary2\ClassLibrary2.csproj" + <TargetFramework = netstandard1.4>}
Reentering project {5: "C:\src\sdk-multiproc\SdkMultiprocTestbed\ClassLibrary1\ClassLibrary1.csproj" + <>} to build targets GetCopyToOutputDirectoryItems
  ClassLibrary2 -> C:\src\sdk-multiproc\SdkMultiprocTestbed\ClassLibrary2\bin\Debug\netstandard1.4\ClassLibrary2.dll
Project {9: "C:\src\sdk-multiproc\SdkMultiprocTestbed\ClassLibrary2\ClassLibrary2.csproj" + <TargetFramework = netstandard1.5>} built by project 4 -- targets 'Build'
Project {10: "C:\src\sdk-multiproc\SdkMultiprocTestbed\ClassLibrary1\ClassLibrary1.csproj" + <ReferringTargetFramework = .NETStandard,Version=v1.5>} built by project 9 -- targets 'GetTargetFrameworkProperties'
Reentering project {5: "C:\src\sdk-multiproc\SdkMultiprocTestbed\ClassLibrary1\ClassLibrary1.csproj" + <TargetFramework = netstandard1.4>} to build targets 
Reentering project {5: "C:\src\sdk-multiproc\SdkMultiprocTestbed\ClassLibrary1\ClassLibrary1.csproj" + <TargetFramework = netstandard1.4>} to build targets GetNativeManifest
Building target 'CoreCompile' in {9: "C:\src\sdk-multiproc\SdkMultiprocTestbed\ClassLibrary2\ClassLibrary2.csproj" + <TargetFramework = netstandard1.5>}
Reentering project {5: "C:\src\sdk-multiproc\SdkMultiprocTestbed\ClassLibrary1\ClassLibrary1.csproj" + <TargetFramework = netstandard1.4>} to build targets GetCopyToOutputDirectoryItems
  ClassLibrary2 -> C:\src\sdk-multiproc\SdkMultiprocTestbed\ClassLibrary2\bin\Debug\netstandard1.5\ClassLibrary2.dll

There are many flavors of invocation of both projects, but CoreCompile is (correctly) invoked only twice per project.

Now I'll try this on the CLI repo where this problem is being observed and see if anything stands out.

[AlgorithmsAreCool]

@rainersigwald I'm still trying to get a minimal repro working, I can tickle the issue but not hit it every time.

My project structure is :

LowerLib (netstandard1.6)
HigherLib (netstandard1.6)
App (netcoreapp1.0)

LowerLib depends on some nugets packages only
HigherLib depends on LowerLib
App Depends on both LowerLib and HigherLib and some other nugets

App.sln includes all 3 projects

What is a TFM btw?
Never mind, I found the answer. I'm build dotnet restore -r win7-x86 followed by dotnet publish app.sln -r win7-x86

Also the error always is HigherLib and is either a hard error of:

CSC : error CS2012: Cannot open 'C:\Dev\Repro\HigherLib\obj\Debug\netstandard1.6\HigherLib.dll' for writing -- 'The process cannot access the file 'C:\Dev\Repro\HigherLib\obj\Debug\netstandard1.6\HigherLib.dll' because it is being used by another process.' [C:\Dev\Repro\HigherLib\HigherLib.csproj]

or a warning of:

C:\Program Files\dotnet\sdk\1.0.0-rc3-004530\Microsoft.Common.CurrentVersion.targets(3935,5): warning MSB3026: Could not copy "obj\Debug\netstandard1.6\higherlib.dll" to "bin\Debug\netstandard1.6\higherlib.dll". Beginning retry 1 in 1000ms. The process cannot access the file 'C:\dev\repro\higherlib\obj\Debug\netstandard1.6\higherlib.dll' because it is being used by another process. [C:\dev\repro\higherlib\higherlib.csproj]

[rainersigwald]

That's a great repro, @AlgorithmsAreCool, thanks!

C:\src\AlgorithmsAreCool-sdk-739>dotnet publish -r win7-x86 /logger:C:\src\ParallelBuildDebuggingLogger\ParallelBuildDebuggingLogger\bin\Debug\netstandard1.4\ParallelBuildDebuggingLogger.dll;CoreCompile /m:1 | findstr /rc:"Building target.*HigherLib.csproj"
Building target 'CoreCompile' in {4: "C:\src\AlgorithmsAreCool-sdk-739\HigherLib\HigherLib.csproj" + <SolutionDir = C:\src\AlgorithmsAreCool-sdk-739\; BuildingSolutionFile = true; Configuration = Debug; Platform = AnyCPU; SolutionFileName = AlgorithmsAreCool-sdk-739.sln; CurrentSolutionConfigurationContents = <SolutionConfiguration>
Building target 'CoreCompile' in {10: "C:\src\AlgorithmsAreCool-sdk-739\HigherLib\HigherLib.csproj" + <TargetFramework = netstandard1.6>}

This shows that the race is between

• The solution-driven invocation of HigherLib
  
  
  
  • New set of global properties is common to all builds invoked in the solution and contains solution metadata
  
  
  • Because this is a single-TFM project, this implicitly is the "inner" build, which does I/O work
  
  
• The ProjectReference-driven invocation of HigherLib for TargetFramework = netstandard1.6
  
  
  
  • New set of global properties is the explicit TFM desired for compatibility with App
  
  
  • Explicit request for the inner build
  
  

That explains a few questions I had ("why only in solutions?" and "why can't I repro this with my multitargeting attempts?").

@nguerrera @natidea I think we can fix this without going all the way to fixing https://github.com/Microsoft/msbuild/issues/1276 (which would probably be ideal but is invasive). Would it be reasonable to respond to the GetTargetFrameworkProperties request with no information if the project has a single TFM (just as we do for legacy projects)? That would collapse the latter request into the former and the engine would just return the previously-built file.

[nguerrera]

We stopped returning "no information" for single targeting because there was a bug where if caller had TargetFramework set as global property (say caller is multi-targeted), then it would be used instead of TargetFramework of callee. Is there something we can do with TreatAsLocal that could help with this instead?

[rainersigwald]

Does RemoveProperties="%(_MSBuildProjectReferenceExistent.GlobalPropertiesToRemove);TargetFramework" not resolve that? I would expect it to.

[nguerrera]

We only remove it from the call to GetTargetFrameworkProperties, but not subsequent targets. Does it work to remove a property and pass it? If so, we could remove it from all calls.

[rainersigwald]

Well this is clear as mud. Looks like RemoveProperties is always considered last, overriding any other way to set properties.

So that won't work. Could we instead unset the property by passing TargetFramework= for single-targeting projects?

[nguerrera]

Will that allow the project to reset it to non empty?

[rainersigwald]

I assumed that would work based on how everything else handles "set to null to unset property", but no it does not 🤦

[rainersigwald]

The TreatAsLocalProperty idea was a very good one, but it doesn't appear to be respected when the <MSBuild> task is considering global properties.

Current notion:

• Remove TargetFramework from the property removals in GetTargetFrameworkProperties
• Restructure GetTargetFrameworkProperties to return no information if both a) there was no passed-in TF and b) there is only one TF in the project. [not totally sure if that's detectable]

Some serious hoop-jumping here, though. Starting to wonder if it'd be better to go with the proposed solution to Microsoft/msbuild#1276 that would allow all existing project and all new single-tfm projects to just return "build me directly", and only invoke inner builds on new multi-tfm projects (which would match the inner builds driven by the call from the sln to the outer build). That is also a scary change, though.

[rainersigwald]

Another notion:

from a single-tfm project, return both the tfm and a flag that tells the consuming project to set the metadata UndefineProperties=TargetFramework for the reference.

[rainersigwald]

I have proof of concept on that latter option.

diff --git "a/c:\\Program Files\\dotnet - Copy\\sdk\\1.0.0-rc4-004689/Microsoft.Common.CurrentVersion.targets" "b/C:\\Program Files\\dotnet\\sdk\\1.0.0-rc4-004689/Microsoft.Common.CurrentVersion.targets"
index da772b0..7f82f2a 100644
--- "a/c:\\Program Files\\dotnet - Copy\\sdk\\1.0.0-rc4-004689/Microsoft.Common.CurrentVersion.targets" 
+++ "b/C:\\Program Files\\dotnet\\sdk\\1.0.0-rc4-004689/Microsoft.Common.CurrentVersion.targets"    
@@ -1527,9 +1527,18 @@ Copyright (C) Microsoft Corporation. All rights reserved.
       <Output TaskParameter="TargetOutputs" PropertyName="_ProjectReferenceTargetFrameworkProperties" />
     </MSBuild>

+    <PropertyGroup>
+      <_RemoveTargetFrameworkFromReference Condition="'$(_ProjectReferenceTargetFrameworkProperties.Contains(`ProjectHasSingleTargetFramework=true`))'">true</_RemoveTargetFrameworkFromReference>
+    </PropertyGroup>
+
     <ItemGroup>
       <_MSBuildProjectReferenceExistent Condition="'%(_MSBuildProjectReferenceExistent.Identity)' == '%(Identity)'">
         <SetTargetFramework>$(_ProjectReferenceTargetFrameworkProperties)</SetTargetFramework>
+
+        <!-- Unconditionally remove the property that was set as a marker to indicate that for this call we should remove TargetFramework -->
+        <UndefineProperties>%(_MSBuildProjectReferenceExistent.UndefineProperties);ProjectHasSingleTargetFramework</UndefineProperties>
+
+        <UndefineProperties Condition="'$(_RemoveTargetFrameworkFromReference)' == 'true'">%(_MSBuildProjectReferenceExistent.UndefineProperties);TargetFramework</UndefineProperties>
       </_MSBuildProjectReferenceExistent>
     </ItemGroup>

diff --git "a/c:\\Program Files\\dotnet - Copy\\sdk\\1.0.0-rc4-004689/Sdks/Microsoft.NET.Sdk/build/Microsoft.NET.Sdk.Common.targets" "b/C:\\Program Files\\dotnet\\sdk\\1.0.0-rc4-004689/Sdks/Microsoft.NET.Sdk/build/Microsoft.NET.Sdk.Common.targets"
index c1c71c2..de62c6c 100644
--- "a/c:\\Program Files\\dotnet - Copy\\sdk\\1.0.0-rc4-004689/Sdks/Microsoft.NET.Sdk/build/Microsoft.NET.Sdk.Common.targets"   
+++ "b/C:\\Program Files\\dotnet\\sdk\\1.0.0-rc4-004689/Sdks/Microsoft.NET.Sdk/build/Microsoft.NET.Sdk.Common.targets"  
@@ -55,13 +55,16 @@ Copyright (c) .NET Foundation. All rights reserved.
     with the referencing project's target framework.
   ============================================================
    -->
-  <Target Name="GetTargetFrameworkProperties" Returns="TargetFramework=$(NearestTargetFramework)">
+  <Target Name="GetTargetFrameworkProperties" Returns="TargetFramework=$(NearestTargetFramework);ProjectHasSingleTargetFramework=$(_HasSingleTargetFramework)">

     <PropertyGroup>
       <!-- If a ReferringTargetFramework was not specified, and we only have one TargetFramework, then don't try to check compatibility -->
       <_SkipNearestTargetFrameworkResolution Condition="'$(TargetFramework)' != '' and '$(ReferringTargetFramework)' == ''">true</_SkipNearestTargetFrameworkResolution>
       <NearestTargetFramework Condition="'$(_SkipNearestTargetFrameworkResolution)' == 'true'">$(TargetFramework)</NearestTargetFramework>

+      <_HasSingleTargetFramework Condition="'$(TargetFramework)' != '' and '$(TargetFrameworks)' == ''">true</_HasSingleTargetFramework>
+      <_HasSingleTargetFramework Condition="'$(_HasSingleTargetFramework)' == ''">false</_HasSingleTargetFramework>
+
       <_PossibleTargetFrameworks Condition="'$(TargetFramework)' != ''">$(TargetFramework)</_PossibleTargetFrameworks>
       <_PossibleTargetFrameworks Condition="'$(TargetFramework)' == ''">$(TargetFrameworks)</_PossibleTargetFrameworks>
     </PropertyGroup>

@nguerrera thoughts?

[rainersigwald]

simplified:

diff --git "a/c:\\Program Files\\dotnet - Copy\\sdk\\1.0.0-rc4-004689/Microsoft.Common.CurrentVersion.targets" "b/C:\\Program Files\\dotnet\\sdk\\1.0.0-rc4-004689/Microsoft.Common.CurrentVersion.targets"
index da772b0..4a60364 100644
--- "a/c:\\Program Files\\dotnet - Copy\\sdk\\1.0.0-rc4-004689/Microsoft.Common.CurrentVersion.targets" 
+++ "b/C:\\Program Files\\dotnet\\sdk\\1.0.0-rc4-004689/Microsoft.Common.CurrentVersion.targets"    
@@ -1530,6 +1530,10 @@ Copyright (C) Microsoft Corporation. All rights reserved.
     <ItemGroup>
       <_MSBuildProjectReferenceExistent Condition="'%(_MSBuildProjectReferenceExistent.Identity)' == '%(Identity)'">
         <SetTargetFramework>$(_ProjectReferenceTargetFrameworkProperties)</SetTargetFramework>
+
+        <UndefineProperties Condition="$(_ProjectReferenceTargetFrameworkProperties.Contains(`ProjectHasSingleTargetFramework=true`))">%(_MSBuildProjectReferenceExistent.UndefineProperties);TargetFramework;ProjectHasSingleTargetFramework</UndefineProperties>
+        <!-- Unconditionally remove the property that was set as a marker to indicate that for this call we should remove TargetFramework -->
+        <UndefineProperties Condition="!$(_ProjectReferenceTargetFrameworkProperties.Contains(`ProjectHasSingleTargetFramework=true`))">%(_MSBuildProjectReferenceExistent.UndefineProperties);ProjectHasSingleTargetFramework</UndefineProperties>
       </_MSBuildProjectReferenceExistent>
     </ItemGroup>

diff --git "a/c:\\Program Files\\dotnet - Copy\\sdk\\1.0.0-rc4-004689/Sdks/Microsoft.NET.Sdk/build/Microsoft.NET.Sdk.Common.targets" "b/C:\\Program Files\\dotnet\\sdk\\1.0.0-rc4-004689/Sdks/Microsoft.NET.Sdk/build/Microsoft.NET.Sdk.Common.targets"
index c1c71c2..de62c6c 100644
--- "a/c:\\Program Files\\dotnet - Copy\\sdk\\1.0.0-rc4-004689/Sdks/Microsoft.NET.Sdk/build/Microsoft.NET.Sdk.Common.targets"   
+++ "b/C:\\Program Files\\dotnet\\sdk\\1.0.0-rc4-004689/Sdks/Microsoft.NET.Sdk/build/Microsoft.NET.Sdk.Common.targets"  
@@ -55,13 +55,16 @@ Copyright (c) .NET Foundation. All rights reserved.
     with the referencing project's target framework.
   ============================================================
    -->
-  <Target Name="GetTargetFrameworkProperties" Returns="TargetFramework=$(NearestTargetFramework)">
+  <Target Name="GetTargetFrameworkProperties" Returns="TargetFramework=$(NearestTargetFramework);ProjectHasSingleTargetFramework=$(_HasSingleTargetFramework)">

     <PropertyGroup>
       <!-- If a ReferringTargetFramework was not specified, and we only have one TargetFramework, then don't try to check compatibility -->
       <_SkipNearestTargetFrameworkResolution Condition="'$(TargetFramework)' != '' and '$(ReferringTargetFramework)' == ''">true</_SkipNearestTargetFrameworkResolution>
       <NearestTargetFramework Condition="'$(_SkipNearestTargetFrameworkResolution)' == 'true'">$(TargetFramework)</NearestTargetFramework>

+      <_HasSingleTargetFramework Condition="'$(TargetFramework)' != '' and '$(TargetFrameworks)' == ''">true</_HasSingleTargetFramework>
+      <_HasSingleTargetFramework Condition="'$(_HasSingleTargetFramework)' == ''">false</_HasSingleTargetFramework>
+
       <_PossibleTargetFrameworks Condition="'$(TargetFramework)' != ''">$(TargetFramework)</_PossibleTargetFrameworks>
       <_PossibleTargetFrameworks Condition="'$(TargetFramework)' == ''">$(TargetFrameworks)</_PossibleTargetFrameworks>
     </PropertyGroup>

[nguerrera]

That looks promising.

[natemcmaster]

If need additional code for repro/verification, this bug happens consistently on https://github.com/aspnet/DependencyInjection. All you have to do is dotnet restore && dotnet build DependencyInjection.sln to hit this.

Using 1.0.0-rc4-004757

[jaredpar]

This bug is marked as fixed in 1.0 RTM yet we are seeing it on Rosyln using 1.01 builds.

• PR: https://github.com/dotnet/roslyn/pull/21167
• Mac runs: https://ci.dot.net/job/dotnet_roslyn/job/master/job/mac_debug_prtest/18/

The problem reproduces locally on some machines, but not others and only on Mac. Our Linux runs which use the same testing scripts don't hit this issue.

CC @khyperia

[jaredpar]

We are hitting this multiple times a day in roslyn. Why is this issue still marked as closed? Is anyone looking into it???

[nguerrera]

It is closed because #797 paired with https://github.com/Microsoft/msbuild/pull/1667 fixed a specific race condition that was very common. I think you've hit a different race with same error message symptom.

[nguerrera]

The fact that it is only occurring on Mac is very different from the original reports.

[nguerrera]

Let's use #475 to track.

[jaredpar]

It is happening on windows too

https://ci.dot.net/job/dotnet_roslyn/job/master/job/windows_determinism_prtest/9548/

[nguerrera]

See https://github.com/dotnet/sdk/issues/475#issuecomment-330993487.