Runtime: nuget.dgspec.json "in use" error during build

[stephentoub]

I just did:

git clean -xdf
git pull upstream master
build.cmd

and got this error soon into the build:

D:\repos\corefx\src\System.Runtime\tests\TestStructs\System.TestStructs.ilproj : error : One or more errors occurred. (The process cannot access the file 'C:\Users\stoub\AppData\Local\Temp\687ea92c-fc3a-417b-8b0f-f858ff41dab5.nuget.dgspec.json' because it is being used by another process.)

cc: @ViktorHofer, @ericstj

[ViktorHofer]

I hit the same issue just now after a clean. This sounds like an issue with the Microsoft.NET.Sdk.IL targets and props (edit: which live in coreclr)

[stephentoub]

Just failed again, this time killing the whole build:

  Restore completed in 15.19 sec for D:\repos\corefx\external\runtime\runtime.depproj.
D:\repos\corefx\src\System.Runtime\tests\TestModule\System.Reflection.TestModule.ilproj : error : One or more errors occurred. (The process cannot access the file 'C:\Users\stoub\AppData\Local\Temp\2cc58865-b052-4168-b75e-848eb7ad92ae.nuget.dgspec.json' because it is being used by another process.)

Build FAILED.

D:\repos\corefx\src\System.Runtime\tests\TestModule\System.Reflection.TestModule.ilproj : error : One or more errors occurred. (The process cannot access the file 'C:\Users\stoub\AppData\Local\Temp\2cc58865-b052-4168-b75e-848eb7ad92ae.nuget.dgspec.json' because it is being used by another process.)
    0 Warning(s)
    1 Error(s)

Time Elapsed 00:00:52.21
Build failed.

d:\repos\corefx>

[ViktorHofer]

Something is locking on the nuget.dgspec.json and not releasing the lock even after the restore/build is done. For now I recommend to do a taskkill /im dotnet.exe /f before building after a clean.

[rrelyea]

Would love to know details if this feels like nuget

[ericstj]

@ViktorHofer do you imagine there's a concurrent restore going on here? What this smells like is a simultaneous restore of two different project evaluations that use the same path. Given our custom configuration system and the differences in behaviors for our external projects vs test projects I think this deserves a first look by us.

[ViktorHofer]

We just talked about this in our standup. This only happens if dotnet instances are still running in the background when doing a clean build. The leftover dotnet instances are locking the dgspec.json in the temp directory but I haven't yet found out why they do so. I quickly glanced over https://github.com/NuGet/NuGet.Client/blob/dev/src/NuGet.Core/NuGet.ProjectModel/DependencyGraphSpec.cs but I couldn't find anything fishy.

Note that we don't directly restore the project that throws the error ("System.TestStructs.ilproj") but transitively as a dependency of "System.Runtime.Tests.csproj".

[stephentoub]

Just hit this again.

D:\repos\corefx\src\System.Runtime\tests\TestStructs\System.TestStructs.ilproj : error : One or more errors occurred. (The process cannot access the file 'C:\Users\stoub\AppData\Local\Temp\01d3ce95-1ea4-431f-96b1-c33a9e7bd4e1.nuget.dgspec.json' because it is being used by another process.)

[ViktorHofer]

@rrelyea could somebody on your team help us diagnose this issue as well? We have a binlog available if that helps: https://dnceng.visualstudio.com/public/_build/results?buildId=333440&view=logs

[Kieranties]

We've been consistently hitting this issue when using additional SDKs specified in the global.json.
E.g. we use https://github.com/Microsoft/MSBuildSdks/tree/master/src/CentralPackageVersions in many projects and will often get:

1>Project "C:\BA\bed11ac6b8240753\xxx.AspNet.sln" on node 1 (Clean target(s)).
[10:24:17] 1>ValidateSolutionConfiguration:
[10:24:17] Building solution configuration "Debug|Any CPU".
[10:24:18] Restoring packages for C:\BuildAgent\tempbuildTmpdb0fd7f7-95fe-4b13-827b-6b3ce61569c6...
[10:24:18] 0>C:\BA\bed11ac6b8240753\test\xxx.Tests\xxx.Tests.csproj : error : One or more errors occurred. (The process cannot access the file 'C:\BuildAgent\tempbuildTmpdb0fd7f7-95fe-4b13-827b-6b3ce61569c6.nuget.dgspec.json' because it is being used by another process.)
[10:24:18] 1>Project "C:\BA\bed11ac6b8240753\xxx.AspNet.sln" (1) is building "C:\BA\bed11ac6b8240753\test\xxx.Tests\xxx.Tests.csproj" (2) on node 1 (Clean target(s)).
[10:24:18] 2>C:\BA\bed11ac6b8240753\test\xxx.Tests\xxx.Tests.csproj : error MSB4236: The SDK 'Microsoft.Build.CentralPackageVersions' specified could not be found.

Pinning the sdk to an older version (e.g. 2.1.503) resolves the issue, but this is not sustainable.

[stephentoub]

This continues to be a real problem for me locally. I've been hitting it maybe every 1 out of 3 clean builds.

[rrelyea]

@zivkan - related to dgspec for feedback? Thoughts?

[zivkan]

I'll try to reproduce the issue myself locally, but for anyone else who does reproduce it locally, would you be able to use the sysinternals tools to verify whether or not any other process has the file open?

NuGet has a pattern of writing files to the temp directory and them moving them to the project's obj/ directory to avoid leaving a partial or inconsistent file in obj/ in case there's a failure part way through. If a virus scanner or other app locks the file, this prevents NuGet from moving the file.

[amis92]

I've been hit by this as well on dotnet restore step in my cloud build:

https://github.com/WarHub/wham/runs/270707280#step:4:23

also:

https://dev.azure.com/warhub/wham/_build/results?buildId=67&view=logs&jobId=12f1170f-54f2-53f3-20dd-22fc7dff55f9&taskId=2b20c7d0-7587-5b0f-aadc-b456e6f0b807&lineStart=155&lineEnd=156&colStart=1&colEnd=1

I'm using Microsoft.Build.CentralPackageVersions pinned in global.json.

[chibanemourad]

@amis92 Same configuration same issue for me

[ericstj]

Folks, if you are hitting this when not building the dotnet/corefx repo then please open an issue with your details in http://github.com/nuget/home. This issue is tracking an issue with CoreFx's build system.

[amis92]

It's already reported: https://github.com/NuGet/Home/issues/8692

@chibanemourad @Kieranties please post your comments there as well :)

[jeffkl]

I have submitted a PR to fix this in NuGet: https://github.com/NuGet/NuGet.Client/pull/3170

[stephentoub]

Thanks, @jeffkl. This appears to be resolved.