Sdk: GenerateDepsFile: The process cannot access the file '...\MyProject.deps.json' because it is being used by another process.

Created on 3 Feb 2019 · 16Comments · Source: dotnet/sdk

UPDATE: While writing this, rather long, post -- I've managed to produce a small reproduction example, see bottom

We've recently upgraded our builds to run on faster hardware with more cpu cores, and are now ~100% of the time seeing the below error in a number of repositories. The error here is from Windows, but our CI servers run on linux and have the same error (see logs below).

We run the following commandlines:

"C:\Program Files\dotnetdotnet.exe" restore C:\ProjectMyProject.sln
"C:\Program Files\dotnetdotnet.exe" build C:\ProjectMyProject.sln --configuration Debug --framework netstandard2.0 --no-restore

C:\Program Files\dotnet\sdk\2.2.101\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.targets(129,5): error MSB4018

The "GenerateDepsFile" task failed unexpectedly. [C:\Project\MyProject.csproj]
System.IO.IOException: The process cannot access the file 'C:\Project\bin\Debug\netstandard2.0\MyProject.deps.json' because it is being used by another process. [C:\Project\MyProject.csproj]
   at System.IO.FileStream.ValidateFileHandle(SafeFileHandle fileHandle) [C:\Project\MyProject.csproj]
   at System.IO.FileStream.CreateFileOpenHandle(FileMode mode, FileShare share, FileOptions options) [C:\Project\MyProject.csproj]
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options) [C:\Project\MyProject.csproj]
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize) [C:\Project\MyProject.csproj]
   at System.IO.File.Create(String path) [C:\Project\MyProject.csproj]
   at Microsoft.NET.Build.Tasks.GenerateDepsFile.ExecuteCore() [C:\Project\MyProject.csproj]
   at Microsoft.NET.Build.Tasks.TaskBase.Execute() [C:\Project\MyProject.csproj]
   at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [C:\Project\MyProject.csproj]
   at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [C:\Project\MyProject.csproj]

Observations:

This fails on my local Windows desktop roughly 33% of the time. Rerunning the build (no cleanup) always succeeds (seems to be a race condition, so HW speeds play a role). On our Linux CI servers, it fails 100% of the time currently
The deps file in question is _always_ the same file for a given repository / builds. The file is for a shared library between multiple projects.
A plain dotnet build MyProject.sln -f netstandard2.0 (no build script) also exhibits this issue.
A dotnet build MyProject.sln seemingly always succeeds (locally and on CI server), at least for three consecutive attempts right now.
All the projects in the MyProject.sln have netstandard2.0 - but a number of them have more targets (ie. netcoreapp)
~We have more projects, f.ex. for tests, but they're in MyProject-Tests.sln. These projects only target netcoreapp2.x - we've split the solutions both for speed when developing and to be able to build artifacts without compiling the tests (since we can't build certain projects from the solution only ... that's another issue entirely)~ _(not relevant)_
Limiting the build to one process (msbuild: /m:1) seems to "solve" the problem.
At work (inaccessible currently), I performed some Procmon captures to see the difference between successful and failing runs
- It turns out, that two dotnet.exe processes attempt to create the .deps.json files
- When succeeding, there is a clear seperation between two CreateFile() calls, where the first creates the file, and the second call (from the second PID) finds the file already there and does nothing (not even reads it)
- When failing, the two processes make calls overlapping each other, leading to both of them discovering the file as missing, and both trying to create it (one obviously failing, producing the stacktrace above) -- classical concurrency issue.

So.

It should be that when two projects reference a third, shared, project .. that this shared one is built exactly once .. right?.. How come two attempts are made at writing the same .deps.json files?
How come this seemingly works fine, when building for all target frameworks?

Versions:

Local, Windows 10 x64, dotnet 2.2.101
Linux CI, docker, dotnet 2.2.103 (we build our own docker images using the dockerfile from the base microsoft/dotnet:2-sdk images)

Reproduction

Project: ci-stresstest-master.zip

In it, I have a solution with 5 projects, all empty (no actual code), but with a targeting setup that mimicks a very small subset of our projects. I've then created four seperate CI builds that did the following:

log (succeeds) dotnet restore ci-stresstest.sln + dotnet build ci-stresstest.sln --no-restore
log (succeeds) dotnet build ci-stresstest.sln
log (fails) dotnet restore ci-stresstest.sln + dotnet build ci-stresstest.sln -f netstandard2.0 --no-restore
log (fails) dotnet build ci-stresstest.sln -f netstandard2.0

Notes on repro.

We run a vanilla docker setup for CI, except for one thing:
- We mount a shared directory for nuget packages, to be able to share downloaded packages between builds

Most helpful comment

For our current use case, we wanted to optimize our builds a bit, because we're targeting two frameworks to allow our developers to test the modules they're working (they have a small program.cs, and a shared library to actually run code -- only available to them when in netcoreapp2.x).

So our builds on the CI would be building for both coreapp and netstandard. Targeting just netstandard (since everything in the slution supports that anyways) did this and shaved a few seconds of our builds.

We've workarounded by not targeting the specific framework, since we found that it was _just_ a few seconds..

However. I'm not at all sure why this is an issue.. All of the projects in question target netstandard2.0..

LordMike on 5 Feb 2019

👍2

All 16 comments

While trying to reproduce a failure on Windows with Procmon, I'm met with this. I've noticed sporadic errors like this before, but never enough of them to make an impression. Given the above, I think there might be _a lot_ of concurrency issues in msbuild/dotnet build ..

Other locking/access errors I've seen:

A copy file to output directory (from library to referencing project) which is retried a few seconds later (this was "just" a warning) (from memory --no pictures / logs)

The process cannot access the file '...\Shared02.dll' because it is being used by another process.

Cleaning the workspace (git clean -xdf) between builds produces this roughly once every two builds.

Log

PS C:\Users\MichaelBisbjerg\Git\ci-stresstest> dotnet build -f netstandard2.0
Microsoft (R) Build Engine version 15.9.20+g88f5fadfbe for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Restoring packages for C:\Users\MichaelBisbjerg\Git\ci-stresstest\Program01\Program01.csproj...
  Restoring packages for C:\Users\MichaelBisbjerg\Git\ci-stresstest\TopLib01\TopLib01.csproj...
  Restoring packages for C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared02\Shared02.csproj...
  Restoring packages for C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared01\Shared01.csproj...
  Restoring packages for C:\Users\MichaelBisbjerg\Git\ci-stresstest\TopLib02\TopLib02.csproj...
  Generating MSBuild file C:\Users\MichaelBisbjerg\Git\ci-stresstest\TopLib02\obj\TopLib02.csproj.nuget.g.props.
  Generating MSBuild file C:\Users\MichaelBisbjerg\Git\ci-stresstest\TopLib01\obj\TopLib01.csproj.nuget.g.props.
  Generating MSBuild file C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared01\obj\Shared01.csproj.nuget.g.props.
  Generating MSBuild file C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared02\obj\Shared02.csproj.nuget.g.props.
  Generating MSBuild file C:\Users\MichaelBisbjerg\Git\ci-stresstest\Program01\obj\Program01.csproj.nuget.g.props.
  Generating MSBuild file C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared01\obj\Shared01.csproj.nuget.g.targets.
  Generating MSBuild file C:\Users\MichaelBisbjerg\Git\ci-stresstest\Program01\obj\Program01.csproj.nuget.g.targets.
  Generating MSBuild file C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared02\obj\Shared02.csproj.nuget.g.targets.
  Generating MSBuild file C:\Users\MichaelBisbjerg\Git\ci-stresstest\TopLib02\obj\TopLib02.csproj.nuget.g.targets.
  Generating MSBuild file C:\Users\MichaelBisbjerg\Git\ci-stresstest\TopLib01\obj\TopLib01.csproj.nuget.g.targets.
  Restore completed in 128,32 ms for C:\Users\MichaelBisbjerg\Git\ci-stresstest\TopLib02\TopLib02.csproj.
  Restore completed in 128,32 ms for C:\Users\MichaelBisbjerg\Git\ci-stresstest\TopLib01\TopLib01.csproj.
  Restore completed in 128,32 ms for C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared02\Shared02.csproj.
  Restore completed in 128,32 ms for C:\Users\MichaelBisbjerg\Git\ci-stresstest\Program01\Program01.csproj.
  Restore completed in 128,32 ms for C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared01\Shared01.csproj.
  Shared01 -> C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared01\bin\Debug\netstandard2.0\Shared01.dll
CSC : error CS2012: Cannot open 'C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared02\obj\Debug\netstandard2.0\Shared02.dll' for writing -- 'The process cannot access the file 'C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared02\obj\Debug\netstandard2.0\Shared02.dll' because it is being used by another process.' [C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared02\Shared02.csproj]
CSC : error CS2012: Cannot open 'C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared01\obj\Debug\netstandard2.0\Shared01.dll' for writing -- 'The process cannot access the file 'C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared01\obj\Debug\netstandard2.0\Shared01.dll' because it is being used by another process.' [C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared01\Shared01.csproj]
  Shared02 -> C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared02\bin\Debug\netstandard2.0\Shared02.dll
  TopLib02 -> C:\Users\MichaelBisbjerg\Git\ci-stresstest\TopLib02\bin\Debug\netstandard2.0\TopLib02.dll
  TopLib01 -> C:\Users\MichaelBisbjerg\Git\ci-stresstest\TopLib01\bin\Debug\netstandard2.0\TopLib01.dll
  TopLib01 -> C:\Users\MichaelBisbjerg\Git\ci-stresstest\TopLib01\bin\Debug\netstandard2.0\TopLib01.dll
  Program01 -> C:\Users\MichaelBisbjerg\Git\ci-stresstest\Program01\bin\Debug\netstandard2.0\Program01.dll

Build FAILED.

CSC : error CS2012: Cannot open 'C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared02\obj\Debug\netstandard2.0\Shared02.dll' for writing -- 'The process cannot access the file 'C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared02\obj\Debug\netstandard2.0\Shared02.dll' because it is being used by another process.' [C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared02\Shared02.csproj]
CSC : error CS2012: Cannot open 'C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared01\obj\Debug\netstandard2.0\Shared01.dll' for writing -- 'The process cannot access the file 'C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared01\obj\Debug\netstandard2.0\Shared01.dll' because it is being used by another process.' [C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared01\Shared01.csproj]
    0 Warning(s)
    2 Error(s)

Time Elapsed 00:00:01.30

Could not read state file "obj\..\TopLib01.csprojAssemblyReference.cache". The process cannot access the file '..\TopLib01.csprojAssemblyReference.cache' because it is being used by another process.

Building the solution _without cleanup_ between builds, this happened 8 out of 17 times I built it.

Log

PS C:\Users\MichaelBisbjerg\Git\ci-stresstest> dotnet build -f netstandard2.0
Microsoft (R) Build Engine version 15.9.20+g88f5fadfbe for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Restore completed in 25,24 ms for C:\Users\MichaelBisbjerg\Git\ci-stresstest\TopLib02\TopLib02.csproj.
  Restore completed in 25,22 ms for C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared02\Shared02.csproj.
  Restore completed in 25,19 ms for C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared01\Shared01.csproj.
  Restore completed in 25,24 ms for C:\Users\MichaelBisbjerg\Git\ci-stresstest\Program01\Program01.csproj.
  Restore completed in 25,19 ms for C:\Users\MichaelBisbjerg\Git\ci-stresstest\TopLib01\TopLib01.csproj.
  Shared02 -> C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared02\bin\Debug\netstandard2.0\Shared02.dll
  Shared02 -> C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared02\bin\Debug\netstandard2.0\Shared02.dll
  Shared01 -> C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared01\bin\Debug\netstandard2.0\Shared01.dll
C:\Program Files\dotnet\sdk\2.2.101\Microsoft.Common.CurrentVersion.targets(2110,5): warning MSB3088: Could not read state file "obj\Debug\netstandard2.0\TopLib01.csprojAssemblyReference.cache". The process cannot access the file 'C:\Users\MichaelBisbjerg\Git\ci-stresstest\TopLib01\obj\Debug\netstandard2.0\TopLib01.csprojAssemblyReference.cache' because it is being used by another process. [C:\Users\MichaelBisbjerg\Git\ci-stresstest\TopLib01\TopLib01.csproj]
  TopLib01 -> C:\Users\MichaelBisbjerg\Git\ci-stresstest\TopLib01\bin\Debug\netstandard2.0\TopLib01.dll
  TopLib02 -> C:\Users\MichaelBisbjerg\Git\ci-stresstest\TopLib02\bin\Debug\netstandard2.0\TopLib02.dll
  TopLib01 -> C:\Users\MichaelBisbjerg\Git\ci-stresstest\TopLib01\bin\Debug\netstandard2.0\TopLib01.dll
  Shared01 -> C:\Users\MichaelBisbjerg\Git\ci-stresstest\Shared01\bin\Debug\netstandard2.0\Shared01.dll
  Program01 -> C:\Users\MichaelBisbjerg\Git\ci-stresstest\Program01\bin\Debug\netstandard2.0\Program01.dll

Build succeeded.

C:\Program Files\dotnet\sdk\2.2.101\Microsoft.Common.CurrentVersion.targets(2110,5): warning MSB3088: Could not read state file "obj\Debug\netstandard2.0\TopLib01.csprojAssemblyReference.cache". The process cannot access the file 'C:\Users\MichaelBisbjerg\Git\ci-stresstest\TopLib01\obj\Debug\netstandard2.0\TopLib01.csprojAssemblyReference.cache' because it is being used by another process. [C:\Users\MichaelBisbjerg\Git\ci-stresstest\TopLib01\TopLib01.csproj]
    1 Warning(s)
    0 Error(s)

Time Elapsed 00:00:00.41

LordMike on 3 Feb 2019

👍2

I have the same problem. Any solutions?

marius-stanescu-archive360 on 5 Feb 2019

The core problem here is that you're specifying a TargetFramework at the solution level. As you note, it works without that, while building all of the TargetFrameworks specified at the individual project level.

We hope to improve the experience around this -- at least providing an informative error, but maybe also doing something smarter. But at the moment, I'm not quite sure what that would be.

Can you describe why you're specifying a TargetFramework and what you expect to happen in that build, as distinct from the no-TF-specified solution build?

rainersigwald on 5 Feb 2019

We've workarounded by not targeting the specific framework, since we found that it was _just_ a few seconds..

However. I'm not at all sure why this is an issue.. All of the projects in question target netstandard2.0..

LordMike on 5 Feb 2019

👍2

Hi,

I've a case where there is no specifying of TargetFramework at the solution level where the dotnet build behaves similar in some cases where TargetFramework at the project level refers to netstandard2.0.

This behavior has just emerged without any notice in dotnet core 2.0. I've also tried to upgrade to dotnet core 2.1, but in vein.

In my case, there seems to be a race condition from my hardware setup, due to the dotnet build leaves few spawned dotnet processes.

Thank you.

Regards
Henrik

HenrikBach1 on 20 Feb 2019

👍1

Note that the spawned processes could be the Dotnet Build Server. You can try shutting them down by running dotnet build-server shutdown. If its that, they shouldn't be a problem (they're meant to speed up future builds). I have an anecdotal feeling that we're worse of for having the build server, but so far it's just that.

LordMike on 20 Feb 2019

Two out of three of them shut down with the dotnet build-server shutdown command:

myagent@3f98f3ab2e96:~/projects/azure/iotedge/scripts/linux$ ps
  PID TTY          TIME CMD
   31 pts/1    00:00:00 bash
10834 pts/1    00:00:04 dotnet
10850 pts/1    00:00:04 dotnet
10866 pts/1    00:00:06 dotnet
11617 pts/1    00:00:00 ps
myagent@3f98f3ab2e96:~/projects/azure/iotedge/scripts/linux$ dotnet build-server shutdown
Shutting down MSBuild server...
Shutting down VB/C# compiler server...
VB/C# compiler server shut down successfully.
MSBuild server shut down successfully.
myagent@3f98f3ab2e96:~/projects/azure/iotedge/scripts/linux$ ps
  PID TTY          TIME CMD
   31 pts/1    00:00:00 bash
10866 pts/1    00:00:07 dotnet
11668 pts/1    00:00:00 ps
myagent@3f98f3ab2e96:~/projects/azure/iotedge/scripts/linux$ ps 10866
  PID TTY      STAT   TIME COMMAND
10866 pts/1    SLl    0:07 /usr/share/dotnet/dotnet /usr/share/dotnet/sdk/2.2.102/MSBuild.dll /usr/share/dotnet/sdk/2.2.102/MSBuild.dll /nologo /nodemode:1 /nodeReuse:true
myagent@3f98f3ab2e96:~/projects/azure/iotedge/scripts/linux$

HenrikBach1 on 20 Feb 2019

That does seem like one of the build processes from the parallel build (the /nodeReuse:true option). That, I have no good answer for.

LordMike on 20 Feb 2019

Ok.

Problem solved by provisioning a new virtual machine.

HenrikBach1 on 22 Feb 2019

We've run into another similar (but not the exact same) race condition:

/usr/share/dotnet/sdk/2.2.103/Microsoft.Common.CurrentVersion.targets(2110,5):
error MSB3248: Parameter "AssemblyFiles" has invalid value
    "/build/Engine/Server/bin/Debug/netcoreapp2.2/Hydra.Engine.Server.dll".
The process cannot access the file
    '/build/Engine/Server/bin/Debug/netcoreapp2.2/Hydra.Engine.Server.dll'
because it is being used by another process. [/build/Server/Map/Hydra.Map.csproj]

I'm not sure if this is caused by a bug in the build system or if there is somehow a conflict in our projects that could be resolved by fixing the project definition. We use standard project layouts with simple SDK style csproj projects.

In the latter case, it would be amazing if the build system could detect the sources of these conflicts and warn or error on them.

anttirt on 25 Mar 2019

👍1

The core problem here is that you're specifying a TargetFramework at the solution level. As you note, it works without that, while building all of the TargetFrameworks specified at the individual project level.

I am getting this error if I simply call dotnet build Main.sln without specifying the target framework.
So I can't believe in this "core problem" explanation.

victor-yarema on 13 Jun 2019

👍1

Currently I am thinking about workaround to simply change build script part from something like

dotnet build Main.sln &&
echo 'Success'

To something like

{
  dotnet build Main.sln ||
  dotnet build Main.sln
} &&
echo 'Success'

There can be more tries if needed.
I already did that trick with MSBuild in year 2013 or 2014 when we were unable to get reliable build of website project (ASP.NET). We had 4 tries back then because 2 tries were not enough.
P.S. The problem with website project was different.

victor-yarema on 13 Jun 2019

😄1

@victor-yarema Can you please file a new issue for that? Please tag me. Because you're not specifying global properties at the solution level, you're seeing a different problem.

If you can attach a binary log of your build (a successful build log is fine), hopefully we can figure out what's going on. Please read the warnings about the information contained in the log. If you cannot upload such a log, please file the bug anyway. I can suggest additional debugging techniques.

rainersigwald on 13 Jun 2019

:+1:

ljmf00 on 22 Aug 2019

FWIW, I've experienced similar issue with GenerateDepsFile phase using .net sdk 3.1.301 - in my case one way to mitigate the issue is to explicitly build the project in question first (using dotnet build) before building projects that depend on it.

theimowski on 15 Jul 2020

👍1

I see builds fail with p5 because of an issue similar to this
our pattern is to start several processes with
dotnet tool restore
then dotnet ef database update --no-build --verbose --context AppDbContext --project "SomeProjectThatMigratesSchemaLocally.csproj"
then dotnet run --no-restore --project ./src/SomeProject.Startup *> console.log
then execute an e2e test to see all is good
we have custom snappy linux azure vm agents with azure yaml pipelines
don't have any lead as to root cause