Home 🚀 - NuGet.targets(498,5): error : Could not find a part of the path '/tmp/NuGetScratch

@joshfree: Did this error occur on a DevOps Pipelines ubuntu-16.04 hosted agent? Because our build just hit this same error on that agent type:

2018-10-24T10:04:59.5271236Z ##[section]Starting: Pack
2018-10-24T10:04:59.5274583Z ==============================================================================
2018-10-24T10:04:59.5274652Z Task         : Command Line
2018-10-24T10:04:59.5274860Z Description  : Run a command line script using cmd.exe on Windows and bash on macOS and Linux.
2018-10-24T10:04:59.5274963Z Version      : 2.136.0
2018-10-24T10:04:59.5275008Z Author       : Microsoft Corporation
2018-10-24T10:04:59.5275055Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkID=613735)
2018-10-24T10:04:59.5275146Z ==============================================================================
2018-10-24T10:05:01.4261872Z Generating script.
2018-10-24T10:05:01.4280191Z Script contents:
2018-10-24T10:05:01.4280847Z dotnet pack --output /home/vsts/work/1/a
2018-10-24T10:05:01.4308839Z [command]/bin/bash --noprofile --norc /home/vsts/work/_temp/db10cd2b-38e8-402d-a8eb-ff9a976ad773.sh
2018-10-24T10:05:09.4566160Z Microsoft (R) Build Engine version 15.8.169+g1ccb72aefa for .NET Core
2018-10-24T10:05:09.4567596Z Copyright (C) Microsoft Corporation. All rights reserved.
2018-10-24T10:05:09.4567846Z 
2018-10-24T10:05:15.6662719Z /usr/share/dotnet/sdk/2.1.403/NuGet.targets(498,5): error : Could not find a part of the path '/tmp/NuGetScratch/6cd713f9-3dde-4d1f-b8cf-ebc068623d7e'. [/home/vsts/work/1/s/KeyVault/Microsoft.Adp.KeyVault/Microsoft.Adp.KeyVault.csproj]
2018-10-24T10:05:15.8885836Z ##[error]Bash exited with code '1'.
2018-10-24T10:05:15.9670564Z ##[section]Finishing: Pack

The agent should be using .NET Core SDK 2.1.403:

https://github.com/Microsoft/azure-pipelines-image-generation/blob/master/images/linux/Ubuntu1604-README.md

mikeharder on 24 Oct 2018

I'm not sure the details on the infra -- @mattgal should know

joshfree on 24 Oct 2018

Speak of the me and I shall appear. Looking.

MattGal on 24 Oct 2018

@mikeharder to answer your question, this specific job runs on a Jenkins agent whose VHD was made by @mmitche a bit over a year ago: (ubuntu1604-20170925-1) . I'm looking at the errors too to see if I can provide more useful data.

MattGal on 24 Oct 2018

@joshfree I can't find any other failures on this build def that both still have their logs available and have this problem. However, one possible cause if we ever get onto such a machine is that tmp can be over-filled on Linux and OSX machines, making further file operations within the /tmp folder fail. It seems very unlikely for the hosted agents though since they are reset every build.

MattGal on 24 Oct 2018

@MattGal: Thanks. It seems more likely this is a general issue with NuGet on Linux, rather than specific to one type of agent.

mikeharder on 24 Oct 2018

@mikeharder I agree and your data definitely makes it seem that way, I was just listing an issue with tmp folders I'd seen.

MattGal on 24 Oct 2018

@MattGal @joshfree @mikeharder Hello, we can see the same issue on macOS 10.13 hosted agent on our Azure DevOps pipeline (please check the failing mac builds):
https://dev.azure.com/EventStoreOSS/EventStore/_build/results?buildId=172&view=logs
https://dev.azure.com/EventStoreOSS/EventStore/_build/results?buildId=178&view=logs

I've seen it occur only on the macOS agent so far.

shaan1337 on 30 Oct 2018

Have same error in Azure DevOps, sometimes it works and sometimes the error is thrown:

2018-11-21T10:24:49.2978416Z ##[section]Starting: Publish debian.8-x64
2018-11-21T10:24:49.2984559Z ==============================================================================
2018-11-21T10:24:49.2984670Z Task         : .NET Core
2018-11-21T10:24:49.2984734Z Description  : Build, test, package, or publish a dotnet application, or run a custom dotnet command. For package commands, supports NuGet.org and authenticated feeds like Package Management and MyGet.
2018-11-21T10:24:49.2984840Z Version      : 2.141.5
2018-11-21T10:24:49.2984883Z Author       : Microsoft Corporation
2018-11-21T10:24:49.2985369Z Help         : [More Information](https://go.microsoft.com/fwlink/?linkid=832194)
2018-11-21T10:24:49.2985479Z ==============================================================================
2018-11-21T10:24:52.1271594Z [command]/usr/bin/dotnet publish /home/vsts/work/1/s/./src/ThemeEditor/ThemeEditor.csproj -r debian.8-x64 -c release -f netcoreapp2.1 --output /home/vsts/work/1/a/ThemeEditor-debian.8-x64
2018-11-21T10:25:02.3781145Z Microsoft (R) Build Engine version 15.8.169+g1ccb72aefa for .NET Core
2018-11-21T10:25:02.3782463Z Copyright (C) Microsoft Corporation. All rights reserved.
2018-11-21T10:25:02.3782560Z 
2018-11-21T10:25:10.5359446Z /usr/share/dotnet/sdk/2.1.403/NuGet.targets(498,5): error : Could not find a part of the path '/tmp/NuGetScratch/eeff630c-dee8-4721-b463-a036362df1b7'. [/home/vsts/work/1/s/src/ThemeEditor.ColorMatch/ThemeEditor.ColorMatch.csproj]
2018-11-21T10:25:10.9354596Z ##[error]Error: /usr/bin/dotnet failed with return code: 1
2018-11-21T10:25:10.9367646Z ##[error]Dotnet command failed with non-zero exit code on the following projects : /home/vsts/work/1/s/./src/ThemeEditor/ThemeEditor.csproj
2018-11-21T10:25:10.9521221Z ##[section]Finishing: Publish debian.8-x64

2018-11-14T11:18:01.9052681Z ##[section]Starting: Build
2018-11-14T11:18:01.9088743Z ==============================================================================
2018-11-14T11:18:01.9088823Z Task         : .NET Core
2018-11-14T11:18:01.9088890Z Description  : Build, test, package, or publish a dotnet application, or run a custom dotnet command. For package commands, supports NuGet.org and authenticated feeds like Package Management and MyGet.
2018-11-14T11:18:01.9088997Z Version      : 2.141.2
2018-11-14T11:18:01.9089041Z Author       : Microsoft Corporation
2018-11-14T11:18:01.9089134Z Help         : [More Information](https://go.microsoft.com/fwlink/?linkid=832194)
2018-11-14T11:18:01.9089191Z ==============================================================================
2018-11-14T11:18:04.9435631Z [command]/usr/bin/dotnet build /home/vsts/work/1/s/ThemeEditor.sln -c release --version-suffix -build475
2018-11-14T11:18:14.0739128Z Microsoft (R) Build Engine version 15.8.169+g1ccb72aefa for .NET Core
2018-11-14T11:18:14.0739913Z Copyright (C) Microsoft Corporation. All rights reserved.
2018-11-14T11:18:14.0740269Z 
2018-11-14T11:18:22.3191598Z /usr/share/dotnet/sdk/2.1.403/NuGet.targets(498,5): error : Could not find a part of the path '/tmp/NuGetScratch/3f61c39c-3ac7-4273-b42c-4766eed97213'. [/home/vsts/work/1/s/src/ThemeEditor.Colors/ThemeEditor.Colors.csproj]
2018-11-14T11:18:22.6100368Z 
2018-11-14T11:18:22.6101225Z Build FAILED.
2018-11-14T11:18:22.6101398Z 
2018-11-14T11:18:22.6102697Z /usr/share/dotnet/sdk/2.1.403/NuGet.targets(498,5): error : Could not find a part of the path '/tmp/NuGetScratch/3f61c39c-3ac7-4273-b42c-4766eed97213'. [/home/vsts/work/1/s/src/ThemeEditor.Colors/ThemeEditor.Colors.csproj]
2018-11-14T11:18:22.6102942Z     0 Warning(s)
2018-11-14T11:18:22.6103276Z     1 Error(s)
2018-11-14T11:18:22.6103366Z 
2018-11-14T11:18:22.6103894Z Time Elapsed 00:00:07.10
2018-11-14T11:18:22.6389494Z ##[error]Error: /usr/bin/dotnet failed with return code: 1
2018-11-14T11:18:22.6469131Z ##[error]Dotnet command failed with non-zero exit code on the following projects : /home/vsts/work/1/s/ThemeEditor.sln
2018-11-14T11:18:22.6686157Z ##[section]Finishing: Build



md5-173dca368c1caebb40f3cb662da4934b



2018-11-14T09:32:29.6394840Z ##[section]Starting: Build
2018-11-14T09:32:29.6406230Z ==============================================================================
2018-11-14T09:32:29.6406390Z Task         : .NET Core
2018-11-14T09:32:29.6406520Z Description  : Build, test, package, or publish a dotnet application, or run a custom dotnet command. For package commands, supports NuGet.org and authenticated feeds like Package Management and MyGet.
2018-11-14T09:32:29.6406700Z Version      : 2.141.2
2018-11-14T09:32:29.6406800Z Author       : Microsoft Corporation
2018-11-14T09:32:29.6406910Z Help         : [More Information](https://go.microsoft.com/fwlink/?linkid=832194)
2018-11-14T09:32:29.6407050Z ==============================================================================
2018-11-14T09:32:30.9736470Z [command]/usr/local/bin/dotnet build /Users/vsts/agent/2.141.1/work/1/s/ThemeEditor.sln -c release --version-suffix -build470
2018-11-14T09:32:31.5584800Z 
2018-11-14T09:32:31.5746640Z Welcome to .NET Core!
2018-11-14T09:32:31.5748670Z ---------------------
2018-11-14T09:32:31.5750210Z Learn more about .NET Core: https://aka.ms/dotnet-docs
2018-11-14T09:32:31.5751670Z Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli-docs
2018-11-14T09:32:31.5752240Z 
2018-11-14T09:32:31.5752840Z Telemetry
2018-11-14T09:32:31.5754110Z ---------
2018-11-14T09:32:31.5755900Z The .NET Core tools collect usage data in order to help us improve your experience. The data is anonymous and doesn't include command-line arguments. The data is collected by Microsoft and shared with the community. You can opt-out of telemetry by setting the DOTNET_CLI_TELEMETRY_OPTOUT environment variable to '1' or 'true' using your favorite shell.
2018-11-14T09:32:31.5766190Z 
2018-11-14T09:32:31.5767380Z Read more about .NET Core CLI Tools telemetry: https://aka.ms/dotnet-cli-telemetry
2018-11-14T09:32:31.5767990Z 
2018-11-14T09:32:31.5768430Z Configuring...
2018-11-14T09:32:31.5769260Z --------------
2018-11-14T09:32:31.5769810Z A command is running to populate your local package cache to improve restore speed and enable offline access. This command takes up to one minute to complete and only runs once.
2018-11-14T09:32:40.3413130Z Decompressing .......... 8726 ms
2018-11-14T09:32:58.5300210Z Expanding .......... 17929 ms
2018-11-14T09:33:00.2265090Z 
2018-11-14T09:33:00.2266040Z ASP.NET Core
2018-11-14T09:33:00.2267440Z ------------
2018-11-14T09:33:00.2268090Z Successfully installed the ASP.NET Core HTTPS Development Certificate.
2018-11-14T09:33:00.2268890Z To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only). For establishing trust on other platforms refer to the platform specific documentation.
2018-11-14T09:33:00.2269110Z For more information on configuring HTTPS see https://go.microsoft.com/fwlink/?linkid=848054.
2018-11-14T09:33:00.5491540Z Microsoft (R) Build Engine version 15.8.169+g1ccb72aefa for .NET Core
2018-11-14T09:33:00.5492040Z Copyright (C) Microsoft Corporation. All rights reserved.
2018-11-14T09:33:00.5492160Z 
2018-11-14T09:33:03.3186030Z /Users/vsts/.dotnet/sdk/2.1.403/NuGet.targets(498,5): error : Could not find a part of the path '/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/NuGetScratch/3a9cebe1-0621-494e-9022-0e6d5de323c3'. [/Users/vsts/agent/2.141.1/work/1/s/src/ThemeEditor.ColorMatch/ThemeEditor.ColorMatch.csproj]
2018-11-14T09:33:03.4405930Z 
2018-11-14T09:33:03.4407290Z Build FAILED.
2018-11-14T09:33:03.4408030Z 
2018-11-14T09:33:03.4410320Z /Users/vsts/.dotnet/sdk/2.1.403/NuGet.targets(498,5): error : Could not find a part of the path '/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/NuGetScratch/3a9cebe1-0621-494e-9022-0e6d5de323c3'. [/Users/vsts/agent/2.141.1/work/1/s/src/ThemeEditor.ColorMatch/ThemeEditor.ColorMatch.csproj]
2018-11-14T09:33:03.4411180Z     0 Warning(s)
2018-11-14T09:33:03.4411830Z     1 Error(s)
2018-11-14T09:33:03.4412620Z 
2018-11-14T09:33:03.4413270Z Time Elapsed 00:00:02.69
2018-11-14T09:33:03.4914590Z ##[error]Error: /usr/local/bin/dotnet failed with return code: 1
2018-11-14T09:33:03.4935490Z ##[error]Dotnet command failed with non-zero exit code on the following projects : /Users/vsts/agent/2.141.1/work/1/s/ThemeEditor.sln
2018-11-14T09:33:03.4988710Z ##[section]Finishing: Build

wieslawsoltes on 21 Nov 2018

Im also seeing this issue:

2018-12-04T13:00:37.8773693Z Microsoft (R) Build Engine version 15.9.20+g88f5fadfbe for .NET Core
2018-12-04T13:00:37.8774410Z Copyright (C) Microsoft Corporation. All rights reserved.
2018-12-04T13:00:37.8774634Z 
2018-12-04T13:00:45.3317432Z /usr/share/dotnet/sdk/2.1.500/NuGet.targets(499,5): error : Could not find a part of the path '/tmp/NuGetScratch/cc7d7c71-8d06-41b3-8285-a1ba4e929d1b'. [/home/vsts/work/1/s/microservices/orders/Ctrade.Orders.Functions/Ctrade.Orders.Functions.csproj]
2018-12-04T13:00:45.7259485Z 
2018-12-04T13:00:45.7260233Z Build FAILED.
2018-12-04T13:00:45.7261293Z 
2018-12-04T13:00:45.7262158Z /usr/share/dotnet/sdk/2.1.500/NuGet.targets(499,5): error : Could not find a part of the path '/tmp/NuGetScratch/cc7d7c71-8d06-41b3-8285-a1ba4e929d1b'. [/home/vsts/work/1/s/microservices/orders/Ctrade.Orders.Functions/Ctrade.Orders.Functions.csproj]
2018-12-04T13:00:45.7262265Z     0 Warning(s)
2018-12-04T13:00:45.7262311Z     1 Error(s)
2018-12-04T13:00:45.7262357Z 
2018-12-04T13:00:45.7262397Z Time Elapsed 00:00:06.60
2018-12-04T13:00:45.7631370Z ##[error]Error: /usr/bin/dotnet failed with return code: 1
2018-12-04T13:00:45.7642832Z ##[error]Dotnet command failed with non-zero exit code on the following projects : /home/vsts/work/1/s/microservices/orders/Ctrade.Orders.sln
2018-12-04T13:00:45.7662644Z ##[section]Finishing: dotnet build

We are building a .net core project on the Azure Devops hosted Ubuntu image. Generally if you re-trigger the build the problem goes away for a bit, but this is not ideal in a ci environment

MarkDordoy on 4 Dec 2018

We are also seeing the same issue. And as @MarkDordoy said re-triggering the build it works, but it's breaking all our CI environment.

2018-12-11T20:35:25.9180039Z /usr/share/dotnet/sdk/2.1.500/NuGet.targets(499,5): error : Could not find a part of the path '/tmp/NuGetScratch/7f908911-2790-486f-81d6-d15c46d49a04'.
2018-12-11T20:35:25.9182545Z
2018-12-11T20:35:25.9184761Z 0 Warning(s)
2018-12-11T20:35:25.9187288Z 1 Error(s)
2018-12-11T20:35:25.9189867Z
2018-12-11T20:35:25.9191718Z Time Elapsed 00:00:07.81
2018-12-11T20:35:25.9893016Z ##[error]Error: /usr/bin/dotnet failed with return code: 1
2018-12-11T20:35:25.9903842Z ##[error]Packages failed to restore
2018-12-11T20:35:26.0016860Z ##[section]Finishing: Restore

rmsryu on 12 Dec 2018

Is it also reproducible on dev box instead if Azure DevOps? and Can someone provide me verbose logs for the issue?

jainaashish on 12 Dec 2018

Could any of you that can reproduce the issue often enough try redirecting TMPDIR to a user-local folder before running restore, and report back to us whether you still see the problem.

nkolev92 on 19 Dec 2018

@nkolev92 I can confirm that that works for me.

(rm -rf /tmp/NuGetScratch, then ln -s /tmp/NuGetScratch <a user-local folder>, then nuget install works)

aveao on 28 Dec 2018

Thanks @aveao
I'd imagine it's a permissions issue.
Do you ever use nuget.exe on mono on the machine in question?

nkolev92 on 28 Dec 2018

I use the aur package "nuget-stable" and use msbuild to build mono stuff. So… I assume that's what it is.

@nkolev92

aveao on 28 Dec 2018

I have a fix for nuget.exe under mono that could be causing this issues in https://github.com/NuGet/NuGet.Client/pull/2691.
I was not able to find any specific issues in the dotnet.exe codepaths.

nkolev92 on 24 Jan 2019

Issue occurs periodically. Run build again solves issue. Using "Hosted Ubuntu 1604" agent.
Message:
2019-01-27T22:22:22.1440448Z /usr/share/dotnet/sdk/2.2.101/NuGet.targets(499,5): error : Could not find a part of the path '/tmp/NuGetScratch/d102ea94-1214-4dd5-a627-d49d3fa8f937'. [/home/vsts/work/1/s/src/SF.DA.BLL/SF.DA.BLL.csproj]
2019-01-27T22:22:22.7437026Z ##[error]Error: /usr/bin/dotnet failed with return code: 1

stas-lucky on 27 Jan 2019

@stas-lucky Have you ever used nuget.exe on mono on the machine where the problems happen?

nkolev92 on 28 Jan 2019

I am seeing this issue on our builds of Azure PowerShell on both the public hosted Linux and macOS workers in ADO. It seems to happen randomly.

2019-01-29T20:41:10.4786930Z /Users/vsts/.dotnet/sdk/2.2.102/NuGet.targets(499,5): error : Could not find a part of the path '/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/NuGetScratch/a88f0d26-af5d-48f7-9acf-bab94e987605'. [/Users/vsts/agent/2.144.2/work/1/s/src/Accounts/Authentication/Authentication.csproj] [/Users/vsts/agent/2.144.2/work/1/s/build.proj]

MiYanni on 29 Jan 2019

👍1

Same issue. Seems to be random, usually works when I rerun it. Using Azure Devops as well.
Does anyone have a fix?

rachelleahr on 30 Jan 2019

@nkolev92 fixed it for Mono in 5.0. Couldn't find anything involving dotnet.exe code paths.
Top suspect is now old tools running on your machine (and yes, we haven't released 5.0 final yet.)
Upgrade all your dotnet sdks...nuget.exes, etc... if you still hit this please let us know.

rrelyea on 11 Feb 2019

We are still hitting this error pretty regularly on Azure DevOps Pipelines Ubuntu Hosted Agents. Here's a failure from this morning.

dotnet pack --output /home/vsts/work/1/a

[command]/bin/bash --noprofile --norc
/home/vsts/work/_temp/045fd6ff-93e9-4d15-8355-70158fa521aa.sh

Microsoft (R) Build Engine version 15.9.20+g88f5fadfbe for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

/usr/share/dotnet/sdk/2.2.103/NuGet.targets(499,5): 
error : Could not find a part of the path 
'/tmp/NuGetScratch/33261173-cbea-49db-b2c3-730de7efa5b9'. 
[/home/vsts/work/1/s/Template/Microsoft.Azure.Template.Test/Microsoft.Azure.Template.Test.csproj]

I believe we are using the latest GA SDK (2.2.103). Would you like us to try a preview to see if it fixes the issue?

Here's the full list of software installed on the agent:

https://github.com/Microsoft/azure-pipelines-image-generation/blob/master/images/linux/Ubuntu1604-README.md

mikeharder on 11 Feb 2019

👍3

we shipped this fix in 16.0-preview3. Our bits with this fix should have been included in the previews that were released when 16.0p3 were released (2.1.600 and 2.2.200 i believe)

rrelyea on 13 Feb 2019

@rrelyea: Thanks, I will try to repro this error with the latest LTS GA (2.1.504), by running a small build in a loop. Then I will change to 2.1.600-preview-009426 and see if it prevents the error.

According to the release notes, SDK 2.1.600-preview-009426 contains NuGet 5.0.0-preview1.5663. Does this sound correct?

mikeharder on 13 Feb 2019

external feedback of similar issue, but it went away: https://developercommunity.visualstudio.com/content/problem/447569/azure-pipelines-nuget-restore-errors-when-trying-t.html

rrelyea on 15 Feb 2019

I attempted to reproduce the failure by repeatedly running dotnet pack (the command we see fail in our main builds) on a hello-world console app, but I wasn't able to repro after hundreds of builds.

In our main builds, we see this fail maybe once a week, and the most recent was 3 days ago on 2/11/19. We'll keep tracking to see if stops happening completely. I suspect this issue is currently still active, but it only repros extremely rarely.

mikeharder on 15 Feb 2019

@mikeharder

Are your build machines shared?

Also I wouldn't necesarilly assume that pack is the problem. I'd guess that the directory was created with bad permissions before pack tried to use it. So the faulty code path could be in restore.

nkolev92 on 15 Feb 2019

@nkolev92: We are using the DevOps Hosted Agents. This is a large pool of agents completely managed by the DevOps service and shared by all customers. Each agent is only used by one build at a time, and agents are completely reset (or re-provisioned) between builds. So every build starts on a clean agent (which should be in exactly the same state every time), and the build has exclusive access to do whatever it wants to the agent.

I agree the faulty code path is most likely in restore (and we might be able to know for sure by finding the error line in NuGet.targets). But I wanted to keep using pack since it's the same command where we see this fail in our main builds.

Nothing else in our build pipeline is running in parallel with restore. It's possible the DevOps Hosted Agent is running something in the background (like trying to clean up /tmp) which causes this, but if this were the case, I would expect it to break many tools which rely on /tmp, not just NuGet.

mikeharder on 15 Feb 2019

This is happening repeatedly right now on one leg of CLI 2.2 CI. @MattGal can you share a machine with @nkolev92 ?

nguerrera on 16 Feb 2019

I can confirm we are seeing this randomly on ADO hosted agents (ubuntu 16.04). I can provide information if needed.

4c74356b41 on 28 Feb 2019

We also see this seemingly randomly on ADO hosted agents (ubuntu 16.04)

sonofaforester on 5 Mar 2019

ASP.NET/EF Core builds have been hitting this a lot on Mac as well.

Here's a publicly available build where this happened: https://dnceng.visualstudio.com/public/_build/results?buildId=115083

It's using this recent 3.0 build: https://dotnetcli.azureedge.net/dotnet/Sdk/3.0.100-preview4-010612/dotnet-sdk-3.0.100-preview4-010612-osx-x64.tar.gz

Eilon on 8 Mar 2019

From @bricelam:

We started seeing NuGet/Home#7341 after upgrading the .NET Core SDK from 3.0.100-preview-010010 to 3.0.100-preview4-010612. The corresponding NuGet versions are 5.0.0-preview1.5663+7bdbabc8a5038373ef972363beff6afff50e2469 and 5.0.0-rtm.5821+cf25cc76caccfa6f68e4c74d7d326fef9f9aa3d4.

Eilon on 13 Mar 2019

Mike's comment suggests 2.2.100 https://github.com/NuGet/Home/issues/7341#issuecomment-462465001, which contains 4.9.2.

So I'd imagine there's an issue in builds earlier than the commit list by Brice.
It could be that we have 2 different issues though.

https://docs.microsoft.com/en-us/nuget/release-notes/nuget-4.9-rtm

@zivkan summarized it well in an e-mail threads.

Things that you can do to help are:

stat /tmp/NuGetScratch
whoami

could be run as part of the build, and when the build fails, to check the output of these commands to see if the build agent account has permissions to write to the directory or not.

If you have a build that already failed and you know what account the build agent runs as, if you can log into the machine to check /tmp/NuGetScratch, you can find the relevant information.

Providing us access to those specific machines also works.

nkolev92 on 13 Mar 2019

It'd also be helpful to run restore /v:d, to allow us to see the full stack trace.

nkolev92 on 14 Mar 2019

Here's a binlog for the error we're seeing on macOS.

Build.zip

System.IO.DirectoryNotFoundException: Could not find a part of the path '/Users/vsts/agent/2.148.1/work/1/s/artifacts/obj/70d88d93-1df8-47b8-8c56-447722545fd1'.
   at System.IO.FileSystem.CreateDirectory(String fullPath)
   at System.IO.Directory.CreateDirectory(String path)
   at NuGet.Common.DirectoryUtility.CreateSingleSharedDirectory(String path)
   at NuGet.Common.DirectoryUtility.CreateSharedDirectory(String path)
   at NuGet.Commands.NoOpRestoreUtilities.PersistDGSpecFile(DependencyGraphSpec spec, RestoreRequest request, ILogger log)
   at NuGet.Commands.NoOpRestoreUtilities.GetHash(RestoreRequest request)
   at NuGet.Commands.RestoreCommand.EvaluateCacheFile()
   at NuGet.Commands.RestoreCommand.ExecuteAsync(CancellationToken token)
   at NuGet.Commands.RestoreRunner.ExecuteAsync(RestoreSummaryRequest summaryRequest, CancellationToken token)
   at NuGet.Commands.RestoreRunner.ExecuteAndCommitAsync(RestoreSummaryRequest summaryRequest, CancellationToken token)
   at NuGet.Commands.RestoreRunner.CompleteTaskAsync(List`1 restoreTasks)
   at NuGet.Commands.RestoreRunner.RunAsync(IEnumerable`1 restoreRequests, RestoreArgs restoreContext, CancellationToken token)
   at NuGet.Commands.RestoreRunner.RunAsync(RestoreArgs restoreContext, CancellationToken token)
   at NuGet.Build.Tasks.RestoreTask.ExecuteAsync(ILogger log)

bricelam on 20 Mar 2019

That stacktrace suggests that NuGet cannot write to the obj folder, which does not make a lot of sense.

@bricelam Would you be willing to patch your dotnet CLI with NuGet bits that contain extra logging information?

fyi @dtivel

nkolev92 on 22 Mar 2019

@stas-lucky Have you ever used nuget.exe on mono on the machine where the problems happen?

I have never used Mono. The problem occurs on Azure DevOps agent, and it seems to be random. Run build again solves the issue

stas-lucky on 22 Mar 2019

Just the give you all an update on the investigation:

The symptoms of the problem are rather generic. The majority of reports are coming in about the /tmp/NuGetScratch folder, but I have not gotten a repro for those.

There's a newer problem that's not related to the /tmp/NuGetScratch directory that @bricelam reported that I have a fix for.

More info at:

https://github.com/NuGet/Home/issues/7908
https://github.com/NuGet/NuGet.Client/pull/2774

To help with the investigation of the /tmp/NuGetScratch failures, please provide logs with /v:d, binlog and the output of the below commands.
stat /tmp/NuGetScratch
whoami

nkolev92 on 25 Mar 2019

👎2

you even have a link to the public build with this error. there is nothing to repro. just do a dotnet build. this all happens on hosted agents, so running those commands after the error is a bit impossible.

4c74356b41 on 26 Mar 2019

👍1

so no luck on the /tmp/nugetscratch issue? it's rather annoying/random and as others have said for the most part as we dont own the build machines this is outside our control. BTW, in our case this reproes during 'dotnet restore'

joseccruz on 27 Mar 2019

@joseccruz
A binlog would be helpful.

I'm digging into some potentially related code on the side, but it's hard to predict whether that's truly the problem without more information or a repro.

nkolev92 on 27 Mar 2019

+1 on this, happens intermittently for us on CI builds on Ubuntu.

rix0rrr on 1 Apr 2019

👍2

Just ran into this _again_. It's been happening for a long time now. Same issue as everyone else: Ubuntu 16.04 Microsoft-hosted build agent.

   2:5>/usr/share/dotnet/sdk/2.2.104/NuGet.targets(499,5): error : Could not find a part of the path '/tmp/NuGetScratch/771dc244-8fa7-4261-9727-b0ff3a1d29eb'. [/home/vsts/work/1/s/src/NetStandard20/NetStandard20.csproj]
   2:5>System.IO.DirectoryNotFoundException: Could not find a part of the path '/tmp/NuGetScratch/771dc244-8fa7-4261-9727-b0ff3a1d29eb'.
   at System.IO.FileSystem.CreateDirectory(String fullPath)
   at System.IO.Directory.CreateDirectory(String path)
   at NuGet.Common.DirectoryUtility.CreateSingleSharedDirectory(String path)
   at NuGet.Common.DirectoryUtility.CreateSharedDirectory(String path)
   at NuGet.Common.ConcurrencyUtilities.get_BasePath()
   at NuGet.Common.ConcurrencyUtilities.FileLockPath(String filePath)
   at NuGet.Common.ConcurrencyUtilities.ExecuteWithFileLocked(String filePath, Action action)
   at NuGet.Configuration.SettingsFile.ExecuteSynchronized(Action ioOperation)
   at NuGet.Configuration.SettingsFile..ctor(String directoryPath, String fileName, Boolean isMachineWide)
   at NuGet.Configuration.Settings.ReadSettings(String settingsRoot, String settingsPath, Boolean isMachineWideSettings)
   at NuGet.Configuration.Settings.LoadUserSpecificSettings(String root, String configFileName, Boolean useTestingGlobalPath)
   at NuGet.Configuration.Settings.LoadSettingsForSpecificConfigs(String root, String configFileName, List`1 validSettingFiles, IMachineWideSettings machineWideSettings, Boolean loadUserWideSettings, Boolean useTestingGlobalPath)
   at NuGet.Configuration.Settings.LoadSettings(String root, String configFileName, IMachineWideSettings machineWideSettings, Boolean loadUserWideSettings, Boolean useTestingGlobalPath)
   at NuGet.Configuration.Settings.LoadDefaultSettings(String root, String configFileName, IMachineWideSettings machineWideSettings)
   at NuGet.Build.Tasks.RestoreSettingsUtils.ReadSettings(String solutionDirectory, String restoreDirectory, String restoreConfigFile, Lazy`1 machineWideSettings)
   at NuGet.Build.Tasks.GetRestoreSettingsTask.Execute()

nathan-alden-sr on 3 Apr 2019

👀1

+1 here

eladb on 3 Apr 2019

looks like is are the only ones not using their hosted agents ;) pretty smart, eh? xD

4c74356b41 on 3 Apr 2019

Update

I have another fix in the related code path in https://github.com/NuGet/NuGet.Client/pull/2795.
Note that while I am confident this will fix at least some of the issues reported here, I have no way of confirming that, so I will rely on feedback from you.

I will update again whenever this fix makes it in the previews of (2.1.700, 2.2.300 & 3.0.100).

nkolev92 on 5 Apr 2019

❤1 👍1

I've just started playing around with ADO hosted agents and I hit this pretty quickly. I don't want to tie up one of my agents we're paying for but I imagine if I just ran the same build on a schedule every 10 minutes with some jitter it would fail several times a day. I encountered it on the 10th build or so.

lukeschlather on 8 Apr 2019

The fix will be in
2.1.700 - preview 2
2.2.300 - preview 2
3.0.100 - preview 4

If/When you upgrade to these versions, let me know if you still see issues.

nkolev92 on 18 Apr 2019

🎉3 ❤1

I'm also seeing this intermittently on Azure Devops.

@nkolev92 are those version numbers of .NET Core? If so, how do I set which version to use with that precision? I'm only aware of how to choose the major and minor version, using the csproj file (<TargetFramework>netcoreapp2.2</TargetFramework>).

johnknoop on 5 May 2019

@johnknoop

That's the .NET Core SDK version. It's what allows you to build the application.
The TargetFramework is about where it can run.

It's something you need to update on the specific machines (Azure Pipelines has an install task + you can use global.json to pin the SDK version).

Hope the following resources help.

https://docs.microsoft.com/en-us/dotnet/core/sdk
https://docs.microsoft.com/en-us/dotnet/core/tools/global-json
https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/tool/dotnet-core-tool-installer?view=azure-devops

nkolev92 on 6 May 2019

👍1

Has anyone seen issues with:

2.1.700
2.2.300
3.0.100

Thanks!

nkolev92 on 7 Jun 2019

At this point, I am closing this issue as fixed.
If you still hit problems, let us know and we'll be happy to investigate.

nkolev92 on 1 Jul 2019

When will azure pipelines use 2.2.300 or higher be default? I don't really want to pin a specific sdk version in all of my repositories

Trolldemorted on 15 Jul 2019

👍1

got the same (similar?) error with monodevelop

https://github.com/mono/monodevelop/issues/8320

lytico on 2 Aug 2019

Why was the issue closed. Pls reopen it. This issue is faced constantly in our ado.

Tenseiga on 27 Sep 2019

Home: NuGet.targets(498,5): error : Could not find a part of the path '/tmp/NuGetScratch

Most helpful comment

All 57 comments

Related issues