Azure-pipelines-agent: Multiple agents on the same machine using tf.exe can lead to getting sources failure

[JamesNK]

A build is intermittently getting this error when fetching source from TFS:

2017-03-23T23:49:31.0591599Z ##[section]Starting: Build solution Roster.WebService.sln
2017-03-23T23:49:31.0591599Z ==============================================================================
2017-03-23T23:49:31.0591599Z Task         : Visual Studio Build
2017-03-23T23:49:31.0591599Z Description  : Build with MSBuild and set the Visual Studio version property
2017-03-23T23:49:31.0591599Z Version      : 1.113.0
2017-03-23T23:49:31.0591599Z Author       : Microsoft Corporation
2017-03-23T23:49:31.0591599Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkID=613727)
2017-03-23T23:49:31.0591599Z ==============================================================================
2017-03-23T23:49:31.6529233Z Unable to determine the workspace. You may be able to correct this by running 'tf workspaces /collection:TeamProjectCollectionUrl'.
2017-03-23T23:49:31.8872843Z ##[error]Exit code 100 returned from process: file name 'tf', arguments 'vc resolvePath "$\My Development\Trunk\src\Rostering\trunk\Roster.WebService.sln" /loginType:OAuth /login:.,******** /noprompt'.
2017-03-23T23:49:31.8872843Z ##[section]Finishing: Build solution Roster.WebService.sln

Copied from https://github.com/Microsoft/vsts-tasks/issues/3853

[ericsciple]

If you can catch it while system.debug is set, the logs will have more details.

[ericsciple]

@JamesNK Does this only happen when using the hosted build agent? And do you have more than one hosted build agent?

[bootrider]

I first installed the agent1, it shows this error, I had to manual perform a "get-latest version" operation in the workspace created by the agent, then I change the workspace to Server (a lot of projects and a lot of files).
then I run the following script in the build server:

Add-type -AssemblyName "Microsoft.TeamFoundation.Client, Version=12.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
Add-type -AssemblyName "Microsoft.TeamFoundation.Common, Version=12.0.0.0, Culture=Neutral, PublicKeyToken=b03f5f7f11d50a3a"
Add-type -AssemblyName "Microsoft.TeamFoundation.VersionControl.Client, Version=12.0.0.0, Culture=Neutral, PublicKeyToken=b03f5f7f11d50a3a"
Add-type -AssemblyName "Microsoft.TeamFoundation.VersionControl.Common, Version=12.0.0.0, Culture=Neutral, PublicKeyToken=b03f5f7f11d50a3a"

# work folder
Push-Location C:\_Agent1\1\s\

$teamProjectCollection="http://tfs.roseninspection.net:8080/tfs/Bogcollection"

$tfsTeamProjectCollection= New-Object Microsoft.TeamFoundation.Client.TfsTeamProjectCollection($teamProjectCollection)

$versioncontrolServer = $tfsTeamProjectCollection.GetService([Microsoft.TeamFoundation.VersionControl.Client.VersionControlServer])

# Afolder or a file in the workspace
$localReference = "C:\_Agent1\1\s\Environment"

[Microsoft.TeamFoundation.VersionControl.Client.Workstation]::Current.EnsureUpdateWorkspaceInfoCache($versionControlServer, $env:USERNAME); 

$workspace = $versioncontrolServer.GetWorkspace($localReference)
[Microsoft.TeamFoundation.VersionControl.Client.Workstation]::Current.GetLocalWorkspaceInfo($localReference)
$workspace.GetServerItemForLocalItem($localReference)

Pop-Location

now is working perfectly

[ericsciple]

@bootrider it sounds like you are hitting a different issue (probably the Get sources not downloading some files authz problem. If that doesn't resolve please open a separate issue.

[ericsciple]

@JamesNK I just merged a fix for a TFVC issue that affects folks using the multiple hosted build agent in parallel. I suspect this may be related to the error you are experiencing.

It would be great if you can confirm whether the bug you are seeing only manifests when using multiple hosted agents in parallel.

The fix will roll out to all scale units within the next couple weeks.

Details about the bug: there was a change to the hosted images so the machine names are not always unique. This introduces a race condition for TFVC when using multiple hosted agents. In your case my suspicion (untested hypothesis) is that a workspace is being deleted by a parallel build, and causing issues the issue.

[JamesNK]

Hi

In our case we are running two agents on the same Azure VM (with different Agent and working directories so they don't conflict with each other).

Is it the same bug?

[ericsciple]

@JamesNK thanks for confirmation. Nope this sounds like a different bug.

I haven't heard of anyone else running into this issue. So my guess is that it has something to do with multiple agents on the same machine (running TFVC builds). I'll talk with folks from the version control team next week to see if they have heard of this issue.

Assuming the issue is related to multiple agents on the same machine... There are a couple changes we want to make for TFVC that might mitigate the issue you are seeing.
1) We want to redirect the local cache folder away from C:\programdata and into the _work folder. Mainly we want to do this for better cleanup. I'll poke through TFVC code a little bit next week to see if this is low hanging fruit.
2) We want to offer the option on the build to instruct the agent to create a server workspace instead of local. Server workspaces makes more sense for CI, but we ran into issues when we tried to switch (some tasks and user scripts assumed files were readonly). So it needs to be opt-in. My thinking is that server workspaces may not have the bug... although it may as well.

Again, I'll talk with TFVC folks next week to see if they know anything about this issue.

[bootrider]

Unfortunately my issue is related to server workspaces​ ( at least i think that) so far i have installed 3 agents in the same machine. Every agent has it's own workspace folder and do not overlap each other

[ericsciple]

@JamesNK I'm trying to think of some workarounds...

Assuming you run the agents as a Windows service (and not interactively from command prompt), one thing that could mitigate is running the agents as different service accounts. My best guess right now is that some race condition is stomping on some local file.

If you don't have complex mappings, another thing you could try is specify the file name as a relative path. The agent will prepend the build sources directory (e.g. C:\Agent1\_work\1\s). Or use $(Build.SourcesDirectory)/relative/path/to/file.ext.

Also I'm planning to add support to opt-in to server workspaces, by setting a variable on the definition (something like Build.TfvcWorkspaceType=Server). I can add that to the next agent release if you want to try it, and see if the issue goes away. My thinking is XAML build supported side-by-side agents and this issue afaik wasn't a problem.

I'll talk with VC folks again to see if they have ideas how to get to the bottom of this issue.

[michha]

We are facing the same error in our on-premise environment. 4 Agents (v2.112.0) are running on the same machine. We have some scheduled builds and two of them are set to run at the same time (6 am). Altough they build the same TFS project, each agent has a different working directory. About twice a week, one of these builds (its not the same build every time) fails with the messages below (system.debug = true).

2017-04-26T04:01:43.0575150Z ##[section]Starting: Apply transforms $(build.sourcesdirectory)\.configs\XXXDev.config => $(build.artifactstagingdirectory)\website\web.config
2017-04-26T04:01:43.0575150Z ==============================================================================
2017-04-26T04:01:43.0575150Z Task         : XDT Transform
2017-04-26T04:01:43.0575150Z Description  : Apply XDT transforms on XML files
2017-04-26T04:01:43.0575150Z Version      : 2.0.0
2017-04-26T04:01:43.0575150Z Author       : Guillaume Rouchon
2017-04-26T04:01:43.0575150Z Help         : v2.0.0, [More Information](https://github.com/qetza/vsts-xdttransform-task#readme)
2017-04-26T04:01:43.0575150Z ==============================================================================
2017-04-26T04:01:43.0575150Z ##[debug]tf vc resolvePath $\XXX\Source /loginType:OAuth /login:.,******** /noprompt
2017-04-26T04:01:43.6043790Z ##[debug]Unable to determine the workspace. You may be able to correct this by running 'tf workspaces /collection:TeamProjectCollectionUrl'.
2017-04-26T04:01:43.6199909Z Unable to determine the workspace. You may be able to correct this by running 'tf workspaces /collection:TeamProjectCollectionUrl'.
2017-04-26T04:01:43.6512403Z ##[error]Exit code 100 returned from process: file name 'tf', arguments 'vc resolvePath $\XXX\Source /loginType:OAuth /login:.,******** /noprompt'.
2017-04-26T04:01:43.6512403Z ##[debug]Microsoft.VisualStudio.Services.Agent.ProcessExitCodeException: Exit code 100 returned from process: file name 'tf', arguments 'vc resolvePath $\XXX\Source /loginType:OAuth /login:.,******** /noprompt'.
   at Microsoft.VisualStudio.Services.Agent.Worker.Build.TfsVCCommandManager.<RunPorcelainCommandAsync>d__28.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.Agent.Worker.Build.TFCommandManager.ResolvePath(String serverPath)
   at Microsoft.VisualStudio.Services.Agent.Worker.Build.TfsVCSourceProvider.GetLocalPath(IExecutionContext executionContext, ServiceEndpoint endpoint, String path)
   at Microsoft.VisualStudio.Services.Agent.Worker.Build.BuildJobExtension.GetRootedPath(IExecutionContext context, String path)
   at Microsoft.VisualStudio.Services.Agent.Worker.TaskRunner.TranslateFilePathInput(String inputValue)
   at Microsoft.VisualStudio.Services.Agent.Worker.TaskRunner.<RunAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.Agent.Worker.StepsRunner.<RunAsync>d__0.MoveNext()
2017-04-26T04:01:43.6512403Z ##[section]Finishing: Apply transforms $(build.sourcesdirectory)\.configs\XXXDev.config => $(build.artifactstagingdirectory)\website\web.config

[ericsciple]

@michha thanks, that gives more evidence this likely a TFVC issue due to multiple agents on the same machine. a quick workaround would be to stagger the times.

[michha]

@ericsciple will this be fixed in the next v2 Agent version?

[TingluoHuang]

agent with fix should be deployed everywhere.

[TingluoHuang]

this haven't been fixed, thought a different issue. :(

[axelheer]

Will this get fixed with TFS 2017 Update 2?

[bryanmacfarlane]

@ericsciple @TingluoHuang Is there a QU2 bug logged for TFSVC? This is an issue in tf.exe and I want to make sure it gets some traction. (tfsvc issues aren't open but at least we can ref progress)

[ericsciple]

@bryanmacfarlane 987895

[ericsciple]

@JamesNK @michha @axelheer please send ersciple a build log if you have a failed build (at microsoft com). I'm working with the VC team and we have an app to run the commands in a loop but so far not getting a repro. We tried create/get/delete/resolvePath in a loop in two separate processes for an hour.

What I'm wondering is what combination of tf commands are running during your builds. For instance, is the workspace created new everytime? If not, are you running a clean build (i.e. is scorch running)?

[michha]

I will try to reproduce it during the next days (we worked around this issue by spreading our build start times) and I will report here again, when the e-mail is sent.

[ericsciple]

@michha sorry i thought i updated this issue. we found the bug. right now i'm waiting until it makes it into a stable branch so i can bundle a new copy of tf and all dependencies. probably a couple weeks before it goes out with an agent release.

[michha]

Thank you for the update. When the new agent is available I will deploy it to our environment and confirm if this bug is fixed.

[jmsvl]

Hello,

I 'm also on a project with several agents on the same build server and with the previous version (116, 117) and this new version (119) I's still getting the messages below every once in a while.

2017-06-29T12:44:20.3873620Z ##[command]tf vc scorch D:\b\a4_work\8\s /recursive /diff /unmapped /loginType:OAuth /login:.,** /noprompt
2017-06-29T12:44:22.6494345Z No appropriate mapping exists for D:\b\a4_work\8\s.

In this build server there's also other older agents that execute XAML Builds.

Can you help?

Thank you

[ericsciple]

Work is in progress to get a new agent that carries the updated tf.exe and dependency DLLs (pr https://github.com/Microsoft/vsts-agent/pull/1056)

[jmsvl]

Thanks for the update

[michha]

After installing TFS 2017 Update 2 and updating the agent from version 2.112.0 to 2.117.2 the problem persists.

@ericsciple: could we have a a minimum TFS agent version or TFS Update number where this fix will be available

[ericsciple]

@michha The fix is in master if you want to build master and try it. One more small change (not a fix, command line interface change) is needed before we release the next agent. The fix will be in the 2.120.0 agent which will be released early next week.

[michha]

Thanks, I will wait for the 120 release.

P.S.: make more use of https://github.com/Microsoft/vsts-agent/milestones to spend less time answering those little questions ;)

[goudarzi]

+1 We are having the exact same issue - more of it recently. Is there a solution around these errors? We keep having random build failures, it's frustrating for the team. Thanks.

Adding more information: We are using VSTS Online (So the on-prem solution might not apply here.)

[ericsciple]

https://github.com/Microsoft/vsts-agent/releases/tag/v2.120.0

[pflangos]

Currently, release 2.120.0 shows as a Pre-release, when will it be considered as a full release i.e. ready for use in production environments?

[ericsciple]

It's being used in production environments now. It just rolled to the first ring. It will continue rolling out to all of the rings over the next several days.

[ericsciple]

...investigating a bug in the 2.120.0 agent. Wait for 2.120.1.

[nero2001]

Hi everyone,

I still have occasional errors in getting sources, while using agent 2.120.1..

"
Exit code 100 returned from process: file name 'tf', arguments 'vc workspace /new /location:local /permission:Public ws_4_69 /collection:https://mediahub.visualstudio.com/ /loginType:OAuth /login:.,** /noprompt'.
"

Thanks!

[ericsciple]

@nero2001 the output (stdout/stderr) from tf workspace /new ... should be in the log. Did it output anything?

[nero2001]

Hi @ericsciple, is this what you want?

2017-08-10T10:46:07.7176994Z ##[section]Starting: Get Sources
2017-08-10T10:46:08.0141146Z Prepending Path environment variable with directory containing 'tf.exe'.
2017-08-10T10:46:08.0141146Z Setting environment variable TFVC_BUILDAGENT_POLICYPATH
2017-08-10T10:46:08.0141146Z Querying workspace information.
2017-08-10T10:51:49.5936306Z ##[command]tf vc scorch D:\b\ag2_work\4\s /recursive /diff /unmapped /loginType:OAuth /login:.,* /noprompt
2017-08-10T10:51:52.1521618Z No appropriate mapping exists for D:\b\ag2_work\4\s.
2017-08-10T10:51:52.1677626Z ##[warning]Exit code 100 returned from process: file name 'tf', arguments 'vc scorch D:\b\ag2_work\4\s /recursive /diff /unmapped /loginType:OAuth /login:.,* /noprompt'.
2017-08-10T10:51:52.1989642Z ##[command]tf vc workspace /delete ws_4_69;93855f07-66c0-4079-9936-550abd63d390 /collection:https://mediahub.visualstudio.com/ /loginType:OAuth /login:.,* /noprompt
2017-08-10T10:56:10.6418170Z ##[command]tf vc workspace /new /location:local /permission:Public ws_4_69 /collection:https://mediahub.visualstudio.com/ /loginType:OAuth /login:.,* /noprompt
2017-08-10T10:57:23.1699362Z The path D:\b\ag2_work\4\s\dev\BIZ\Components\Bam is already mapped in workspace ws_4_69.
2017-08-10T10:57:23.2323394Z ##[error]Exit code 100 returned from process: file name 'tf', arguments 'vc workspace /new /location:local /permission:Public ws_4_69 /collection:https://mediahub.visualstudio.com/ /loginType:OAuth /login:.,** /noprompt'.
2017-08-10T10:57:23.2323394Z ##[section]Finishing: Get Sources

[ericsciple]

Did the computer name change? tf workspace /delete was successful, so server registration should be gone. It additionally should remove the workspace metadata on your local computer.

[nero2001]

No, it did not change.. And the error only occurs sometimes..

Em 10/08/2017 17:37, "ericsciple" [e-mail] escreveu:

Did the computer name change? tf workspace /delete was successful, so
server registration should be gone. It additionally should remove the
workspace metadata on your local computer.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/vsts-agent/issues/898#issuecomment-321605875,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAg7frgo3aSGVrREt953n6reBWBodoEiks5sWzGtgaJpZM4MyGX1
.

[ericsciple]

@nero2001 thanks I just discussed with a coworker. We think we know where to look for the specific issue you hit (sounds like another race condition since you are running multiple agents on the same box). You are running multiple agents on the same machine, correct?

[nero2001]

Yes indeed.

Em 11/08/2017 22:30, "ericsciple" [e-mail] escreveu:

@nero2001 https://github.com/nero2001 thanks I just discussed with a
coworker. We think we know where to look for the specific issue you hit
(sounds like another race condition since you are running multiple agents
on the same box). You are running multiple agents on the same machine,
correct?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/vsts-agent/issues/898#issuecomment-321923612,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAg7fg-C_Q0H7P5uIcHJWwfbgjxbdaTZks5sXMf4gaJpZM4MyGX1
.

[ericsciple]

@nero2001 did you update all agents your machine to 2.120.1?

[greadtm]

Does 2.120.2 address all known issues with running multiple agents on a single server?

[ericsciple]

@greadtm v2.120.2 carries a new version of git. I still have an outstanding question whether all agents on the machine were updated to 2.1201. or not (if not then that could explain the issue).

[nero2001]

Yes i did.

Em 14/08/2017 18:52, "ericsciple" [e-mail] escreveu:

@nero2001 https://github.com/nero2001 did you update all agents your
machine to 2.120.1?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/vsts-agent/issues/898#issuecomment-322260450,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAg7fovi4-4-inWrSUHivlerG9nhRAfJks5sYIligaJpZM4MyGX1
.

[greadtm]

Is this issue still being investigated?

[ericsciple]

@garimasi

[GarimaSi]

The tf workspaces issue for multiple agents is being looked into. We will get back here once we know more and/or if we have any follow up questions. Thanks for your patience!

[greadtm]

I wanted to follow up and see if there is any progress on this fix? We are interested in getting this fix to our development user base.

[jmsvl]

Hi, we're also waiting for follow up on this matter. Thanks

[syedhassanabbas]

last Saturday I upgraded to TFS 2017 Update 2 with new agents on build server and Dev teams reported exact same issue which is random issue. I will wait for this fix as this is a major productivity impediment for our development teams.

[GarimaSi]

@greadtm, @jmsvl, @syedhassanabbas
My apologies if you have already answered the questions below, but this thread has gotten a little convoluted for us to keep track.

Can you please reply with the following:

1. Have you updated all your agents to 2.120.1?
2. What version of VS is installed on the agents?
3. Are all agents at the same version, with the same VS version?
4. Build step details: How many agents are you running on the same machine? How often do these agents run in parallel?
5. If this is a regression, what was the last working version #?
6. Latest build logs that carry the error

Also, would you be willing to potentially run debug bits to help us root cause.
Thanks!

[jmsvl]

Reply inline

1. Have you updated all your agents to 2.120.1? [jmsvl] All agents
   updated to 2.120.2
2. What version of VS is installed on the agents? [jmsvl] VS 2010, 2012
   and 2015
3. Are all agents at the same version, with the same VS version?
   [jmsvl] All agents on the server are the same version with multiple VS
   versions
4. Build step details: How many agents are you running on the same
   machine? How often do these agents run in parallel? [jmsvl] 4 agents. All
   checkins trigger multiple builds.
5. If this is a regression, what was the last working version #?
   [jmsvl] Certainly a version previous to 2.117 but I cannot be sure of the
   number
6. Latest build logs that carry the error [jmsvl] logs with different
   errors attached

On Tue, Aug 29, 2017 at 4:09 PM, Garima Singh [e-mail]
wrote:

@greadtm https://github.com/greadtm, @jmsvl https://github.com/jmsvl,
@syedhassanabbas https://github.com/syedhassanabbas
My apologies if you have already answered the questions below, but this
thread has gotten a little convoluted for us to keep track.

Can you please reply with the following:

1. Have you updated all your agents to 2.120.1?
2. What version of VS is installed on the agents?
3. Are all agents at the same version, with the same VS version?
4. Build step details: How many agents are you running on the same
   machine? How often do these agents run in parallel?
5. If this is a regression, what was the last working version #?
6. Latest build logs that carry the error

Also, would you be willing to potentially run debug bits to help us root
cause.
Thanks!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/vsts-agent/issues/898#issuecomment-325694937,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGnnXPXRPC_jcRK8LxKneNKn4ax3gJ4Eks5sdCmngaJpZM4MyGX1
.

[GarimaSi]

@jmsvl Can you try reattaching the logs please. It looks like they didn't get through the first time.

[Cristie]

Deployment Groups are not available in Team Foundation. Learn more, https://www.visualstudio.com/en-us/docs/build/concepts/definitions/release/deployment-groups/#deploy-agents-to-a-deployment-group

[jmsvl]

I sent via email reply and it didn't work. Via site now.

logs_57650.zip
logs_57664.zip

[GarimaSi]

@Cristie Wrong thread?

[Cristie]

@GarimaSi No that link is valid

[GarimaSi]

@Cristie I know the link is valid. I was just wondering if it is on the wrong thread. If it isn't, can you please expand on how it pertains to the issue being discussed here.

[Cristie]

Build Agent Pools, Azure VMs, Team Foundation Servers, etc. are experiencing the same related issues. Related threads #1159 #1141 #1163

[ericsciple]

those are separate issues. let's keep this thread focused.

[Cristie]

@ericsciple I strongly disagree. A build failing to deploy on a Team Foundation server is a server side issue wherein Deployment Groups are not available (i.e. supported) by the server.

[bryanmacfarlane]

@Cristie - this issue is only tracking the error mentioned at the beginning of this issue

From a build, get sources with multiple agents can cause:

Unable to determine the workspace. You may be able to correct this by running 'tf workspaces /collection:TeamProjectCollectionUrl'.
2017-03-23T23:49:31.8872843Z ##[error]Exit code 100 returned from process

That is an issue in tf.exe shipped with the agent

[syedhassanabbas]

@GarimaSi

Sorry for the late response. Below are the answer for your questions for the issue "/loginType:OAuth /login:.,** /noprompt'" while building. We have also noticed we are not able to deploy as windows copy file task is unable to find the network path, (it says path dose not exist). while running the same release with old agent it works ok. FYI, we have enable the old agents (shipped with TFS 2017 Update 1 R2) and currently only using the old one.

Have you updated all your agents to 2.120.1?
Yes

What version of VS is installed on the agents?
2013 and 2015

Are all agents at the same version, with the same VS version?
No, the latest one dose not work but all old agents are working.

Build step details: How many agents are you running on the same machine? How often do these agents run in parallel?
We have 3 machines. One machine is running 2 old agents, one machine have 2 new agents (agents are disabled because of the issue), the main build server have 9 agents installed (two agents are new (disabled) and other 7 are old)

If this is a regression, what was the last working version #?
Please find attached.
logs_8215.zip
ReleaseLogs_1693.zip

Latest build logs that carry the error
Not sure how can I share as I don't see attachment option here. Do you want me to upload it somewhere?

Also, would you be willing to potentially run debug bits to help us root cause.
Yes, based on my and my team's availability.

[syedhassanabbas]

FYI the version i see in file names of old agents is vsts-agent-win7-x64-2.111.1. and the version of new agents (that have this issue) is vsts-agent-win7-x64-2.117.2.

[greadtm]

We have 2.120.2 but have yet to deploy it our production environment, We are still on agent 2.105. We held off on pushing the newest agent out as user nero2001 claimed they were still experiencing the issue with the newest agent. GarimaSi then responded the issue was still being researched. I have yet to see a definitive answer on whether this specific bug was resolved:

From a build, get sources with multiple agents can cause:
Unable to determine the workspace. You may be able to correct this by running 'tf workspaces /collection:TeamProjectCollectionUrl'.
2017-03-23T23:49:31.8872843Z ##[error]Exit code 100 returned from process

when a server has multiple agents.

[GarimaSi]

@syedhassanabbas, @greadtm Thanks for letting us know the state the agents are in. The old agents (prior to 2.120.1) have a bug that causes this problem in some cases which is why we need you to upgrade to a newer agent. When I said we are still researching and investigating the issue, we meant we are trying to root cause a different bug that may be causing similar symptoms. My apologies for the confusion. Please upgrade at your earliest convenience and let us know if you are still seeing issues.

[syedhassanabbas]

@GarimaSi I have installed 2.120.2 version of agents and it looks like the workspace error is not coming anymore but we still have issues with publish artefacts task. Error below. Not sure If this is the right thread to discuss this new issue though.

System.IO.DirectoryNotFoundException: Could not find a part of the path 'G:\agent3_work\20\a_PublishedWebsites'.

System.Management.Automation.ParameterBindingValidationException: Cannot bind argument to parameter 'Path' because it is null.
PowerShell script completed with 2 errors.

[ericsciple]

@syedhassanabbas please open a separate issue

[syedhassanabbas]

Please ignore the previous error as it was executed with old agent. Sorry my mistake. I just executed the build on new agents and got same original error.

Starting: Get Sources

---

Prepending Path environment variable with directory containing 'tf.exe'.
Setting environment variable TFVC_BUILDAGENT_POLICYPATH
Querying workspace information.
tf vc workspace /new /location:local /permission:Public ws_2_56 /collection:http://wcs-tfs.comfin.ge.com:8080/tfs/Factorlink/ /loginType:OAuth /login:.,* /noprompt
The path F:\agent2_work\2\s is already mapped in workspace ws_2_53;Build\ecf431a2-806c-44bb-8caf-79f0f4b87699 [http://wcs-tfs.comfin.ge.com:8080/tfs/WCSOnline].
Exit code 100 returned from process: file name 'tf', arguments 'vc workspace /new /location:local /permission:Public ws_2_56 /collection:http://wcs-tfs.comfin.ge.com:8080/tfs/Factorlink/ /loginType:OAuth /login:.,* /noprompt'.

[ericsciple]

2.120.2 fixes a couple of the code paths that can cause the race condition. One more code path was identified today and the version control team is aware and will provide a fix.

[syedhassanabbas]

Thanks Eric, I will keep an eye on this thread.

[syedhassanabbas]

Hi Eric,

Any idea when can we expect this fix?

[ericsciple]

I'm hoping to have a build ready tomorrow available for download. I'm planning to start rolling the agent out through the rings tomorrow will continue to roll over the next week or so.

[metaphysico]

On version 123 I am still having the issue, but I can add some details and a workaround that may be helpful to find this problem and get people back up and running again.

So here is the scenario as I have pinned down.

1. Remove all agents
2. Remove all workspaces with tf workspaces /remove:*
3. Run a build, get the error
4. Run Developer command Prompt
5. Navigate to folder inside of workspace
6. Run tf workfold > get same error
   8a. Run tf workspaces /s:serverurl > a workspace shows up
   8b. Alternative to 8a, open visual studio and open manage workspaces, this makes tf workfold show details as well. I think it calls the same workspaces call when it opens.
7. Run tf workfold > now the workspace resolves.
8. From this point the builds will work when they run in that workspace, unless the build has clean enabled, in which will delete the workspace then add it back with the same issue.

I too though it was random until I tracked the issue. Here is what mine did.

1. Build1 ran on Agent1 and failed for error, I went to build server and looked at the workspaces which fixed that build.
2. Build2 ran on Agent1 with same problem and I went to VS and opened workspaces again
3. at this point build1 and build2 had "Fixed" workspaces on that agent.
4. Now Build1 and Build2 run find as they always get queued on agent1.
5. Run Build1 and Build2 in Parrallel, now agent1 does build1 and agent2 takes build2 at which time build2 now gets the error.
6. Opening workspaces will fix Agent2s workspace for build2, or if you run again after build1 finishes it will probably start on Agent1 again which was "Fixed' before.

So my assumption here is the workspace is being created but not attached to something. I think it's the cache based on
MSDN Blog
which states "If the workspace mapping exists on the server, then your mappings cache file may need to be refreshed. Try running this command to populate/refresh the cache: tf workspaces /s:http://tfs-server:8080" Which is why I tried 8a mentioned earlier and is the workaround if implemented as below.

WORKAROUND:

In the powershell call "tf workspaces /s:http://tfs-server.com" before calling tf whatever.

The only other current "Workaround" we have is,

1. Disable clean workspace in all build definitions.
2. Every time a new workspace is created (because a build runs on an agent it has not run on yet), manually run what is mentioned in 8a. From there you have to do this for every workspace, which in my case is 42 worst case, which is not feasible.

Here are the details for the questions asked.

1. Have you updated all your agents to 2.120.1? on 123
2. What version of VS is installed on the agents? 2015.2
3. Are all agents at the same version, with the same VS version? Not sure how the versions relate. Looks like agents are using ddls for 15 and VS is 14, so I assume they are different. But VS15 compared to 2.123 do not seem to relate well.
4. Build step details: How many agents are you running on the same machine? How often do these agents run in parallel? 6 agents 7 builds, 27 releases, so 204 workspaces max. Run parallel a lot.
5. If this is a regression, what was the last working version #? from old agents 14.102...
6. Latest build logs that carry the error supplied. I would have to go through all of the files and details to not violate our strict NDA and security agreements. But needed info is:
   VERBOSE: setting location to: C:\Builds\Agent1_work\1\s\mycode
   VERBOSE: exe: C:\Program Files (x86)\Microsoft Visual Studio 14.0common7\ide\tf.exe
   VERBOSE: args: checkout SolutionInfo.cs
   Calling Cmd: C:\Program Files (x86)\Microsoft Visual Studio 14.0common7\ide\tf.exe checkout SolutionInfo.cs
   VERBOSE: stdout:
   stderr: Unable to determine the workspace. You may be able to correct this by running 'tf workspaces /collection:TeamProjectCollectionUrl'.
   VERBOSE: exit code 100

Now I am not sure if the issue is caused by having VS2015 installed while using agents for 2017, looking at the prior messages, it looks like everyone having this issue is using this scenario, so that may be the root problem. Several people have said they have VS2015, and I am running tf from version 14 (which is VS2015, so confusing). That may be the key to replicate the issue.

This is my setup for tf.exe $tfsTool = "C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\TF.exe" so directly using 14.

Sorry for the long message, but I hope this is detailed enough and contains enough information to fix or workaround the problem. Personally I think if this is an issue using 2015 TF and 2017 agents, then there may need to be a workaround posted and not necessarily a fix, but a fix would be nice.

[jmsvl]

Any idea when will the 123 version be released? Thank you

[TingluoHuang]

@jmsvl it should already shows in your VSTS account's download agent pages.
https://github.com/Microsoft/vsts-agent/releases/tag/v2.123.0

[jmsvl]

Thanks. Already updated and I will be on the lookout for the usual "get sources" error.

[jmsvl]

Now, with all agents updated on our 2 build servers, the number of errors is almost zero. But now and again there is still one error 100 on "'vc workfold /map". Is anyone getting this also?

[fiveshotsofespresso]

We also experience this issue -- TFS 2017 U2 w/ TFS Agent 2.124.0.

Interestingly, the collisions only occur cross-collection, so far as we can tell. Still grasping at straws for how to fix this or help our devs who experience this issue.

[Prachiti9]

@jmsvl Good to hear that error count has been gone down for you. Can you share logs for error 100 you are still getting?
@fiveshotsofespresso The fixes we have made so far are suppose to work cross-collection as well. Are you getting exact same error? If yes, it would be great if you can share logs.

[fiveshotsofespresso]

@Prachiti9 Sure.

Quick overview: TFS 2017 Update 2, Running vsts-agent version 2.124.0 on Windows 2012 R2 machines. 4 agents per machine, separate service names. Happens on both VS2015 and VS2017 agents.

2017-11-27T16:12:42.4031263Z ##[section]Starting: Get Sources
2017-11-27T16:12:42.4500063Z Prepending Path environment variable with directory containing 'tf.exe'.
2017-11-27T16:12:42.4500063Z Setting environment variable TFVC_BUILDAGENT_POLICYPATH
2017-11-27T16:12:42.4500063Z Querying workspace information.
2017-11-27T16:12:47.0751037Z ##[command]tf vc workspace /new /location:local /permission:Public ws_5_268 /collection:https://tfs.company.com/tfs/DefaultCollection2/ /loginType:OAuth /login:.,* /noprompt
2017-11-27T16:12:47.9501230Z The path F:\tfs\1_work\5\s is already mapped in workspace ws_5_268;Build\4e01aebd-d1a6-4a0d-8ee3-e5a901209e9a [https://tfs.company.com/tfs/DefaultCollection].
2017-11-27T16:12:48.0282432Z ##[error]Exit code 100 returned from process: file name 'tf', arguments 'vc workspace /new /location:local /permission:Public ws_5_268 /collection:https://tfs.company.com/tfs/DefaultCollection2/ /loginType:OAuth /login:.,* /noprompt'.
2017-11-27T16:12:48.0282432Z ##[section]Finishing: Get Sources

The errors are always about a build directory sourced from the other collection, we've seen no instance of conflicts within the same collection.

[Prachiti9]

@fiveshotsofespress looking at your log snippet (line - The path F:\tfs\1_work\5\s is already mapped in workspace) this issue seems different than one we are tracing with @jmsvl . It might be confusing as error looks same on surface but can you please open a new issue and share full logs? Thank you!

[fiveshotsofespresso]

@Prachiti9 Sure. #1311 Created.

[Allann]

I just received this error. We run on VSTS but build on a local agent. This was straight after a successful build/deploy. We ran the exact same build again (with clean set to true). And received the error, except it doesn't appear to be a login error:

2017-12-11T23:24:29.6002476Z ##[command]tf vc undo /recursive D:\agent_work\1\s /loginType:OAuth /login:.,* /noprompt
2017-12-11T23:24:30.4440202Z TF400024: The change on D:\agent_work\1\s\Dev\Jbssa.FreightRates\src\Jbssa.FreightRates.Web\wwwroot\bundle\bootstrap.css cannot be undone because a file already exists at D:\agent_work\1\s\Dev\Jbssa.FreightRates\src\Jbssa.FreightRates.Web\wwwroot\bundle\bootstrap.css. The file must be deleted from disk for the undo to succeed.
2017-12-11T23:24:30.5221498Z ##[error]Exit code 100 returned from process: file name 'tf', arguments 'vc undo /recursive D:\agent_work\1\s /loginType:OAuth /login:.,* /noprompt'.
2017-12-11T23:24:30.5377742Z ##[debug]Microsoft.VisualStudio.Services.Agent.ProcessExitCodeException: Exit code 100 returned from process: file name 'tf', arguments 'vc undo /recursive D:\agent_work\1\s /loginType:OAuth /login:.,** /noprompt'.
at Microsoft.VisualStudio.Services.Agent.ProcessInvoker.d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.Agent.Worker.Build.TfsVCCommandManager.d__26.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.Agent.Worker.Build.TFCommandManager.d__25.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.Agent.Worker.Build.TfsVCSourceProvider.d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.Agent.Worker.Build.BuildJobExtension.d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.Agent.Worker.JobExtensionRunner.d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.Agent.Worker.StepsRunner.d__1.MoveNext()
2017-12-11T23:24:30.5377742Z ##[section]Finishing: Get Sources

Happy to supply more details if needed, but attached the full debug log file if that helps
logs_376.zip

Update: manually deleted all the files in the Agent work folder and now it builds fine again. There were no locks on the file. It was a clash between the VSTS and the file system.

[TingluoHuang]

I have read through the thread and i think folks either get their problem fixed already or opened a separate issue, i am going to close this extremely long thread.