Azure-pipelines-tasks: Get Sources Task intermittently fails with tf.exe returning 100

[JustinKaffenberger]

Required Information

Entering this information will route you directly to the right team and expedite traction.

Question, Bug, or Feature?
Type: Bug

Enter Task Name: Get Sources

Environment

• Server - Azure Pipelines

  • Organization: Exacta-E6
  • Team Project: Zeta
  • Build Definition Name: AgentTest-JustinK
  • Build Definition Number: 33110

• Agent - Hosted

  • Pool: Azure Pipelines
  • Image: vs2017-win2016
  • Agent: Azure Pipelines 3

Issue Description

The build definition that I've linked here is a sample that I created to quickly reproduce and troubleshoot issues we've been seeing with the Get Sources task. We recently migrated our on-premise TFS to Azure DevOps, and have purchased 6 parallel jobs. Across the board, we are seeing intermittent, unpredictable failures with the Get Sources task, all ending with the tf.exe returning 100.

Task logs

logs_33110.zip

Troubleshooting

We have tried multiple agent image types. We've also tried local AND server workspace mappings, but the errors persist.

Error logs

##[error]Exit code 100 returned from process: file name 'tf', arguments 'vc workspace /new /location:local /permission:Public ws_1_45 /collection:https://dev.azure.com/Exacta-E6/ /loginType:OAuth /login:.,*** /noprompt'.
##[debug]Processed: ##vso[task.logissue type=error;]Exit code 100 returned from process: file name 'tf', arguments 'vc workspace /new /location:local /permission:Public ws_1_45 /collection:https://dev.azure.com/Exacta-E6/ /loginType:OAuth /login:.,*** /noprompt'.
##[debug]Processed: ##vso[task.complete result=Failed;]
##[debug]   at Microsoft.VisualStudio.Services.Agent.Util.ProcessInvoker.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, InputQueue`1 redirectStandardIn, Boolean inheritConsoleHandler, Boolean keepStandardInOpen, Boolean highPriorityProcess, CancellationToken cancellationToken)
   at Agent.Plugins.Repository.TfsVCCliManager.RunCommandAsync(FormatFlags formatFlags, Boolean quiet, String[] args)
   at Agent.Plugins.Repository.TFCliManager.WorkspaceNewAsync()
   at Agent.Plugins.Repository.TfsVCSourceProvider.GetSourceAsync(AgentTaskPluginExecutionContext executionContext, RepositoryResource repository, CancellationToken cancellationToken)
   at Agent.Plugins.Repository.CheckoutTask.RunAsync(AgentTaskPluginExecutionContext executionContext, CancellationToken token)
   at Agent.PluginHost.Program.Main(String[] args)
Finishing: Checkout Zeta@ to s

[stephenmichaelf]

@jeschu1 Could you help us determine if this is a tf issue or an Agent issue? Thanks!

[jeschu1]

@JustinKaffenberger, Do you only see this issue when 2 concurrent builds are running?

As the error states, it thinks two processes are accessing the same workspace at the same time.

We're doing some additional investigation on our side as well.

[JustinKaffenberger]

@jeschu1 we have up to 6 parallel agents operating against the same TFVC codebase. However, each pipeline tends to operate on its own TF branch. It's safe to say that this issue shows up with a higher rate the more parallel builds that we have running.

[colewerner]

@jeschu1 Do you have an update on this? This causes us quite a few heartaches because it's causing a lot of different build pipelines to fail and also at critical times.

It seems like a potential fix was put in place even though I haven't seen any updates on this because we are now getting the following on the Get Sources step in our build pipeline:

TF14061: The workspace ws_1_8;Project Collection Build Service (Exacta-E6) does not exist.
Sleeping for 200 ms
Retrying. Attempt $2/$3
##[error]_proc should be null. (Parameter '_proc')

Unfortunately this is not the only place (Get Sources Step) where we run into this problem either. We also have pipelines that check out / check in files where we see the same original error posted. Those errors look like this:

TF: Checking in changes to src/WixInstallerAttributes.wxi.
Checkin Arguments: vc checkin "src/WixInstallerAttributes.wxi" /comment:"Updating WiX Installer Version Number ***NO_CI***" /notes:"Code Reviewer=N/A;Changes Made=Updating WiX Installer Version Number;Version Number = src/WixInstallerAttributes.wxi: 1.218.49.0" /loginType:OAuth /login:.,*** /noprompt
TF14061: The workspace ws_1_8;Project Collection Build Service (Exacta-E6) does not exist.

[jeschu1]

@MaximMalygin I believe your team has been looking into similar issues. Can you please advise?

[ajainguru]

We have also started getting below error with Hosted ubuntu agent.
Retrying. Attempt $2/$3

[error]_proc should be null. (Parameter '_proc')

Any idea what can fix this? Seems to be issue on Azure agent. It is not able to find/create workspace.

[RuslanRasulovAkv]

Hi @jeschu1 ! I found in logs that 2.165 version of the Agent was used and may be we should update it to the latest one. A retry logic was added In Agent's version 2.169.1 which can influence this step.

[colewerner]

@RuslanRasulovAkv If you are looking at the original logs posted here in this issue those are dated at this point. I posted an update on ~7/1/2020 above showing where this retry logic looks like it was implemented and is failing.

[github-actions[bot]]

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

[JustinKaffenberger]

Our in-house testing over the past few months has shown us that this issue has been resolved.