Powershell: Pipelines cannot be run concurrently

[MS-syh2qs]

Customers are running into System.Management.Automation.PSInvalidOperationException: The pipeline was not run because a pipeline is already running. Pipelines cannot be run concurrently error when calling Add-AzVhd method from Az.Compute module. Looking at the stack trace, it's coming from the following method in Automation SDK: System.Management.Automation.Runspaces.PipelineBase.DoConcurrentCheck(Boolean syncCall, Object syncObject, Boolean isInLock)

However, we are not able to reproduce it every time. We are only able to reproduce it successfully around 20% of the time. We are not using any piping in our PS script. Could you guys help us out in determining what the problem might be?

Steps to reproduce

No reliable reproduce method. Personally, I am unable to reproduce it, however another engineer is able to and he has around 20% repro rate. 

# Cosine IOT Services Azure Internal Consumption
$subscriptionId='f9cb372a-8903-4370-9ce0-3042e3bb2cc8'

$rgSeed = "akstest" # TODO customize
$vmSeed = "0430c101a"
$vhdFolder = "C:\Users\jiria\Downloads"
$vhdFileName = "aks-vhd-2.0.20200815.0745.vhd"
$storageAccountName = "storage" + $rgSeed

$location = "westus2"

# $vmSize = "Standard_A2"
$vmSize = "Standard_NC6"

.\vm-create.ps1 $subscriptionId $rgSeed $vmSeed $vhdFolder $vhdFileName $storageAccountName $location $vmSize

Expected behavior

No error

Actual behavior

error: Exception - System.Management.Automation.PSInvalidOperationException: The pipeline was not run because a pipeline is already running. Pipelines cannot be run concurrently. 

Environment data

[iSazonov]

Please ask in https://github.com/Azure/azure-powershell

GitHub

Azure/azure-powershell

Microsoft Azure PowerShell. Contribute to Azure/azure-powershell development by creating an account on GitHub.

[MS-syh2qs]

@iSazonov, I was told by Azure-PS team to come here. They weren't sure what the best way to instantiate PowerShell class. What is the difference between calling Create() vs Create(RunspaceMode.NewRunSpace)?

[iSazonov]

@MS-syh2qs Please add more info as our issue template ask. Also please add a reference to your discussion with Azure-PS team.

[rjmholt]

We are not using any piping in our PS script

A pipeline here is what runs the commands. The error message means you're trying to run multiple commands at the same time on the same thread (i.e. in the same runspace).

In the snippet and error message you've shown, there isn't enough information to establish what's going on. In particular we don't know what cmdlets are being used, where the error occurs or any other details about the error. We need:

• The contents of vm-create.ps1 and any other scripts being called -- we need to know the whole program you're running
• The full stack trace of the error, which you can get with Get-Error or $error[0] | fl * -force after you reproduce

[vexx32]

I think they mentioned the cmdlet they ran; Add-AzVhd -- I suspect the cmdlet is doing something... unwise... internally if that's all they ran.

[MS-syh2qs]

@rjmholt,

Here's the script:
vm-create.txt

This is the full stack trace:

DEBUG: AzureQoSEvent: CommandName - Add-AzVhd; IsSuccess - False; Duration - 00:56:52.8136956;; Exception - System.Management.Automation.PSInvalidOperationException: The pipeline was not
run because a pipeline is already running. Pipelines cannot be run concurrently.
   at System.Management.Automation.Runspaces.PipelineBase.DoConcurrentCheck(Boolean syncCall, Object syncObject, Boolean isInLock)
   at System.Management.Automation.Runspaces.RunspaceBase.DoConcurrentCheckAndAddToRunningPipelines(PipelineBase pipeline, Boolean syncCall)
   at System.Management.Automation.Runspaces.PipelineBase.CoreInvoke(IEnumerable input, Boolean syncCall)
   at System.Management.Automation.Runspaces.PipelineBase.Invoke(IEnumerable input)
   at System.Management.Automation.PowerShell.Worker.ConstructPipelineAndDoWork(Runspace rs, Boolean performSyncInvoke)
   at System.Management.Automation.PowerShell.Worker.CreateRunspaceIfNeededAndDoWork(Runspace rsToUse, Boolean isSync)
   at System.Management.Automation.PowerShell.CoreInvokeHelper[TInput,TOutput](PSDataCollection`1 input, PSDataCollection`1 output, PSInvocationSettings settings)
   at System.Management.Automation.PowerShell.CoreInvoke[TInput,TOutput](PSDataCollection`1 input, PSDataCollection`1 output, PSInvocationSettings settings)
   at System.Management.Automation.PowerShell.Invoke(IEnumerable input, PSInvocationSettings settings)
   at Microsoft.Azure.Commands.Compute.Models.PSSyncOutputEvents.LogProgressComplete(Int32 activityId, String activity)
   at Microsoft.Azure.Commands.Compute.Models.PSSyncOutputEvents.ProgressUploadComplete(TimeSpan elapsed)
   at Microsoft.WindowsAzure.Commands.Sync.ProgressTracker.Dispose(Boolean disposing)
   at Microsoft.WindowsAzure.Commands.Sync.ProgressTracker.Dispose()
   at Microsoft.WindowsAzure.Commands.Sync.Upload.BlobSynchronizer.Synchronize()
   at Microsoft.Azure.Commands.Compute.Models.VhdUploaderModel.Upload(UploadParameters uploadParameters)
   at Microsoft.Azure.Commands.Compute.StorageServices.AddAzureVhdCommand.ExecuteCmdlet()
   at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord();

[rjmholt]

Thanks for that.

Here is the relevant code in Azure PowerShell.

This is expected behaviour from the PowerShell engine; trying to run multiple PowerShells at the same time on the same runspace will result in this error.

Azure PowerShell needs to create a nested PowerShell instance to do this when calling from the pipeline thread -- this can be done, since we are under the ProcessRecord() call stack. In cases where they are calling from a different thread, they need to come up with another way to synchronise their use of the original runspace.

I suggest if you have the original Azure PowerShell issue, we should move this back to there and reopen it, or otherwise open a new issue on Azure PowerShell.

[rjmholt]

I was told by Azure-PS team to come here

Apologies, I misunderstood this as "the AzPS team told me the issue lies with PowerShell" rather than "I'm trying to help the AzPS team fix this and I'm here for answers".

Basically the fix depends on:

• How this method is being called/needs to be called (is it always from within the scope of a cmdlet's process methods, or can it be from other threads?)
• What is needed from the existing context and why the original runspace is being use

Basically, if this is always being called from the original cmdlet thread under the callstack of BeginProcessing()/ProcessRecord()/EndProcessing(), then it's enough just to use a nested PowerShell instance:

using (var pwsh = PowerShell.Create(RunspaceMode.CurrentRunspace))
{
    pwsh.AddCommand("Write-Progress")
        .AddParameter("Id", activityId)
        .AddParameter("Activity", activity)
        .AddParameter("Status", Rsrc.PSSyncOutputEventsLogProgressCompleteCompleted)
        .AddParameter("Completed")
        .Invoke();
}

However, if this is being called from multiple threads as I suspect, then we need to understand the requirements better to come up with a better mechanism to synchronise usage or to otherwise manage the concurrent use of PowerShell resources. Some avenues available are:

• Creating a new runspace to support background operations, or even using a runspace pool
• Using appropriate PSThreadOptions to create that runspace
• Using PowerShell eventing for background tasks
• Creating a more customised concurrency management model, like a consumer thread

In this particular case though, we're trying to write progress, which is a user-facing UI task (i.e. very much a foreground operation). In PowerShell, you don't need to create and execute a PowerShell instance for this. Instead, you can use the Host's UI property ($Host.UI, also available in PSCmdlets with the Host property) to write things whenever you like -- no synchronisation required. But this is a bad idea, because writing output to the screen while the user is doing something else is at best a strange experience and at worst totally mangles their terminal.

So my recommendation is:

• We need to understand the requirements of how PSSyncOutputEvents is being used.
  
  
  
  • Is it just being used to write to the UI, or does it need to actually execute commands?
  
  
  • If it is executing commands etc, does it need the current runspace to do it?
  
  
  • Under what circumstances are other threads calling into PSSyncOutputEvents and why?
  
  
• Based on our understanding of those requirements, we should
  
  
  
  • Turn purely UI-facing methods into well synchronised calls into the current Host UI, rather than creating a new PowerShell instance
  
  
  • Come up with a robust thread model for any remaining invocations that need to run PowerShell
  
  

[MS-syh2qs]

@rjmholt,

Apologies, I misunderstood this as "the AzPS team told me the issue lies with PowerShell" rather than "I'm trying to help the AzPS team fix this and I'm here for answers".

Yeah sorry about that, I wasn't very clear.

Thank you for the detailed response, my team will take a look where you've pointed to.