Powershell: Feature Request: Multithreaded "Process" block

[JamesLear92]

Summary of the new feature/enhancement

When passing parameters from the pipeline, the process block is repeated for each object. It would be fantastic if you could run these process blocks in parallel in scenarios where processing in order does not matter. You could get massive performance increases in a very concise and elegant way.

It would also be fantastic if we could specify a parameter flag that treats an input as if it was passed via the pipeline so that regardless of whether we're passing an array into the pipeline, or an array into a parameter, the process block is looped foreach object. At the moment if we have a function that has an array input parameter which can come from the pipeline, we have to have a foreach loop in the process block for when not piping which is not needed when we are piping.

Proposed technical implementation details (optional)

I suspect that the easiest way to implement this is by using a runspace pool, and as pipeline objects arrive they are put into a queue and are processed by a runspace when one becomes available.
"-ProcessInParallel" and "-ThrottleLimit" could be enabled by CmdletBinding flags.
I am guessing that non-serialisable objects wouldn't be allowed in the "Begin" block.

Example

function Invoke-Example
{
    [CmdletBinding(SupportsProcessInParallel = $true)]
    param
    (
        [parameter(ValueFromPipeline = $true, TreatValueAsPipeline = $true)]
        [double[]]$InputArray
    )
    BEGIN { }
    PROCESS
    {
        Write-Verbose -Message ('InputLength: {0} ThreadID: {1}' -f $InputArray.Count, [Threading.Thread]::CurrentThread.ManagedThreadId)
    }
    END { }
}

So we'd see something a little like this:

1..5 | Invoke-Example -ProcessInParallel
VERBOSE: InputLength: 1 ThreadID: 1
VERBOSE: InputLength: 1 ThreadID: 2
VERBOSE: InputLength: 1 ThreadID: 3
VERBOSE: InputLength: 1 ThreadID: 4
VERBOSE: InputLength: 1 ThreadID: 5

Invoke-Example -InputArray (1..5) -ProcessInParallel
VERBOSE: InputLength: 1 ThreadID: 1
VERBOSE: InputLength: 1 ThreadID: 2
VERBOSE: InputLength: 1 ThreadID: 3
VERBOSE: InputLength: 1 ThreadID: 4
VERBOSE: InputLength: 1 ThreadID: 5

Instead of what would be in current versions:

1..5 | Invoke-Example
VERBOSE: InputLength: 1 ThreadID: 1
VERBOSE: InputLength: 1 ThreadID: 1
VERBOSE: InputLength: 1 ThreadID: 1
VERBOSE: InputLength: 1 ThreadID: 1
VERBOSE: InputLength: 1 ThreadID: 1

Invoke-Example -InputArray (1..5)
VERBOSE: InputLength: 5 ThreadID: 1

[edison-moreland]

I would love this feature! I actually had a similar idea while porting some scripts to use the new ForEach-Object -Parallel.

It would make life a lot easier when you just have one function you need to run in parallel.

[w3bward]

I was just thinking about this while working on a function in a module. I could easily set it up to accept the output of another function in the same module from the pipeline, except it needs to be run in parallel. Right now I'm debating adding the Foreach-Object -parallel inside the function (I really don't like this from a design standpoint), or just resigning to wrapping the function at runtime. It would be great to have the best of both worlds!

I like the idea of making it an attribute of CmdletBinding. I would suggest that when SupportsProcessInParallel or SupportsParallelProcessing or whatever it's called is $true then the following common parameters should be available to control it:

• -Parallel (switch to enable at runtime)
• -ThrottleLimit (same as Foreach-Object)
• -AsJob (same as Foreach-Object)