Powershell: Parallel Foreach Thread locking

[Smorgan05]

Summary of the new feature/enhancement

PowerShell 7 implemented Parallel Foreach but has not added a means to do Sync lock. In C# the way to do this is as follows:

object sync = new object();
Parallel.ForEach(List,
item =>
{
lock (sync){ Locked item to execute in multiple threads}
});

Please implement the ability to lock the script block as follows:

$Sync = New-Object -TypeName System.Object
$List | ForEach-Object -Parallel {
//Locked Resource for each Thread
lock($Sync){
Write-Host $_
}}

Currently the way this is done is by the following:

$List | ForEach-Object -Parallel {
[System.Threading.Monitor]::Enter($_)
Write-Host $_
[System.Threading.Monitor]::Exit($_)
} -ThrottleLimit 5

[iSazonov]

It looks like overhead for PowerShell to have a lock.

/cc @PaulHigin

[PaulHigin]

There is no need to take a lock before running a script block. And in fact doing so serializes script block execution, completely losing any benefit of using -Parallel. PowerShell can't know what script will be executed in the -Parallel block and so it is up to the script author to use thread synchronization objects as needed, just like in C#. But it is recommended to use thread safe types (such as ConcurrentDictionary) so that explicit use of sync objects is not needed.

[Smorgan05]

@PaulHigin I see your point but if you guys are going to implement Multi-threading then you should leave it up to the user how they execute such operations like how C# handles it. The performance impact is minimal as it simply ensures thread safety when executing commands. The performance gain of Parallel vs non is still there so what is the performance impact on PowerShell implementing this?

[PaulHigin]

I am not convinced this is necessary. As you point out, it is already possible to use C# sync objects if needed. But needing sync objects seems rare enough where it is not necessary to create a new PowerShell language 'lock' keyword.

BTW, it is not necessary to protect 'Write-Host' inside a lock.

It seems to me the most likely scenario is for collecting data, but that can be covered by C# thread safe types or the PowerShell pipeline.

$dictionary = [System.Collections.Concurrent.ConcurrentDictionary[[int],[string]]]::new()
1..1000 | ForEach-Object -Parallel {
    $dict = $using:dictionary
    $null = $dict.TryAdd($_, "Item $_")
    Write-Host "Item $_"
}
$dictionary[35]
Item 35

$Items = 1..1000 | ForEach-Object -Parallel {
    Write-Output "Item $_"
    Write-Host "Item $_"
}
$Items[35]
Item 36

BTW, PowerShell has supported running multi-threaded script blocks for over 10 years via API. The new ForeEach-Object -Parallel feature just makes it easier to use, similar to ThreadJobs.

[Smorgan05]

If we follow through with this logic then the user should always use concurrent data structures when working with Parallel foreach. The user is not going to want to use concurrent data structures all the time because of the overhead involved with such processing. Additionally, there should be a means to initialize a Concurrent Data Structure without using .net objects to make this happen.

I think we need to cut to the chase here. The lock statement should be implemented so that the end user has both options. Having a flag means that it won't be turned on by default thus nailing your concerns.

[SeeminglyScience]

If we follow through with this logic then the user should always use concurrent data structures when working with Parallel foreach.
(...)
Additionally, there should be a means to initialize a Concurrent Data Structure without using .net objects to make this happen.

The majority of use cases won't actually need concurrent data structures nor locking. Most of the time folks just use the pipeline like in @PaulHigin's example.

The user is not going to want to use concurrent data structures all the time because of the overhead involved with such processing.

Typically the concurrent types are a lot faster than old school sync objects.

The lock statement should be implemented so that the end user has both options. Having a flag means that it won't be turned on by default thus nailing your concerns.

There's only going to be a very small handful of cases where a sync object makes sense. In those scenarios, using Monitor.Enter and Exit will work just fine. For something to be worth adding a whole extra keyword and hooking up the parser/compiler to handle it, it's gotta have pretty high demand.

[iSazonov]

Really PowerShell implements many magic things hided from users. Thus conhost has internal locks and users have no needs to lock/sync console outputs.
I think we should follow the best practice and enhance internal PowerShell engine if we need to cover a new scenario.

[msftbot[bot]]

This issue has been marked as answered and has not had any activity for 1 day. It has been closed for housekeeping purposes.