Powershell: For ForEach-Object -Parallel, nested loops improvment

[PrzemyslawKlys]

While you're at improving ForEach-Object -Parallel please address this:

Doesn't work

$OhMy = 'Test'
(1..100) | ForEach-Object -Parallel {
    $Test = $Using:OhMy
    (1..50) | ForEach-Object -Parallel {
        $Test5 = $Using:Test
    }
}

Works:

$OhMy = 'Test'
$Test = $null
(1..100) | ForEach-Object -Parallel {
    $Test = $Using:OhMy
    (1..50) | ForEach-Object -Parallel {
        $Test5 = $Using:Test
    }
}

[mklement0]

Just to make it more explicit what you're experiencing: there is an _inconsistency_ in that:

• _nesting_ of $using: references _is_ effectively supported

• but the variable-existence check behaves as if it weren't.

# Fails, because the `$using:` check looks for a $test variable in the *script* scope, 
# not the *enclosing one*.
1..2 | % -Parallel { $test = 'actual';  % -parallel { $using:test } }

# By formally satisfying the script-scope check, 
# the command is allowed to run, and the enclosing scope's value *is* used.
PS> $test = 'dummy'; 1..2 | % -Parallel { $test = 'actual';  % -parallel { $using:test } }
actual
actual

[iSazonov]

@PaulHigin Could you please look the issue?

[PaulHigin]

This is by design. The $using: keyword is effective only for variables defined in the current scope. The help document should be updated to make this clear.

[vexx32]

I believe the request is that the design be _improved_ to accommodate managing nested jobs more easily from a script, no? 🙂

[PaulHigin]

I wonder if there are any real world cases for running foreach -parallel nested like this. If so then I question whether it is being used effectively. This seems like a nice to have feature. But I'll add the committee review tag so it can be considered.

[PrzemyslawKlys]

I did use it - that's why I know about it.

https://github.com/EvotecIT/Graphimo/blob/master/Examples/Example-DeletingCalendarEvents.ps1

Basically I was calling MSGraph and iterating all users in Office 365 and all their calendars (each user having 3-20 calendars) and each calendar having 10k to 50k events and deleting those I didn't want. I couldn't easily overcome the issue of Graph by doing proper reconnection to avoid timeout as I am not so proficient with it. Instead, I just initiated new sessions for each calendar, rather than per user.

If it's supported, and it works and the only negative thing about it is the need to define a variable at the top I would like it to be fixed - as surely it's better to define variable only if it's necessary.

GitHub

EvotecIT/Graphimo

Contribute to EvotecIT/Graphimo development by creating an account on GitHub.

[mklement0]

This may be too naive a solution, but setting searchNestedScriptBlocks to false in the following code at least gets rid of the overzealous variable-existence check (which is the real problem here).

https://github.com/PowerShell/PowerShell/blob/f6a897331702a8f11f7447d3a77b46fd21286a5d/src/System.Management.Automation/engine/runtime/ScriptBlockToPowerShell.cs#L176-L183

[PaulHigin]

Sorry, I just now spent time looking into this more closely. I thought this was passing one scope variable to other scopes. But I agree that this should work:

$Test = "Test1"
1..2 | % -parallel {
  "Here is first: $using:Test"
  $Test2 = "Test2"
  1..2 | % -parallel {
    "Here is second: $using:Test2"
  }
}

And yes, I think @mklement0 fix is probably correct for Foreach-Object -Parallel case. The using variable map should only apply to current scriptblock and not nested scriptblocks. This may be an easy fix.

[iSazonov]

I wonder if there are any real world cases for running foreach -parallel nested like this.

Language allows this. So options are:

• explicitly block the nested scenario
• fix the scenario.

This also brings general question: if user create nested job in a job is this supported?

[PaulHigin]

@iSazonov
There are a lot of things in PowerShell language that can cause trouble or is not necessarily good to use in particular cases. I don't think restricting the language is the right answer, but enlightening users about pros and cons of patterns is much better.

Running nested ForEach -Parallel can greatly magnify resource usage and end up being far slower than expected, hence my question. But as long as users are aware, nested foreach -Parallel may be an optimal solution. It sounds like @PrzemyslawKlys has a legitimate need for the pattern.

Anyway, I was in a hurry and misunderstood this issue (my bad and I do apologize), and this is indeed a bug. I am working on a fix.